EXP Test Framework

Introduction

EXP is a new open-source test framework from Tesults (https://github.com/tesults/exp).

Here is what a simple test file containing three tests looks like in EXP

module.exports = {
  suite: "Test Suite Example",
  cases: [
    {
      name: "Test 1",
      test: function () {
        console.log('testing here');
        // return nothing (undefined) for a pass result
      }
    },
    {
      name: "Test 2",
      test: function () {
        console.log('testing here too');
        // return the string "unknown" for an unknown result
      }
    },
    {
      name: "Test 3",
      test: function () {
        console.log('and here');
        // return anything else for a fail result
      }
    }
  ] 
}

Why introduce a new test framework?

Over the years the Tesults team has gained considerable experience with a range of test frameworks, both as users and as developers of integrations and plugins for transmitting test results to Tesults. Overtime we have come to realize there is a need for a new test framework that overcomes some of the issues we see with what is currently available.

We do not expect or recommend anyone with a large body of existing tests suites to migrate to using EXP. Other test frameworks are great, and we will always be developing integrations and plugins for other test frameworks. In some cases other test frameworks are actually better suited to the task at hand. For example if you are performing automated front-end testing via the browser there are frameworks out there that have considerable capabilities to make that process smoother. EXP is a general purpose test framework, it can certainly be used to facilitate front-end testing via a browser but you would have to code the browser driver and selenium setup yourself. That said, if you're looking for a framework to generally test and starting a new project, EXP is worth serious consideration.

EXP is primarily intended for end-to-end automated testing, but can be utilized for unit and integration testing too, and requires the use of Node.js. Test cases are written in JavaScript. In the future we may create versions for other languages but usually for end-to-end testing of front-end and back-end systems the program at test and the test framework testing it can utilize different languages.

EXP is named after experience points commonly gained in RPG video games as characters gain experience, often abbreviated as EXP or exp. This test framework was developed after years of experience points gained using and developing for other test frameworks.

Quick Setup

Ensure Node.js is installed, version 10 or later. Create a Node.js project using npm init.

cd into your node project and do:

npm install exp-tf --save

In your package.json file, add this test command in the scripts block:

...

"scripts": {
  "test": "exp dir=/full/path/to/tests tesults-target=token"
}

...

The dir arg should be the full path to where you test files are located. You can provide multiple dir params (dir=path1 dir=path2 dir=path3) if you have tests in separate locations, but in most cases you can supply the a parent directory and EXP will search all sub directories for test files. If tests you expect to be found are not being found it is likely due to an error with the test file, in particular check appropriate required modules can be loaded. The tesults-target arg is optional, provide it if you want to report results to Tesults. The token value can be found from the configuration menu in Tesults.

Run your tests using:

npm test

Test Suite

A test suite is a collection of test cases. You generally have one test suite per file.

At a minimum a valid test file must export an object with a cases property consisting of an array of test cases, all other properties are optional:

module.exports = { cases: [] }

To specify the suite name, add the suite property. You can name your test file anything you want but keeping the name close to the suite name can be useful for organizational purposes.

module.exports = { cases: [], suite: "Test Suite A" }

Use the hooks property to declare test hooks. Test hooks are functions that run before and after all of the test cases in the test file or before and after each test case. Hooks can be useful if you want to initialize a state for example and then tear things down after a test has completed. EXP supports four test hooks: beforeAll, afterAll, beforeEach, afterEach.

module.exports = {
  suite: "Test Suite A",
  hooks: {
    beforeAll: function () {
       // runs before all test cases in this file
    },
    beforeEach: function () {
       // runs before each test case in this file
    },
    afterEach: function () {
       // runs after each test cases in this file
    },
    afterAll: function () {
       // runs after all test cases in this file
    }
  },
  cases: [ ... ] 
}

Use the timeout property to specify a value for timing out a test case in milliseconds (ms) and failing it if the test case does not end on its own before that duration. By default EXP uses a timeout of 60000ms or one minute. You can override this value by using timeout. Caveat: if your test case blocks the Node.js event loop then timeout will not succeed and EXP will appear to hang. You can identify which test is responsible using EXP console output to see which test is running. Although EXP could be designed to not have this limitation, this would put considerable restrictions on how you can write tests. This way you have great flexibility and convenience around writing tests and only have to ensure you guard against event loop blocking.

module.exports = {
  suite: "Test suite name (optional)", 
  hooks: { /* test hooks */ }, 
  cases: [ /* tests cases here */ ] 
  timeout: 60000 
}

Test Case

A test case is an object containing properties that define a single test including a test function that is executed to run your test.

{
  name: "Test 1",
  test: function () {...}
}

Add test cases to the cases array inside a test suite object:

module.exports = {
  cases: [
    {
      name: "Test 1",
      test: function () {...}
    },
    {
      name: "Test 2",
      test: function () {...}
    }
  ]
}

The name property is used to name the test case.

The test property value should contain the code for the test in an anonymous function.

{
  name: "Test 1",
  test: function () {
    const expected = 1;
    const actual = valueFromSomewhere();
    if (expected !== actual) {
      return 'actual value: ' + actual + ' does not match expected: ' + expected;
    }
  }
}

If the test function returns nothing (undefined) then the test case is considered to be a pass. If you return the string 'unknown' then test result is unknown. If you return anything else or the test throws an exception then the test result is a fail. In the example above, if actual has a value of 1 then the result will be pass because nothing is returned, otherwise it will be a fail because the message about the actual and expected value not matching is returned. In this example the function runs synchronously. Your test function can be asynchronous using callbacks or async/await/promises, see below for details on how to do this.

For each test case, the name and test are the only required properties, all others are optional.

Use the timeout property to set a timeout value in millseconds that changes the default EXP value of 60000ms (1 minute). Note that this can be applied at the test suite / test file level as well. If you set a timeout for the suite and also set a timeout for the test case then the test case value overrides the suite value for the test case. Caveat: if your test case blocks the Node.js event loop then timeout will not succeed and EXP will appear to hang. You can identify which test is responsible using EXP console output to see which test is running. Although EXP could be designed to not have this limitation, this would put considerable restrictions on how you can write tests. This way you have great flexibility and convenience around writing tests and only have to ensure you guard against event loop blocking.

Use the desc property to provide a description of the test case. You can explain what the test case does beyond the context the name provides. This can be useful for reporting and sharing results with others who may be unfamiliar with your test cases.

Usually the suite is defined at the test suite (test file) level. You can also define it at a test case level. If a suite is defined at both levels then the test case value overrides the suite value for the test case.

module.exports = {
  suite: "Test Suite A", 
  cases: [ {    {
      name: "Test 1",
      test: function () {
        console.log('Test 1 running');
      }
    },
    {
      name: "Test 2",
      suite: "Test Suite B",
      test: function () {
        console.log('Test 2 running');
      }
    }
} ] 
}

Add an unlimited number of custom fields to a test case by declaring properties with an underscore (_) prefix:

{
  name: "Test 1",
  test: function () {
    // test code 
    }
  }
  _Custom1: "Some value",
  _Custom2: "Another value"
}

Use test steps to gain additional detail beyond an overall test pass or fail. Each test step can have the same fields a test case can but generally you utilize the name and result properties.

{
  name: "Login",
  suite: "Authentication",
  desc: "Open up the app and login."
  _User: "Test User 1",
  steps: [
    {name: "Open app", result: "unknown"},
    {name: "Enter email", result: "unknown"},
    {name: "Enter password", result: "unknown"}
    {name: "Click login button", result: "unknown"}
    {name: "Confirm logged in", result: "unknown"}
  ],
  test: function () {
    const steps = module.exports.context.steps;
    let step = 0;
    // code to open app
    steps[step++].result = "pass";
    // code to enter email
    steps[step++].result = "pass";
    // code to enter password
    steps[step++].result = "pass";
    // code to click login button
    steps[step++].result = "pass";
    // code to confirm login
    steps[step++].result = "pass";
  }
}

Use paramsList to create parameterized test cases. In this example, the test case will be run twice, once with x = 0 and y = 1 and once with x = 1 and y = 2. Use params from context in the test function to access the parameters for the current test case.

{
  name: "Test with params",
  desc: "Demonstrating a parameterized test case"
  _custom: "Custom fields start with an underscore (_) and can have any string value.",
  paramsList: [{"x": "0", "y":"1"}, {"x": "1", "y":"2"}],
  test: function () {
    const params = module.exports.context.params;
    // Can now use params:
    exp.log("x: " + params.x + ", y: " + params.y);
  }
}

To retrieve params and steps, the examples above made use of context. The context is set by EXP as tests run and gives your test function access to properties of the currently running test case and test suite.

{
  name: "Test 1",
  desc: "Demonstrating context"
  test: function () {
    const context = module.exports.context;
    // Use context to access params, steps and all other test case properties:
    exp.log("Name of this test is: " + context.name);
  }
}

Synchronous Test

EXP will interpret test functions as synchronous and not wait for any asynchronous operations to complete if they have the format below. As with all test functions there are three possible outcomes: pass, fail, unknown. If the test function returns nothing (undefined) then the result is pass. If the function returns the string 'unknown' then the result is unknown. Anything else is a fail.

{
  name: "Test 1",
  test: function () {
    // return 'some error' 
  }
}

Asynchronous (Async/Await/Promise) Test

To have EXP wait for asynchronous operations to complete where you are using async/await or promises ensure you add the async keyword to your function declaration.

{
  name: "Test 1",
  test: async function () {
    const result = await someAsyncTask();
    // either return undefined for pass or return anything else for a fail
  }
}

Asynchronous (Callback) Test

To have EXP wait for asynchronous operations to complete where you are using a callback, have the function take a callback argument. The callback function must use the typical Node.js convention of having the first parameter be error (err) otherwise EXP may not interpret the result correctly.

{
  name: "Test 1",
  test: function (callback) {
    doSomething(function (err, result) {
      if (err) {
        callback(err);
      } else {
        callback();
      }
    });
  }
}

EXP Utility Functions

EXP provides two utility functions that make reporting test data easier and more powerful with Tesults.

Use the log function to have EXP create a log file for each test case and log the output to console.

const exp = require('exp-tf');

module.exports = {
  suite: "Test Suite A", 
  cases: [
    {
      name: "Test 1",
      test: function () {
        exp.log('Test 1 running');
      }
    }
  ] 
}

Use the file function to have EXP save the file and include it as part of the test case details information when reviewing results.

const exp = require('exp-tf');

module.exports = {
  suite: "Test Suite A", 
  cases: [
    {
      name: "Test 1",
      test: function () {
        const screenshot = takeScreenshot(); // takeScreenshot is an example
        exp.file(screenshot);
      }
    }
  ] 
}

Use wait in an async test function to wait without blocking the Node.js event loop. There can occassionally be a reason to do this in end-to-end tests but it should be avoided in favor of a wait threshold. Pass in time duration in milliseconds.

const exp = require('exp-tf');

module.exports = {
  suite: "Test Suite A", 
  cases: [
    {
      name: "Test 1",
      test: async function () {
        exp.log('Here');
        await exp.wait(10000);
        exp.log('... then here 10 seconds later');
      }
    }
  ] 
}

Setup - Advanced

The most basic way to setup and run EXP is by adding a test script to your package.json as demonstrated in quick setup:

npm install exp-tf --save

In your package.json file, in the scripts block add:

"scripts": {
  "test": "exp dir=/full/path/to/tests tesults-target=token"
}

The dir arg should be the full path to where you test files are located. You can provide multiple dir params (dir=path1 dir=path2 dir=path3) if you have tests in separate locations, but in most cases you can supply the parent directory and EXP will search all sub directories for test files. If tests you expect to be found are not being found it is likely due to an error with the test file, in particular check appropriate required modules can be loaded. The tesults-target arg is optional, provide it if you want to report results to Tesults, the token value is availble from your projects configuration menu in Tesults.

Run your tests using:

npm test

In many cases you have multiple test jobs (or tasks) and you can add multiple scripts to handle different tasks and name them however you want:

"scripts": {
  "test-job-1": "exp dir=/full/path/to/tests1 tesults-target=token1",
  "test-job-1": "exp dir=/full/path/to/tests2 tesults-target=token2",
  "test-job-1": "exp dir=/full/path/to/tests3 tesults-target=token3",
  "test-job-1": "exp dir=/full/path/to/tests3 tesults-target=token4"
}

Then run your tests using:

npm run test-job-1

You can also run jobs on the command line within your Node.js project using exp dir=[dir] tesults-target=[token]

exp dir=/full/path/to/tests tesults-target=token

You can also run EXP using the programmatic interface. In a JS file in your Node.js project:

const exp = require('exp-tf');

exp.tesults.enabled = true;
exp.tesults.target = 'token';
exp.dirs = ['/full/path/to/tests'];
exp.start();

Future

There is a lot planned for future updates including better specificiation of test files to run and support for other reporters besides Tesults. EXP is open source, with the MIT license. For any questions related to EXP please contact support at Tesults.