Tesults JSON data standard
The Tesults JSON data standard defines formatting for transmitting test results data over a network. The format is intended to be utilized bi-directionally for both recording data and for retrieval for consumption.
Results
The format represents test results data for a single test run. The results field is the only top-level entry consisting of an object with one field named cases. The value for cases is an array containing test cases and their results. There can be any number of test cases within a test run and all of them are contained within the cases array.
{
"results": { "cases": [] }
}
Test cases
Each test case object contained in the cases array has two required fields, name and result:
{
"name": "Test name",
"result": "pass"
}
This is valid results data containing two test cases:
{
"results": {
"cases": [
{
"name": "Test 1",
"result": "pass"
},
{
"name": "Test 2",
"result": "fail"
}
]
}
}
Test case fields
All fields besides name and result for a test case are optional, including an unlimited number of custom fields. This object shows a complete test case data object using every possible field and is followed by a section detailing each field:
{
"name": "Test name",
"result": "pass",
"suite": "Test suite",
"desc": "Test description",
"reason": "Test failure reason",
"params": { "Test param 1": "Value 1", "Test param 2": "Value 2", ... },
"files": ["/full/path/to/file", "/full/path/screenshot.png", "/full/path/test.log"],
"steps": [{"name": "Step 1", "result": "pass"}, {"name": "Step 2", "result": "pass"}],
"start": <milliseconds since Unix epoch>,
"end": <milliseconds since Unix epoch>,
"duration": <milliseconds>,
"rawResult": "Raw result value to use with a result interpretation map",
"_Custom": "A custom field. Any field starting with an underscore is a custom field.",
}
name
The name of the test case. In the context of a test framework this is the name of a single test case function or scenario.
result ["pass", "fail", "unknown"]
The result of the test case. The value for this must be one of “pass”, “fail”, or “unknown”. Any other result status can be supplied using the optional rawResult field, detailed below, but in the case of the result field mapping to one of the "pass", "fail" or "unknown" status standardizes the outcome of a test from a range of test frameworks.
rawResult
Any result status value. Raw here refers to the internal test framework status value prior to being mapped to one of "pass", "fail" or "unknown" for the result field described above. As an example some test cases use the status "passed" or "success" to indicate a passing result. In this case the value for result would be "pass" and the rawResult would be "passed" or "success" depending on the test framework. Similarly a test case may use "failed" or "failure" for a failing result, in this case the value for result would be "fail" and rawResult would be "failed" or "failure" depending on the test framework. Similarly, a range of values can be mapped to an "unknown" result including rawResult values such as "skipped" and "skip".
suite
The suite the test case belongs to. In the context of a test framework this is the name of the parent container of a group of test cases.
desc
A description of a single test case. Used to provide more information about a test case and what it does beyond the name.
reason
Test case failure reason, such as the failing assertion output or stack trace.
params
For test frameworks supporting parameterized tests where the name of the test does not change when running with different parameters, the params field can be used to distinguish parameterized test case runs. For example, a test case that is run with two parameters that change change, x, y, can be specified as a params value: {x: "x value", y: "y value"}.
files
An array of files that were generated by the test case. Each value is a string to the full path of the file on the host system running the test case. Tesults language libraries supporting files will automatically upload files if specified.
start
The time the test case started, represented as a number in milliseconds since Unix epoch.
end
The time the test case ended, represented as a number in milliseconds since Unix epoch.
duration
The duration of the test case in milliseconds. If start and end are supplied, this is superflous.
_Custom
Any field beginning with an underscore character is a custom field. Test framework authors can add any number of custom fields.
Test framework authors can support test results transmission to Tesults easily by formatting test results data in the Tesults JSON data standard. After the data is the correct format only a single function call is required to upload results. The language library documentation explains how to do this for each language.
Tesults expects test results data in the Tesults JSON data standard as described above, with an added target field shown below. The token string is generated from the UI or from the Tesults API.
{
"target": "token",
"results": { "cases": [] }
}
The Tesults API makes it possible to retrieve test results data too and the data format in the response matches the test cases fields defined in the Tesults JSON data standard. See the /results API documentation for more details.
The Tesults team is here to support integration developers. Contact help@tesults.com for answers to any questions you may have.