Introduction
Duette supports importing results from QTP, RFT, Selenium (HTML format) and unit testing frameworks.
For unit test frameworks ET supports a number of different formats (Gallio, JSUnit,JUnit, Microsoft MSTest, Parasoft C++test and PHPUnit). Prior to being imported, ET transforms each of these formats into the JUnit XML results format (a popular Java unit test framework).
If you have bespoke unit testing data that you want to import into Enterprise Tester, you can create bespoke tooling to transform your results into JUnit XML format. Once your results are transformed, they can then be imported into Duette.
This page documents the JUnit XML format itself.
Basic Structure
Here is an example of a JUnit output file, showing a skip and failed result, as well as a single passed result.
<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
<testsuite name="JUnitXmlReporter" errors="0" tests="0" failures="0" time="0" timestamp="2013-05-24T10:23:58" />
<testsuite name="JUnitXmlReporter.constructor" errors="0" skipped="1" tests="3" failures="1" time="0.006" timestamp="2013-05-24T10:23:58">
<properties>
<property name="java.vendor" value="Sun Microsystems Inc." />
<property name="compiler.debug" value="on" />
<property name="project.jdk.classpath" value="jdk.classpath.1.6" />
</properties>
<testcase classname="JUnitXmlReporter.constructor" name="should default path to an empty string" time="0.006">
<failure message="test failure">Assertion failed</failure>
</testcase>
<testcase classname="JUnitXmlReporter.constructor" name="should default consolidate to true" time="0">
<skipped />
</testcase>
<testcase classname="JUnitXmlReporter.constructor" name="should default useDotNotation to true" time="0" />
</testsuite>
</testsuites>
Below is the documented structure of a typical JUnit XML report. Notice that a report can contain 1 or more test suite. Each test suite has a set of properties (recording environment information). Each test suite also contains 1 or more test case and each test case will either contain a skipped, failure or error node if the test did not pass. If the test case has passed, then it will not contain any nodes. For more details of which attributes are valid for each node please consult the following section "Schema".
<testsuites> => the aggregated result of all junit testfiles
<testsuite> => the output from a single TestSuite
<properties> => the defined properties at test execution
<property> => name/value pair for a single property
...
</properties>
<error></error> => optional information, in place of a test case - normally if the tests in the suite could not be found etc.
<testcase> => the results from executing a test method
<system-out> => data written to System.out during the test run
<system-err> => data written to System.err during the test run
<skipped/> => test was skipped
<failure> => test failed
<error> => test encountered an error
</testcase>
...
</testsuite>
...
</testsuites>
Schema
The JUnit XML Report output comes from a build tool called Nant, as opposed to the JUnit project itself - thus it can be a little tricky to nail down an official spec for the format, even though it's widely adopted and used. There have been a number of attempts to codify the schema, first off there is an XSD for JUnit:
XSD
<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" elementFormDefault="qualified" attributeFormDefault="unqualified">
<xs:annotation>
<xs:documentation xml:lang="en">JUnit test result schema for the Apache Ant JUnit and JUnitReport tasks
Copyright © 2011, Windy Road Technology Pty. Limited
The Apache Ant JUnit XML Schema is distributed under the terms of the GNU Lesser General Public License (LGPL) http://www.gnu.org/licenses/lgpl.html
Permission to waive conditions of this license may be requested from Windy Road Support (http://windyroad.org/support).</xs:documentation>
</xs:annotation>
<xs:element name="testsuite" type="testsuite" />
<xs:simpleType name="ISO8601_DATETIME_PATTERN">
<xs:restriction base="xs:dateTime">
<xs:pattern value="[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}" />
</xs:restriction>
</xs:simpleType>
<xs:element name="testsuites">
<xs:annotation>
<xs:documentation xml:lang="en">Contains an aggregation of testsuite results</xs:documentation>
</xs:annotation>
<xs:complexType>
<xs:sequence>
<xs:element name="testsuite" minOccurs="0" maxOccurs="unbounded">
<xs:complexType>
<xs:complexContent>
<xs:extension base="testsuite">
<xs:attribute name="package" type="xs:token" use="required">
<xs:annotation>
<xs:documentation xml:lang="en">Derived from testsuite/@name in the non-aggregated documents</xs:documentation>
</xs:annotation>
</xs:attribute>
<xs:attribute name="id" type="xs:int" use="required">
<xs:annotation>
<xs:documentation xml:lang="en">Starts at '0' for the first testsuite and is incremented by 1 for each following testsuite</xs:documentation>
</xs:annotation>
</xs:attribute>
</xs:extension>
</xs:complexContent>
</xs:complexType>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
<xs:complexType name="testsuite">
<xs:annotation>
<xs:documentation xml:lang="en">Contains the results of exexuting a testsuite</xs:documentation>
</xs:annotation>
<xs:sequence>
<xs:element name="properties">
<xs:annotation>
<xs:documentation xml:lang="en">Properties (e.g., environment settings) set during test execution</xs:documentation>
</xs:annotation>
<xs:complexType>
<xs:sequence>
<xs:element name="property" minOccurs="0" maxOccurs="unbounded">
<xs:complexType>
<xs:attribute name="name" use="required">
<xs:simpleType>
<xs:restriction base="xs:token">
<xs:minLength value="1" />
</xs:restriction>
</xs:simpleType>
</xs:attribute>
<xs:attribute name="value" type="xs:string" use="required" />
</xs:complexType>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
<xs:element name="testcase" minOccurs="0" maxOccurs="unbounded">
<xs:complexType>
<xs:choice minOccurs="0">
<xs:element name="error">
<xs:annotation>
<xs:documentation xml:lang="en">Indicates that the test errored. An errored test is one that had an unanticipated problem. e.g., an unchecked throwable; or a problem with the implementation of the test. Contains as a text node relevant data for the error, e.g., a stack trace</xs:documentation>
</xs:annotation>
<xs:complexType>
<xs:simpleContent>
<xs:extension base="pre-string">
<xs:attribute name="message" type="xs:string">
<xs:annotation>
<xs:documentation xml:lang="en">The error message. e.g., if a java exception is thrown, the return value of getMessage()</xs:documentation>
</xs:annotation>
</xs:attribute>
<xs:attribute name="type" type="xs:string" use="required">
<xs:annotation>
<xs:documentation xml:lang="en">The type of error that occured. e.g., if a java execption is thrown the full class name of the exception.</xs:documentation>
</xs:annotation>
</xs:attribute>
</xs:extension>
</xs:simpleContent>
</xs:complexType>
</xs:element>
<xs:element name="failure">
<xs:annotation>
<xs:documentation xml:lang="en">Indicates that the test failed. A failure is a test which the code has explicitly failed by using the mechanisms for that purpose. e.g., via an assertEquals. Contains as a text node relevant data for the failure, e.g., a stack trace</xs:documentation>
</xs:annotation>
<xs:complexType>
<xs:simpleContent>
<xs:extension base="pre-string">
<xs:attribute name="message" type="xs:string">
<xs:annotation>
<xs:documentation xml:lang="en">The message specified in the assert</xs:documentation>
</xs:annotation>
</xs:attribute>
<xs:attribute name="type" type="xs:string" use="required">
<xs:annotation>
<xs:documentation xml:lang="en">The type of the assert.</xs:documentation>
</xs:annotation>
</xs:attribute>
</xs:extension>
</xs:simpleContent>
</xs:complexType>
</xs:element>
</xs:choice>
<xs:attribute name="name" type="xs:token" use="required">
<xs:annotation>
<xs:documentation xml:lang="en">Name of the test method</xs:documentation>
</xs:annotation>
</xs:attribute>
<xs:attribute name="classname" type="xs:token" use="required">
<xs:annotation>
<xs:documentation xml:lang="en">Full class name for the class the test method is in.</xs:documentation>
</xs:annotation>
</xs:attribute>
<xs:attribute name="time" type="xs:decimal" use="required">
<xs:annotation>
<xs:documentation xml:lang="en">Time taken (in seconds) to execute the test</xs:documentation>
</xs:annotation>
</xs:attribute>
</xs:complexType>
</xs:element>
<xs:element name="system-out">
<xs:annotation>
<xs:documentation xml:lang="en">Data that was written to standard out while the test was executed</xs:documentation>
</xs:annotation>
<xs:simpleType>
<xs:restriction base="pre-string">
<xs:whiteSpace value="preserve" />
</xs:restriction>
</xs:simpleType>
</xs:element>
<xs:element name="system-err">
<xs:annotation>
<xs:documentation xml:lang="en">Data that was written to standard error while the test was executed</xs:documentation>
</xs:annotation>
<xs:simpleType>
<xs:restriction base="pre-string">
<xs:whiteSpace value="preserve" />
</xs:restriction>
</xs:simpleType>
</xs:element>
</xs:sequence>
<xs:attribute name="name" use="required">
<xs:annotation>
<xs:documentation xml:lang="en">Full class name of the test for non-aggregated testsuite documents. Class name without the package for aggregated testsuites documents</xs:documentation>
</xs:annotation>
<xs:simpleType>
<xs:restriction base="xs:token">
<xs:minLength value="1" />
</xs:restriction>
</xs:simpleType>
</xs:attribute>
<xs:attribute name="timestamp" type="ISO8601_DATETIME_PATTERN" use="required">
<xs:annotation>
<xs:documentation xml:lang="en">when the test was executed. Timezone may not be specified.</xs:documentation>
</xs:annotation>
</xs:attribute>
<xs:attribute name="hostname" use="required">
<xs:annotation>
<xs:documentation xml:lang="en">Host on which the tests were executed. 'localhost' should be used if the hostname cannot be determined.</xs:documentation>
</xs:annotation>
<xs:simpleType>
<xs:restriction base="xs:token">
<xs:minLength value="1" />
</xs:restriction>
</xs:simpleType>
</xs:attribute>
<xs:attribute name="tests" type="xs:int" use="required">
<xs:annotation>
<xs:documentation xml:lang="en">The total number of tests in the suite</xs:documentation>
</xs:annotation>
</xs:attribute>
<xs:attribute name="failures" type="xs:int" use="required">
<xs:annotation>
<xs:documentation xml:lang="en">The total number of tests in the suite that failed. A failure is a test which the code has explicitly failed by using the mechanisms for that purpose. e.g., via an assertEquals</xs:documentation>
</xs:annotation>
</xs:attribute>
<xs:attribute name="errors" type="xs:int" use="required">
<xs:annotation>
<xs:documentation xml:lang="en">The total number of tests in the suite that errorrd. An errored test is one that had an unanticipated problem. e.g., an unchecked throwable; or a problem with the implementation of the test.</xs:documentation>
</xs:annotation>
</xs:attribute>
<xs:attribute name="time" type="xs:decimal" use="required">
<xs:annotation>
<xs:documentation xml:lang="en">Time taken (in seconds) to execute the tests in the suite</xs:documentation>
</xs:annotation>
</xs:attribute>
</xs:complexType>
<xs:simpleType name="pre-string">
<xs:restriction base="xs:string">
<xs:whiteSpace value="preserve" />
</xs:restriction>
</xs:simpleType>
</xs:schema>
Relax NG Compact Syntax
There is also a Relax NG Compact Syntax Schema:
junit.rnc:
#----------------------------------------------------------------------------------
start = testsuite
property = element property {
attribute name {text},
attribute value {text}
}
properties = element properties {
property*
}
failure = element failure {
attribute message {text},
attribute type {text},
text
}
testcase = element testcase {
attribute classname {text},
attribute name {text},
attribute time {text},
failure?
}
testsuite = element testsuite {
attribute errors {xsd:integer},
attribute failures {xsd:integer},
attribute hostname {text},
attribute name {text},
attribute tests {xsd:integer},
attribute time {xsd:double},
attribute timestamp {xsd:dateTime},
properties,
testcase*,
element system-out {text},
element system-err {text}
}
#----------------------------------------------------------------------------------
and junitreport.rnc
#----------------------------------------------------------------------------------
include "junit.rnc" {
start = testsuites
testsuite = element testsuite {
attribute errors {xsd:integer},
attribute failures {xsd:integer},
attribute hostname {text},
attribute name {text},
attribute tests {xsd:integer},
attribute time {xsd:double},
attribute timestamp {xsd:dateTime},
attribute id {text},
attribute package {text},
properties,
testcase*,
element system-out {text},
element system-err {text}
}
}
testsuites = element testsuites {
testsuite*
}
Sourcecode
The JUnit XML report format originates the JUnit ANT task - this is the definitive source for the JUnit Report XML format - and source code can be found on the apache SVN repository here:
This can be useful if attempting to do anything "tricky" with JUnit output where you need to confirm it's compliant.
Stack Overflow Discussions
Here are some additional topics on stack overflow that discuss the JUnit XML format which can be useful for learning about the schema for the JUnit XML results format, as well to get hints about what minimum subset is normally suitable for most tools such as Jenkins (the same rules apply to ET generally as well).
- http://stackoverflow.com/questions/442556/spec-for-junit-xml-output
- http://stackoverflow.com/questions/428553/unable-to-get-hudson-to-parse-junit-test-output-xml
- http://stackoverflow.com/questions/411218/hudson-c-and-unittest
- http://stackoverflow.com/questions/4922867/junit-xml-format-specification-that-hudson-supports
- http://stackoverflow.com/questions/11530955/about-jenkins-junit-xml-format