subunit: A streaming protocol for test results Copyright (C) 2005 Robert Collins This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA Subunit ------- Subunit is a streaming protocol for test results. The protocol is human readable and easily generated and parsed. By design all the components of the protocol conceptually fit into the xUnit TestCase->TestResult interaction. Subunit comes with command line filters to process a subunit stream and language bindings for python, C, C++ and shell. Bindings are easy to write for other languages. A number of useful things can be done easily with subunit: * Test aggregation: Tests run separately can be combined and then reported/displayed together. For instance, tests from different languages can be shown as a seamless whole. * Test archiving: A test run may be recorded and replayed later. * Test isolation: Tests that may crash or otherwise interact badly with each other can be run seperately and then aggregated, rather than interfering with each other. * Grid testing: subunit can act as the necessary serialisation and deserialiation to get test runs on distributed machines to be reported in real time. Subunit supplies the following filters: * tap2subunit - convert perl's TestAnythingProtocol to subunit. * subunit2pyunit - convert a subunit stream to pyunit test results. * subunit-filter - filter out tests from a subunit stream. * subunit-ls - list the tests present in a subunit stream. * subunit-stats - generate a summary of a subunit stream. * subunit-tags - add or remove tags from a stream. Integration with other tools ---------------------------- Subunit's language bindings act as integration with various test runners like 'check', 'cppunit', Python's 'unittest'. Beyond that a small amount of glue (typically a few lines) will allow Subunit to be used in more sophisticated ways. Python ====== As a TestResult, Subunit can translate method calls from a test run into a Subunit stream:: # Get a TestSuite or TestCase to run suite = make_suite() # Create a stream (any object with a 'write' method) stream = file('tests.log', 'wb') # Create a subunit result object which will output to the stream result = subunit.TestProtocolClient(stream) # Run the test suite reporting to the subunit result object suite.run(result) # Close the stream. stream.close() As a TestCase, subunit can read from a stream and inform a TestResult of the activity from the stream:: # Get a stream (any object with a readline() method), in this case example the # stream output by the example before. stream = file('tests.log', 'rb') # Create a subunit ProtocolTestCase which will read from the stream and emit # activity to a result when run() is called. suite = subunit.ProtocolTestCase(stream) # Create a result object to show the contents of the stream. result = unittest._TextTestResult(sys.stdout) # 'run' the tests - process the stream and feed its contents to result. suite.run(result) stream.close() Subunit has support for non-blocking usage too, for use with asyncore or Twisted. See the TestProtocolServer class for more details. Building on these foundations, Subunit also offers some convenience tools. The ``IsolatedTestSuite`` class is a decorator that will fork() before running the decorated item, and gather the results from the child process via subunit. This is useful for handlings tests that mutate global state, or are testing C extensions that could crash the VM. Similarly, ``IsolatedTestCase`` is a base class which can be subclassed to get tests that will fork() before the test is run. Finally, ``ExecTestCase`` is a convenience wrapper for running an external program to get a subunit stream and then report that back to an arbitrary result object:: class AggregateTests(subunit.ExecTestCase): def test_script_one(self): """./bin/script_one""" def test_script_two(self): """./bin/script_two""" # Normally your normal test loading would take of this automatically, # It is only spelt out in detail here for clarity. suite = unittest.TestSuite([AggregateTests("test_script_one"), AggregateTests("test_script_two")]) # Create any TestResult class you like. result = unittest._TextTestResult(sys.stdout) # And run your suite as normal, subunit will exec each external script as # needed and report to your result object. suite.run(result) C = Subunit has C bindings to emit the protocol, and comes with a patch for 'check' which has been nominally accepted by the 'check' developers. See 'c/README' for more details. C++ === C++ uses the C bindings and includes a patch for cppunit. See 'c++/README' for details. shell ===== Similar to C, the shell bindings consist of simple functions to output protocol elements, and a patch for adding subunit output to the 'ShUnit' shell test runner. See 'shell/README' for details. The protocol ------------ Sample subunit wire contents ---------------------------- The following:: test: test foo works success: test foo works. test: tar a file. failure: tar a file. [ .. ].. space is eaten. foo.c:34 WARNING foo is not defined. ] a writeln to stdout When run through subunit2pyunit:: .F a writeln to stdout ======================== FAILURE: tar a file. ------------------- .. ].. space is eaten. foo.c:34 WARNING foo is not defined. Subunit protocol description ============================ test|testing|test:|testing: test label success|success:|successful|successful: test label success|success:|successful|successful: test label [ ... ] failure test label failure: test label failure test label [ ... ] failure: test label [ ... ] error: test label error: test label [ ] skip[:] test label skip[:] test label [ ] xfail[:] test label xfail[:] test label [ ] tags: [-]TAG ... time: YYYY-MM-DD HH:MM:SSZ unexpected output on stdout -> stdout. exit w/0 or last test -> error Tags given outside a test are applied to all following tests Tags given after a test: line and before the result line for the same test apply only to that test, and inheric the current global tags. A '-' before a tag is used to remove tags - e.g. to prevent a global tag applying to a single test, or to cancel a global tag. In Python, tags are assigned to the .tags attribute on the RemoteTest objects created by the TestProtocolServer. The time element acts as a clock event - it sets the time for all future events. Currently this is not exposed at the python API layer. The skip result is used to indicate a test that was found by the runner but not fully executed due to some policy or dependency issue. This is represented in python using the addSkip interface that testtools (https://edge.launchpad.net/testtools) defines. When communicating with a non skip aware test result, the test is reported as an error. The xfail result is used to indicate a test that was expected to fail failing in the expected manner. As this is a normal condition for such tests it is represented as a successful test in Python. In future, skip and xfail results will be represented semantically in Python, but some discussion is underway on the right way to do this.