+++ /dev/null
-
- subunit: A streaming protocol for test results
- Copyright (C) 2005-2009 Robert Collins <robertc@robertcollins.net>
-
- Licensed under either the Apache License, Version 2.0 or the BSD 3-clause
- license at the users choice. A copy of both licenses are available in the
- project source as Apache-2.0 and BSD. You may not use this file except in
- compliance with one of these two licences.
-
- Unless required by applicable law or agreed to in writing, software
- distributed under these licenses is distributed on an "AS IS" BASIS, WITHOUT
- WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
- license you chose for the specific language governing permissions and
- limitations under that license.
-
- See the COPYING file for full details on the licensing of Subunit.
-
- subunit reuses iso8601 by Michael Twomey, distributed under an MIT style
- licence - see python/iso8601/LICENSE for details.
-
-Subunit
--------
-
-Subunit is a streaming protocol for test results. The protocol is human
-readable and easily generated and parsed. By design all the components of
-the protocol conceptually fit into the xUnit TestCase->TestResult interaction.
-
-Subunit comes with command line filters to process a subunit stream and
-language bindings for python, C, C++ and shell. Bindings are easy to write
-for other languages.
-
-A number of useful things can be done easily with subunit:
- * Test aggregation: Tests run separately can be combined and then
- reported/displayed together. For instance, tests from different languages
- can be shown as a seamless whole.
- * Test archiving: A test run may be recorded and replayed later.
- * Test isolation: Tests that may crash or otherwise interact badly with each
- other can be run separately and then aggregated, rather than interfering
- with each other.
- * Grid testing: subunit can act as the necessary serialisation and
- deserialiation to get test runs on distributed machines to be reported in
- real time.
-
-Subunit supplies the following filters:
- * tap2subunit - convert perl's TestAnythingProtocol to subunit.
- * subunit2csv - convert a subunit stream to csv.
- * subunit2pyunit - convert a subunit stream to pyunit test results.
- * subunit2gtk - show a subunit stream in GTK.
- * subunit2junitxml - convert a subunit stream to JUnit's XML format.
- * subunit-diff - compare two subunit streams.
- * subunit-filter - filter out tests from a subunit stream.
- * subunit-ls - list info about tests present in a subunit stream.
- * subunit-stats - generate a summary of a subunit stream.
- * subunit-tags - add or remove tags from a stream.
-
-Integration with other tools
-----------------------------
-
-Subunit's language bindings act as integration with various test runners like
-'check', 'cppunit', Python's 'unittest'. Beyond that a small amount of glue
-(typically a few lines) will allow Subunit to be used in more sophisticated
-ways.
-
-Python
-======
-
-Subunit has excellent Python support: most of the filters and tools are written
-in python and there are facilities for using Subunit to increase test isolation
-seamlessly within a test suite.
-
-One simple way to run an existing python test suite and have it output subunit
-is the module ``subunit.run``::
-
- $ python -m subunit.run mypackage.tests.test_suite
-
-For more information on the Python support Subunit offers , please see
-``pydoc subunit``, or the source in ``python/subunit/__init__.py``
-
-C
-=
-
-Subunit has C bindings to emit the protocol, and comes with a patch for 'check'
-which has been nominally accepted by the 'check' developers. See 'c/README' for
-more details.
-
-C++
-===
-
-The C library is includable and usable directly from C++. A TestListener for
-CPPUnit is included in the Subunit distribution. See 'c++/README' for details.
-
-shell
-=====
-
-Similar to C, the shell bindings consist of simple functions to output protocol
-elements, and a patch for adding subunit output to the 'ShUnit' shell test
-runner. See 'shell/README' for details.
-
-Filter recipes
---------------
-
-To ignore some failing tests whose root cause is already known::
-
- subunit-filter --without 'AttributeError.*flavor'
-
-
-The protocol
-------------
-
-Sample subunit wire contents
-----------------------------
-
-The following::
- test: test foo works
- success: test foo works.
- test: tar a file.
- failure: tar a file. [
- ..
- ].. space is eaten.
- foo.c:34 WARNING foo is not defined.
- ]
- a writeln to stdout
-
-When run through subunit2pyunit::
- .F
- a writeln to stdout
-
- ========================
- FAILURE: tar a file.
- -------------------
- ..
- ].. space is eaten.
- foo.c:34 WARNING foo is not defined.
-
-
-Subunit protocol description
-============================
-
-This description is being ported to an EBNF style. Currently its only partly in
-that style, but should be fairly clear all the same. When in doubt, refer the
-source (and ideally help fix up the description!). Generally the protocol is
-line orientated and consists of either directives and their parameters, or
-when outside a DETAILS region unexpected lines which are not interpreted by
-the parser - they should be forwarded unaltered.
-
-test|testing|test:|testing: test LABEL
-success|success:|successful|successful: test LABEL
-success|success:|successful|successful: test LABEL DETAILS
-failure: test LABEL
-failure: test LABEL DETAILS
-error: test LABEL
-error: test LABEL DETAILS
-skip[:] test LABEL
-skip[:] test LABEL DETAILS
-xfail[:] test LABEL
-xfail[:] test LABEL DETAILS
-uxsuccess[:] test LABEL
-uxsuccess[:] test LABEL DETAILS
-progress: [+|-]X
-progress: push
-progress: pop
-tags: [-]TAG ...
-time: YYYY-MM-DD HH:MM:SSZ
-
-LABEL: UTF8*
-DETAILS ::= BRACKETED | MULTIPART
-BRACKETED ::= '[' CR UTF8-lines ']' CR
-MULTIPART ::= '[ multipart' CR PART* ']' CR
-PART ::= PART_TYPE CR NAME CR PART_BYTES CR
-PART_TYPE ::= Content-Type: type/sub-type(;parameter=value,parameter=value)
-PART_BYTES ::= (DIGITS CR LF BYTE{DIGITS})* '0' CR LF
-
-unexpected output on stdout -> stdout.
-exit w/0 or last test completing -> error
-
-Tags given outside a test are applied to all following tests
-Tags given after a test: line and before the result line for the same test
-apply only to that test, and inherit the current global tags.
-A '-' before a tag is used to remove tags - e.g. to prevent a global tag
-applying to a single test, or to cancel a global tag.
-
-The progress directive is used to provide progress information about a stream
-so that stream consumer can provide completion estimates, progress bars and so
-on. Stream generators that know how many tests will be present in the stream
-should output "progress: COUNT". Stream filters that add tests should output
-"progress: +COUNT", and those that remove tests should output
-"progress: -COUNT". An absolute count should reset the progress indicators in
-use - it indicates that two separate streams from different generators have
-been trivially concatenated together, and there is no knowledge of how many
-more complete streams are incoming. Smart concatenation could scan each stream
-for their count and sum them, or alternatively translate absolute counts into
-relative counts inline. It is recommended that outputters avoid absolute counts
-unless necessary. The push and pop directives are used to provide local regions
-for progress reporting. This fits with hierarchically operating test
-environments - such as those that organise tests into suites - the top-most
-runner can report on the number of suites, and each suite surround its output
-with a (push, pop) pair. Interpreters should interpret a pop as also advancing
-the progress of the restored level by one step. Encountering progress
-directives between the start and end of a test pair indicates that a previous
-test was interrupted and did not cleanly terminate: it should be implicitly
-closed with an error (the same as when a stream ends with no closing test
-directive for the most recently started test).
-
-The time directive acts as a clock event - it sets the time for all future
-events. The value should be a valid ISO8601 time.
-
-The skip, xfail and uxsuccess outcomes are not supported by all testing
-environments. In Python the testttools (https://launchpad.net/testtools)
-library is used to translate these automatically if an older Python version
-that does not support them is in use. See the testtools documentation for the
-translation policy.
-
-skip is used to indicate a test was discovered but not executed. xfail is used
-to indicate a test that errored in some expected fashion (also know as "TODO"
-tests in some frameworks). uxsuccess is used to indicate and unexpected success
-where a test though to be failing actually passes. It is complementary to
-xfail.
-
-Hacking on subunit
-------------------
-
-Releases
-========
-
-* Update versions in configure.ac and python/subunit/__init__.py.
-* Make PyPI and regular tarball releases. Upload the regular one to LP, the
- PyPI one to PyPI.
-* Push a tagged commit.
-
git.samba.org / samba.git / blobdiff