s4-rootdse: improved operations error messages

[amitay/samba.git] / selftest / README

diff --git a/selftest/README b/selftest/README
index 83a3e6fd491164608c718821e132a1954847ea3a..752b07bf640d91c2071b4c9c78b0d731734a8875 100644 (file)
--- a/selftest/README
+++ b/selftest/README
@@ -3,10 +3,19 @@
 This directory contains test scripts that are useful for running a
 bunch of tests all at once. 
 
+There are two parts to this: 
+
+ * The test runner (selftest/selftest.pl)
+ * The test formatter
+
+selftest.pl simply outputs subunit, which can then be formatted or analyzed 
+by tools that understand the subunit protocol. One of these tools is 
+format-subunit, which is used by default as part of "make test".
+
 Available testsuites
 ====================
 The available testsuites are obtained from a script, usually 
-selftest/samba{3,4}_tests.sh. This script should for each testsuite output 
+source{3,4}/selftest/tests.sh. This script should for each testsuite output 
 the name of the test, the command to run and the environment that should be 
 provided. Use the included "plantest" function to generate the required output.
 
@@ -28,61 +37,39 @@ This protocol works by writing simple messages to standard output. Any
 messages that can not be interpreted by this protocol are considered comments 
 for the last announced test.
 
-Accepted commands are:
-
-test
-~~~~
-test: <NAME>
-
-Announce that a new test with the specified name is starting
-
-success
-~~~~~~~
-success: <NAME> 
-
-Announce that the test with the specified name is done and ran successfully.
+For a full description of the subunit protocol, see ../lib/subunit/README.
 
-failure
-~~~~~~~
-failure: <NAME> 
-failure: <NAME> [ REASON ]
+The following commands are Samba extensions to Subunit:
 
-Announce that the test with the specified name failed. Optionally, it is 
-possible to specify a reason it failed.
+testsuite-count
+~~~~~~~~~~~~~~~
+testsuite-count: number
 
-The alias "fail" will also work.
+Announce the number of tests that is going to be run.
 
-xfail
-~~~~~
-xfail: <NAME>
-xfail: <NAME> [ REASON ]
+start-testsuite
+~~~~~~~~~~~~~~~
+start-testsuite: name
 
-Announce that the test with the specified name failed but that the failure
-was expected, e.g. it's a test for a known bug that hasn't been fixed yet.
-Alternatively it is also possible to simply return "failure:" here but 
-specify in the samba4-knownfailures file that it is failing.
+The testsuite name is used as prefix for all containing tests.
 
-skip
-~~~~
-skip: <NAME>
-skip: <NAME> [ REASON ]
+skip-testsuite
+~~~~~~~~~~~~~~
+skip-testsuite: name
 
-Announce that the test with the specified name was skipped. Optionally a 
-reason can be specified.
+Mark the testsuite with the specified name as skipped.
 
-time
-~~~~
-time: YYYY-MM-DD HH:mm:ssZ
+testsuite-success
+~~~~~~~~~~~~~~~~~
+testsuite-success: name
 
-Announce the current time. This may be used to calculate the duration of 
-various tests.
+Indicate that the testsuite has succeeded successfully.
 
-prefix
-~~~~~~
-prefix: Prefix
+testsuite-fail
+~~~~~~~~~~~~~~
+testsuite-fail: name
 
-Change the global prefix for test names. To unset the prefix, specify an empty 
-string.
+Indicate that a testsuite has failed.
 
 Environments
 ============