# vim: ft=rst
This directory contains test scripts that are useful for running a
-bunch of tests all at once.
+bunch of tests all at once.
+
+There are two parts to this:
+
+ * The test runner (selftest/selftest.pl)
+ * The test formatter
+
+selftest.pl simply outputs subunit, which can then be formatted or analyzed
+by tools that understand the subunit protocol. One of these tools is
+format-subunit, which is used by default as part of "make test".
Available testsuites
====================
-The available testsuites are obtained from a script, usually
-selftest/samba{3,4}_tests.sh. This script should for each testsuite output
-the name of the test, the command to run and the environment that should be
+The available testsuites are obtained from a script, usually
+source{3,4}/selftest/tests.py. This script should for each testsuite output
+the name of the test, the command to run and the environment that should be
provided. Use the included "plantest" function to generate the required output.
Testsuite behaviour
-================================
+===================
Exit code
------------
-The testsuites should exit with a non-zero exit code if at least one
+The testsuites should exit with a non-zero exit code if at least one
test failed. Skipped tests should not influence the exit code.
Output format
-------------
-Testsuites can simply use the exit code to indicate whether all of their
-tests have succeeded or one or more have failed. It is also possible to
-provide more granular information using the Subunit protocol.
+Testsuites can simply use the exit code to indicate whether all of their
+tests have succeeded or one or more have failed. It is also possible to
+provide more granular information using the Subunit protocol.
-This protocol works by writing simple messages to standard output. Any
-messages that can not be interpreted by this protocol are considered comments
+This protocol works by writing simple messages to standard output. Any
+messages that can not be interpreted by this protocol are considered comments
for the last announced test.
-Accepted commands are:
+For a full description of the subunit protocol, see the README file in the subunit
+repository at http://github.com/testing-cabal/subunit.
-test
-~~~~~~~~~~~~
-test: <NAME>
+The following commands are Samba extensions to Subunit:
-Announce that a new test with the specified name is starting
-
-success
+start-testsuite
~~~~~~~~~~~~~~~
-success: <NAME>
+start-testsuite: name
-Announce that the test with the specified name is done and ran successfully.
+The testsuite name is used as prefix for all containing tests.
-failure
-~~~~~~~~~~~~~~~
-failure: <NAME>
-failure: <NAME> [ REASON ]
+skip-testsuite
+~~~~~~~~~~~~~~
+skip-testsuite: name
-Announce that the test with the specified name failed. Optionally, it is
-possible to specify a reason it failed.
+Mark the testsuite with the specified name as skipped.
-skip
-~~~~~~~~~~~~
-skip: <NAME>
-skip: <NAME> [ REASON ]
+testsuite-success
+~~~~~~~~~~~~~~~~~
+testsuite-success: name
-Announce that the test with the specified name was skipped. Optionally a
-reason can be specified.
+Indicate that the testsuite has succeeded successfully.
-knownfail
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-knownfail: <NAME>
-knownfail: <NAME> [ REASON ]
+testsuite-fail
+~~~~~~~~~~~~~~
+testsuite-fail: name
-Announce that the test with the specified name was run and failed as expected.
-Alternatively it is also possible to simply return "failure:" here but
-specify in the samba4-knownfailures file that it is failing.
+Indicate that a testsuite has failed.
Environments
============
-Tests often need to run against a server with particular things set up,
-a "environment". This environment is provided by the test "target": Samba 3,
+Tests often need to run against a server with particular things set up,
+a "environment". This environment is provided by the test "target": Samba 3,
Samba 4 or Windows.
-The following environments are currently available:
+The environments are currently available include
- none: No server set up, no variables set.
- - dc: Domain controller set up. The following environment variables will
+ - dc,s3dc: Domain controller set up. The following environment variables will
be set:
* USERNAME: Administrator user name
- * PASSWORD: Administrator password
- * DOMAIN: Domain name
- * REALM: Realm name
- * SERVER: DC host name
- * SERVER_IP: DC IPv4 address
- * NETBIOSNAME: DC NetBIOS name
- * NETIOSALIAS: DC NetBIOS alias
-
- - member: Domain controller and member server that is joined to it set up. The
+ * PASSWORD: Administrator password
+ * DOMAIN: Domain name
+ * REALM: Realm name
+ * SERVER: DC host name
+ * SERVER_IP: DC IPv4 address
+ * SERVER_IPV6: DC IPv6 address
+ * NETBIOSNAME: DC NetBIOS name
+ * NETIOSALIAS: DC NetBIOS alias
+
+ - member,s4member,s3member: Domain controller and member server that is joined to it set up. The
following environment variables will be set:
* USERNAME: Domain administrator user name
- * PASSWORD: Domain administrator password
- * DOMAIN: Domain name
- * REALM: Realm name
- * SERVER: Name of the member server
+ * PASSWORD: Domain administrator password
+ * DOMAIN: Domain name
+ * REALM: Realm name
+ * SERVER: Name of the member server
+See Samba.pm, Samba3.pm and Samba4.pm for the full list.
Running tests
=============
make test
-To run a quick subset (aiming for about 1 minute of testing) run::
+To run a quicker subset run::
make quicktest
git.samba.org / kai / samba-autobuild / .git / blobdiff