# vim: ft=rst
This directory contains test scripts that are useful for running a
-bunch of tests all at once.
+bunch of tests all at once.
-There are two parts to this:
+There are two parts to this:
* The test runner (selftest/selftest.pl)
* The test formatter
-selftest.pl simply outputs subunit, which can then be formatted or analyzed
-by tools that understand the subunit protocol. One of these tools is
-format-subunit.pl, which is used by default as part of "make test".
+selftest.pl simply outputs subunit, which can then be formatted or analyzed
+by tools that understand the subunit protocol. One of these tools is
+format-subunit, which is used by default as part of "make test".
Available testsuites
====================
-The available testsuites are obtained from a script, usually
-source{3,4}/selftest/tests.sh. This script should for each testsuite output
-the name of the test, the command to run and the environment that should be
+The available testsuites are obtained from a script, usually
+source{3,4}/selftest/tests.py. This script should for each testsuite output
+the name of the test, the command to run and the environment that should be
provided. Use the included "plantest" function to generate the required output.
Testsuite behaviour
Exit code
------------
-The testsuites should exit with a non-zero exit code if at least one
+The testsuites should exit with a non-zero exit code if at least one
test failed. Skipped tests should not influence the exit code.
Output format
-------------
-Testsuites can simply use the exit code to indicate whether all of their
-tests have succeeded or one or more have failed. It is also possible to
-provide more granular information using the Subunit protocol.
+Testsuites can simply use the exit code to indicate whether all of their
+tests have succeeded or one or more have failed. It is also possible to
+provide more granular information using the Subunit protocol.
-This protocol works by writing simple messages to standard output. Any
-messages that can not be interpreted by this protocol are considered comments
+This protocol works by writing simple messages to standard output. Any
+messages that can not be interpreted by this protocol are considered comments
for the last announced test.
-Accepted commands are:
+For a full description of the subunit protocol, see the README file in the subunit
+repository at http://github.com/testing-cabal/subunit.
-test
-~~~~
-test: <NAME>
-
-Announce that a new test with the specified name is starting
-
-success
-~~~~~~~
-success: <NAME>
-
-Announce that the test with the specified name is done and ran successfully.
-
-failure
-~~~~~~~
-failure: <NAME>
-failure: <NAME> [ REASON ]
-
-Announce that the test with the specified name failed. Optionally, it is
-possible to specify a reason it failed.
-
-The alias "fail" will also work.
-
-xfail
-~~~~~
-xfail: <NAME>
-xfail: <NAME> [ REASON ]
-
-Announce that the test with the specified name failed but that the failure
-was expected, e.g. it's a test for a known bug that hasn't been fixed yet.
-Alternatively it is also possible to simply return "failure:" here but
-specify in the samba4-knownfailures file that it is failing.
-
-skip
-~~~~
-skip: <NAME>
-skip: <NAME> [ REASON ]
-
-Announce that the test with the specified name was skipped. Optionally a
-reason can be specified.
-
-time
-~~~~
-time: YYYY-MM-DD HH:mm:ssZ
-
-Announce the current time. This may be used to calculate the duration of
-various tests.
-
-The following are Samba extensions to Subunit:
-
-testsuite-count
-~~~~~~~~~~~~~~~
-testsuite-count: number
-
-Announce the number of tests that is going to be run.
+The following commands are Samba extensions to Subunit:
start-testsuite
-~~~~~~~~~
+~~~~~~~~~~~~~~~
start-testsuite: name
The testsuite name is used as prefix for all containing tests.
Environments
============
-Tests often need to run against a server with particular things set up,
-a "environment". This environment is provided by the test "target": Samba 3,
+Tests often need to run against a server with particular things set up,
+a "environment". This environment is provided by the test "target": Samba 3,
Samba 4 or Windows.
-The following environments are currently available:
+The environments are currently available include
- none: No server set up, no variables set.
- - dc: Domain controller set up. The following environment variables will
+ - dc,s3dc: Domain controller set up. The following environment variables will
be set:
* USERNAME: Administrator user name
- * PASSWORD: Administrator password
- * DOMAIN: Domain name
- * REALM: Realm name
- * SERVER: DC host name
- * SERVER_IP: DC IPv4 address
- * NETBIOSNAME: DC NetBIOS name
- * NETIOSALIAS: DC NetBIOS alias
-
- - member: Domain controller and member server that is joined to it set up. The
+ * PASSWORD: Administrator password
+ * DOMAIN: Domain name
+ * REALM: Realm name
+ * SERVER: DC host name
+ * SERVER_IP: DC IPv4 address
+ * SERVER_IPV6: DC IPv6 address
+ * NETBIOSNAME: DC NetBIOS name
+ * NETIOSALIAS: DC NetBIOS alias
+
+ - member,s4member,s3member: Domain controller and member server that is joined to it set up. The
following environment variables will be set:
* USERNAME: Domain administrator user name
- * PASSWORD: Domain administrator password
- * DOMAIN: Domain name
- * REALM: Realm name
- * SERVER: Name of the member server
+ * PASSWORD: Domain administrator password
+ * DOMAIN: Domain name
+ * REALM: Realm name
+ * SERVER: Name of the member server
+See Samba.pm, Samba3.pm and Samba4.pm for the full list.
Running tests
=============
make test
-To run a quick subset (aiming for about 1 minute of testing) run::
+To run a quicker subset run::
make quicktest