Martin Schwenke [Mon, 8 Oct 2018 02:41:39 +0000 (13:41 +1100)]
ctdb-tests: Drop tests that only start and stop daemons
Integration test cases now start and shut down the cluster.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Martin Schwenke [Tue, 9 Oct 2018 04:09:38 +0000 (15:09 +1100)]
ctdb-tests: Move enabling of event scripts to setup_ctdb()
This is for the real cluster case.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Martin Schwenke [Mon, 8 Oct 2018 02:39:30 +0000 (13:39 +1100)]
ctdb-tests: Improve signal handling trap
Interrupting a test run currently moves on to the next test. It
should exit.
Follow the practice of exiting with 128 + signal number.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Martin Schwenke [Mon, 8 Oct 2018 02:38:10 +0000 (13:38 +1100)]
ctdb-tests: Drop cleanup_handler()
Running testsuite-specific code here isn't a good option.
Daemons are now shut down in ctdb_test_exit(), even when testing is
interrupted.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Martin Schwenke [Mon, 8 Oct 2018 02:30:55 +0000 (13:30 +1100)]
ctdb-tests: Start daemons in ctdb_test_init(), stop them in ctdb_test_exit()
This makes tests self-contained. They can also now be individually
looped, if necessary.
Most tests (all but 1 complex, more than 50% of simple) restart the
daemons anyway, so this simplification is worth it.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Martin Schwenke [Fri, 2 Nov 2018 05:21:26 +0000 (16:21 +1100)]
ctdb-tests: Ignore SIGPIPE during simple test cleanup
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Martin Schwenke [Mon, 8 Oct 2018 02:55:08 +0000 (13:55 +1100)]
ctdb-tests: Drop setting of unused variable scriptname
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Martin Schwenke [Mon, 8 Oct 2018 01:59:33 +0000 (12:59 +1100)]
ctdb-tests: Drop use of confusing testfailures variable
Exit on first test failure instead of setting a variable. The bizarre
logic in ctdb_test_exit() makes this worth dropping.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Martin Schwenke [Mon, 8 Oct 2018 01:07:38 +0000 (12:07 +1100)]
ctdb-tests: Drop useless "ctdb version" test
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Martin Schwenke [Mon, 8 Oct 2018 01:05:42 +0000 (12:05 +1100)]
ctdb-tests: Rationalise tunable simple tests
These 3 tests duplicate various checks and can easily be handled as a
single test.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Martin Schwenke [Mon, 8 Oct 2018 00:33:08 +0000 (11:33 +1100)]
ctdb-tests: Rationalise ctdb stop/continue/disable/enable simple tests
The "continue" and "enable" tests are just extensions of the "stop"
and "disable" tests, so drop the latter 2.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Martin Schwenke [Mon, 8 Oct 2018 00:13:15 +0000 (11:13 +1100)]
ctdb-tests: Use wait_until_node_has_no_ips() in some tests
This strengthens those tests to ensure that released IPs aren't
replaced with others.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Martin Schwenke [Mon, 8 Oct 2018 00:05:41 +0000 (11:05 +1100)]
ctdb-tests: Add function wait_until_node_has_no_ips()
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Jeremy Allison [Tue, 6 Nov 2018 00:57:51 +0000 (16:57 -0800)]
lib: ldb: Remove use of talloc_autofree_context().
Just use NULL in test case. talloc_autofree_context() is deprecated.
Signed-off-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Jeremy Allison [Tue, 6 Nov 2018 00:56:05 +0000 (16:56 -0800)]
lib: talloc: Mark talloc_autofree_context() as deprecated.
Signed-off-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Martin Schwenke [Fri, 19 Oct 2018 06:27:15 +0000 (17:27 +1100)]
ctdb-event: Only run talloc report if CTDB_INTERACTIVE is set
This is only really wanted for interactive testing when logging to
stderr.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Martin Schwenke [Fri, 19 Oct 2018 05:18:13 +0000 (16:18 +1100)]
ctdb-event: Never fork to become daemon in eventd
This stops ctdbd from being able to shut down eventd, since the PID it
records will be invalid. There's no need for eventd to fork.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Martin Schwenke [Thu, 18 Oct 2018 06:50:09 +0000 (17:50 +1100)]
ctdb-daemon: Improve documentation for -i option
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Martin Schwenke [Thu, 18 Oct 2018 06:47:55 +0000 (17:47 +1100)]
ctdb-daemon: Don't set log_to_stdout for become_daemon()
ctdbd logs to stderr in interactive mode, not stdout. This way stdout
is always closed.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Martin Schwenke [Wed, 17 Oct 2018 10:24:07 +0000 (21:24 +1100)]
ctdb-daemon: Avoid unnecessarily spamming the logs when in test mode
Logging the logging location to syslog can be useful on production
systems when the configuration goes unexpectedly missing. However, in
test mode this just adds noise to the logs on the test system.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Martin Schwenke [Fri, 12 Oct 2018 03:41:42 +0000 (14:41 +1100)]
ctdb-tools: Detect unknown node number
If there aren't enough addresses in the list then the shift will
silently fail and the printed address will be the unshifted value of
$1, which is incorrect/unexpected. So, sanity check the node number.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Martin Schwenke [Thu, 5 Jul 2018 09:04:37 +0000 (19:04 +1000)]
ctdb-tests: README updates
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Martin Schwenke [Thu, 10 May 2018 06:48:31 +0000 (16:48 +1000)]
ctdb-tests: Remove export of CTDB_SOCKET
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Martin Schwenke [Tue, 28 Aug 2018 22:23:51 +0000 (08:23 +1000)]
ctdb-tests: Use path_socket() in dummy client
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Martin Schwenke [Fri, 11 May 2018 11:55:55 +0000 (21:55 +1000)]
ctdb-tests: Drop incorrect comment, unused function
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Martin Schwenke [Thu, 10 May 2018 07:36:04 +0000 (17:36 +1000)]
ctdb-tests: Drop setting of CTDB_SOCKET and CTDB_PIDFILE
The local daemons ssh stub doesn't need to do this because the ctdbd
and the ctdb tool now only need CTDB_TEST_MODE and CTDB_BASE for local
daemon tests.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Martin Schwenke [Thu, 10 May 2018 07:34:19 +0000 (17:34 +1000)]
ctdb-daemon: Drop unused function ctdb_set_socketname()
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Martin Schwenke [Thu, 10 May 2018 07:32:33 +0000 (17:32 +1000)]
ctdb-daemon: Use path functions for socket and PID file
Drop the use of ctdb_set_sockname() because it complicates the memory
allocation and this is the only place it is used. Just assign to the
relevant pointer.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Martin Schwenke [Fri, 6 Jul 2018 05:30:31 +0000 (15:30 +1000)]
ctdb-tests: Use path_socket() in test client tools
Just leak the memory allocated by path_socket(). This is only used in
short-lived test programs, so it isn't worth the hassle of plumbing a
talloc context through several layers to get here.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Martin Schwenke [Thu, 10 May 2018 06:46:18 +0000 (16:46 +1000)]
ctdb-tools: Use path_socket() in ctdb tool
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Martin Schwenke [Thu, 10 May 2018 03:27:28 +0000 (13:27 +1000)]
ctdb-tests: Use ctdb-path for fake_ctdbd directory setup
This needs to be done before any of the code changes are made,
including updating the ctdb tool.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Martin Schwenke [Fri, 20 Apr 2018 03:18:09 +0000 (13:18 +1000)]
ctdb-tests: Use ctdb-path-like values for local daemons socket and PID file
However, don't use ctdb-path itself because some tests use nested
instances of onnode. The outermost instance would set CTDB_SOCKET and
any inner instance would pick up that value, regardless of CTDB_BASE.
This is a temporary measure to avoid breaking testing while use of the
path functions is added to ctdbd and the ctdb tool. When this is
complete these variables can be removed altogether because the code
will just depend on CTDB_TEST_MODE and CTDB_BASE.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Martin Schwenke [Tue, 18 Sep 2018 06:19:40 +0000 (16:19 +1000)]
ctdb-common: Allow path_socket() to use $CTDB_SOCKET
Use of CTDB_SOCKET is being generally removed. However, this override
is being added to allow test code outside of ctdb/ to be able to
specify the socket, if desired.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Tim Beale [Mon, 5 Nov 2018 04:01:55 +0000 (17:01 +1300)]
drs_utils: Avoid invalid dereference of v8 requests
req.more_flags only exists for v10 requests, so we throw an exception if
we try to dereference that field on a v8 (or v5) request. Unfortunately,
we were checking that we support v10 *after* we had tried to access the
more_flags. This patch fixes up the order of the checks.
This may be a problem trying to replicate with an older Windows DC
(pre-2008R2), and was reported on the samba mailing-list at one point:
https://lists.samba.org/archive/samba/2018-June/216541.html
Unfortunately this patch doesn't help the overall situation at all (the
join will fail because we can't resolve the link target and we can't use
GET_TGT). But it now gives you a more meaningful error, i.e.
ERROR(runtime): uncaught exception - (8639, "Failed to process 'chunk'
of DRS replicated objects: DOS code 0x000021bf"
instead of:
ERROR(<type 'exceptions.AttributeError'>): uncaught exception -
'drsuapi.DsGetNCChangesRequest8' object has no attribute 'more_flags'
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Tim Beale <timbeale@samba.org>
Autobuild-Date(master): Tue Nov 6 07:15:33 CET 2018 on sn-devel-144
Tim Beale [Mon, 5 Nov 2018 03:54:05 +0000 (16:54 +1300)]
drs_utils: Fix some long lines
Tweak the code slightly to avoid some 80+ character lines.
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Tim Beale [Mon, 5 Nov 2018 03:34:15 +0000 (16:34 +1300)]
libnet: Reset debug counters after replicating critical objects
Reset the debug counters once we have finished replicating a given
partition. This helps if we replicate the same partition immediately
afterward with different options.
This helps the DC join debug look less weird. Because it replicates the
critical objects first, and then the base partition, previously it
always ended up overcounting, e.g.
Partition[DC=addom,DC=samba,DC=example,DC=com] objects[314/218]
linked_values[48/24]
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Tim Beale [Mon, 5 Nov 2018 01:30:14 +0000 (14:30 +1300)]
join: Fix join large-DB timeout against Windows
The LDAP connection can also timeout when trying to join a Windows DC
with a very large database. However, in this case Windows gives a
slightly different error message (NT_STATUS_CONNECTION_RESET instead of
NT_STATUS_CONNECTION_DISCONNECTED).
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13612
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Noel Power [Tue, 4 Sep 2018 19:33:35 +0000 (20:33 +0100)]
s4/selftest: enable samba.tests.samba_tool.gpo for PY3
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Autobuild-User(master): Noel Power <npower@samba.org>
Autobuild-Date(master): Mon Nov 5 23:04:48 CET 2018 on sn-devel-144
Noel Power [Wed, 5 Sep 2018 16:01:17 +0000 (17:01 +0100)]
python/samba/gp_parse: PY2/PY3 Decode only when necessary
In python2 we decode str types in load_xml, in python3 these are
str class(s) which we cannot decode.
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Noel Power [Wed, 5 Sep 2018 15:14:32 +0000 (16:14 +0100)]
python/samba/tests/samba_tool: PY2/PY3 compat port for test
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Noel Power [Wed, 5 Sep 2018 14:23:01 +0000 (15:23 +0100)]
python/samba/gp_parse: Fix mulitple encode step with write_section
In python2 as far as I can see GptTmplInfParser.write_binary more
or less works by accident.
write_binary creates a writer for the 'utf8' codec, such a writer
should consume unicode and emit utf8 encoded bytes. This writer
is passed to each of the sections managed by GptTmplInfParser as
follows
def write_binary(self, filename):
with codecs.open(filename, 'wb+',
self.encoding) as f:
for s in self.sections:
self.sections[s].write_section(s, f)
And each section type itself is encoding its result to 'utf-16-le'
e.g.
class UnicodeParam(AbstractParam):
def write_section(self, header, fp):
fp.write(u'[Unicode]\r\nUnicode=yes\r\n'.encode(self.encoding)
But this makes little sense, it seems like sections are encoded to one
encoding but the total file is supposed to be encoded as ut8??? Also
having an encoding per ParamType doesn't seem correct.
Bizarely in PY2 this works and it actually encodes the whole file as utf-16le
In PY3 you can't do this as the writer wants to deal with strings not bytes
(after the extra encode phase in 'write_section'.
So, changes here are to remove the unnecessary encoding in each 'write_section'
method, additionally in GptTmplInfParser.write_binary the
codecs.open call now uses the correct codec (e.g. 'utf-16-le') to write
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Noel Power [Wed, 5 Sep 2018 13:54:24 +0000 (14:54 +0100)]
python/samba/netcmd: misc PY2/PY3 compat changes for gpo.py
Fixes:
1) various ldb.bytes that should be displayed as strings in PY3
2) sorting of lists of xml Element in PY3
3) various 'open' need to be opened in binary mode (to accept binary
data)
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Noel Power [Wed, 5 Sep 2018 13:39:11 +0000 (14:39 +0100)]
python/samab/gp_parse: remove unused code
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Noel Power [Wed, 5 Sep 2018 13:18:16 +0000 (14:18 +0100)]
python/samba/gp_parse: Use csv.reader for parsing cvs files
The previous version here was using UnicodeReader which was
wrapping the UTF8Recoder class and passing that to csv.reader.
It looks like the intention was to read a bytestream in a
certain encoding and then reencode it to a different encoding.
And then UnicodeReader creates unicode from the newly encoded stream.
This is unnecssary, we know the encoding of the bytesstream and
codec.getreader will happily consume the bytstream and give back
unicode. The unicode can be fed directly into csv.writer.
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Noel Power [Wed, 5 Sep 2018 12:00:59 +0000 (13:00 +0100)]
python/samba/gp_parse: PY2/PY3 compat porting for gp_init.py
Fixes
1) use compat versions of ConfigParser and StringIO
2) fix sort list of XML Elements
3) open file needs to be opened in binary mode as write_pretty_xml
routing uses BytesIO() object.
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Noel Power [Wed, 5 Sep 2018 11:52:30 +0000 (12:52 +0100)]
python/samba/gp_parse: PY3 open file non-binary mode for write_binary
Although this is unintuitive it's because we are writing unicode
not bytes (both in PY2 & PY3). using the 'b' mode causes an error in
PY3.
In PY3 we can define the encoding, but not in PY2.
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Noel Power [Wed, 5 Sep 2018 11:46:44 +0000 (12:46 +0100)]
python/samba/gp_parse: PY3 file -> open
'file' no longer exists in PY3 replace with 'open'
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Noel Power [Wed, 5 Sep 2018 11:36:00 +0000 (12:36 +0100)]
python/samba/gp_parse: PY2/PY3 compat changes for __init__.py
Fixes.
1) sorting of xml.etree.ElementTree.Element, in PY2 sort
seems to sort lists of these. In PY3 this no longer works.
Choosing tag as the sort key for py3 so at least in python3
there is a consistent sort (probably won't match how it is
sorted in PY2 but nothing seems to depend on that)
2) md5 requires bytes
3) tostring returns bytes in PY3, adjust code for that
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Martin Schwenke [Mon, 29 Oct 2018 03:33:08 +0000 (14:33 +1100)]
ctdb-recovery: Ban a node that causes recovery failure
... instead of applying banning credits.
There have been a couple of cases where recovery repeatedly takes just
over 2 minutes to fail. Therefore, banning credits expire between
failures and a continuously problematic node is never banned,
resulting in endless recoveries. This is because it takes 2
applications of banning credits before a node is banned, which
generally involves 2 recovery failures.
The recovery helper makes up to 3 attempts to recover each database
during a single run. If a node causes 3 failures then this is really
equivalent to 3 recovery failures in the model that existed before the
recovery helper added retries. In that case the node would have been
banned after 2 failures.
So, instead of applying banning credits to the "most failing" node,
simply ban it directly from the recovery helper.
If multiple nodes are causing recovery failures then this can cause a
node to be banned more quickly than it might otherwise have been, even
pre-recovery-helper. However, 90 seconds (i.e. 3 failures) is a long
time to be in recovery, so banning earlier seems like the best
approach.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13670
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Mon Nov 5 06:52:33 CET 2018 on sn-devel-144
Tim Beale [Tue, 30 Oct 2018 03:14:33 +0000 (16:14 +1300)]
traffic: Machine accounts were generated as critical objects
Due to the userAccountControl flags we were specifying, the machine
accounts were all created as critical objects. When trying to populate
1000s of machine accounts in a DB, this makes replication unnecessarily
slow (because it has to replicate them all twice).
This patch changes it so when we're just creating machine accounts for
the purpose of populating a semi-realistic DB, we jsut use the default
WORKSTATION_TRUST_ACCOUNT flag.
Note that for the accounts used for traffic-replay, we apparently need
the existing flags in order for the DC to accept certain requests.
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Autobuild-User(master): Tim Beale <timbeale@samba.org>
Autobuild-Date(master): Mon Nov 5 03:43:24 CET 2018 on sn-devel-144
Tim Beale [Mon, 22 Oct 2018 22:16:31 +0000 (11:16 +1300)]
traffic_replay: Generate machine accounts as well as users
Currently the tool only generates the machine accounts needed for
traffic generation. However, this isn't realistic if we're trying to use
the tool to generate users to simulate a large network.
This patch generates machine accoutns along with the user accounts.
Note we assume there will be more computer accounts than users in a real
network (e.g. work laptops, servers, etc), so generate slightly more
computer accounts.
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Tim Beale [Mon, 22 Oct 2018 21:46:17 +0000 (10:46 +1300)]
traffic_replay: Improve user generation debug
When creating 1000s of users you currently get a lot of debug, but at
the same time you have no idea how far through creating the users you
actually are.
Instead of logging every single user account that's created, log every
50th (as well as how far through the overall generation we are).
Logger already includes timestamps, so we can remove generating the
timestamp diff manually. User creation is the slowest operation - adding
groups/memberships is much faster, so we don't need to log as
frequently.
Note that there is a usability trade-off on how frequently we log
depending on whether the user is using the slower (but more common)
method of going via LDAP, vs the much faster (but more obscure) method
of writing directly to sam.ldb with ldb:nosync=true. In my tests, we end
up logging every ~30-ish secs with LDAP, and every ~3 seconds with
direct file writes.
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Tim Beale [Mon, 22 Oct 2018 21:24:51 +0000 (10:24 +1300)]
traffic_replay: Convert print() to logger.info()
Using logger is more helpful here because it includes timestamps, so we
can see how long things are taking. It's also more consistent with the
rest of the traffic_replay logging.
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Tim Beale [Wed, 31 Oct 2018 20:42:33 +0000 (09:42 +1300)]
traffic_replay: Write group memberships once per group
Each user-group membership was being written to the DB in a single
operation. With large numbers of users (e.g. 10,000 in average 15 groups
each), this becomes a lot of operations (e.g. 150,000). This patch
reworks the code so that we write the memberships for a group in
one operation. E.g. instead of 150,000 DB operations, we might make
1,500. This makes writing the group memberships several times
faster.
Note that rthere is a performance vs memory tradeoff. When we hit
10,000+ members in a group, memory-usage in the underlying DB modify
operation becomes very inefficient/costly. So we avoid potential memory
usage problems by writing no more than 1,000 users to a group at once.
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Tim Beale [Wed, 31 Oct 2018 03:50:27 +0000 (16:50 +1300)]
traffic_replay: Re-organize assignments to be group-based
We can speed up writing the group memberships by adding multiple users
to a group in a single DB modify operation.
To do this, we first need to reorganize the assignments so instead
of being a set of tuples, it's a dictionary where key=group and
value=list-of-users-in-group.
add_users_to_groups() now iterates through the users/groups slightly
differently, but mostly it's just indentation changes. We haven't
changed the number of DB operations yet - we'll do that in the next
patch.
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Tim Beale [Tue, 16 Oct 2018 03:01:25 +0000 (16:01 +1300)]
traffic_replay: Prevent users having 1000+ memberOf links
When adding 10,000 users, one user would end up in over 1000 groups.
With 100,000 users, it would be more like 10,000 groups. While it makes
sense to have groups with large numbers of users, having a single user
in 1000s of groups is probably less realistic.
This patch changes the shape of the Pareto distribution that we use to
assign users to groups. The aim is to cap users at belonging to at most
~500 groups. Increasing the shape of the Pareto distribution pushes the
user assignments so they're closer to the average, and the tail (with
users in lots of groups) is not so large).
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Tim Beale [Mon, 15 Oct 2018 21:57:29 +0000 (10:57 +1300)]
traffic_replay: Change user distribution to use Pareto Distribution
The current probability we were assigning to users roughly approximates
the Pareto Distribution (with shape=1.0). This means the code now uses a
documented algorithm (i.e. explanation on Wikipedia). It also allows us
to vary the distribution by changing the shape parameter.
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Tim Beale [Mon, 15 Oct 2018 03:24:00 +0000 (16:24 +1300)]
traffic_replay: Improve assign_groups() performance with large domains
When assigning 10,000 users to 15 groups each (on average),
assign_groups() would take over 30 seconds. This did not include any DB
operations whatsoever. This patch improves things, so that it takes less
than a second in the same situation.
The problem was the code was looping ~23 million times where the
'random.random() < probability * 10000' condition was not met. The
problem is individual group/user probabilities get lower as the number
of groups/users increases. And so with large numbers of users, most of
the time the calculated probability was very small and didn't meet the
threshold.
This patch changes it so we can select a user/group in one go, avoiding
the need to loop multiple times.
Basically we distribute the users (or groups) between 0.0 and 1.0, so
that each user has their own 'slice', and this slice is proporational to
their weighted probability. random.random() generates a value between
0.0 and 1.0, so we can use this to pick a 'slice' (or rather, we use
this as an index into the list, using .bisect()). Users/groups with
larger probabilities end up with larger slices, so are more likely to
get picked.
The end result is roughly the same distribution as before, although the
first 10 or so user/groups seem to get picked more frequently, so the
weighted-probability calculations may need tweaking some more.
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Tim Beale [Tue, 16 Oct 2018 23:54:03 +0000 (12:54 +1300)]
traffic_replay: Split out random group membership generation logic
This doesn't change functionality at all. It just moves the probability
calculations out into separate functions.
We want to tweak the logic/implementation behind this code, but the
rest of assign_groups() doesn't really care how the underlying
probabilities are worked out, so long as it gets a suitably random
user/group membership each time round the loop.
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Tim Beale [Thu, 18 Oct 2018 03:36:44 +0000 (16:36 +1300)]
traffic_replay: Add helper class for group assignments
Wrap up the group assignment calculations in a helper class. We're going
to tweak the internals a bit in subsequent patches, but the rest of the
code doesn't really need to know about these changes.
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Christof Schmitt [Fri, 2 Nov 2018 19:07:58 +0000 (12:07 -0700)]
selftest: Run smb2.delete-on-close-perms also with "delete readonly = yes"
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13673
Signed-off-by: Christof Schmitt <cs@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Autobuild-User(master): Christof Schmitt <cs@samba.org>
Autobuild-Date(master): Sat Nov 3 05:55:45 CET 2018 on sn-devel-144
Christof Schmitt [Fri, 2 Nov 2018 19:03:51 +0000 (12:03 -0700)]
selftest: Add share to test "delete readonly" option
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13673
Signed-off-by: Christof Schmitt <cs@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Christof Schmitt [Fri, 2 Nov 2018 19:08:23 +0000 (12:08 -0700)]
smbd: Fix DELETE_ON_CLOSE behaviour on files with READ_ONLY attribute
MS-FSA states that a CREATE with FILE_DELETE_ON_CLOSE on an existing
file with READ_ONLY attribute has to return STATUS_CANNOT_DELETE. This
was missing in smbd as the check used the DOS attributes from the CREATE
instead of the DOS attributes on the existing file.
We need to handle the new file and existing file cases separately.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13673
Signed-off-by: Christof Schmitt <cs@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Christof Schmitt [Fri, 2 Nov 2018 17:49:53 +0000 (10:49 -0700)]
smbtorture: Add test for DELETE_ON_CLOSE on files with READ_ONLY attribute
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13673
Signed-off-by: Christof Schmitt <cs@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Volker Lendecke [Fri, 26 Oct 2018 12:09:32 +0000 (14:09 +0200)]
lib: Make dom_sid_string_buf static
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Sat Nov 3 00:23:31 CET 2018 on sn-devel-144
Volker Lendecke [Fri, 26 Oct 2018 06:25:14 +0000 (08:25 +0200)]
lib: Use dom_sid_str_buf
This is the one place where we have to do another strpcy. Many of the
sid_to_fstring calls should be replacable by dom_sid_str_buf, so this
will get less.
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Volker Lendecke [Fri, 26 Oct 2018 06:25:14 +0000 (08:25 +0200)]
auth: Use dom_sid_str_buf
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Volker Lendecke [Fri, 26 Oct 2018 06:25:14 +0000 (08:25 +0200)]
audit_logging: Use dom_sid_str_buf
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Volker Lendecke [Fri, 26 Oct 2018 06:25:14 +0000 (08:25 +0200)]
passdb: Use dom_sid_str_buf
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Volker Lendecke [Fri, 26 Oct 2018 06:25:14 +0000 (08:25 +0200)]
lib: Use dom_sid_str_buf
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Volker Lendecke [Fri, 26 Oct 2018 06:25:14 +0000 (08:25 +0200)]
libsmb: Use dom_sid_str_buf
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Volker Lendecke [Fri, 26 Oct 2018 06:25:14 +0000 (08:25 +0200)]
smbd: Use dom_sid_str_buf
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Volker Lendecke [Fri, 26 Oct 2018 06:25:14 +0000 (08:25 +0200)]
winbindd: Use dom_sid_str_buf
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Volker Lendecke [Fri, 26 Oct 2018 06:25:14 +0000 (08:25 +0200)]
winbindd: Use dom_sid_str_buf
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Volker Lendecke [Thu, 25 Oct 2018 19:45:05 +0000 (21:45 +0200)]
auth4: Use dom_sid_str_buf
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Volker Lendecke [Thu, 25 Oct 2018 19:26:38 +0000 (21:26 +0200)]
audit_tests: Use dom_sid_str_buf
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Volker Lendecke [Thu, 18 Oct 2018 04:18:22 +0000 (06:18 +0200)]
net: Use dom_sid_str_buf
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Volker Lendecke [Thu, 18 Oct 2018 04:08:32 +0000 (06:08 +0200)]
idmap: Use dom_sid_str_buf
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Volker Lendecke [Thu, 18 Oct 2018 04:08:19 +0000 (06:08 +0200)]
auth: Use dom_sid_str_buf
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Volker Lendecke [Thu, 18 Oct 2018 03:57:26 +0000 (05:57 +0200)]
dsdb: Use dom_sid_str_buf
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Volker Lendecke [Thu, 18 Oct 2018 03:55:24 +0000 (05:55 +0200)]
rpc_server4: Use dom_sid_str_buf
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Volker Lendecke [Thu, 18 Oct 2018 03:55:04 +0000 (05:55 +0200)]
nbt_server: Use dom_sid_str_buf
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Ralph Boehme [Wed, 31 Oct 2018 09:21:31 +0000 (10:21 +0100)]
s3:smbd: remove now unused check if fsp is NULL
This was used internally to mark an aio request as cancelled. As the aio
cancellation functionality has been removed, we can now also remove this
check.
Bug: https://bugzilla.samba.org/show_bug.cgi?id=13667
Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Ralph Boehme [Sun, 28 Oct 2018 18:35:59 +0000 (19:35 +0100)]
s3:smbd: fix SMB2 aio cancelling
As we currently don't attempt to cancel the internal aio request, we
must ignore the SMB2 cancel request and continue to process the SMB2
request, cf MS-SM2 3.3.5.16:
If the target request is not successfully canceled, processing of the
target request MUST continue and no response is sent to the cancel
request.
Bug: https://bugzilla.samba.org/show_bug.cgi?id=13667
Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Ralph Boehme [Sun, 28 Oct 2018 18:29:26 +0000 (19:29 +0100)]
s4:torture/smb2/read: add test for cancelling SMB aio
Bug: https://bugzilla.samba.org/show_bug.cgi?id=13667
Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Ralph Boehme [Sun, 28 Oct 2018 18:28:42 +0000 (19:28 +0100)]
vfs_delay_inject: implement pread_send and pwrite_send
Bug: https://bugzilla.samba.org/show_bug.cgi?id=13667
Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Ralph Boehme [Sun, 28 Oct 2018 01:05:45 +0000 (02:05 +0100)]
s4:libcli/smb2: reapply request endtime
tevent_req_finish() removed a possible request timeout, make sure to
reinstall it. This happened when an interim SMB2 response was received.
Bug: https://bugzilla.samba.org/show_bug.cgi?id=13667
Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Ralph Boehme [Sun, 28 Oct 2018 01:04:51 +0000 (02:04 +0100)]
libcli: fill endtime if smbXcli_req_create() timeout is non-zero
Bug: https://bugzilla.samba.org/show_bug.cgi?id=13667
Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Ralph Boehme [Sun, 28 Oct 2018 01:03:28 +0000 (02:03 +0100)]
libcli: add smbXcli_req_endtime
Bug: https://bugzilla.samba.org/show_bug.cgi?id=13667
Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Volker Lendecke [Thu, 18 Oct 2018 03:46:37 +0000 (05:46 +0200)]
lib: Add dom_sid_str_buf
This is modeled after server_id_str_buf, which as an API to me is easier to
use: I can rely on the compiler to get the buffer size right.
It is designed to violate README.Coding's "Make use of helper variables", but
as this API is simple enough and the output should never be a surprise at all,
I think that's worth it.
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Fri Nov 2 20:11:11 CET 2018 on sn-devel-144
Volker Lendecke [Thu, 1 Nov 2018 10:11:17 +0000 (11:11 +0100)]
lib: Add error checks in dom_sid_string_buf
Also, avoid casts by using PRIxxx macros
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Noel Power [Thu, 1 Nov 2018 12:53:01 +0000 (12:53 +0000)]
tests/py/rodc_rwdc: Fix py2/py3 .next compat issues
Python 3 does not have .next(), which we rely on, change the
code slightly so it works in py2/py3 (using builtin next function)
Pair-programmed-with: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Autobuild-User(master): Douglas Bagnall <dbagnall@samba.org>
Autobuild-Date(master): Fri Nov 2 06:56:24 CET 2018 on sn-devel-144
Douglas Bagnall [Sat, 27 Oct 2018 22:12:48 +0000 (11:12 +1300)]
s4/scripting/*: py3 compatible print
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Noel Power <noel.power@suse.com>
Douglas Bagnall [Sat, 27 Oct 2018 21:53:49 +0000 (10:53 +1300)]
selftesthelpers: fix py3 tests with extra_path
If a test was supplied with extra_path, a PYTHONPATH= env variable was
prepended to the args list, but the py3_compatible clause assumed the
first args element was /usr/bin/python.
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Noel Power <noel.power@suse.com>
Douglas Bagnall [Fri, 26 Oct 2018 06:33:48 +0000 (19:33 +1300)]
python dbcheck: don't use mutable default args
In this code
def f(a, b=[]):
b.append(a)
return b
all single argument calls to f() will affect the same copy of b.
In the controls case, controls=None has the same effect as
controls=[].
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Noel Power <noel.power@suse.com>
Gary Lockyer [Wed, 19 Sep 2018 03:01:22 +0000 (15:01 +1200)]
WHATSNEW: KDC prefork support
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Autobuild-User(master): Gary Lockyer <gary@samba.org>
Autobuild-Date(master): Fri Nov 2 03:47:57 CET 2018 on sn-devel-144
Gary Lockyer [Mon, 27 Aug 2018 19:46:59 +0000 (07:46 +1200)]
kdc: Update debug calls
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Gary Lockyer [Wed, 22 Aug 2018 23:26:40 +0000 (11:26 +1200)]
source4 smbd kdc: allow the kdc to run in prefork
Modify the kdc to allow it to run in the prefork process model. The
task_init function has been split up and code moved into the post_fork
function.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Gary Lockyer [Wed, 22 Aug 2018 21:35:52 +0000 (09:35 +1200)]
source4 smdb: Add a post fork hook to the service API
Add a post fork hook to the service API this will be called:
- standard process model
immediately after the task_init.
- single process model
immediately after the task_init
- prefork process model, inhibit_pre_fork = true
immediately after the task_init
- prefork process model, inhibit_pre_fork = false
after each service worker has forked. It is not run on the service
master process.
The post fork hook is not called in the standard model if a new process
is forked on a new connection. It is instead called immediately after
the task_init.
The task_init hook has been changed to return an error code. This ensures
the post_fork code is only run if the task_init code completed successfully.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Gary Lockyer [Wed, 22 Aug 2018 21:29:56 +0000 (09:29 +1200)]
source4 smbd: Make the service_details structure constant.
Make the service_details structure a static const.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>