python/samba/gp_parse: PY2/PY3 compat changes for __init__.py

Fixes.

1) sorting of xml.etree.ElementTree.Element, in PY2 sort
   seems to sort lists of these. In PY3 this no longer works.
   Choosing tag as the sort key for py3 so at least in python3
   there is a consistent sort (probably won't match how it is
   sorted in PY2 but nothing seems to depend on that)
2) md5 requires bytes
3) tostring returns bytes in PY3, adjust code for that

Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

ctdb-recovery: Ban a node that causes recovery failure

... instead of applying banning credits.

There have been a couple of cases where recovery repeatedly takes just
over 2 minutes to fail.  Therefore, banning credits expire between
failures and a continuously problematic node is never banned,
resulting in endless recoveries.  This is because it takes 2
applications of banning credits before a node is banned, which
generally involves 2 recovery failures.

The recovery helper makes up to 3 attempts to recover each database
during a single run.  If a node causes 3 failures then this is really
equivalent to 3 recovery failures in the model that existed before the
recovery helper added retries.  In that case the node would have been
banned after 2 failures.

So, instead of applying banning credits to the "most failing" node,
simply ban it directly from the recovery helper.

If multiple nodes are causing recovery failures then this can cause a
node to be banned more quickly than it might otherwise have been, even
pre-recovery-helper.  However, 90 seconds (i.e. 3 failures) is a long
time to be in recovery, so banning earlier seems like the best
approach.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13670

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Mon Nov  5 06:52:33 CET 2018 on sn-devel-144

traffic: Machine accounts were generated as critical objects

Due to the userAccountControl flags we were specifying, the machine
accounts were all created as critical objects. When trying to populate
1000s of machine accounts in a DB, this makes replication unnecessarily
slow (because it has to replicate them all twice).

This patch changes it so when we're just creating machine accounts for
the purpose of populating a semi-realistic DB, we jsut use the default
WORKSTATION_TRUST_ACCOUNT flag.

Note that for the accounts used for traffic-replay, we apparently need
the existing flags in order for the DC to accept certain requests.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Autobuild-User(master): Tim Beale <timbeale@samba.org>
Autobuild-Date(master): Mon Nov  5 03:43:24 CET 2018 on sn-devel-144

traffic_replay: Generate machine accounts as well as users

Currently the tool only generates the machine accounts needed for
traffic generation. However, this isn't realistic if we're trying to use
the tool to generate users to simulate a large network.

This patch generates machine accoutns along with the user accounts.
Note we assume there will be more computer accounts than users in a real
network (e.g. work laptops, servers, etc), so generate slightly more
computer accounts.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

traffic_replay: Improve user generation debug

When creating 1000s of users you currently get a lot of debug, but at
the same time you have no idea how far through creating the users you
actually are.

Instead of logging every single user account that's created, log every
50th (as well as how far through the overall generation we are).

Logger already includes timestamps, so we can remove generating the
timestamp diff manually. User creation is the slowest operation - adding
groups/memberships is much faster, so we don't need to log as
frequently.

Note that there is a usability trade-off on how frequently we log
depending on whether the user is using the slower (but more common)
method of going via LDAP, vs the much faster (but more obscure) method
of writing directly to sam.ldb with ldb:nosync=true. In my tests, we end
up logging every ~30-ish secs with LDAP, and every ~3 seconds with
direct file writes.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

traffic_replay: Convert print() to logger.info()

Using logger is more helpful here because it includes timestamps, so we
can see how long things are taking. It's also more consistent with the
rest of the traffic_replay logging.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

traffic_replay: Write group memberships once per group

Each user-group membership was being written to the DB in a single
operation. With large numbers of users (e.g. 10,000 in average 15 groups
each), this becomes a lot of operations (e.g. 150,000). This patch
reworks the code so that we write the memberships for a group in
one operation. E.g. instead of 150,000 DB operations, we might make
1,500. This makes writing the group memberships several times
faster.

Note that rthere is a performance vs memory tradeoff. When we hit
10,000+ members in a group, memory-usage in the underlying DB modify
operation becomes very inefficient/costly. So we avoid potential memory
usage problems by writing no more than 1,000 users to a group at once.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

traffic_replay: Re-organize assignments to be group-based

We can speed up writing the group memberships by adding multiple users
to a group in a single DB modify operation.

To do this, we first need to reorganize the assignments so instead
of being a set of tuples, it's a dictionary where key=group and
value=list-of-users-in-group.

add_users_to_groups() now iterates through the users/groups slightly
differently, but mostly it's just indentation changes. We haven't
changed the number of DB operations yet - we'll do that in the next
patch.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

traffic_replay: Prevent users having 1000+ memberOf links

When adding 10,000 users, one user would end up in over 1000 groups.
With 100,000 users, it would be more like 10,000 groups. While it makes
sense to have groups with large numbers of users, having a single user
in 1000s of groups is probably less realistic.

This patch changes the shape of the Pareto distribution that we use to
assign users to groups. The aim is to cap users at belonging to at most
~500 groups. Increasing the shape of the Pareto distribution pushes the
user assignments so they're closer to the average, and the tail (with
users in lots of groups) is not so large).

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

traffic_replay: Change user distribution to use Pareto Distribution

The current probability we were assigning to users roughly approximates
the Pareto Distribution (with shape=1.0). This means the code now uses a
documented algorithm (i.e. explanation on Wikipedia). It also allows us
to vary the distribution by changing the shape parameter.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

traffic_replay: Improve assign_groups() performance with large domains

When assigning 10,000 users to 15 groups each (on average),
assign_groups() would take over 30 seconds. This did not include any DB
operations whatsoever. This patch improves things, so that it takes less
than a second in the same situation.

The problem was the code was looping ~23 million times where the
'random.random() < probability * 10000' condition was not met. The
problem is individual group/user probabilities get lower as the number
of groups/users increases. And so with large numbers of users, most of
the time the calculated probability was very small and didn't meet the
threshold.

This patch changes it so we can select a user/group in one go, avoiding
the need to loop multiple times.

Basically we distribute the users (or groups) between 0.0 and 1.0, so
that each user has their own 'slice', and this slice is proporational to
their weighted probability. random.random() generates a value between
0.0 and 1.0, so we can use this to pick a 'slice' (or rather, we use
this as an index into the list, using .bisect()). Users/groups with
larger probabilities end up with larger slices, so are more likely to
get picked.

The end result is roughly the same distribution as before, although the
first 10 or so user/groups seem to get picked more frequently, so the
weighted-probability calculations may need tweaking some more.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

traffic_replay: Split out random group membership generation logic

This doesn't change functionality at all. It just moves the probability
calculations out into separate functions.

We want to tweak the logic/implementation behind this code, but the
rest of assign_groups() doesn't really care how the underlying
probabilities are worked out, so long as it gets a suitably random
user/group membership each time round the loop.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

traffic_replay: Add helper class for group assignments

Wrap up the group assignment calculations in a helper class. We're going
to tweak the internals a bit in subsequent patches, but the rest of the
code doesn't really need to know about these changes.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

selftest: Run smb2.delete-on-close-perms also with "delete readonly = yes"

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13673

Signed-off-by: Christof Schmitt <cs@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Autobuild-User(master): Christof Schmitt <cs@samba.org>
Autobuild-Date(master): Sat Nov  3 05:55:45 CET 2018 on sn-devel-144

selftest: Add share to test "delete readonly" option

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13673

Signed-off-by: Christof Schmitt <cs@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

smbd: Fix DELETE_ON_CLOSE behaviour on files with READ_ONLY attribute

MS-FSA states that a CREATE with FILE_DELETE_ON_CLOSE on an existing
file with READ_ONLY attribute has to return STATUS_CANNOT_DELETE. This
was missing in smbd as the check used the DOS attributes from the CREATE
instead of the DOS attributes on the existing file.

We need to handle the new file and existing file cases separately.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13673

Signed-off-by: Christof Schmitt <cs@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

smbtorture: Add test for DELETE_ON_CLOSE on files with READ_ONLY attribute

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13673

Signed-off-by: Christof Schmitt <cs@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

lib: Make dom_sid_string_buf static

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Sat Nov  3 00:23:31 CET 2018 on sn-devel-144

lib: Use dom_sid_str_buf

This is the one place where we have to do another strpcy. Many of the
sid_to_fstring calls should be replacable by dom_sid_str_buf, so this
will get less.

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

auth: Use dom_sid_str_buf

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

audit_logging: Use dom_sid_str_buf

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

passdb: Use dom_sid_str_buf

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

lib: Use dom_sid_str_buf

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

libsmb: Use dom_sid_str_buf

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

smbd: Use dom_sid_str_buf

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

winbindd: Use dom_sid_str_buf

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

winbindd: Use dom_sid_str_buf

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

auth4: Use dom_sid_str_buf

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

audit_tests: Use dom_sid_str_buf

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

net: Use dom_sid_str_buf

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

idmap: Use dom_sid_str_buf

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

auth: Use dom_sid_str_buf

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

dsdb: Use dom_sid_str_buf

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

rpc_server4: Use dom_sid_str_buf

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

nbt_server: Use dom_sid_str_buf

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

s3:smbd: remove now unused check if fsp is NULL

This was used internally to mark an aio request as cancelled. As the aio
cancellation functionality has been removed, we can now also remove this
check.

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13667

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

s3:smbd: fix SMB2 aio cancelling

As we currently don't attempt to cancel the internal aio request, we
must ignore the SMB2 cancel request and continue to process the SMB2
request, cf MS-SM2 3.3.5.16:

  If the target request is not successfully canceled, processing of the
  target request MUST continue and no response is sent to the cancel
  request.

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13667

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

s4:torture/smb2/read: add test for cancelling SMB aio

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13667

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

vfs_delay_inject: implement pread_send and pwrite_send

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13667

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

s4:libcli/smb2: reapply request endtime

tevent_req_finish() removed a possible request timeout, make sure to
reinstall it. This happened when an interim SMB2 response was received.

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13667

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

libcli: fill endtime if smbXcli_req_create() timeout is non-zero

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13667

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

libcli: add smbXcli_req_endtime

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13667

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

lib: Add dom_sid_str_buf

This is modeled after server_id_str_buf, which as an API to me is easier to
use: I can rely on the compiler to get the buffer size right.

It is designed to violate README.Coding's "Make use of helper variables", but
as this API is simple enough and the output should never be a surprise at all,
I think that's worth it.

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Fri Nov  2 20:11:11 CET 2018 on sn-devel-144

lib: Add error checks in dom_sid_string_buf

Also, avoid casts by using PRIxxx macros

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

tests/py/rodc_rwdc: Fix py2/py3 .next compat issues

Python 3 does not have .next(), which we rely on, change the
code slightly so it works in py2/py3 (using builtin next function)

Pair-programmed-with: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Autobuild-User(master): Douglas Bagnall <dbagnall@samba.org>
Autobuild-Date(master): Fri Nov  2 06:56:24 CET 2018 on sn-devel-144

s4/scripting/*: py3 compatible print

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Noel Power <noel.power@suse.com>

selftesthelpers: fix py3 tests with extra_path

If a test was supplied with extra_path, a PYTHONPATH= env variable was
prepended to the args list, but the py3_compatible clause assumed the
first args element was /usr/bin/python.

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Noel Power <noel.power@suse.com>

python dbcheck: don't use mutable default args

In this code

def f(a, b=[]):
    b.append(a)
    return b

all single argument calls to f() will affect the same copy of b.

In the controls case, controls=None has the same effect as
controls=[].

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Noel Power <noel.power@suse.com>

WHATSNEW: KDC prefork support

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Autobuild-User(master): Gary Lockyer <gary@samba.org>
Autobuild-Date(master): Fri Nov  2 03:47:57 CET 2018 on sn-devel-144

kdc: Update debug calls

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>

source4 smbd kdc: allow the kdc to run in prefork

Modify the kdc to allow it to run in the prefork process model. The
task_init function has been split up and code moved into the post_fork
function.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>

source4 smdb: Add a post fork hook to the service API

Add a post fork hook to the service API this will be called:

 - standard process model
   immediately after the task_init.

- single process model
  immediately after the task_init

- prefork process model, inhibit_pre_fork = true
  immediately after the task_init

- prefork process model, inhibit_pre_fork = false
  after each service worker has forked. It is not run on the service
  master process.

The post fork hook is not called in the standard model if a new process
is forked on a new connection. It is instead called immediately after
the task_init.

The task_init hook has been changed to return an error code. This ensures
the post_fork code is only run if the task_init code completed successfully.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>

source4 smbd: Make the service_details structure constant.

Make the service_details structure a static const.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>

windbindd: reword error message

Reword the asprintf() out of memory message to make it clear where the
issue is.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>

replmd: Single DB operation per source object during link processing

Move the source object checks and DB modify operation up a level, so we
only do them once per source object rather than once per link.

This allows LMDB joins to succeed with ~15,000 members in a group.
Previously LMDB would fail with the error:

 Failed to apply linked attribute change '(-30792) - MDB_MAP_FULL:
  Environment mapsize limit reached at ../lib/ldb/ldb_mdb/ldb_mdb.c:203'

Rewriting the same object ~15000 times seemed to completely fill up
the LMDB 8Gb buffer. Presumably this was because LMDB is 'copy on
write', so it was storing ~15,000 copies of the same object. Strangely,
we don't see this problem writing the backlinks (which this patch won't
have helped with at all, because that's modifying the target object).

Note uSNChanged was only being added to the msg once, so the code has
been modified to replace the usnChanged each time (i.e. remove it and
re-add it).

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Thu Nov  1 23:48:21 CET 2018 on sn-devel-144

replmd: Move talloc context one level up

Eventually we want to combine multiple link attributes, that apply to the
same source object, into a single DB 'modify' operation. This will mean
the memory context needs to hang around until we have performed the DB
operation (instead of allocating a temporary context for each link).

This patch moves the talloc context one level up, so a temp context gets
allocated for each link *group*, instead of for each link *attribute*.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

replmd: Split apart source/target checks for links

We've grouped the linked attributes by source-object. Next, we want to
avoid duplicated processing for the source object, i.e. we only need to
check the source object exists once, not once per link.

Before we can do this, we need to tease apart
replmd_extract_la_entry_details(), which is doing both source and target
object processing. Split out extracting the target DSDB-DN so that it's
done separately.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

replmd: Group together link attribute processing by source object

Instead of processing each link attribute one at a time, we want to
group them together by source object. This will mean we only have to
look-up the source object once, and only perform one DB 'modify'
operation. With groups with 1000s of members, this will help improve
performance.

This patch takes the first step of group together the links by
source-object. A new 'la_group' struct is added to help track what links
belong to the same source object. The la_list essentially becomes a
'list of lists' now.

Note that only related links *in the same chunk* are only grouped together.
While it is trivial to groups together links that span different
replication chunks, this would be a fairly insignificant efficiency gain,
but seems to have a fairly detrimental memory overhead, once you get
into groups with 10,000+ members.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

replmd: Add more debug for replicating links

During a join of a large DB, processing the linked attributes can take a
long time. The join hangs in 'Committing SAM database' for many minutes
with no indication of whether it's making progress or not.

This patch adds some extra debug to show how far through processing the
linked attributes we are, when there are many thousands of links.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

torture: Fix the clang build

It's used uninitialized if an early torture_assert fails

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
Autobuild-User(master): Volker Lendecke <vl@samba.org>
Autobuild-Date(master): Thu Nov  1 17:34:31 CET 2018 on sn-devel-144

selftest: Remove unnecessary code for backup testenvs

setup_namespaces() already gets done for the backupfromdc's domain, so
this step is unnecessary for the restoredc and offlinebackupdc testenvs
(which are based off the backupfromdc's database).

The setup_namespaces() step is still necessary for the renamedc/labdc,
as these don't have the UPN/SPN suffixes for the new realm yet.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Thu Nov  1 13:06:01 CET 2018 on sn-devel-144

dnsupdate: Skip kerberos step if use-file specified

If there's a problem in get_credentials() (getting the machine account
Kerberos credentials), then we fallback to use_samba_tool (essentially
ignoring use-file). However, there's no need to do this, as use-file
shouldn't require Kerberos credentials.

This was making bootstrapping issues starting a testenv harder to debug.
Obviously, Kerberos is dependent on DNS functioning correctly, but
running dnsupdate was also dependent on having a working Kerberos KDC.
In my case, the testenv had a bad krb5.conf file, but the problem
appeared as resolv-wrapper errors (due to a missing RESOLV_WRAPPER_HOSTS
file, which should've been generated by dnsupdate).

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Pair-Programmed-With: Garming Sam <garming@catalyst.net.nz>

dnsupdate: Pass smb.conf through to samba-tool commands

If you call samba_dnsupdate with a --configfile option, this wasn't
passed through to the samba-tool commands the script tries to run.
Normally, samba_dnsupdate would only be run on the DC itself, so it
shouldn't be a big deal, however, this may be a problem if you install
the samba database into a non-default location (i.e. not
/usr/local/samba).

This patch passes through the smb.conf file, if one was specified.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

selftest: Add new customdc testenv that can load any backup-file

This adds a new testenv that can be used for sandpit/manual testing.
This testenv can be based off any backup-file that you like.

The main use case is large databases. Populating 1000s of users is
time-consuming (it can take hours to create a really large DB). Instead
of having to manually add users to the testenv every time you want to
try something, this allows you to populate the users just once, take a
backup/snapshot of the DB, and then spin up the backup multiple times.

In theory this testenv could be useful for other situations too, e.g.
dealing with a corrupted database, testing DB migration (e.g. 4.7 -->
4.8), or if (for some reason) you wanted to create a realistic
lab-domain within a testenv.

To run-up the testenv you need to specify a BACKUP_FILE environment
variable (the same way we specify the SELFTEST_TESTENV), e.g.
  BACKUP_FILE=/files/backup-10k-ad_dc.tar.bz2 \
    SELFTEST_TESTENV=customdc make testenv

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

pytests: allow blackbox subcommands without a shell

When given a list, it will use the list directly as an argument list,
avoiding shell-expansion and the intermediatory process.

This removes shell expansion trouble, and saves the machine a little
bit of work.

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

samba-tool ntacl: remove unused imports and variables

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Noel Power <noel.power@suse.com>
Autobuild-User(master): Douglas Bagnall <dbagnall@samba.org>
Autobuild-Date(master): Thu Nov  1 09:40:02 CET 2018 on sn-devel-144

python/tests/docs: remove unused import, variables

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Noel Power <noel.power@suse.com>

python/join: remove unused imports and variables

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

python/upgradehelpers: remove unused provision imports

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Noel Power <noel.power@suse.com>

samba-tool processes: remove unused imports and variables

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Noel Power <noel.power@suse.com>

samba-tool domain: remove unused variables

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Noel Power <noel.power@suse.com>

py/tests/dcerpc_integer: remove dup tests

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Noel Power <noel.power@suse.com>

autobuild: remove unused variables

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Noel Power <noel.power@suse.com>

python/tests/source: remove useless local variable

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Noel Power <noel.power@suse.com>

python/tests/credentials: remove unused variables

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Noel Power <noel.power@suse.com>

python/xattr: remove unused variable

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Noel Power <noel.power@suse.com>

python/upgradehelpers: remove unused variable

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Noel Power <noel.power@suse.com>

samba-tool dbcheck: remove unused variable

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Noel Power <noel.power@suse.com>

dbchecker: remove unused variables

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

samba-tool spn: remove unused variables

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Noel Power <noel.power@suse.com>

samba-tool rodc: remove unused variable

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Noel Power <noel.power@suse.com>

python/samba3: import passdb in the manner it is used

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Noel Power <noel.power@suse.com>

python/kcc/graph: import KCCError, which is used somewhere

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Noel Power <noel.power@suse.com>

tests/rodc_rwdc: fix message format

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Noel Power <noel.power@suse.com>

traffic_replay: Exception has no .message

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Noel Power <noel.power@suse.com>

py/tests/dcerpc_rpc: Py3 compat integer types

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Noel Power <noel.power@suse.com>

waflib: fix syntax error in string format

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Noel Power <noel.power@suse.com>

waflib: add necessary imports

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Noel Power <noel.power@suse.com>

python/remove_dc: fix Exception construction

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Noel Power <noel.power@suse.com>

python/remove_dc: fix formating syntax (missing %s)

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Noel Power <noel.power@suse.com>

provision/backend: LDAPBackendResult takes no creds

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Noel Power <noel.power@suse.com>

provision/backend: fix formating syntax

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Noel Power <noel.power@suse.com>

provision: fix string formatting (number of args)

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Noel Power <noel.power@suse.com>

provision: fix string format syntax

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Noel Power <noel.power@suse.com>

Python provision: remove "ExistingBackend"

This was unused and broken. e.g. here:

-    def init(self):
-        # Check to see that this 'existing' LDAP backend in fact exists
-        ldapi_db = Ldb(self.ldapi_uri)

there is no attribute self.ldapi_uri, so this would always raise an
exception.

It was being left around in case it became useful, but that doesn't
seem to be happening.

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Noel Power <noel.power@suse.com>

python dbcheck: use real exception name

and conventional indent

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Noel Power <noel.power@suse.com>

python: avoid useless work in dsdb_Dn.__cmp__

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Noel Power <noel.power@suse.com>

python join: use the sd_utils we imported

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Noel Power <noel.power@suse.com>

s4/dsdb/pytest/sites: do not use variables out of scope

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Noel Power <noel.power@suse.com>

ldb/tests/py/index: remove dup value in dictionary

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Noel Power <noel.power@suse.com>