vfs_fruit: move FinderInfo lenght check to ad_convert()

The final step in consolidating all conversion related work in
ad_convert(). No change in behaviour.

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13649

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

vfs_fruit: move FinderInfo conversion to helper function and call it from ad_convert()

No change in behaviour.

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13649

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

vfs_fruit: move storing of modified struct adouble to ad_convert()

ad_convert() modified it, so let ad_convert() also save it to disk. No
change in behaviour.

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13649

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

vfs_fruit: remove unneeded fd argument from ad_convert()

Use the struct adouble member ad_fd instead of passing it as an
argument. Who did that in the first place? :)

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13649

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

vfs_fruit: do direct return from error checks in ad_convert()

Subsequent commits will move the mmap() into the subfunctions. This
change just prepares for that.

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13649

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

vfs_fruit: move setting ADEID_FINDERI length to ad_convert_xattr()

ad_convert_xattr() does the conversion of the xattr data in the
AppleDouble file, so we should update it's size there and should not
defer it to the caller.

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13649

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

vfs_fruit: store filler bytes from AppleDouble file header in struct adouble

This can later be used to distinguish between macOS created AppleDouble
files and AppleDouble files created by Samba or Netatalk.

macOS:    "Mac OS X        "
Samba:    "Netatalk        "
Netatalk: "Netatalk        "

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13649

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

vfs_fruit: fix two comments

Thanks to the recent addition of ad_convert_xattr() we now correctly
handle this case.

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13649

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

s4:torture: FinderInfo conversion test with AppleDouble without xattr data

This testcase demonstrates that the AppleDouble conversion in vfs_fruit
doesn't correctly convert the FinderInfo data from the AppleDouble file
to a stream.

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13649

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

tests/python/ldap: use int instead of long for time_t

Python int is at least a C long; Python long disappears in Py3.

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Noel Power <nopower@suse.com>
Autobuild-User(master): Douglas Bagnall <dbagnall@samba.org>
Autobuild-Date(master): Wed Oct 10 09:28:20 CEST 2018 on sn-devel-144

python/upgradehelpers: use int not long for PY3

int works OK for py2 also.

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Noel Power <nopower@suse.com>

pyauth: Remove imessaging_ctx parameter to new

The pyauth code assumes the messaging context code is a py_talloc
object.  But the code in pymessaging returns a wrapped talloc object.
Removing the parameter as it's not currently used by any code.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Noel Power <nopower@suse.com>

popt_common_creds: actually use the ignore_missing_conf flag

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Pair-programmed-with: Olly Betts <olly@survex.com>
Reviewed-by: Noel Power <nopower@suse.com>

netcmd: apply the new get_logger to cmds

This is an example of how to use the new logger.

Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

netcmd: modify Command.get_logger to use get_samba_logger

By doing this, we don't need to repeat the log level convert code any more.
Also, logs have colors now.

Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

script/traffic_replay: apply new logger to replace print

Use logger to replace print

Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

script/traffic_replay: get debug level via api

The -d option will set samba global debug level automatically.
We should not parse and use the passed in value.

Use samba.get_debug_level instead.

Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

script/traffic_replay: print packets data to stderr

This is debug info, should print to stderr.
Otherwise it will flood stdout.

Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

emulate/traffic: apply new logger to replace print

These print are actually progress infomation, should use logger to
print to stderr, other than stdout.

Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

samba/logger: add logger module for python

We need a consitent way for logging in Samba Python code.

This module provides a factory method `get_samba_logger` to create logger,
with a reasonable default format and optional color.

Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

emulate/traffic: allow traffic_replay to run users and groups generate multiple times

When we run `traffic_replay --generate-users-only`, if we cancel it or
it breaks in middle, it won't do anything when we try to run it again.

This is because the code will check the first user/group to create. If
it's already there, then it thought task already done, and break the loop.

This commit change the behavior:
We search existing users/groups first, skip existing ones, and
create non-existing ones. So we can run it multi-times to make sure the
expected users and groups are actually created.

Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

lib: Remove lib/crypto/crc32.[ch]

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Wed Oct 10 01:41:52 CEST 2018 on sn-devel-144

drsuapi: Use the zlib version of crc32

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

auth: Use the zlib version of crc32

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

build: Multi-line deps

We'll add one in the next commit

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

pdb: Reduce code duplication in make_user_info()

10 lines less and a few hundred (-O0) bytes .text less

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Tue Oct  9 01:22:53 CEST 2018 on sn-devel-144

pdb: Fix some "(ret == true)" to just "(ret)"

"ret" is a boolean, so this should not change semantics

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

pdb: Use "sid_compose" where appropriate

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

lib: Avoid the use of open_memstream in tevent_req_profile_string

Solaris does not have it.

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13629
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

tdb: Basic test for circular freelist fix

Try to store a record for which the (circular) freelist does not have
any entry.

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

tdb: Basic test for circular hash chain fix

This just walks tdb_find by searching for a nonexistent record

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

tdb: Make the freelist walk circular-safe

We can't really do the full check while the freelist is modified on the
fly. As long as we don't merge any freelist entries, we should be good
to apply this check.

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

tdb: Align integer types

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

tdb: Make get_hash_length circular-safe

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

tdb: Make tdb_find_dead circular-safe

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

tdb: Make tdb_dump_chain circular-list safe

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

tdb: Make tdb_find circular-safe

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

tdb: Add tdb_chainwalk_check

This captures the tdb_rescue protection against circular hash chains
with a slow pointer updated only on every other record traverse

If a hash chain has a loop, eventually the next_ptr
will cycle around and be identical to the 'slow' pointer.

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

registry: Don't use an uninitialized value

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

registry: Print failure of regdb_unpack_values

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

registry: Add error checks to regdb_unpack_values

This makes "regdb_unpack_values" take a size_t as buflen. The only
caller calls it with TDB_DATA.dsize, which *is* size_t. Convert the
internal "len" variable to the unsigned size_t as well and add overflow
checks. This depends on tdb_unpack to either return -1 or a positive
value less than or equal to the passed-in "size_t" buflen;

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

registry: Add error checks to regdb_fetch_keys_internal

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

registry: Fix a typo

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

tdb_unpack: Protect against overflow

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

tdb_unpack: Correct "len" arg for "B" format

All but one of the users of the "B" format specifier passed in a pointer
to uint32_t instead of what tdb_unpack expected, an "int". Because this
is a purely internal API, change the tdb_unpack function and adjust that
one caller.

To reviewers: Please check carefully, thanks :-)

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

tdb_unpack: Convert to size_t for internal calculations

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

gpo: Test process_group_policy in gp_sec_ext

Signed-off-by: David Mulder <dmulder@suse.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Aurelien Aptel <aaptel@suse.com>
Autobuild-User(master): Aurélien Aptel <aaptel@samba.org>
Autobuild-Date(master): Mon Oct  8 21:25:59 CEST 2018 on sn-devel-144

gpo: test the get_deleted_gpos_list() function

Signed-off-by: David Mulder <dmulder@suse.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Aurelien Aptel <aaptel@suse.com>

gpo: Test the new get_applied functions

Signed-off-by: David Mulder <dmulder@suse.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Aurelien Aptel <aaptel@suse.com>

gpo PEP8: balance whitespace around equals

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: David Mulder <dmulder@suse.com>
Reviewed-by: Aurelien Aptel <aaptel@suse.com>

gpupdate: test the new --force option

Signed-off-by: David Mulder <dmulder@suse.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Aurelien Aptel <aaptel@suse.com>

gpupdate: Add the --force option

This option forces the reapplication of policy,
and works the same as MS 'gpupdate /force'

Signed-off-by: David Mulder <dmulder@suse.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Aurelien Aptel <aaptel@suse.com>

gpo: Remove unused apply_log_pop() and list() funcs

Signed-off-by: David Mulder <dmulder@suse.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Aurelien Aptel <aaptel@suse.com>

gpo: add unapply to the gp_sec_ext

Signed-off-by: David Mulder <dmulder@suse.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Aurelien Aptel <aaptel@suse.com>

gpo: Calculate deleted gpos and unapply them

Signed-off-by: David Mulder <dmulder@suse.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Aurelien Aptel <aaptel@suse.com>

gpo: Use the new process_group_policy() for unapply

Signed-off-by: David Mulder <dmulder@suse.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Aurelien Aptel <aaptel@suse.com>

gpo: avoid quadratic behaviour in guid retrieval

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: David Mulder <dmulder@suse.com>
Reviewed-by: Aurelien Aptel <aaptel@suse.com>

gpo: Create a function for returning applied settings

This returns a list of guids for gpos applied
plus settings applied and their previous values.

Signed-off-by: David Mulder <dmulder@suse.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Aurelien Aptel <aaptel@suse.com>

gpo: apply_map should not be required for gp_ext

The apply_map function should not be a requirement
to implement the gp_ext class, since only the
gp_sec_ext uses it now.

Signed-off-by: David Mulder <dmulder@suse.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Aurelien Aptel <aaptel@suse.com>

gpo: remove unreached non-DC branch in gp_sec_ext.apply_map()

We don't get this far if we are not a DC, and if somehow we do the
errors will be no more informative due to this special case.

Signed-off-by: David Mulder <dmulder@suse.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Aurelien Aptel <aaptel@suse.com>

gpo: Move policy application to the gp_ext

Policy specific setting application should be
handled by the group policy extension, not the
read/parse handler.

Signed-off-by: David Mulder <dmulder@suse.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Aurelien Aptel <aaptel@suse.com>

gpo: Remove unused gp_ext.list() function

Signed-off-by: David Mulder <dmulder@suse.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Aurelien Aptel <aaptel@suse.com>

gpo: Implement process_group_policy() gp_ext func

MS spec describes the policy callback as a
function called ProcessGroupPolicy which accepts
a pDeletedGPOList and a pChangedGPOList param.
The Group Policy Client Side Extension then
iterates over the deleted, then the changed gpo
lists and applies/unapplies policy. We should do
this also.

Signed-off-by: David Mulder <dmulder@suse.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Aurelien Aptel <aaptel@suse.com>

gpo: Remove unused methods from gp_sec_ext

These functions were added by Luke, but have
never actually done anything. If/when we
read from these *.pol files, we won't need these
separate functions to do it.

Signed-off-by: David Mulder <dmulder@suse.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Aurelien Aptel <aaptel@suse.com>

gpo: Initialize gp_ext variables in constructor

Initialize variables for the gp_ext in the
constructor instead of passing them via the parse
function.
This is a dependency of the "gpo: Implement
process_group_policy() gp_ext func" patch, since
the parse() function is now called by the ext,
instead of by gpupdate within apply_gp(). The
parse() function should only take the path
variable, to simplify writing Client Side
Extensions.

Signed-off-by: David Mulder <dmulder@suse.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Aurelien Aptel <aaptel@suse.com>

gpupdate: Remove the unnecessary url parameter

The samdb object isn't initialized here anymore,
but in the gp_sec_ext, so this parameter to
gpupdate does nothing.

Signed-off-by: David Mulder <dmulder@suse.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Aurelien Aptel <aaptel@suse.com>

gpo: gp_sec_ext should check whether to apply

Whether an extension should apply should be
determined by the extension, not by the
calling script.

Signed-off-by: David Mulder <dmulder@suse.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Aurelien Aptel <aaptel@suse.com>

gpo: Initialize SamDB in the gp_sec_ext

The SamDB is only used by the gp_sec_ext, and
isn't needed elsewhere, so initialize it where
we need it and avoid passing it around
everywhere.
It makes the most sense to put this in the setter
class that uses it, so pass our creds down so we
have access to it, then initialize it there.

Signed-off-by: David Mulder <dmulder@suse.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Aurelien Aptel <aaptel@suse.com>

gpo: abstract methods are defined in the parent class

These methods don't need redefined in the child
class.

Signed-off-by: David Mulder <dmulder@suse.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Aurelien Aptel <aaptel@suse.com>

ctdb-tests: Improve counting of database records

Record counts are sometimes incomplete for large databases when
relevant tests are run on a real cluster.

This probably has something to do with ssh, pipes and buffering, so
move the filtering and counting to the remote end.  This means that
only the count comes across the pipe, instead of all the record data.

Instead of explicitly excluding the key for persistent database
sequence numbers, just exclude any key starting with '_'.  Such keys
are not used in tests.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Mon Oct  8 05:36:11 CEST 2018 on sn-devel-144

ctdb-tests: Add extra debug to large database recovery test

This test sometimes fails, probably because the test is flakey.
Either the records aren't being added correctly or the counting of
records loses records.  Try to debug both possibilities.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-tests: Shut down transaction_loop clients more cleanly

A transaction_loop client can exit with a transaction active when its
time limit expires.  This causes a recovery and causes problems with
the test cleanup, which detects unwanted recoveries and fails.

Set a flag when the time limit expires and exit cleanly before the
next transaction is started.

Pair-programmed-with: Amitay Isaacs <amitay@gmail.com>
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-tools: Have onnode pass -n option even when regular ssh not in use

ONNODE_SSH is really a test hook, so it doesn't need to support
completely random values.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-tests: Support closing of stdin in local daemons ssh stub

Not sure this is needed but this makes it behave the same as ssh.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-tests: Be more careful when building public IP addresses

The goal is to allow more local daemons by expanding the address range
rather than generating invalid addresses.

For IPv6, use a separate address space instead of an offset for the
2nd address.

For IPv4, use the last 2 octets with addresses starting at
192.168.100.1 and 192.168.200.1.  Avoid addresses with 0 and 255 in
the last octet by using a maximum of 100 addresses per "subnet"
starting at .1.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-tests: Be more careful when building node addresses

The goal is to allow more local daemons by expanding the address range
rather than generating invalid addresses.

For IPv6, use all 4 trailing hex digits.

For IPv4, use the last 2 octets.  Although 127.0.0.0 is a /8 network,
avoid unexpected issues due to 0 and 255 in the last octet.  Use a
maximum of 100 addresses per "subnet" starting at .1.  Keep the first
group of addresses in 127.0.0.0/24 to continue to allow a reasonable
number of nodes to be tested with socket-wrapper.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-tests: Don't format IPv4 octets as hex digits

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-tests: Be more efficient about starting/stopping local daemons

Don't loop, just use onnode all.

For shutting down, use onnode -p all.  This results in a significant
time saving for stopping many deamons because "ctdb shutdown" is now
synchronous.

onnode -p all can be used to start daemons directly because they
daemonize.  However, this does not work under valgrind because the
valgrind process does not exit, so onnode will wait forever for it.
In this case, use onnode without the -p option.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-tests: Do not use ctdbd_wrapper in local daemon tests

Run the daemon directly and shut it down using ctdb shutdown.

The wrapper waits for ctdbd to reach >=FIRST_RECOVERY runstate within
a timeout period and shuts ctdbd down if that doesn't happen.  This is
only really used to ensure that ctdbd doesn't exit early after an
apparently successful start.  There are no known cases where ctdbd
will continue running but fail to reach >=FIRST_RECOVERY runstate.

When ctdbd is started in tests, the test code will wait until ctdbd is
in a healthy state on all nodes before proceeding, so there is
effectively no change in behaviour.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-tests: Don't remove non-existent test database directory

This directory is no longer used.  Lack of removal doesn't seem to
cause a problem.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-tests: Drop unused function maybe_stop_ctdb()

There are too many functions to start/stop daemons.  Simplify this.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-tests: Explicitly check for local daemons when shutting down

This is clearer if the logic is explicit...  and...

There are too many functions to start/stop daemons.  Simplify this.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-tests: Drop functions daemons_start(), daemons_stop()

There are too many functions to start/stop daemons.  Simplify this.

Inline the functionality into ctdb_start_all() and ctdb_stop_all().

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-tests: Don't used daemons_start()/daemons_stop() directly in tests

There are too many functions to start/stop daemons.  Simplify this.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-tests: Rename _ctdb_start_all() -> ctdb_start_all()

There are too many functions to start/stop daemons.  Simplify this.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-tests: Rename ctdb_start_all() -> ctdb_init()

There are too many functions to start/stop daemons.  Simplify this.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-tests: Drop ps_ctdbd()

This was used for debugging tests by ensuring that the arguments to
ctdbd were as expected.  It no longer outputs anything useful because
ctdbd is now started without arguments.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-tests: Drop code for RECEIVE_RECORDS control

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13641

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-protocol: Drop marshalling code for RECEIVE_RECORDS control

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13641

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-protocol: Mark RECEIVE_RECORDS control obsolete

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13641

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-daemon: Drop implementation of RECEIVE_RECORDS control

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13641

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-vacuum: Remove unnecessary check for zero records in delete list

Since no records are deleted from RB tree during step 1, there is no
need for the check.  Run step 2 unconditionally.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13641

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-vacuum: Fix the incorrect counting of remote errors

If a node fails to delete a record in TRY_DELETE_RECORDS control during
vacuuming, then it's possible that other nodes also may fail to delete a
record.  So instead of deleting the record from RB tree on first failure,
keep track of the remote failures.

Update delete_list.remote_error and delete_list.left statistics only
once per record during the delete_record_traverse.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13641

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-vacuum: Simplify the deletion of vacuumed records

The 3-phase deletion of vacuumed records was introduced to overcome
the problem of record(s) resurrection during recovery.  This problem
is now handled by avoiding the records from recently INACTIVE nodes in
the recovery process.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13641

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-tests: Add recovery record resurrection test for volatile databases

Ensure that deleted records and vacuumed records are not resurrected
from recently inactive nodes.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13641

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-daemon: Invalidate records if a node becomes INACTIVE

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13641

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-daemon: Don't pull any records if records are invalidated

This avoids unnecessary work during recovery to pull records from nodes
that were INACTIVE just before the recovery.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13641

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-daemon: Add invalid_records flag to ctdb_db_context

If a node becomes INACTIVE, then all the records in volatile databases
are invalidated.  This avoids the need to include records from such
nodes during subsequent recovery after the node comes out INACTIVE state.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13641

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdbd_conn: Generalise inaccurate error message

Seeing:

  ctdb_read_packet failed: Cannot allocate memory
  [...,  0] ../source3/lib/ctdbd_conn.c:121(cluster_fatal)
  cluster fatal event: ctdbd died

The error is due to a memory allocation failure rather than ctdbd
dying.  However, the error message makes people wonder why ctdbd died.

Another alternative would be to wrap cluster_fatal() and have the
wrapper interpret the return value from ctdb_read_packet() to choose
from a set of more precise messages to pass to cluster_fatal().  For a
memory allocation it isn't strictly necessary to call cluster_fatal(),
but all is probably lost and it is still probably better to try to
exit cleanly as soon as possible instead of crashing somewhere.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Volker Lendecke <vl@samba.org>
Autobuild-User(master): Volker Lendecke <vl@samba.org>
Autobuild-Date(master): Fri Oct  5 16:16:29 CEST 2018 on sn-devel-144

s4/auth/tests: Fix kerberos test string size

>>> len("user0@samba.example.com")
23

But the string definition does not take a final '\0' into account.
As per Volker's suggestion, use compiler's support to allocate
the string properly.

Signed-off-by: Alexander Bokovoy <ab@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>