smbd: Use leases_db in try_lease_upgrade()

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Christof Schmitt <cs@samba.org>

smbd: Split up grant_fsp_lease()

Simple refactoring into simpler routines. View best with "git show -b"

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Christof Schmitt <cs@samba.org>

smbd: Add update_share_mode_lease_from_db()

This is an interim function supposed to be around for just a few patches as
long as we have both the leases.tdb entries and the leases[] in
share_mode_entries around. It makes it easier to transition to just use
leases.tdb while keeping the code running.

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Christof Schmitt <cs@samba.org>

smbd: Don't pass up lease_idx from grant_fsp_lease

The only reason for grant_fsp_lease to return the lease_idx was to pass it down
to set_share_mode. That does not need it anymore.

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Christof Schmitt <cs@samba.org>

smbd: Don't pass lease_idx down to set_share_mode()

Temporary patch to keep the code running. The new code in set_share_mode() will
leave again once the patchset to remove share_mode_lease and thus the lease_idx
in share_mode_entry goes away.

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Christof Schmitt <cs@samba.org>

utils: Use leases_db in smbstatus

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Christof Schmitt <cs@samba.org>

smbd: Use leases_db in fsp_lease_update()

Remove a reference to "struct share_mode_lease"

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Christof Schmitt <cs@samba.org>

smbd: Use leases_db in get_lease_type()

Remove a reference to share_mode_data->leases[]

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Christof Schmitt <cs@samba.org>

smbd: Use leases_db in delay_for_oplock()

Remove a reference to share_mode_data->leases[]

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Christof Schmitt <cs@samba.org>

smbd: Use share_mode_entry's lease data in lease_match()

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Christof Schmitt <cs@samba.org>

smbd: Use share_mode_entry's lease data in vfs_default_durable_reconnect()

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Christof Schmitt <cs@samba.org>

smbd: Use share_mode_entry's lease data in remove_share_mode_lease()

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Christof Schmitt <cs@samba.org>

smbd: Use share_mode_entry's lease data in remove_share_mode_lease()

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Christof Schmitt <cs@samba.org>

smbd: Use share_mode_entry's lease data in is_same_lease()

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Christof Schmitt <cs@samba.org>

smbd: Use share_mode_entry's lease data in delay_rename_for_lease_break()

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Christof Schmitt <cs@samba.org>

smbd: Use share_mode_entry's lease data in delay_for_oplock()

This was the last "share_mode_lease" reference in this function, remove
variable "l".

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Christof Schmitt <cs@samba.org>

smbd: Add lease key to share_mode_entry

Instead of indexing into the leases[] array, put the lease_db reference into
the share_mode_entry. For simplicity, put in the client guid as well. We
*might* be able to retrieve that from somewhere else, but as other smbd
processes have to look at the lease values, put in the full leases_db index
data.

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Christof Schmitt <cs@samba.org>

smbd: Use leases_db_set()

Whenever we update the share_mode_lease struct, also update the leases.tdb
entry.

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Christof Schmitt <cs@samba.org>

leases_db: Add getter/setter for share_mode_lease metadata

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Christof Schmitt <cs@samba.org>

leases_db: Add share_mode_lease info to leases.tdb

This is the data stored in share_mode_lease inside the leases[] array in
locking.tdb. This and all the following patches move all leases array to
looking at the leases.tdb.

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Christof Schmitt <cs@samba.org>

smbd: Pass lease parameters explicitly to find_fsp_lease

This avoids a use of "struct share_mode_lease"

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Christof Schmitt <cs@samba.org>

smbd: Introduce a helper variable in delay_for_oplock()

This removes a few explicit share_mode_lease dereferences

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Christof Schmitt <cs@samba.org>

smbd: Introduce a helper variable in delay_for_oplock()

Why? I am preparing a patchset that will remove
"share_mode_lease". This patch is a micro-step towards that, removing
a set of references to this struct.

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Christof Schmitt <cs@samba.org>

smbd: Small optimization for break_to_none

We don't need to memzero the struct when we can struct-initialize it a
bit later, implicitly initializing the rest to zero.

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Christof Schmitt <cs@samba.org>

.gitlab-ci.yml: show the system state also as after_script

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
Autobuild-User(master): Stefan Metzmacher <metze@samba.org>
Autobuild-Date(master): Fri Apr 12 20:42:27 UTC 2019 on sn-devel-144

.gitlab-ci.yml: print out /proc/swaps

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>

.gitlab-ci.yml: print out /etc/os-release

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>

.gitlab-ci.yml: remove before_script section of .private_template

It's already inherited from .shared_template.

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>

docs/vfs_ceph: describe new ACL behaviour

vfs_ceph now explicitly enables libcephfs POSIX ACL support.

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13896

Signed-off-by: David Disseldorp <ddiss@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Fri Apr 12 19:40:25 UTC 2019 on sn-devel-144

vfs_ceph: explicitly enable libcephfs POSIX ACL support

libcephfs disables ACL support by default and returns -EOPNOTSUPP in the
POSIX ACL get/setxattr paths as a result. Enable support by setting the
following Ceph config parameters during mount:
        client acl type = posix_acl
        fuse default permissions = false

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13896

Signed-off-by: David Disseldorp <ddiss@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

build: add explicit cephfs include path for vfs_ceph builds

Needed if building with a custom --with-libcephfs path.

Signed-off-by: David Disseldorp <ddiss@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

ctdb-common: Avoid race between fd and signal events

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13895

In run_proc, there was an implicit assumption that when a process exits,
fd event (pipe between parent and child) would be processed first and
signal event (SIGCHLD for the child) would be processed later.

However, that is not the case.  SIGCHLD can be received asynchronously
any time even when the pipe data has not fully been read.  This causes
run_proc to miss some of the output from child process in tests.

When SIGCHLD is being processed, if the pipe between parent and child is
still open, then do an explict read from the pipe to ensure we read any
data still in the pipe before closing the pipe.

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Fri Apr 12 08:19:29 UTC 2019 on sn-devel-144

ctdb-daemon: Revert "We can not assume that just because we could complete a TCP handshake"

We also can not assume that nodes can be marked as connected via only
the keepalive mechanism.  Keepalives are not sent to disconnected
nodes so, in the absence of other packets (e.g. broadcasts), 2 nodes
may never become marked as connected to each other.

Revert to marking nodes as connected in the TCP transport code.  If a
connection is to a non(-operational) ctdbd then it will revert to
disconnected after a short while and may actually flap.  This should
be rare.

This reverts commit 66919db3d7ab1e091223faf515b183af8bfddc83.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13888

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

Revert "ctdb-scripts: Do not "correct" number of nfsd threads when it is 0"

I thought this was being triggered during automated testing.
However, it appears that a poor choice of fixed ports for NFS RPC
services was the real problem.  Revert, since the original behaviour
may be useful.

This reverts commit f1a1c300e192d43f5c9faf9450ffbf16341a2661.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

selftest: rename schemaupgrade_dc (+pair) to schema_dc

This is needed because the name of the autobuild job and
the name of the selftest env end up in the socket path
for ncalrpc sockets.

The challenge is that (for example)
/memdisk/autobuild/fl/b2424063/samba-schemaupgrade/bin/ab/schemaupgrade_pair_dc/ncalrpc/np/protected_storage
does not fit in a struct sockaddr_un.

Signed-off-by: Garming Sam <garming@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Fri Apr 12 05:41:36 UTC 2019 on sn-devel-144

ndrdump: change behaviour of flags to operate as flags

These are called flags because that is what they become to the ndr_pull function,
but to avoid total confusion treat them as flags generally even if the values are
always exclusive (at the moment).

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

ndrdump: Remove local variables for pipes

There's no need for the local variables as the NDR call structure
pointer is kept around anyway.

Signed-off-by: Michael Hanselmann <public@hansmi.ch>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

ldb: Avoid calling talloc_get_type() in ldb_kv_parse_data_unpack()

We have the ldb_kv in the caller, just fill it into the context and
so avoid the cost of the talloc_get_type().

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

utils: Move conn_tdb.c to utils/

That's a wrapper that only smbstatus and net status use by now.

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Fri Apr 12 00:37:05 UTC 2019 on sn-devel-144

smbd: Remove some unused includes

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

build: Move smbstatus definition to source3/utils/wscript_build

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

lib: remove duplicate check

This check was supposed to be removed by c9f4b92a613.

Signed-off-by: Swen Schillig <swen@linux.ibm.com>
Reviewed-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Christof Schmitt <cs@samba.org>
Autobuild-User(master): Christof Schmitt <cs@samba.org>
Autobuild-Date(master): Thu Apr 11 23:34:51 UTC 2019 on sn-devel-144

source4: Update error check for new string conversion wrapper

The new string conversion wrappers detect and flag errors
which occured during the string to integer conversion.
Those modifications required an update of the callees
error checks.

Signed-off-by: Swen Schillig <swen@linux.ibm.com>
Reviewed-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Christof Schmitt <cs@samba.org>

libcli: Update error check for new string conversion wrapper

The new string conversion wrappers detect and flag errors
which occured during the string to integer conversion.
Those modifications required an update of the callees
error checks.

Signed-off-by: Swen Schillig <swen@linux.ibm.com>
Reviewed-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Christof Schmitt <cs@samba.org>

common-lib: Update error check for new string conversion wrapper

The new string conversion wrappers detect and flag errors
which occured during the string to integer conversion.
Those modifications required an update of the callees
error checks.

Signed-off-by: Swen Schillig <swen@linux.ibm.com>
Reviewed-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Christof Schmitt <cs@samba.org>

ctdb-tools: Update error check for new string conversion wrapper

The new string conversion wrappers detect and flag errors
which occured during the string to integer conversion.
Those modifications required an update of the callees
error checks.

Signed-off-by: Swen Schillig <swen@linux.ibm.com>
Reviewed-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Christof Schmitt <cs@samba.org>

ctdb-protocol: Update error check for new string conversion wrapper

The new string conversion wrappers detect and flag errors
which occured during the string to integer conversion.
Those modifications required an update of the callees
error checks.

Signed-off-by: Swen Schillig <swen@linux.ibm.com>
Reviewed-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Christof Schmitt <cs@samba.org>

modules: Update error check for new string conversion wrapper

The new string conversion wrappers detect and flag errors
which occured during the string to integer conversion.
Those modifications required an update of the callees
error checks.

Signed-off-by: Swen Schillig <swen@linux.ibm.com>
Reviewed-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Christof Schmitt <cs@samba.org>

utils: Update error check for new string conversion wrapper

The new string conversion wrappers detect and flag errors
which occured during the string to integer conversion.
Those modifications required an update of the callees
error checks.

Signed-off-by: Swen Schillig <swen@linux.ibm.com>
Reviewed-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Christof Schmitt <cs@samba.org>

lib: Update error check for new string conversion wrapper

The new string conversion wrappers detect and flag errors
which occured during the string to integer conversion.
Those modifications required an update of the callees
error checks.

Signed-off-by: Swen Schillig <swen@linux.ibm.com>
Reviewed-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Christof Schmitt <cs@samba.org>

lib: modify string conversion wrapper to handle invalid strings

The standard string conversion routines convert a "non-number string"
to zero and do not flag an error.
This is changed now by returning EINVAL if no conversion occured.

Signed-off-by: Swen Schillig <swen@linux.ibm.com>
Reviewed-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Christof Schmitt <cs@samba.org>

tests/ldb_kv: Add another case for completeness

Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Thu Apr 11 05:25:02 UTC 2019 on sn-devel-144

ldb_kv: Avoid memdup of database records in the case of base searches

This makes LDAP bind significantly faster in the case of having many
members, due to large size of these records (with tens of thousands of
member links). During the nested group calculation, you are only
interested in memberOf not the member links.

(We add a bit-field to determine whether or not the backend actually
supports pointing into database memory. For some reason TDB pointers
aren't stable, so for now we set this option just on LMDB backends.)

Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

ldb_mdb: Change function declaration as per README.coding

Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

ldb_kv: Remove unnecessary space

Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

selftest: split schemaupgrade testenv out

Schemaupgrade tests are particularly resource intensive and are causing
runners to hit their memory and CPU limits, so we need to split them
out.

Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

repl: test for schema object and LA repl across chunks

During replication, transmission of objects and linked attributes are
split into chunks.  These two tests check behavioural consistency across
chunks for regular schema objects and linked attributes.

Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

selftest: tagging tests for new schemaupgrade_dc target

Tagging schema tests against schemaupgrade_dc test target and fixing
some DN assertions to be more generic.

Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

selftest: Add new 2-DC testenv for live schema upgrade

This adds a new 2-DC testenv that:
1. Provisions an AD DC with 2008R2 schema
2. Joins another AD DC with 2008R2 schema
3. Starts Samba
4. Performs a live schema upgrade on the PDC
Testenv targetting in tests.py files for this testenv required that we
extend the environment dependencies system to include optional post-startup
dependencies specified in ENV_DEPS_POST maps.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

dsdb/repl: we need to replicate the whole schema before we can apply it

Otherwise we may not be able to construct a working schema that's
required to apply the changes.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=12204

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

dsdb:samdb: schemainfo update with relax control

Currently schema info's revision field isn't incremented if relax
control is present.  This is so that no increment is done during
provision, but we need the relax control in other situations where
the increment is desired, so we should use the provision control instead
to disable schema info update.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13799

Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

python/provision: use provision and relax controls for schema provision

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13799

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

s4:provision: split out provision_self_join_modify_schema.ldif

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13799

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

ldapcmp: ignore 'schemaInfo' if two domains are compared

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13799

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

samdb: test for schemainfo update with relax control

Currently schema info's revision field isn't incremented if relax
control is present.  This is so that no increment is done during
provision, but we need the relax control in other situations where the
increment is desired.  This patch adds a failing test to expose the
problem.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13799

Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

drsuapi.idl: add DRSUAPI_ATTID_schemaInfo

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13799

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

ldb_kv: Skip @ records early in a search full scan

@ records like @IDXLIST are only available via a base search on the specific name
but the method by which they were excluded was expensive, after the unpack the
DN is exploded and ldb_match_msg_error() would reject it for failing to match the
scope.

This uses the fact that @ records have the DN=@ prefix on their TDB/LMDB key
to quickly exclude them from consideration.

Based on analysis by Garming Sam.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13893

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Wed Apr 10 06:23:39 UTC 2019 on sn-devel-144

vfs_full_audit: Fix logging of get_real_filename output

result == 0 indicated success. In that case log the available
found_name.

Signed-off-by: Christof Schmitt <cs@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Wed Apr 10 01:17:28 UTC 2019 on sn-devel-144

ctdb-test: Adding test case to verify queue resizeing

If a data packet arrives which exceeds the queue's current buffer size,
the buffer needs to be increased to hold the full packet. Once the packet
is processed the buffer size should be decreased to its standard size again.
This test case verifies this process.

Signed-off-by: Swen Schillig <swen@linux.ibm.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Christof Schmitt <cs@samba.org>
Autobuild-User(master): Christof Schmitt <cs@samba.org>
Autobuild-Date(master): Wed Apr 10 00:17:37 UTC 2019 on sn-devel-144

ctdb-test: Adding test case verifying data in buffer move

Signed-off-by: Swen Schillig <swen@linux.ibm.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Christof Schmitt <cs@samba.org>

ctdb-test: Modify ctdb_io_test test_setup to provide queue reference

Some test scenarios require access to the created queue.
Prepare the test_setup function to provide it as additional parameter.

Signed-off-by: Swen Schillig <swen@linux.ibm.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Christof Schmitt <cs@samba.org>

leases_db: Make leases_db_del use leases_db_do_locked

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Tue Apr  9 19:31:09 UTC 2019 on sn-devel-144

leases_db: Make leases_db_add use leases_db_do_locked

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

leases_db: Make leases_db_rename atomic

Do the rename under one lock to protect against potential races while
we don't hold it.

Factor out the NDR marshalling into leases_db_do_locked(), leaving the
rename function pretty simple.

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

smbd: Factor out map_lease_type_to_oplock

grant_fsp_oplock_type has enough complex logic, make this a bit shorter

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

lib: Initialize variables in parse_resolvconf_fp

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

lib: Initialize getline() arguments

Keep "len" valid across the loop iterations for getline to consume

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13892

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

CVE-2019-3880 s3: rpc: winreg: Remove implementations of SaveKey/RestoreKey.

Remove the now unused code implementations of
registry file io.

As reported by Michael Hanselmann.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13851

Signed-off-by: Jeremy Allison <jra@samba.org>
Autobuild-User(master): Karolin Seeger <kseeger@samba.org>
Autobuild-Date(master): Mon Apr  8 11:43:31 UTC 2019 on sn-devel-144

CVE-2019-3880 s3: rpc: winreg: Remove implementations of SaveKey/RestoreKey.

The were not using VFS backend calls and could only work
locally, and were unsafe against symlink races and other
security issues.

If the incoming handle is valid, return WERR_BAD_PATHNAME.

[MS-RRP] states "The format of the file name is implementation-specific"
so ensure we don't allow this.

As reported by Michael Hanselmann.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13851

Signed-off-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

CVE-2019-3870 pysmbd: Ensure a zero umask is set for smbd.mkdir()

mkdir() is the other call that requires a umask of 0 in Samba.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13834

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

CVE-2019-3870 pysmbd: Move umask manipuations as close as possible to users

Umask manipulation was added to pysmbd with e146fe5ef96c1522175a8e81db15d1e8879e5652 in 2012
and init_files_struct was split out in 747c3f1fb379bb68cc7479501b85741493c05812 in 2018 for
Samba 4.9. (It was added to assist the smbd.create_file() routine used in the backup and
restore tools, which needed to write files with full metadata).

This in turn avoids leaving init_files_struct() without resetting the umask to
the original, saved, value.

Per umask(2) this is required before open() and mkdir() system calls (along
side other file-like things such as those for Unix domain socks and FIFOs etc).

Therefore for safety and clarify the additional 'belt and braces' umask
manipuations elsewhere are removed.

mkdir() will be protected by a umask() bracket, for correctness, in the next patch.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13834

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

CVE-2019-3870 pysmbd: Include tests to show the outside umask has no impact

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13834

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

CVE-2019-3870 tests: Add test to check file-permissions are correct after provision

This provisions a new DC and checks there are no world-writable
files in the new DC's private directory.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13834

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

CVE-2019-3870 tests: Extend smbd tests to check for umask being overwritten

The smbd changes the umask - if the code fails to restore the umask to
what it was, then this is very bad. Add an extra check to every
smbd-related test that the umask at the end of the test is the same as
what it was at the beginning (i.e. if the smbd code changed the umask
then it correctly restored the value afterwards).

As the selftest sets the umask for all tests to zero, it makes it hard
to detect this problem, so the test setUp() needs to set it to something
else first.

This extra checking is added to the setUp()/tearDown() so that it
applies to all test-cases. However, any failure that occur with this
approach will not be able to be known-failed.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13834

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

selftest: Correct name of flapping smb2.notify test

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Mon Apr  8 03:09:42 UTC 2019 on sn-devel-144

ldb: version 2.0.0

* Version bump for adding index_format_fn to the schema syntax structure.
* Range index support added, allowing <= and >= operations to be indexed
* Improved reindex performance by setting the in-memory TDB hash size correctly

Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

ldb_mdb: Add some warnings about poorly constructed callbacks

Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

ldb: tests for <= and >= integer indexing with duplicates

We need to make sure that duplicates are correctly returned (uSNChanged
for instance is UNIQUE but, we should be able to index on attributes
which are not unique).

Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

ldb: tests for <= and >= integer indexing

Testing max, min and negative values for indexed 32 and 64 bit types.
This has to be done in two different files because the 64 bit type is
LDB_SYNTAX_INTEGER which is implemented at the ldb level, while the 32
bit is added in the ldb-samba module.  Schema syntax binding added for
ldb-samba.

We also need to make sure that full scans are not invoked for LMDB.

Pair-programmed-with: Garming Sam <garming@catalyst.net.nz>

Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

ldb: Add ORDERED_INTEGER to the proto-schema handling

Adding ordered integer proto schema handling in kv index cache.  This
allows ordered 64 bit integers to be used in cached fields like
@ATTRIBUTES

Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

schema_syntax: Add comments for our index format functions

We had to devise our own scheme for writing integers in a human readable
format which also sorted correctly numerically. This might look a bit
confusing to outsiders, so here's a large comment as a peace offering.

Pair-programmed-with: Tim Beale <timbeale@catalyst.net.nz>

Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

ldb: activating <= and >= indexing for integers

Activating <= and >= mdb indexing in samba for int32 and int64 attributes by:
1. Adding index_format_fn to LDB_SYNTAX_SAMBA_INT32 in ldb_samba
2. Cloning the 64bit LDB_SYNTAX_INTEGER type as LDB_SYNTAX_ORDERED_INTEGER
3. Adding index_format_fn to the new type
4. Modifying LargeInteger use the new type in samba schema
5. Bumping the index version to trigger reindexing

Pair-programmed-with: Garming Sam <garming@catalyst.net.nz>

Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

ldb_kv_index: Add a giant comment in regards to index_format_fn

The reason we needed it in the first place was that the original
canonicalize is being used for non-index functions and it never produced
the right order originally (at least for integers).

Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

ldb_kv_index: Make the edge keys slightly cleaner and generic

It makes no difference in our standard case because \0 will always go
before any value for our index_format_fn, but this is better for
correctness (in case we do mess up our NUL terminations elsewhere).

Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

ldb: <= and >= indexed searching

Full implementation of <= and >= indexed searching using iterate_range
backend operation.  Adds index_format_fn to ldb_schema_syntax so
requires an ABI version bump.  The function must be provided for any
type for which <= and >= indexing is required, and must return a
lexicographically ordered canonicalization of a value.  This causes
index entries to be written in correct order to the database, so
iterate_range on the index DNs can be used.

ldb_kv_index_key is modified to return an index DN with attribute name
but without value if an empty value is provided.  This is needed for
constructing keys that match the beginning or end of an index DN range.

Pair-programmed-with: Garming Sam <garming@catalyst.net.nz>

Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

lmdb: iterate_range cmocka testing

Cmocka testing for LMDB iterate_range operation added in previous commit.

Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

lmdb: iterate_range implementation

Adding iterate_range to LDB API and implementing in LMDB.  This
operation takes a start_key and end_key and returns all records between
the two, inclusive of both.  This will be used to implementing indexing
for <= and >= expressions.

Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

libnet vampire: NULL access bug fix

NULL pointer access bug fix

Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

ctdb: Fix a typo

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Autobuild-User(master): Martin Schwenke <martins@samba.org>
Autobuild-Date(master): Sat Apr  6 11:51:55 UTC 2019 on sn-devel-144

ctdb: Slightly simplify ctdb_ltdb_lock_fetch_requeue

Reduce indentation with an early return

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Martin Schwenke <martin@meltin.net>