amitay/samba-autobuild/.git
9 days agoctdb-common: Avoid race between fd and signal events master
Amitay Isaacs [Tue, 9 Apr 2019 04:44:04 +0000 (14:44 +1000)]
ctdb-common: Avoid race between fd and signal events

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13895

In run_proc, there was an implicit assumption that when a process exits,
fd event (pipe between parent and child) would be processed first and
signal event (SIGCHLD for the child) would be processed later.

However, that is not the case.  SIGCHLD can be received asynchronously
any time even when the pipe data has not fully been read.  This causes
run_proc to miss some of the output from child process in tests.

When SIGCHLD is being processed, if the pipe between parent and child is
still open, then do an explict read from the pipe to ensure we read any
data still in the pipe before closing the pipe.

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
9 days agoctdb-daemon: Revert "We can not assume that just because we could complete a TCP...
Martin Schwenke [Fri, 5 Apr 2019 05:17:35 +0000 (16:17 +1100)]
ctdb-daemon: Revert "We can not assume that just because we could complete a TCP handshake"

We also can not assume that nodes can be marked as connected via only
the keepalive mechanism.  Keepalives are not sent to disconnected
nodes so, in the absence of other packets (e.g. broadcasts), 2 nodes
may never become marked as connected to each other.

Revert to marking nodes as connected in the TCP transport code.  If a
connection is to a non(-operational) ctdbd then it will revert to
disconnected after a short while and may actually flap.  This should
be rare.

This reverts commit 66919db3d7ab1e091223faf515b183af8bfddc83.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13888

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
9 days agoRevert "ctdb-scripts: Do not "correct" number of nfsd threads when it is 0"
Martin Schwenke [Thu, 4 Apr 2019 07:21:49 +0000 (18:21 +1100)]
Revert "ctdb-scripts: Do not "correct" number of nfsd threads when it is 0"

I thought this was being triggered during automated testing.
However, it appears that a poor choice of fixed ports for NFS RPC
services was the real problem.  Revert, since the original behaviour
may be useful.

This reverts commit f1a1c300e192d43f5c9faf9450ffbf16341a2661.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
9 days agoselftest: rename schemaupgrade_dc (+pair) to schema_dc
Garming Sam [Fri, 12 Apr 2019 01:31:29 +0000 (03:31 +0200)]
selftest: rename schemaupgrade_dc (+pair) to schema_dc

This is needed because the name of the autobuild job and
the name of the selftest env end up in the socket path
for ncalrpc sockets.

The challenge is that (for example)
/memdisk/autobuild/fl/b2424063/samba-schemaupgrade/bin/ab/schemaupgrade_pair_dc/ncalrpc/np/protected_storage
does not fit in a struct sockaddr_un.

Signed-off-by: Garming Sam <garming@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Fri Apr 12 05:41:36 UTC 2019 on sn-devel-144

9 days agondrdump: change behaviour of flags to operate as flags
Andrew Bartlett [Fri, 12 Apr 2019 03:10:35 +0000 (15:10 +1200)]
ndrdump: change behaviour of flags to operate as flags

These are called flags because that is what they become to the ndr_pull function,
but to avoid total confusion treat them as flags generally even if the values are
always exclusive (at the moment).

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
9 days agondrdump: Remove local variables for pipes
Michael Hanselmann [Wed, 3 Apr 2019 22:04:23 +0000 (00:04 +0200)]
ndrdump: Remove local variables for pipes

There's no need for the local variables as the NDR call structure
pointer is kept around anyway.

Signed-off-by: Michael Hanselmann <public@hansmi.ch>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
9 days agoldb: Avoid calling talloc_get_type() in ldb_kv_parse_data_unpack()
Andrew Bartlett [Thu, 11 Apr 2019 23:26:01 +0000 (11:26 +1200)]
ldb: Avoid calling talloc_get_type() in ldb_kv_parse_data_unpack()

We have the ldb_kv in the caller, just fill it into the context and
so avoid the cost of the talloc_get_type().

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
9 days agoutils: Move conn_tdb.c to utils/
Volker Lendecke [Thu, 11 Apr 2019 11:03:30 +0000 (13:03 +0200)]
utils: Move conn_tdb.c to utils/

That's a wrapper that only smbstatus and net status use by now.

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Fri Apr 12 00:37:05 UTC 2019 on sn-devel-144

9 days agosmbd: Remove some unused includes
Volker Lendecke [Thu, 11 Apr 2019 10:58:05 +0000 (12:58 +0200)]
smbd: Remove some unused includes

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
9 days agobuild: Move smbstatus definition to source3/utils/wscript_build
Volker Lendecke [Thu, 11 Apr 2019 09:36:56 +0000 (11:36 +0200)]
build: Move smbstatus definition to source3/utils/wscript_build

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
9 days agolib: remove duplicate check
Swen Schillig [Thu, 11 Apr 2019 07:52:05 +0000 (09:52 +0200)]
lib: remove duplicate check

This check was supposed to be removed by c9f4b92a613.

Signed-off-by: Swen Schillig <swen@linux.ibm.com>
Reviewed-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Christof Schmitt <cs@samba.org>
Autobuild-User(master): Christof Schmitt <cs@samba.org>
Autobuild-Date(master): Thu Apr 11 23:34:51 UTC 2019 on sn-devel-144

9 days agosource4: Update error check for new string conversion wrapper
Swen Schillig [Wed, 6 Mar 2019 09:11:39 +0000 (10:11 +0100)]
source4: Update error check for new string conversion wrapper

The new string conversion wrappers detect and flag errors
which occured during the string to integer conversion.
Those modifications required an update of the callees
error checks.

Signed-off-by: Swen Schillig <swen@linux.ibm.com>
Reviewed-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Christof Schmitt <cs@samba.org>
9 days agolibcli: Update error check for new string conversion wrapper
Swen Schillig [Wed, 6 Mar 2019 09:06:35 +0000 (10:06 +0100)]
libcli: Update error check for new string conversion wrapper

The new string conversion wrappers detect and flag errors
which occured during the string to integer conversion.
Those modifications required an update of the callees
error checks.

Signed-off-by: Swen Schillig <swen@linux.ibm.com>
Reviewed-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Christof Schmitt <cs@samba.org>
9 days agocommon-lib: Update error check for new string conversion wrapper
Swen Schillig [Wed, 6 Mar 2019 09:02:53 +0000 (10:02 +0100)]
common-lib: Update error check for new string conversion wrapper

The new string conversion wrappers detect and flag errors
which occured during the string to integer conversion.
Those modifications required an update of the callees
error checks.

Signed-off-by: Swen Schillig <swen@linux.ibm.com>
Reviewed-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Christof Schmitt <cs@samba.org>
9 days agoctdb-tools: Update error check for new string conversion wrapper
Swen Schillig [Wed, 6 Mar 2019 08:48:24 +0000 (09:48 +0100)]
ctdb-tools: Update error check for new string conversion wrapper

The new string conversion wrappers detect and flag errors
which occured during the string to integer conversion.
Those modifications required an update of the callees
error checks.

Signed-off-by: Swen Schillig <swen@linux.ibm.com>
Reviewed-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Christof Schmitt <cs@samba.org>
9 days agoctdb-protocol: Update error check for new string conversion wrapper
Swen Schillig [Wed, 6 Mar 2019 08:43:53 +0000 (09:43 +0100)]
ctdb-protocol: Update error check for new string conversion wrapper

The new string conversion wrappers detect and flag errors
which occured during the string to integer conversion.
Those modifications required an update of the callees
error checks.

Signed-off-by: Swen Schillig <swen@linux.ibm.com>
Reviewed-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Christof Schmitt <cs@samba.org>
9 days agomodules: Update error check for new string conversion wrapper
Swen Schillig [Wed, 6 Mar 2019 08:34:10 +0000 (09:34 +0100)]
modules: Update error check for new string conversion wrapper

The new string conversion wrappers detect and flag errors
which occured during the string to integer conversion.
Those modifications required an update of the callees
error checks.

Signed-off-by: Swen Schillig <swen@linux.ibm.com>
Reviewed-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Christof Schmitt <cs@samba.org>
9 days agoutils: Update error check for new string conversion wrapper
Swen Schillig [Wed, 6 Mar 2019 08:29:13 +0000 (09:29 +0100)]
utils: Update error check for new string conversion wrapper

The new string conversion wrappers detect and flag errors
which occured during the string to integer conversion.
Those modifications required an update of the callees
error checks.

Signed-off-by: Swen Schillig <swen@linux.ibm.com>
Reviewed-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Christof Schmitt <cs@samba.org>
9 days agolib: Update error check for new string conversion wrapper
Swen Schillig [Wed, 6 Mar 2019 08:07:13 +0000 (09:07 +0100)]
lib: Update error check for new string conversion wrapper

The new string conversion wrappers detect and flag errors
which occured during the string to integer conversion.
Those modifications required an update of the callees
error checks.

Signed-off-by: Swen Schillig <swen@linux.ibm.com>
Reviewed-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Christof Schmitt <cs@samba.org>
9 days agolib: modify string conversion wrapper to handle invalid strings
Swen Schillig [Wed, 6 Mar 2019 08:03:27 +0000 (09:03 +0100)]
lib: modify string conversion wrapper to handle invalid strings

The standard string conversion routines convert a "non-number string"
to zero and do not flag an error.
This is changed now by returning EINVAL if no conversion occured.

Signed-off-by: Swen Schillig <swen@linux.ibm.com>
Reviewed-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Christof Schmitt <cs@samba.org>
10 days agotests/ldb_kv: Add another case for completeness
Garming Sam [Tue, 9 Apr 2019 01:22:01 +0000 (13:22 +1200)]
tests/ldb_kv: Add another case for completeness

Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Thu Apr 11 05:25:02 UTC 2019 on sn-devel-144

10 days agoldb_kv: Avoid memdup of database records in the case of base searches
Garming Sam [Thu, 7 Mar 2019 03:45:46 +0000 (16:45 +1300)]
ldb_kv: Avoid memdup of database records in the case of base searches

This makes LDAP bind significantly faster in the case of having many
members, due to large size of these records (with tens of thousands of
member links). During the nested group calculation, you are only
interested in memberOf not the member links.

(We add a bit-field to determine whether or not the backend actually
supports pointing into database memory. For some reason TDB pointers
aren't stable, so for now we set this option just on LMDB backends.)

Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
10 days agoldb_mdb: Change function declaration as per README.coding
Garming Sam [Thu, 11 Apr 2019 00:53:45 +0000 (12:53 +1200)]
ldb_mdb: Change function declaration as per README.coding

Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
10 days agoldb_kv: Remove unnecessary space
Garming Sam [Mon, 8 Apr 2019 02:41:43 +0000 (14:41 +1200)]
ldb_kv: Remove unnecessary space

Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
10 days agoselftest: split schemaupgrade testenv out
Aaron Haslett [Mon, 1 Apr 2019 21:45:36 +0000 (10:45 +1300)]
selftest: split schemaupgrade testenv out

Schemaupgrade tests are particularly resource intensive and are causing
runners to hit their memory and CPU limits, so we need to split them
out.

Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
10 days agorepl: test for schema object and LA repl across chunks
Aaron Haslett [Tue, 19 Feb 2019 01:33:33 +0000 (14:33 +1300)]
repl: test for schema object and LA repl across chunks

During replication, transmission of objects and linked attributes are
split into chunks.  These two tests check behavioural consistency across
chunks for regular schema objects and linked attributes.

Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
10 days agoselftest: tagging tests for new schemaupgrade_dc target
Aaron Haslett [Tue, 12 Feb 2019 00:31:57 +0000 (13:31 +1300)]
selftest: tagging tests for new schemaupgrade_dc target

Tagging schema tests against schemaupgrade_dc test target and fixing
some DN assertions to be more generic.

Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
10 days agoselftest: Add new 2-DC testenv for live schema upgrade
Tim Beale [Thu, 17 Jan 2019 04:18:48 +0000 (17:18 +1300)]
selftest: Add new 2-DC testenv for live schema upgrade

This adds a new 2-DC testenv that:
1. Provisions an AD DC with 2008R2 schema
2. Joins another AD DC with 2008R2 schema
3. Starts Samba
4. Performs a live schema upgrade on the PDC
Testenv targetting in tests.py files for this testenv required that we
extend the environment dependencies system to include optional post-startup
dependencies specified in ENV_DEPS_POST maps.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
10 days agodsdb/repl: we need to replicate the whole schema before we can apply it
Stefan Metzmacher [Fri, 8 Feb 2019 14:49:55 +0000 (14:49 +0000)]
dsdb/repl: we need to replicate the whole schema before we can apply it

Otherwise we may not be able to construct a working schema that's
required to apply the changes.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=12204

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
10 days agodsdb:samdb: schemainfo update with relax control
Aaron Haslett [Wed, 3 Apr 2019 03:34:42 +0000 (16:34 +1300)]
dsdb:samdb: schemainfo update with relax control

Currently schema info's revision field isn't incremented if relax
control is present.  This is so that no increment is done during
provision, but we need the relax control in other situations where
the increment is desired, so we should use the provision control instead
to disable schema info update.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13799

Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
10 days agopython/provision: use provision and relax controls for schema provision
Stefan Metzmacher [Fri, 8 Mar 2019 10:28:42 +0000 (11:28 +0100)]
python/provision: use provision and relax controls for schema provision

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13799

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
10 days agos4:provision: split out provision_self_join_modify_schema.ldif
Stefan Metzmacher [Fri, 8 Mar 2019 10:27:14 +0000 (11:27 +0100)]
s4:provision: split out provision_self_join_modify_schema.ldif

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13799

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
10 days agoldapcmp: ignore 'schemaInfo' if two domains are compared
Stefan Metzmacher [Thu, 21 Feb 2019 08:20:48 +0000 (09:20 +0100)]
ldapcmp: ignore 'schemaInfo' if two domains are compared

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13799

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
10 days agosamdb: test for schemainfo update with relax control
Aaron Haslett [Thu, 4 Apr 2019 01:39:41 +0000 (14:39 +1300)]
samdb: test for schemainfo update with relax control

Currently schema info's revision field isn't incremented if relax
control is present.  This is so that no increment is done during
provision, but we need the relax control in other situations where the
increment is desired.  This patch adds a failing test to expose the
problem.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13799

Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
10 days agodrsuapi.idl: add DRSUAPI_ATTID_schemaInfo
Stefan Metzmacher [Fri, 22 Feb 2019 23:14:31 +0000 (00:14 +0100)]
drsuapi.idl: add DRSUAPI_ATTID_schemaInfo

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13799

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
11 days agoldb_kv: Skip @ records early in a search full scan
Andrew Bartlett [Thu, 4 Apr 2019 21:46:50 +0000 (10:46 +1300)]
ldb_kv: Skip @ records early in a search full scan

@ records like @IDXLIST are only available via a base search on the specific name
but the method by which they were excluded was expensive, after the unpack the
DN is exploded and ldb_match_msg_error() would reject it for failing to match the
scope.

This uses the fact that @ records have the DN=@ prefix on their TDB/LMDB key
to quickly exclude them from consideration.

Based on analysis by Garming Sam.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13893

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Wed Apr 10 06:23:39 UTC 2019 on sn-devel-144

11 days agovfs_full_audit: Fix logging of get_real_filename output
Christof Schmitt [Tue, 9 Apr 2019 20:57:36 +0000 (13:57 -0700)]
vfs_full_audit: Fix logging of get_real_filename output

result == 0 indicated success. In that case log the available
found_name.

Signed-off-by: Christof Schmitt <cs@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Wed Apr 10 01:17:28 UTC 2019 on sn-devel-144

11 days agoctdb-test: Adding test case to verify queue resizeing
Swen Schillig [Mon, 18 Mar 2019 14:25:54 +0000 (15:25 +0100)]
ctdb-test: Adding test case to verify queue resizeing

If a data packet arrives which exceeds the queue's current buffer size,
the buffer needs to be increased to hold the full packet. Once the packet
is processed the buffer size should be decreased to its standard size again.
This test case verifies this process.

Signed-off-by: Swen Schillig <swen@linux.ibm.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Christof Schmitt <cs@samba.org>
Autobuild-User(master): Christof Schmitt <cs@samba.org>
Autobuild-Date(master): Wed Apr 10 00:17:37 UTC 2019 on sn-devel-144

11 days agoctdb-test: Adding test case verifying data in buffer move
Swen Schillig [Mon, 18 Mar 2019 14:22:19 +0000 (15:22 +0100)]
ctdb-test: Adding test case verifying data in buffer move

Signed-off-by: Swen Schillig <swen@linux.ibm.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Christof Schmitt <cs@samba.org>
11 days agoctdb-test: Modify ctdb_io_test test_setup to provide queue reference
Swen Schillig [Mon, 18 Mar 2019 14:15:25 +0000 (15:15 +0100)]
ctdb-test: Modify ctdb_io_test test_setup to provide queue reference

Some test scenarios require access to the created queue.
Prepare the test_setup function to provide it as additional parameter.

Signed-off-by: Swen Schillig <swen@linux.ibm.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Christof Schmitt <cs@samba.org>
11 days agoleases_db: Make leases_db_del use leases_db_do_locked
Volker Lendecke [Mon, 8 Apr 2019 13:38:01 +0000 (15:38 +0200)]
leases_db: Make leases_db_del use leases_db_do_locked

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Tue Apr  9 19:31:09 UTC 2019 on sn-devel-144

11 days agoleases_db: Make leases_db_add use leases_db_do_locked
Volker Lendecke [Mon, 8 Apr 2019 13:33:30 +0000 (15:33 +0200)]
leases_db: Make leases_db_add use leases_db_do_locked

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
11 days agoleases_db: Make leases_db_rename atomic
Volker Lendecke [Mon, 8 Apr 2019 13:18:31 +0000 (15:18 +0200)]
leases_db: Make leases_db_rename atomic

Do the rename under one lock to protect against potential races while
we don't hold it.

Factor out the NDR marshalling into leases_db_do_locked(), leaving the
rename function pretty simple.

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
11 days agosmbd: Factor out map_lease_type_to_oplock
Volker Lendecke [Tue, 18 Sep 2018 08:53:23 +0000 (10:53 +0200)]
smbd: Factor out map_lease_type_to_oplock

grant_fsp_oplock_type has enough complex logic, make this a bit shorter

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
11 days agolib: Initialize variables in parse_resolvconf_fp
Volker Lendecke [Tue, 9 Apr 2019 10:49:00 +0000 (12:49 +0200)]
lib: Initialize variables in parse_resolvconf_fp

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
11 days agolib: Initialize getline() arguments
Volker Lendecke [Tue, 9 Apr 2019 10:47:13 +0000 (12:47 +0200)]
lib: Initialize getline() arguments

Keep "len" valid across the loop iterations for getline to consume

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13892

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
13 days agoCVE-2019-3880 s3: rpc: winreg: Remove implementations of SaveKey/RestoreKey.
Jeremy Allison [Wed, 27 Mar 2019 19:51:27 +0000 (12:51 -0700)]
CVE-2019-3880 s3: rpc: winreg: Remove implementations of SaveKey/RestoreKey.

Remove the now unused code implementations of
registry file io.

As reported by Michael Hanselmann.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13851

Signed-off-by: Jeremy Allison <jra@samba.org>
Autobuild-User(master): Karolin Seeger <kseeger@samba.org>
Autobuild-Date(master): Mon Apr  8 11:43:31 UTC 2019 on sn-devel-144

13 days agoCVE-2019-3880 s3: rpc: winreg: Remove implementations of SaveKey/RestoreKey.
Jeremy Allison [Thu, 21 Mar 2019 21:51:30 +0000 (14:51 -0700)]
CVE-2019-3880 s3: rpc: winreg: Remove implementations of SaveKey/RestoreKey.

The were not using VFS backend calls and could only work
locally, and were unsafe against symlink races and other
security issues.

If the incoming handle is valid, return WERR_BAD_PATHNAME.

[MS-RRP] states "The format of the file name is implementation-specific"
so ensure we don't allow this.

As reported by Michael Hanselmann.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13851

Signed-off-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
13 days agoCVE-2019-3870 pysmbd: Ensure a zero umask is set for smbd.mkdir()
Andrew Bartlett [Thu, 21 Mar 2019 04:24:14 +0000 (17:24 +1300)]
CVE-2019-3870 pysmbd: Ensure a zero umask is set for smbd.mkdir()

mkdir() is the other call that requires a umask of 0 in Samba.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13834

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
13 days agoCVE-2019-3870 pysmbd: Move umask manipuations as close as possible to users
Andrew Bartlett [Thu, 14 Mar 2019 05:20:06 +0000 (18:20 +1300)]
CVE-2019-3870 pysmbd: Move umask manipuations as close as possible to users

Umask manipulation was added to pysmbd with e146fe5ef96c1522175a8e81db15d1e8879e5652 in 2012
and init_files_struct was split out in 747c3f1fb379bb68cc7479501b85741493c05812 in 2018 for
Samba 4.9. (It was added to assist the smbd.create_file() routine used in the backup and
restore tools, which needed to write files with full metadata).

This in turn avoids leaving init_files_struct() without resetting the umask to
the original, saved, value.

Per umask(2) this is required before open() and mkdir() system calls (along
side other file-like things such as those for Unix domain socks and FIFOs etc).

Therefore for safety and clarify the additional 'belt and braces' umask
manipuations elsewhere are removed.

mkdir() will be protected by a umask() bracket, for correctness, in the next patch.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13834

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
13 days agoCVE-2019-3870 pysmbd: Include tests to show the outside umask has no impact
Andrew Bartlett [Thu, 21 Mar 2019 04:21:58 +0000 (17:21 +1300)]
CVE-2019-3870 pysmbd: Include tests to show the outside umask has no impact

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13834

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
13 days agoCVE-2019-3870 tests: Add test to check file-permissions are correct after provision
Tim Beale [Fri, 15 Mar 2019 00:52:50 +0000 (13:52 +1300)]
CVE-2019-3870 tests: Add test to check file-permissions are correct after provision

This provisions a new DC and checks there are no world-writable
files in the new DC's private directory.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13834

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
13 days agoCVE-2019-3870 tests: Extend smbd tests to check for umask being overwritten
Tim Beale [Fri, 15 Mar 2019 02:20:21 +0000 (15:20 +1300)]
CVE-2019-3870 tests: Extend smbd tests to check for umask being overwritten

The smbd changes the umask - if the code fails to restore the umask to
what it was, then this is very bad. Add an extra check to every
smbd-related test that the umask at the end of the test is the same as
what it was at the beginning (i.e. if the smbd code changed the umask
then it correctly restored the value afterwards).

As the selftest sets the umask for all tests to zero, it makes it hard
to detect this problem, so the test setUp() needs to set it to something
else first.

This extra checking is added to the setUp()/tearDown() so that it
applies to all test-cases. However, any failure that occur with this
approach will not be able to be known-failed.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13834

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
13 days agoselftest: Correct name of flapping smb2.notify test
Andrew Bartlett [Mon, 8 Apr 2019 02:04:08 +0000 (14:04 +1200)]
selftest: Correct name of flapping smb2.notify test

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Mon Apr  8 03:09:42 UTC 2019 on sn-devel-144

13 days agoldb: version 2.0.0
Aaron Haslett [Wed, 20 Mar 2019 00:52:16 +0000 (13:52 +1300)]
ldb: version 2.0.0

* Version bump for adding index_format_fn to the schema syntax structure.
* Range index support added, allowing <= and >= operations to be indexed
* Improved reindex performance by setting the in-memory TDB hash size correctly

Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
13 days agoldb_mdb: Add some warnings about poorly constructed callbacks
Garming Sam [Wed, 3 Apr 2019 23:04:47 +0000 (12:04 +1300)]
ldb_mdb: Add some warnings about poorly constructed callbacks

Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
13 days agoldb: tests for <= and >= integer indexing with duplicates
Garming Sam [Wed, 3 Apr 2019 03:12:35 +0000 (16:12 +1300)]
ldb: tests for <= and >= integer indexing with duplicates

We need to make sure that duplicates are correctly returned (uSNChanged
for instance is UNIQUE but, we should be able to index on attributes
which are not unique).

Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
13 days agoldb: tests for <= and >= integer indexing
Aaron Haslett [Mon, 11 Mar 2019 03:39:13 +0000 (16:39 +1300)]
ldb: tests for <= and >= integer indexing

Testing max, min and negative values for indexed 32 and 64 bit types.
This has to be done in two different files because the 64 bit type is
LDB_SYNTAX_INTEGER which is implemented at the ldb level, while the 32
bit is added in the ldb-samba module.  Schema syntax binding added for
ldb-samba.

We also need to make sure that full scans are not invoked for LMDB.

Pair-programmed-with: Garming Sam <garming@catalyst.net.nz>

Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
13 days agoldb: Add ORDERED_INTEGER to the proto-schema handling
Garming Sam [Mon, 25 Mar 2019 00:52:42 +0000 (13:52 +1300)]
ldb: Add ORDERED_INTEGER to the proto-schema handling

Adding ordered integer proto schema handling in kv index cache.  This
allows ordered 64 bit integers to be used in cached fields like
@ATTRIBUTES

Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
13 days agoschema_syntax: Add comments for our index format functions
Garming Sam [Thu, 4 Apr 2019 21:22:28 +0000 (10:22 +1300)]
schema_syntax: Add comments for our index format functions

We had to devise our own scheme for writing integers in a human readable
format which also sorted correctly numerically. This might look a bit
confusing to outsiders, so here's a large comment as a peace offering.

Pair-programmed-with: Tim Beale <timbeale@catalyst.net.nz>

Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
13 days agoldb: activating <= and >= indexing for integers
Aaron Haslett [Thu, 14 Mar 2019 05:05:23 +0000 (18:05 +1300)]
ldb: activating <= and >= indexing for integers

Activating <= and >= mdb indexing in samba for int32 and int64 attributes by:
1. Adding index_format_fn to LDB_SYNTAX_SAMBA_INT32 in ldb_samba
2. Cloning the 64bit LDB_SYNTAX_INTEGER type as LDB_SYNTAX_ORDERED_INTEGER
3. Adding index_format_fn to the new type
4. Modifying LargeInteger use the new type in samba schema
5. Bumping the index version to trigger reindexing

Pair-programmed-with: Garming Sam <garming@catalyst.net.nz>

Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
13 days agoldb_kv_index: Add a giant comment in regards to index_format_fn
Garming Sam [Thu, 4 Apr 2019 02:02:15 +0000 (15:02 +1300)]
ldb_kv_index: Add a giant comment in regards to index_format_fn

The reason we needed it in the first place was that the original
canonicalize is being used for non-index functions and it never produced
the right order originally (at least for integers).

Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
13 days agoldb_kv_index: Make the edge keys slightly cleaner and generic
Garming Sam [Thu, 4 Apr 2019 01:36:08 +0000 (14:36 +1300)]
ldb_kv_index: Make the edge keys slightly cleaner and generic

It makes no difference in our standard case because \0 will always go
before any value for our index_format_fn, but this is better for
correctness (in case we do mess up our NUL terminations elsewhere).

Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
13 days agoldb: <= and >= indexed searching
Aaron Haslett [Mon, 4 Mar 2019 06:06:31 +0000 (19:06 +1300)]
ldb: <= and >= indexed searching

Full implementation of <= and >= indexed searching using iterate_range
backend operation.  Adds index_format_fn to ldb_schema_syntax so
requires an ABI version bump.  The function must be provided for any
type for which <= and >= indexing is required, and must return a
lexicographically ordered canonicalization of a value.  This causes
index entries to be written in correct order to the database, so
iterate_range on the index DNs can be used.

ldb_kv_index_key is modified to return an index DN with attribute name
but without value if an empty value is provided.  This is needed for
constructing keys that match the beginning or end of an index DN range.

Pair-programmed-with: Garming Sam <garming@catalyst.net.nz>

Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
13 days agolmdb: iterate_range cmocka testing
Aaron Haslett [Mon, 4 Mar 2019 02:08:25 +0000 (15:08 +1300)]
lmdb: iterate_range cmocka testing

Cmocka testing for LMDB iterate_range operation added in previous commit.

Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
13 days agolmdb: iterate_range implementation
Garming Sam [Sun, 3 Mar 2019 23:50:24 +0000 (12:50 +1300)]
lmdb: iterate_range implementation

Adding iterate_range to LDB API and implementing in LMDB.  This
operation takes a start_key and end_key and returns all records between
the two, inclusive of both.  This will be used to implementing indexing
for <= and >= expressions.

Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
13 days agolibnet vampire: NULL access bug fix
Garming Sam [Mon, 11 Mar 2019 22:16:38 +0000 (11:16 +1300)]
libnet vampire: NULL access bug fix

NULL pointer access bug fix

Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2 weeks agoctdb: Fix a typo
Volker Lendecke [Thu, 4 Apr 2019 14:33:22 +0000 (16:33 +0200)]
ctdb: Fix a typo

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Autobuild-User(master): Martin Schwenke <martins@samba.org>
Autobuild-Date(master): Sat Apr  6 11:51:55 UTC 2019 on sn-devel-144

2 weeks agoctdb: Slightly simplify ctdb_ltdb_lock_fetch_requeue
Volker Lendecke [Fri, 18 Jan 2019 15:50:22 +0000 (16:50 +0100)]
ctdb: Slightly simplify ctdb_ltdb_lock_fetch_requeue

Reduce indentation with an early return

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Martin Schwenke <martin@meltin.net>
2 weeks agotorture: Add test for talloc size accounting in memcache
Christof Schmitt [Thu, 28 Mar 2019 17:46:43 +0000 (10:46 -0700)]
torture: Add test for talloc size accounting in memcache

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13865

Signed-off-by: Christof Schmitt <cs@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Sat Apr  6 06:08:42 UTC 2019 on sn-devel-144

2 weeks agomemcache: Increase size of default memcache to 512k
Christof Schmitt [Fri, 5 Apr 2019 22:43:21 +0000 (15:43 -0700)]
memcache: Increase size of default memcache to 512k

With the fixed accounting of talloc objects, the default cache size
needs to increase. The exact increase required depends on the workloads,
going form 256k to 512k seems like a reasonable guess.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13865

Signed-off-by: Christof Schmitt <cs@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
2 weeks agomemcache: Properly track the size of talloc objects
Christof Schmitt [Mon, 1 Apr 2019 23:23:35 +0000 (16:23 -0700)]
memcache: Properly track the size of talloc objects

With memcache_add_talloc, the talloc object becomes part of the pool and
the memcache_element stores a pointer to the talloc object. The
size of the the talloc object was not used when tracking the used space,
allowing the cache to grow larger than defined in the memcache_init
call.

Fix this by adding the size of the talloc object to the used space.

Also record the initial size of the talloc object for proper adjustment
of the used space in the cache later. This is in case the size of the
talloc object is modified while being owned by the cache (e.g.
allocating talloc child objects). This should never happen, but better
be safe than ending up with a broken cache usage counter.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13865

Signed-off-by: Christof Schmitt <cs@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
2 weeks agomemcache: Introduce struct for storing talloc pointer
Christof Schmitt [Mon, 1 Apr 2019 22:38:59 +0000 (15:38 -0700)]
memcache: Introduce struct for storing talloc pointer

This allows extending the additional data stored for talloced objects
later.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13865

Signed-off-by: Christof Schmitt <cs@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
2 weeks agonetcmd: Fix passwordsettings --max-pwd-age command
Tim Beale [Tue, 2 Apr 2019 20:10:55 +0000 (09:10 +1300)]
netcmd: Fix passwordsettings --max-pwd-age command

The min_pwd_age and max_pwd_age parameters are both optional and default
to None. However, if we just set the max-pwd-age, then the check
'min_pwd_age >= max_pwd_age' will throw a Python exception because it's
trying to compare an int to NoneType (min_pwd_age). This works on Python 2
but is a problem on Python 3.

We could just add a check that min_pwd_age is not None, but that defeats
the point of having the check if you're only setting either the min or
max age indepedently.

This patch gets the current min/max password age from the DB (in ticks).
If either setting is changed, the ticks will be updated. Then at the end
we check the min is still less than the max (to do this, we convert the
ticks back to days in the interests of readability).

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13873

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Fri Apr  5 08:03:08 UTC 2019 on sn-devel-144

2 weeks agonetcmd: Add some timestamp conversion helper functions
Tim Beale [Mon, 1 Apr 2019 22:10:41 +0000 (11:10 +1300)]
netcmd: Add some timestamp conversion helper functions

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13873

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2 weeks agonetcmd: Use python constant for -0x8000000000000000
Tim Beale [Mon, 1 Apr 2019 03:42:32 +0000 (16:42 +1300)]
netcmd: Use python constant for -0x8000000000000000

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13873

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2 weeks agotests: Add test for setting min/maxPwdAge
Tim Beale [Mon, 1 Apr 2019 03:32:27 +0000 (16:32 +1300)]
tests: Add test for setting min/maxPwdAge

Currently setting maxPwdAge doesn't work at all.

While we're adding a test, we might as well assert that minPwdAge
can't be greater than maxPwdAge as well.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13873

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2 weeks agoldb_kv_search: avoid handling uninitialised dn
Douglas Bagnall [Thu, 4 Apr 2019 02:08:18 +0000 (15:08 +1300)]
ldb_kv_search: avoid handling uninitialised dn

If ldb_kv_filter_attrs() fails, we don't know that the dn of filtered_msg
is OK.

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Fri Apr  5 05:46:55 UTC 2019 on sn-devel-144

2 weeks agoautobuild: attempt authenticated email if environment suggests it
Douglas Bagnall [Wed, 3 Apr 2019 00:17:17 +0000 (13:17 +1300)]
autobuild: attempt authenticated email if environment suggests it

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2 weeks agodsdb/modules: minor comment typos in samba_dsdb
Douglas Bagnall [Sun, 31 Mar 2019 21:25:28 +0000 (10:25 +1300)]
dsdb/modules: minor comment typos in samba_dsdb

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2 weeks agodsdb mods/extended_dn_store: used the ldb we already have
Douglas Bagnall [Thu, 28 Mar 2019 03:06:56 +0000 (16:06 +1300)]
dsdb mods/extended_dn_store: used the ldb we already have

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2 weeks agospell "recursive"
Douglas Bagnall [Wed, 27 Mar 2019 02:37:54 +0000 (15:37 +1300)]
spell "recursive"

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2 weeks agoperf-tests: rename paged search test for regex disambiguation
Douglas Bagnall [Wed, 3 Apr 2019 21:36:37 +0000 (10:36 +1300)]
perf-tests: rename paged search test for regex disambiguation

We like to use "TESTS=medley" for the old ad_dc_medley.

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2 weeks agorpc: Convert npa_tstream.c to use tstream_u32_read_send
Volker Lendecke [Mon, 1 Apr 2019 13:23:11 +0000 (15:23 +0200)]
rpc: Convert npa_tstream.c to use tstream_u32_read_send

This avoids a bit of code duplication. Overall the last two commits
add a few lines, but that also contains the header file and another GPL
header for tstream_u32_read.c.

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Fri Apr  5 01:00:48 UTC 2019 on sn-devel-144

2 weeks agorpc: Add tstream_u32_read
Volker Lendecke [Thu, 21 Mar 2019 18:41:28 +0000 (19:41 +0100)]
rpc: Add tstream_u32_read

In npa_tstream.c we have two next_vector functions reading a big
endian uin32_t length and then the blob described by the length. This
factors that next_vector out into a central routine.

Why? I'll add another NPA protocol in the future, and this would add
yet another two copies of that next_vector code

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
2 weeks agos3-messages: add mallinfo() information to pool-usage report
Ralph Wuerthner [Fri, 29 Mar 2019 11:44:50 +0000 (12:44 +0100)]
s3-messages: add mallinfo() information to pool-usage report

Signed-off-by: Ralph Wuerthner <ralph.wuerthner@de.ibm.com>
Reviewed-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Christof Schmitt <cs@samba.org>
Autobuild-User(master): Christof Schmitt <cs@samba.org>
Autobuild-Date(master): Thu Apr  4 23:39:25 UTC 2019 on sn-devel-144

2 weeks agos3-messages: modify msg_pool_usage() to allow enhanced memory reports
Ralph Wuerthner [Fri, 29 Mar 2019 11:30:45 +0000 (12:30 +0100)]
s3-messages: modify msg_pool_usage() to allow enhanced memory reports

Signed-off-by: Ralph Wuerthner <ralph.wuerthner@de.ibm.com>
Reviewed-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Christof Schmitt <cs@samba.org>
2 weeks agopython join: Set index transaction cache size.
Gary Lockyer [Mon, 1 Apr 2019 03:49:38 +0000 (16:49 +1300)]
python join: Set index transaction cache size.

The default value is too small for joining a large domain.  So we specify a
size of 200,000 which is suitable for domains with up to 100,000 users.

At a later date this could be added as a parameter to the join, but
200,000 should be suitable for now.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Thu Apr  4 07:45:03 UTC 2019 on sn-devel-144

2 weeks agolib ldb key_value: Set index cache size on open
Gary Lockyer [Mon, 1 Apr 2019 03:33:52 +0000 (16:33 +1300)]
lib ldb key_value: Set index cache size on open

Set the default index cache from the passed option
"transaction_index_cache_size" on open.  This allows the default cache
size to be overridden when processing large transactions i.e. joining a
large domain.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2 weeks agolib ldb key_value: set the cache size for re-indexing
Gary Lockyer [Mon, 1 Apr 2019 02:28:31 +0000 (15:28 +1300)]
lib ldb key_value: set the cache size for re-indexing

Set the index cache size to the number of records in the databse when
reindexing.

This significantly improves reindex performance.  For a domain with
100,000 users the reindex times are reduced from 17 minutes to 45
seconds.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2 weeks agolib ldb key_value: Add get_size method
Gary Lockyer [Mon, 1 Apr 2019 02:27:32 +0000 (15:27 +1300)]
lib ldb key_value: Add get_size method

Add the get_size method to the ldb_key_value layer, this will allow the
reindexing code to get an estimate of the number of records in the
database.

The lmdb backend returns an accurate count of the number of records in
the database withe the mdb_env_stat call.

The tdb backend does not provide a low cost method to determine the
number of records on the database.  It does provide a tdb_summary call
however this this walks the entire database.

So for tdb we use the map size divided by 500, this over estimates the counts
for small domains, but the extra memory allocated for the cache should
not be significant.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2 weeks agolib ldb key_value: Pass index cache size
Gary Lockyer [Mon, 1 Apr 2019 01:10:10 +0000 (14:10 +1300)]
lib ldb key_value: Pass index cache size

Pass the index cache size to ldb_kv_index_transaction_start.  This will
allow it to be set for reindex and join operations, where the current
defaults result in a significant performance penalty on large databases.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2 weeks agolib ldb key_value: Remove index cache lazy initialisation
Gary Lockyer [Mon, 1 Apr 2019 00:12:20 +0000 (13:12 +1300)]
lib ldb key_value: Remove index cache lazy initialisation

Remove the lazy initialisation of the index cache. This make setting
the size of the cache for re-indexing easier, which will be done in
later commits.

Performance testing shows that the removal of lazy initialisation makes
no appreciable difference to performance.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2 weeks agoflapping tests: Add samba3.smb2.notify
Gary Lockyer [Wed, 3 Apr 2019 00:03:56 +0000 (13:03 +1300)]
flapping tests: Add samba3.smb2.notify

Add samba3.smb2.notify until Metze gets time to follow it up.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2 weeks agos3-libnet_join: allow fallback to NTLMSSP auth in libnet_join
Günther Deschner [Tue, 2 Apr 2019 11:16:55 +0000 (13:16 +0200)]
s3-libnet_join: allow fallback to NTLMSSP auth in libnet_join

When a non-DNS and non-default admin domain is provided during the join
sometimes we might not be able to kinit with 'user@SHORTDOMAINNAME'
(e.g. when the winbind krb5 locator is not installed). In that case lets
fallback to NTLMSSP, like we do in winbind.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13861

Guenther

Signed-off-by: Guenther Deschner <gd@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
Autobuild-User(master): Andreas Schneider <asn@cryptomilk.org>
Autobuild-Date(master): Wed Apr  3 18:57:31 UTC 2019 on sn-devel-144

2 weeks agos3-libnet_join: setup libnet join error string when AD connect fails
Günther Deschner [Tue, 2 Apr 2019 11:16:11 +0000 (13:16 +0200)]
s3-libnet_join: setup libnet join error string when AD connect fails

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13861

Guenther

Signed-off-by: Guenther Deschner <gd@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
2 weeks agos3-libnet_join: always pass down admin domain to ads layer
Günther Deschner [Tue, 2 Apr 2019 11:14:06 +0000 (13:14 +0200)]
s3-libnet_join: always pass down admin domain to ads layer

Otherwise we could loose the information that a non-default domain name
has been used for admin creds.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13861

Guenther

Signed-off-by: Guenther Deschner <gd@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
2 weeks agorpc: Fix a typo
Volker Lendecke [Mon, 1 Apr 2019 19:14:05 +0000 (21:14 +0200)]
rpc: Fix a typo

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Wed Apr  3 17:53:30 UTC 2019 on sn-devel-144

2 weeks agoauth: Add necessary decoration to auth/auth_util.h
Volker Lendecke [Wed, 3 Apr 2019 10:16:50 +0000 (12:16 +0200)]
auth: Add necessary decoration to auth/auth_util.h

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
2 weeks agorpc: Don't crash if npa_accept fails
Volker Lendecke [Tue, 2 Apr 2019 14:37:39 +0000 (16:37 +0200)]
rpc: Don't crash if npa_accept fails

We can only rely on session_info_transport to be filled correctly upon
success of the accept_existing_recv function

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>