tdb: version 1.3.18

* Fix build problems with older python versions.
* C99 build fixes.
* Fix standalone build of tdb.

Signed-off-by: Stefan Metzmacher <metze@samba.org>

Search for location of waf script

When calling make from the ldb, talloc, tdb, and
tevent bundles, we need to first find the
location of the waf script. Currently the build
fails since it can't find waf.

Fixes regression caused by a660b7f.

Signed-off-by: David Mulder <dmulder@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

CVE-2019-3824 ldb: Release ldb 1.6.1

* CVE-2019-3824 out of bounds read in wildcard compare (bug 13773)

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13773

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
Autobuild-User(master): Gary Lockyer <gary@samba.org>
Autobuild-Date(master): Mon Feb 25 22:54:13 CET 2019 on sn-devel-144

CVE-2019-3824 ldb: Add tests for ldb_wildcard_match

Add cmocka tests for ldb_wildcard_match.

Running test_wildcard_match under valgrind reproduces
 CVE-2019-3824 out of bounds read in wildcard compare (bug 13773)

 valgrind --suppressions=lib/ldb/tests/ldb_match_test.valgrind\
          bin/ldb_match_test

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13773

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

CVE-2019-3824 ldb: wildcard_match end of data check

ldb_handler_copy and ldb_val_dup over allocate by one and add a trailing '\0'
to the data, to make them safe to use the C string functions on.

However testing for the trailing '\0' is not the correct way to test for
the end of a value, the length should be checked instead.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13773

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

CVE-2019-3824 ldb: wildcard_match check tree operation

Check the operation type of the passed parse tree, and return
LDB_INAPPROPRIATE_MATCH if the operation is not LDB_OP_SUBSTRING.

A query of "attribute=*" gets parsed as LDB_OP_PRESENT, checking the
operation and failing ldb_wildcard_match should help prevent confusion
writing tests.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13773

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

CVE-2019-3824 ldb: ldb_parse_tree use talloc_zero

Initialise the created ldb_parse_tree with talloc_zero, this ensures
that it is correctly initialised if inadvertently passed to a function
expecting a different operation type.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13773

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

CVE-2019-3824 ldb: Improve code style and layout in wildcard processing

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13773

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>

CVE-2019-3824 ldb: Extra comments to clarify no pointer wrap in wildcard processing

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13773

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>

CVE-2019-3824 ldb: Out of bound read in ldb_wildcard_compare

There is valgrind error in few tests tests/test-generic.sh
 91 echo "Test wildcard match"
 92 $VALGRIND ldbadd $LDBDIR/tests/test-wildcard.ldif  || exit 1
 93 $VALGRIND ldbsearch '(cn=test*multi)'  || exit 1
 95 $VALGRIND ldbsearch '(cn=*test_multi)'  || exit 1
 97 $VALGRIND ldbsearch '(cn=test*multi*test*multi)'  || exit 1

e.g.
  ==3098== Memcheck, a memory error detector
  ==3098== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
  ==3098== Using Valgrind-3.14.0 and LibVEX; rerun with -h for copyright info
  ==3098== Command: ./bin/ldbsearch (cn=test*multi)
  ==3098==
  ==3098== Invalid read of size 1
  ==3098==    at 0x483CEE7: memchr (vg_replace_strmem.c:890)
  ==3098==    by 0x49A9073: memmem (in /usr/lib64/libc-2.28.9000.so)
  ==3098==    by 0x485DFE9: ldb_wildcard_compare (ldb_match.c:313)
  ==3098==    by 0x485DFE9: ldb_match_substring (ldb_match.c:360)
  ==3098==    by 0x485DFE9: ldb_match_message (ldb_match.c:572)
  ==3098==    by 0x558F8FA: search_func (ldb_kv_search.c:549)
  ==3098==    by 0x48C78CA: ??? (in /usr/lib64/libtdb.so.1.3.17)
  ==3098==    by 0x48C7A60: tdb_traverse_read (in /usr/lib64/libtdb.so.1.3.17)
  ==3098==    by 0x557B7C4: ltdb_traverse_fn (ldb_tdb.c:274)
  ==3098==    by 0x558FBFA: ldb_kv_search_full (ldb_kv_search.c:594)
  ==3098==    by 0x558FBFA: ldb_kv_search (ldb_kv_search.c:854)
  ==3098==    by 0x558E497: ldb_kv_callback (ldb_kv.c:1713)
  ==3098==    by 0x48FCD58: tevent_common_invoke_timer_handler (in /usr/lib64/libtevent.so.0.9.38)
  ==3098==    by 0x48FCEFD: tevent_common_loop_timer_delay (in /usr/lib64/libtevent.so.0.9.38)
  ==3098==    by 0x48FE14A: ??? (in /usr/lib64/libtevent.so.0.9.38)
  ==3098==  Address 0x4b4ab81 is 0 bytes after a block of size 129 alloc'd
  ==3098==    at 0x483880B: malloc (vg_replace_malloc.c:309)
  ==3098==    by 0x491048B: talloc_strndup (in /usr/lib64/libtalloc.so.2.1.15)
  ==3098==    by 0x48593CA: ldb_casefold_default (ldb_utf8.c:59)
  ==3098==    by 0x485F68D: ldb_handler_fold (attrib_handlers.c:64)
  ==3098==    by 0x485DB88: ldb_wildcard_compare (ldb_match.c:257)
  ==3098==    by 0x485DB88: ldb_match_substring (ldb_match.c:360)
  ==3098==    by 0x485DB88: ldb_match_message (ldb_match.c:572)
  ==3098==    by 0x558F8FA: search_func (ldb_kv_search.c:549)
  ==3098==    by 0x48C78CA: ??? (in /usr/lib64/libtdb.so.1.3.17)
  ==3098==    by 0x48C7A60: tdb_traverse_read (in /usr/lib64/libtdb.so.1.3.17)
  ==3098==    by 0x557B7C4: ltdb_traverse_fn (ldb_tdb.c:274)
  ==3098==    by 0x558FBFA: ldb_kv_search_full (ldb_kv_search.c:594)
  ==3098==    by 0x558FBFA: ldb_kv_search (ldb_kv_search.c:854)
  ==3098==    by 0x558E497: ldb_kv_callback (ldb_kv.c:1713)
  ==3098==    by 0x48FCD58: tevent_common_invoke_timer_handler (in /usr/lib64/libtevent.so.0.9.38)
  ==3098==
  # record 1
  dn: cn=test_multi_test_multi_test_multi,o=University of Michigan,c=TEST
  cn: test_multi_test_multi_test_multi
  description: test multi wildcards matching
  objectclass: person
  sn: multi_test
  name: test_multi_test_multi_test_multi
  distinguishedName: cn=test_multi_test_multi_test_multi,o=University of Michiga
   n,c=TEST

  # returned 1 records
  # 1 entries
  # 0 referrals

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13773

Signed-off-by: Lukas Slebodnik <lslebodn@fedoraproject.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>

libcli: Pass buf/len to smb2_negotiate_context_add

Every caller did a data_blob_const() right before calling
smb2_negotiate_context_add(). Avoid that.

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
Autobuild-User(master): Andreas Schneider <asn@cryptomilk.org>
Autobuild-Date(master): Mon Feb 25 21:07:22 CET 2019 on sn-devel-144

libsmb: Resolve special _recv handling in cli_ntcreate

cli_smb2_create_fnum_recv will gain output create blobs soon and thus
differ from the NT1 function.

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>

smbd: Align integer types

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>

torture: Use GUID_zero()

10 lines less...

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>

s3: smbd: filenames - ensure we replace the missing '/' if we error in an intermediate POSIX path.

Previous regression test ensures we still return the correct
error code for POSIX pathname operations.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13803

Signed-off-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>
Autobuild-User(master): Volker Lendecke <vl@samba.org>
Autobuild-Date(master): Mon Feb 25 09:33:27 CET 2019 on sn-devel-144

s3: torture: Add additional POSIX mkdir tests.

Ensure that if POSIX_foo exists as a file
we return the correct error code NT_STATUS_OBJECT_PATH_NOT_FOUND
if we try and traverse it as a directory.

Also ensure creation/deletion of POSIX_foo/foo fails
for directories and files with NT_STATUS_OBJECT_PATH_NOT_FOUND
if the directory POSIX_foo/ doesn't exist.

knownfail is back :-).

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13803

Signed-off-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>

ctdb-cluster-mutex: Separate out command and file handling

This code is difficult to read and there really is no common code
between the 2 cases.  For example, there is no need to split a
filename into words.  Separating each of the 2 cases into its own
function makes the logic much easier to understand.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13800

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Mon Feb 25 03:40:16 CET 2019 on sn-devel-144

ctdb-tests: Add a test for configuring the recovery lock as a command

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13800

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-tests: Add -R option for local daemons to use recovery lock command

Under the covers, a command is always used.  However, there is no way
of testing of the code path where a command is explicitly configured.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13800

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-tests: Force test failure if local daemon setup fails

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13800

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-recoverd: Time out attempt to take recovery lock after 120s

Currently this will wait forever.  It really needs a timeout in case
the cluster filesystem (or other lock mechanism) is completely wedged.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13800

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-recoverd: Ban node on unknown error when taking recovery lock

We really shouldn't see unknown errors.  They probably represent a
misconfigured recovery lock or similar.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13800

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-recoverd: Make recoverd context available in recovery lock handle

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13800

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-recoverd: Clean up logging on failure to take recovery lock

Add an explicit case for a timeout and clean up the other messages.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13800

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-recoverd: Free cluster mutex handler on failure to take lock

If nested events occur while the file descriptor handler is still
active then chaos can ensue.  For example, if a node is banned and the
lock is explicitly cancelled (e.g. due to election loss) then
double-talloc-free()s abound.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13800

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-config: Change example recovery lock setting to one that fails

ctdbd will start without a recovery lock configured.  It will log a
message saying that this is not optimal.  However, a careless user may
overlook both this message and the importance of setting a recovery
lock.  If the existing example configuration is uncommented then the
directory containing it will be created (by 01.reclock.script) and the
failure (i.e. multiple nodes able to take the lock) will be confusing.

Instead, change the example setting to one that will result in banned
nodes, encouraging users to consciously configure (or deconfigure) the
recovery lock.  Tweak the corresponding comment.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13790

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

smbd: unix_convert: Ensure we don't call get_real_filename on POSIX paths.

For posix_pathnames don't blunder into the name_has_wildcard OR
get_real_filename() codepaths as they may be doing case insensitive lookups.
So when creating a new POSIX directory 'Foo' they might
match on name 'foo'.

Remove POSIX-MKDIR from knownfail.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13803

Signed-off-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>
Autobuild-User(master): Volker Lendecke <vl@samba.org>
Autobuild-Date(master): Sun Feb 24 14:04:14 CET 2019 on sn-devel-144

smbd: SMB1-POSIX: Add missing info-level SMB_POSIX_PATH_OPEN for UCF_UNIX_NAME_LOOKUP flag.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13803

Signed-off-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>

s3: smbtorture3: Add POSIX-MKDIR test for posix_mkdir case sensitive bug.

Test does:

mkdir POSIX_foo
mkdir POSIX_Foo
mkdir POSIX_foo/Foo
mkdir POSIX_foo/foo
mkdir POSIX_Foo/Foo
mkdir POSIX_Foo/foo

Which should pass a SMB1 POSIX extensions server
as posix mkdir should always be case sensitive
no matter what the share is set to.

Mark as knownfail for now.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13803

Signed-off-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>

winbindd: set idmap cache entries as the last step in async wb_xids2sids

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13802

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>
Autobuild-User(master): Volker Lendecke <vl@samba.org>
Autobuild-Date(master): Sat Feb 23 09:23:22 CET 2019 on sn-devel-144

winbindd: track whether a result from xid2sid was coming from the cache

This is needed in preparation of moving the step to update the idmap
cache from the per-idmap-domain callback wb_xids2sids_dom_done() to the
top-level callback wb_xids2sids_done().

Currently the sequence of action is:

* check cache, if not found:
  * ask backends
  * cache result from backend
* return results

Iow, if we got something from the cache, we don't write the cache.

The next commit defers updating the cache to the top-level callback, so
the sequence becomes

* check cache, if not found:
  * ask backends
* cache results
* return results

This has two problems:

* it needlessly writes to the cache what we just got from it

* it possibly overwrites the ID_TYPE_BOTH for a SID-to-xid mapping in
  the following case:

  - existing ID_TYPE_BOTH mapping in the cache, eg:

    IDMAP/SID2XID/S-1-5-21-2180672342-2513613279-2566592647-512 -> Value: 3000000:B

  - someone calls wb_xids2sids_send() with xid.id=3000000,xid.type=ID_TYPE_GID

  - cache lookup with idmap_cache_find_gid2sid() succeeds

  - when caching results we'd call idmap_cache_set_sid2unixid() with the
    callers xid.type=ID_TYPE_GID, so idmap_cache_set_sid2unixid() will
    overwrite the SID-to-xid mapping with ID_TYPE_GID

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13802

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>

winbindd: switch send-next/done order

In preparation of adding more logic to the done step. No change in
behaviour.

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13802

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>

winbindd: update xid in wb_xids2sids_state->xids with what we got

In preparation of priming the idmap cache in the top-level
wb_xids2sids_done(), not in the per-idmap-domain callback
wb_xids2sids_dom_done().

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13802

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>

winbindd: convert id to a pointer in wb_xids2sids_dom_done()

No change in behaviour.

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13802

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>

winbindd: make xids a const argument to wb_xids2sids_send()

The previous commit made an internal copy of xids, this commit makes it
more obvious that we must not mess with the xids argument but treat it as
an in-parameter and don't write to it.

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13802

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>

winbindd: make a copy of xid's in wb_xids2sids_send()

This is in preparation of setting the result of the mapping in the top-
level callback wb_xids2sids_done(), not in the per-idmap-domain callback
wb_xids2sids_dom_done().

When caching the mapping we need the id-type from the backend, so we
need a way to pass up that information from wb_xids2sids_dom_done() up
to wb_xids2sids_done()

The xids array copy gets passed from wb_xids2sids_send() to
wb_xids2sids_dom_send(), so wb_xids2sids_dom_done() can then directly
update the top-level copy.

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13802

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>

s3:winbindd: Remove unused arcfour.h from PAM handling

Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Fri Feb 22 23:16:40 CET 2019 on sn-devel-144

s3:rpc_server: Remove unused arcfour.h from netlogon

Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

s3:auth: Remove unused arcfour.h from auth_util.c

Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

s3:auth: Remove unused arcfour.h header from server_info.c

Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

s4:dns_crypto: Remove unused include of hmac_md5.h

Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

libsmb: Fix a resource leak in cli_posix_mkdir

smbd does posix_mkdir if the wire flags are exactly

if (wire_open_mode == (SMB_O_CREAT|SMB_O_DIRECTORY))

open_flags_to_wire however adds a SMB_O_RDONLY, so that we enter the
normal open routine which happens to create a directory as well. The
main difference is that posix_mkdir does *NOT* return an open
handle. As we did not enter this code path due to the SMB_O_RDONLY we
leak a SMB1 fd per cli_posix_mkdir call.

Pretty hard to test automatically, this would be an interaction with
smbstatus.

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

libsmb: Pull up wire_flags calculation from open_internal

This avoids passing down a boolean

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

libsmb: Convert cli_posix_open to normal tevent_req pattern

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

set caller allocation units in statvfs f_bavail

Signed-off-by: Mark Niggemann <mark.niggemann@ge.com>
Reviewed-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Bjoern Jacke <bjacke@samba.org>
Autobuild-User(master): Björn Jacke <bjacke@samba.org>
Autobuild-Date(master): Fri Feb 22 15:04:55 CET 2019 on sn-devel-144

notifyd: Fix SIGBUS on sparc

Problem is the structure "struct notify_instance" can lay in message buffer on
address not dividable by 8 but this structure begins by uint_64 (secs in
time-stamp). Structure should be re-packed to standalone buffer before the use.

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13704
Signed-off-by: jiri.sasek@oracle.com
Reviewed-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Autobuild-User(master): Björn Jacke <bjacke@samba.org>
Autobuild-Date(master): Fri Feb 22 12:30:11 CET 2019 on sn-devel-144

ctdb-tests: Add test for ctdb_io.c

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13791

Signed-off-by: Christof Schmitt <cs@samba.org>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Autobuild-User(master): Martin Schwenke <martins@samba.org>
Autobuild-Date(master): Fri Feb 22 03:51:37 CET 2019 on sn-devel-144

ctdb: buffer write beyond limits

In order to calculate the number of bytes correctly which
are to be read into the buffer, the buffer.offset must be taken
into account.

This patch fixes a regression introduced by 382705f495dd.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13791

Signed-off-by: Swen Schillig <swen@linux.ibm.com>
Reviewed-by: Christof Schmitt <cs@samba.org>
Reviewed-by: Martin Schwenke <martin@meltin.net>

selftest: Confirm new and old SDDL strings differ after a samba-tool dsacl set

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Thu Feb 21 05:37:31 CET 2019 on sn-devel-144

selftest: Add test for samba-tool dsacl get, cross-checked with samba-tool dsacl set

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

samba-tool dsacl: Mark old and new descriptor output correctly

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

Add command "samba-tool dsacl get" This code is very equal to "samba-tool dsacl set", except it only prints out the current sddl of an object.

Signed-off-by: Martin Krämer <mk.maddin@gmail.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

Update dsacl.py - add_ace to handle/verify sddl parameter correct

Test for samba-tool dsacl set --sddl parmeter

Update tests.py - add dsacl (dsacl.py / samba-tool dsacl set) test

Signed-off-by: <Martin Krämer mk.maddin@gmail.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

bootstrap/README.md: add README.md

Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

bootstrap/dists: add rendered files for dists

Add these into repo, then we can link it to samba wiki,
for people to get a latest and precise pkg list

Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

bootstrap/docker.py: add cli to build/tag/push docker images

Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

bootstrap/template.py: add cli to render templates

- bootstrap for each dist
- Dockerfile for each dist
- Vagrantfile all in one

Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

bootstrap/config.py: define package lists and templates

Define default pkg list, and allow to override for each dist.
Also define bootstrap/Dockerfile/Vagrantfile templates.

Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

selftest/tests: add smoketests for dbcheck --quick-membership-checks

Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

selftest/tests: add helper method to simplify plantestsuite

Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

dbcheck: skip reverse member link checks when cli option specified

currently dbcheck cmd tooks about 1 day to finish on a 100k user database.
We can skip member reverse link checks to speed it up dramatically.
A new cli option is added to enable the skipping.

Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

s4/param: Fix provision_get_schema leaking python object

provision_get_schema returns a ldb_context object which is stored
in a python object. As a result the parent python object is never
decrefed and probably not released ever.

Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

vfs_ceph: refactor if-error-return-else logic

vfs_ceph has quite a few occurrences of:
if (result < 0) {
WRAP_RETURN(result); /* calls return */
} else {
...
}

This change drops the superfluous else {} encapsulation and also removes
duplication of ceph statx debug messages.

Signed-off-by: David Disseldorp <ddiss@samba.org>
Reviewed-by: Guenther Deschner <gd@samba.org>
Autobuild-User(master): Günther Deschner <gd@samba.org>
Autobuild-Date(master): Wed Feb 20 13:56:09 CET 2019 on sn-devel-144

vfs_glusterfs: check for VFS_ADD_FSP_EXTENSION() failure

Signed-off-by: David Disseldorp <ddiss@samba.org>
Reviewed-by: Guenther Deschner <gd@samba.org>

WHATSNEW: winbind authentication logging

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Wed Feb 20 07:43:10 CET 2019 on sn-devel-144

winbind: Log PAM and NTLM authentications.

Generate JSON authentication messages for winbind PAM_AUTH and
PAM_AUTH_CRAP requests.  The logon_id in these messages can be used to
link them to the SamLogon messages.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

winbind: Generate and pass logon ID

Generate a random logon_id and pass it in the SamLogon calls.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

kdc hdb: Generate and pass logon ID

Generate and pass the logon_id in SamLogon calls

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

s4 rpc netlogon: Pass logon_id to auth logging

Pass the logon_id passed in the netlogon identity information to
auth_logging.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

auth log: Log the netlogon logon id.

Add code to log the logonId in the JSON Authentication messages.

The version number for Authentication messages changes from 1.1 to 1.2
to reflect this.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

librpc idl: netlogon netr_identity_info logon_id to 64 bit

Fold the two 32 bit values logon_id_high and logon_id_low into a single
64 bit logon_id in netr_identity_info.  This will be used to tie
together winbind and SamLogon requests in audit logging.

Summary of the of the Query and Response from Microsoft on it's usage.

[REG:119013019612095] [MS-NRPC]: NETLOGON_LOGON_IDENTITY_INFO: Does
the Reserved field have LogonId meaning?

Questions:
  In NetrLogonSamLogonEx does the Reserved field
  (of NETLOGON_LOGON_IDENTITY_INFO) have LogonId meaning?

  What is a valid LogonID, and does have any audit usage?

  Samba is sending a constant "deadbeef" in hex and would like to
  understand any usage of this field.

Response:
  The NRPC spec is accurate in defining the field as Reserved, and without
  protocol significance. In the header file in our source code, it is
  defined as LogonId and commented as such, but it’s effectively not used.
  This is probably why the API structure has that field name. It may have
  been intended as such but it’s not used.

Samba will send a random value in this field.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

lib util: Add function to generate random uint64_t

Generate a random uint64_t , which will be used for the netlogon
logon_id.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

s3 auth: Create messaging and lp contexts.

If 'auth event notifications' are enabled create an imessaging_context
and a loadparm_context that can be passed to log_authentication_event.

This will allow the generated authentication messages to be tested.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

s3 winbind auth_log: Tests for logon id logging.

Tests to validate that winbind generates a random logon_id and passes it
in the netlogon call.

This will allow the linking of the windbind authentication requests and
the SamLogon request on the DC.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

wbinfo: fix --ntlmv1 option

Currently using the --ntlmv1 option fails with an unknown option error.
This patch ensures that the option is correctly supported.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

auth_log tests: Allow the remote address to be None

Allow self.remoteAddress to be None, remote address filtering is not
required for the winbind auth logging tests.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

selftest: Add basic sanity-check tests for nopython target

Previously we were only checking samba compiled OK with
--disable-python, not that it actually ran.

The main problem is all the make test framework is based around
subunit/smbtorture, neither of which we seem to build with
disable-python. However, for just a simple sanity-check, we can just
bypass all the subunit-filter work and just call the Perl code directly.
This works OK as long as it's just simple shell script tests that we're
running, as we can check the script's exit code directly.

The main thing that we really want to test is that we can start up the
smbd testenv and connect to it (i.e. a simple smbclient test).

This patch adds a new 'make test-nopython' target. This disables the
subunit filtering, and runs a small test-list that was generated manually.

Note that currently this has the limitation that it doesn't support known
failures or flapping tests. However, just checking that smd starts up OK
is probably OK for now.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Wed Feb 20 02:10:00 CET 2019 on sn-devel-144

selftest: Abort if we fail to startup testenv with '--one' option

The --one selftest.pl option means abort when the first test fails.
However, when 'make test' fails to startup a testenv, it'll try to
continue and run other tests by default. When '--one' is used,
selftest.pl can just die() at that point.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

s3:tests: Set/return $failed in test_smbclient_auth.sh

Update the test so the shell script returns pass/fail as the exit code.

Note that subunit is just looking for 'failure:' in the test output for
whether the test passed or failed, so setting $failed isn't strictly
required.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

nssswitch: Log user access to kerberos

Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Noel Power <npower@samba.org>
Autobuild-User(master): Andreas Schneider <asn@cryptomilk.org>
Autobuild-Date(master): Mon Feb 18 13:01:12 CET 2019 on sn-devel-144

.gitlab-ci.yml: merge .gitlab-ci-private.yml

`.gitlab-ci.yml` support conditional jobs with `only` and `except`.
And variables can be read from repo CI/CD settings as condition:

    build_samba:
      script: ...
      only:
        variables:
          - $SUPPORT_PRIVATE_TEST == 'yes'

Instead of having 2 copies of yml file, we can use this feature to
trigger private jobs only when a var like `SUPPORT_PRIVATE_TEST` is defined.

I've already added above var to our repos.
Once merged, we can remove custom CI config file in
gitlab repo settings, and remove .gitlab-ci-private.yml file from code.

Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Mon Feb 18 10:54:19 CET 2019 on sn-devel-144

wafsamba/symbols: change regex to match both rpath and runpath for different readelf output

In `wafsamba.dumplicate_symbols` test, it will use Popen to call:

    readelf --dynamic bin/default/source3/lib/netapi/examples/netlogon/netlogon_control2

then try to find rpath lib lines from output with regex:

    re_rpath     = re.compile(b'Library rpath: \[(.*)\]')

In ubuntu 14.04 docker image, which current CI is using, the actual output
from `readelf` is `runpath` instead of 'rpath':

    ...
    Library runpath: [/home/gitlab-runner/samba/bin/shared:/home/gitlab-runner/samba/bin/shared/private]\n'
    ...

So the regex never matched, and hide a bug.

In Ubuntu 1604 docker image, the output changes to `rpath` and matched the
regex, which expose the error in previous commit.

Improve the regex to match both `rpath` and `runpath`.

Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Alexander Bokovoy <ab@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

wafsamba/symbols: always split Popen output by bytes

In py3, `wafsamba.duplicate_symbols` test may fail:

    ...
    elfpipe = subprocess.Popen(['readelf', '--dynamic', binname], stdout=subprocess.PIPE).stdout
    ...

    File "./buildtools/wafsamba/symbols.py", line 153, in get_libs
        rpath.extend(m.group(1).split(":"))
    TypeError: a bytes-like object is required, not 'str'

Because Popen will always return bytestr even in py3, and ":" is a
unicode str in py3.  Change ":" to b":" to fix.

Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Alexander Bokovoy <ab@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

selftest: Initialize DC_SERVER/etc variables in one place

It's simpler to do setup DC_SERVER/etc in the same place we set
SERVER/etc. (Rather than initializing them for every single testenv,
like we were doing).

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Mon Feb 18 07:24:05 CET 2019 on sn-devel-144

selftest: Add helper function to avoid repeated {DC_SERVER}=x

For every 2-DC testenv we setup, we copy the $DC_SERVER value of the
dependent testenv (i.e. the PDC) into the env variables for the new
testenv. This means DC_SERVER always points to the PDC (or first DC).

This adds a helper function to avoid repeating this code for every 2-DC
environment we setup.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

selftest: Remove need for $RODC_DC_SERVER env variable

Same deal as earlier patch - we can use the $SERVER env variable instead
and avoid the need for this extra variable.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

selftest: Remove unused environment variables

We only really want generic environment variables. For 2+ DC
environments, we have the $SERVER and $DC_SERVER (aka PDC) variables.
However, lots of testenvs also export really specific environment
variables, e.g. VAMPIRE_2000_DC_SERVER_IP (despite that testenv being
only used for a single test case).

Previously the <testenv>_SERVER variable was used for DRS tests, but we
can avoid the need to do this now. The other variables are not used at
all.

The RODC and TRUST environment variables are still used by a few tests.
SUBDOM_DC_SERVER is only used within Samba4.pm and not exported.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

tests: Tweak DRS tests to avoid unnecessary env variables

Each DC used in a DRS test has its own '<testenv>_SERVER' environment
variable, e.g. VAMPIRE_DC_SERVER. These variables are only used by
test.py for DRS, but they're not actually needed.

The $SERVER environment variable holds the same information, so we can
just use this in test.py instead.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

selftest: Export DC conf path for special cases

In a few rare cases, a test needs to assert aspects both client-side and
server-side aspects. A typical example would be the audit logging, which
is exercising client-side behaviour, but also asserting the server-side
logging.

Usually this has involved a kludge in tests.py to either use
socket-wrapper explicitly, or hardcode in the server smb.conf path.

This patch exposes the existing SERVERCONFFILE env variable to the
tests. DC_SERVERCONFFILE has been added for 2 DC testenvs, where we need
the PDC's smb.conf.

The benefit of doing this way is the filepath/testenv-dependency logic
is all self-contained with the Perl code, and it doesn't bleed out into
tests.py as well.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

selftest: Add common function to return cmd environment variables

We prefix samba-tool commands with a bunch of WRAPPER/CONF environment
variables, in order for the command to work properly. These variables
are duplicated all over the place in the selftest code. This patch adds
a helper function to return the variables, so we can reduce the required
code down to a single line in a lot of places.

A couple of exceptions I've left alone:
- drs replicate, which omits the RESOLV_WRAPPER_CONF/_HOSTS variables
  (I'm not sure whether that's deliberate or not).
- create_backup(), which uses the backupfromdc's krb5.conf rather than
  the new testenv.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

third_party/nss_wrapper/wscript: check for libnsl and libsocket

this is needed as there are HAVE_LIBNSL and HAVE_LIBSOCKET in the code and
Samba fails to build in a terrible obscure way on Solaris if this is not
working inside nss_wrapper here.

Signed-off-by: Bjoern Jacke <bjacke@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
Autobuild-User(master): Björn Jacke <bjacke@samba.org>
Autobuild-Date(master): Sun Feb 17 15:05:20 CET 2019 on sn-devel-144

waf: add library dependency for sendfile on Solaris

Signed-off-by: Bjoern Jacke <bjacke@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>

third_party/nss_wrapper/wscript: fix check for gethostbyname

Signed-off-by: Bjoern Jacke <bjacke@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>

waf: fix setting of RPATH_ST variable

Signed-off-by: Bjoern Jacke <bjacke@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>

wafsamba: we should also remove stale symlinks here

Signed-off-by: Bjoern Jacke <bjacke@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>

waf: print the library name in which we search for a function

Signed-off-by: Bjoern Jacke <bjacke@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>

waf: use the correct WERROR_CFLAGS in CHECK_CODE

all the non gcc version were incorrectly set here till now

Signed-off-by: Bjoern Jacke <bjacke@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>

waf: remove redundant WERROR flag

CHECK_CFLAGS always uses WERROR flags

Signed-off-by: Bjoern Jacke <bjacke@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>

waf: remove duplicate WERROR cflags

WERROR flags are already added by the strict=True switch.

Signed-off-by: Bjoern Jacke <bjacke@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>

waf: fix compiler warnings in configure checks

the studio compiler issued here:

warning: statement not reached

Signed-off-by: Bjoern Jacke <bjacke@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>