s3:messages: check reg->refcount == 0 before accessing other elements

Pair-Programmed-With: Stefan Metzmacher <metze@samba.org>

Signed-off-by: Ralph Boehme <slow@samba.org>
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>

winbind: Remove an unused struct declaration

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Tue Apr 24 04:44:30 CEST 2018 on sn-devel-144

dbwrap: Remove dependency on samba-hostconfig

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

dbwrap: Remove calls to loadparm

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

libcli: Call dbwrap_local_open with the correct tdb flags

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

libcli: Call dbwrap_local_open with the correct hash size

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

dbwrap3: Call dbwrap_local_open with the correct tdb flags

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

dbwrap3: Call dbwrap_local_open with the correct hash size

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

cluster4: Call dbwrap_local_open with the correct tdb_flags

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

cluster4: Call dbwrap_local_open with the correct hash size

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

credentials: Call dbwrap_local_open with the correct tdb_flags

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

credentials: Call dbwrap_local_open with the correct hash size

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

libcli: Call dbwrap_local_open with the correct tdb_flags

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

libcli: Call dbwrap_local_open with the correct hash size

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

s3:vfs_aio_pthread: make use of pthreadpool_tevent instead of pthreadpool_pipe

pthreadpool_tevent provides a much simpler api and avoids an extra
pipe for the completion notification.

This means we now have just one thread pool, that's shared for
all async pread, pwrite, fsync and openat() calls, instead of having
an extra pool for openat() with the same possible number of threads.

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>
Autobuild-User(master): Volker Lendecke <vl@samba.org>
Autobuild-Date(master): Mon Apr 23 13:30:06 CEST 2018 on sn-devel-144

selftest: add some basic testing for aio_pthread

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>

docs-xml: rewrite the vfs_aio_pthread manpage to reflect the >= 4.0.0 behavior

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>

s3:smbd: call pthreadpool_tevent_init() already in smbd_process()

pthreadpool_tevent_init() doesn't start any thread yet, it only
allocates a bit of memory.

It's easier to start this in a central place, so that it's
available to all VFS modules.

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>

build: fix build without JSON audit support

Autobuild-User(master): Björn Baumbach <bbaumbach@samba.org>
Autobuild-Date(master): Fri Apr 20 21:12:33 CEST 2018 on sn-devel-144

s3/security.c undefined value

s3/security.c had an NTSTATUS status that was undefined and with the configure
option --address-sanitizer this caused uninitialised value error.

Signed-off-by: William Brown <william@blackhats.net.au>
Reviewed-by: Alexander Bokovoy <ab@samba.org>
Reviewed-by: Simo Sorce <idra@samba.org>
Autobuild-User(master): Alexander Bokovoy <ab@samba.org>
Autobuild-Date(master): Fri Apr 20 18:18:25 CEST 2018 on sn-devel-144

ldb_wrap.c missing header

ldb_wrap.c was missing unistd.h causing implicit symbol declaration and error
during compilation.

Signed-off-by: William Brown <william@blackhats.net.au>
Reviewed-by: Alexander Bokovoy <ab@samba.org>
Reviewed-by: Simo Sorce <idra@samba.org>

s3:passdb: Do not return OK if we don't have pinfo set up

This prevents a crash in fill_mem_keytab_from_secrets()

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13376

Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Autobuild-User(master): Andreas Schneider <asn@cryptomilk.org>
Autobuild-Date(master): Thu Apr 19 00:32:22 CEST 2018 on sn-devel-144

build: add option to choose to build with or without JSON audit support

Add a new configure option:

  --with-json-audit
            Build with JSON auth audit support (default=auto). This requires
    the jansson devel package.

Signed-off-by: Björn Baumbach <bb@sernet.de>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
Autobuild-User(master):  <bbaumbach@samba.org>
Autobuild-Date(master): Wed Apr 18 15:25:42 CEST 2018 on sn-devel-144

nbt_server: Align integer types

sizeof returns size_t

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Autobuild-User(master): Volker Lendecke <vl@samba.org>
Autobuild-Date(master): Wed Apr 18 00:11:51 CEST 2018 on sn-devel-144

nbt_server: Factor out nbtd_node_status_reply_packet

Separate packet creation from sending out the packet. This way packet
creation can be used elsewhere in the future.

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

nbt_server: Factor out nbtd_name_query_reply_packet

Separate packet creation from sending out the packet. This way packet
creation can be used elsewhere in the future.

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

tstream: Fix CID 1167981 Unchecked return value

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Autobuild-User(master): Volker Lendecke <vl@samba.org>
Autobuild-Date(master): Mon Apr 16 19:09:56 CEST 2018 on sn-devel-144

tstream: Fix CID 1167982 Unchecked return value

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

libdgram: Fix an error path memleak

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Autobuild-User(master): Stefan Metzmacher <metze@samba.org>
Autobuild-Date(master): Fri Apr 13 21:04:28 CEST 2018 on sn-devel-144

libnbt: Align data types

ARRAY_SIZE returns size_t

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

libnbt: Add an explicit "mem_ctx" to name_request_send

Implicitly hanging requests off nbtsock is too inflexible for future use

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

traffic_relay: bulk port print to modern py3 style

Change print to function and avoid the ugly `print >>sys.stderr`.

Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
Autobuild-User(master): Douglas Bagnall <dbagnall@samba.org>
Autobuild-Date(master): Fri Apr 13 10:36:32 CEST 2018 on sn-devel-144

s4/webserver: initialise optional parameter

OK, this is unused and unimplemented.

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Alexander Bokovoy <ab@samba.org>

s4/lib/py-registry: initialize optional parameters for open_* functions

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Alexander Bokovoy <ab@samba.org>

s3/py_passdb: initialize optional parameters earlier

It is just a bit easier to see what is happening.

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Alexander Bokovoy <ab@samba.org>

nbt/pynbt: initialize optional parameter in nbt_name_refresh

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Alexander Bokovoy <ab@samba.org>

nbt/pynbt: initialize optional parameter in nbt_name_register

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Alexander Bokovoy <ab@samba.org>

ldb/pyldb: initialize optional parameter in ldb_connect()

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Alexander Bokovoy <ab@samba.org>

kcc/graph: add __hash__ to InternalEdge for py3

In py3, if a class defines `__eq__()` but not `__hash__()`, its instances will
not be usable as items in hashable collections, e.g.: set.

Add `__hash__()` to InternalEdge, so it can be added to a set in py3.

Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

kcc: fix sort for py3

py2:

    list.sort(cmp=None, key=None, reverse=False)
    sorted(iterable[, cmp[, key[, reverse]]])

py3:

    list.sort(key=None, reverse=False)
    sorted(iterable, *, key=None, reverse=False)

The `cmp` arg was removed in py3, make use of `key` arg to work around.

Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

kcc/kcc_utils: fix divide for py3

`/` will return float other than int in py3.
Use `//` to keep consistent with py2.

Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

kcc/kcc_utils: convert dict.keys to list

In py3, `dict.keys()` will return a iterator not a list.
Convert it to list to support both py2 and py3.

Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

kcc/graph_utils: port string.translate for py3

In py3, `str.translate` removed the second positional argument
`deletechars`, which means you can not use it to delete chars from str.
Use `replace` for this case.

Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

selftest/graph: enable py3 for samba.tests.graph

Changes are made separatedly in previous commits.
No change needed here.

Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

python/tests/graph: actually test graphs, don't print

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

graph: fix sort for py3

`sorted` can not sort `None` with str in py3, use the `key` arg to fix.
Sort None as ''.

Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

graph: fix divide for py3

`/` will return float other than int in py3.
Use `//` to keep consistent with py2.

Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

selftest: enable py3 for samba.tests.krb5_credentials

Fix encoding issue.

Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

selftest: enable py3 for samba.tests.docs

Popen methods will return bytes.
Decode output to string before using.

Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

selftest: enable py3 for samba.tests.source

In py2, `open` has no `encoding` arg, python guesses file encoding from
locale. This could be wrong.

Use `io.open` to open a file, so we can specify encoding in both py2 and
py3.

Also, open file with `r` instead of `rb` for py3.

Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

selftest: enable py3 for samba.tests.lsa_string

No change needed.

Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

selftest: enable py3 for samba.tests.dcerpc.registry

No changes needed.

Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

selftest: enable py3 for samba.tests.dcerpc.rpcecho

Prefix `b` for bytes.

Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

selftest: enable py3 for samba.tests.dcerpc.bare

Prefix b to bytes.

Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

selftest: enable py3 for samba.tests.dcerpc.sam

1. Fix invalid declaration syntax for toArray
2. Simplify toArray implementation with list comprehension.
3. Remove ending L for long integer.

Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

python: fix unicode escape in doc string

The doc string has `\u` mark inside, which will cause encoding error in
py3. prefix `r` to doc string to fix.

Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

python: bulk convert zip to list

In py3, zip will return a iterator other than a list.
Convert it to a list to support both py2 and py3.

Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

python: bulk replace file to open for py3

The builtin function `file` was removed in py3. Use `open` instead.

Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

python: bulk replace dict.itervalues to values for py3

In py3, iterxxx methods are removed.

Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

python: bulk port tdb iterkeys for py3

In py3, `dict.iterkeys()` is removed, we need to use `keys()` instead.
This is compatible with py2 since `dict.keys()` exists for py2.

tdb pretents to be a dict, however, not completely.
It provides `iterkeys()` for py2 only, and `keys()` for py3 only,
which means replace `iterkeys()` to `keys()` will break py2.

In python, iter a dict will implicitly iter on keys.
Use this feature to work around.

Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

python: bulk replace dict.iteritems to items for py3

In py3, iterxxx methods are removed.

Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

python/wscript: build grouping library for py2/py3 (--extra-python)

Grouping library is not used for any delivered libraries, however
it is used internally when building local libraries used when
"make test TESTS='blah'" is invoked. Failure to provide the grouping
library results in missing symbols (and cores) when running tests

Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

s3/lib/policy/wscript_build: build samba_policy lib for extra-python/py3

pypolicy module needs appropriate samba_policy library for
extra-python/py3 therefore we need to build it for it to be available

Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

s3/libnet/wscript: build samba-net lib for extra-python/py3

python_net needs to link appropriate samba-net library for extra-py/py3
version so we need to build it for extra-python/py3

Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

s3/param/wscript: build PROVISION subsytem for extra-python/py3

Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

wscript_build: make sure we link extra-python versions of libraries

Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

python: create test for pysmb module.

Signed-off-by: David Mulder <dmulder@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

s4/libcli: python3 port for smb module

Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

dbwrap: Fix "use mmap = no"

Mutexes require mmap.

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
Autobuild-User(master): Volker Lendecke <vl@samba.org>
Autobuild-Date(master): Thu Apr 12 23:04:18 CEST 2018 on sn-devel-144

torture: Test compound request request counters

This will send an unfixed smbd into the

SMB_ASSERT(op->request_count > 0);

in smbd_smb2_request_reply_update_counts

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13215
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Autobuild-User(master): Volker Lendecke <vl@samba.org>
Autobuild-Date(master): Thu Apr 12 14:38:39 CEST 2018 on sn-devel-144

s3:smb2_server: correctly maintain request counters for compound requests

If a session expires during a compound request chain,
we exit smbd_smb2_request_dispatch() with
'return smbd_smb2_request_error(req, ...)' before
calling smbd_smb2_request_dispatch_update_counts().

As req->request_counters_updated was only reset
within smbd_smb2_request_dispatch_update_counts(),
smbd_smb2_request_reply_update_counts() was called
twice on the same request, which triggers
SMB_ASSERT(op->request_count > 0);

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13215

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>

gpo: python chardet is not a dep of samba

Signed-off-by: David Mulder <dmulder@suse.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Douglas Bagnall <dbagnall@samba.org>
Autobuild-Date(master): Thu Apr 12 11:27:03 CEST 2018 on sn-devel-144

libgpo: gpo_copy_file() shouldn't explicitly call smb1

Don't call cli_openx directly to open a file this
calls smb1 code explicitly, which fails if we did
a multi-protocol negotiate and negotiated smb2+.
Use the higher level cli_open() instead.

Signed-off-by: David Mulder <dmulder@suse.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

param: Add python binding for lpcfg_cache_path

Signed-off-by: David Mulder <dmulder@suse.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

gpo: Create a gp_sec_ext module

Move the gp_sec_ext into it's own module,
which is how new gp_ext's will be created.

Signed-off-by: David Mulder <dmulder@suse.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

gpo: Move implementation from samba_gpoupdate

The implementation of group policy apply should
not be in the application script. One reason is
to implement user apply, we can call these
functions via the python c-api, (passing creds
via the command line will expose them via ps).
Another reason for this is if some overrides
the smb.conf "gpo update command" option, it
would be useful to have these functions.

Signed-off-by: David Mulder <dmulder@suse.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

gpo: Create base class gp_inf_ext

Abstract the process of reading from an ini file,
since other extensions will be reading gpos this
way.

Signed-off-by: David Mulder <dmulder@suse.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

gpo: Move the file parse function to gp_ext

A file will always be read from the sysvol the
same way, but the data will be read differently.
This patch moves the parse function to gp_ext,
and requires subclasses to implement the read()
function to interpret the data.

Signed-off-by: David Mulder <dmulder@suse.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

gpo: Move gp_sec_ext __init__ to base class

For this class to be extensible, the constructor
should be available to subclasses.

Signed-off-by: David Mulder <dmulder@suse.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

gpo: Rename the inf_to class to gp_ext_setter

This class will be subclassed and used for more
than just inf settings application.

Signed-off-by: David Mulder <dmulder@suse.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

Check "auth event notification" param in log_json

When Samba is compiled with JSON support it logs an error message
at log level 3 by default for every authentication event:

get_auth_event_server: Failed to find 'auth_event' registered on the
message bus to send JSON authentication events to:
NT_STATUS_OBJECT_NAME_NOT_FOUND

This patch adjusts the log_json function to consider the smb.conf
parameter "auth event notification" before calling auth_message_send.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13381

Signed-off-by: Arvid Requate <requate@univention.de>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Thu Apr 12 08:12:39 CEST 2018 on sn-devel-144

ldb-samba: require pid match for cached ldb

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

ldb_wrap: Remove the magic cache of database handles except for sam.ldb

sam.ldb is handled in samdb_connect_url(), not this function.

This cache caused issues when "private dir" was changed in a testing script, but also
just generates many-owner shared mutable state that is frowned upon these days.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

dsdb: Use talloc_get_type_abort() in schema_load_{start,end}_transaction

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13379

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

dsdb: ensure we take out a read lock during the dsdb_init

We have to also take it out in the partitions code when we load the
partition backends.

This ensures that the init handlers hold a whole-db lock just as the
search code does.

To ensure the locking count in schema_load is balanced, the
private data is now created in the first lock_read() call.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13379

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

dsdb: Rework schema reload during the read lock

Rather than refusing the reload based on making cached sequence numbers match
just load it once at the time the DB is globally locked, if required.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13379

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

dsdb: Load schema during the read_lock() hook, not the search

This should trigger slightly less often and is the more correct place, as
we only load it during the first lock when not in a transaction.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13379

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

dsdb: Allow search before init() is called in extended_dn_out

This matches the earlier check of p && p->normalise.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13379

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

dsdb: Move ldb_set_default_dns() into rootdse_get_private_data()

This call needs to be done at the very first chance, in this case
during the first call to the lock_read() hook, otherwise the
schema_data module can't find the schema.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13379

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

dsdb: Create rootdse_get_private_data()

This will get the private data on the first call, allowing that not to be
the init() hook.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13379

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

selftest: Make a transaction before @INDEXLIST etc is checked in dsdb_schema_attributes.py

This helps us remove the write to the database from the (soon to be
read locked) init code.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13379

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

dsdb: Wait until a transaction starts to call dsdb_schema_set_indices_and_attributes()

This avoids starting a transaction in schema_load_init() and allows it
to operate with a read lock held, which will avoid locking issues
(deadlock detected due to lock odering if we do not have a global
read lock).

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13379

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

dsdb: Allow search before init() call in encrypted_secrets

Simply do not decrypt anything until the init call is run.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13379

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

dsdb: Do not create a transaction in partition_init()

This will allow us to lock the databases for read during all of the Samba init
hooks.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13379

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

dsdb: Ensure to cancel the transaction if we fail to save the prefixMap

This rare error case forgot to call ldb_transaction_cancel()

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

ldb_wrap: Remove ldb_transaction_cancel_noerr from ldb_wrap_fork_hook()

Writing to a TDB, without locks (these are per-process) in a forked child is never going to
end well, if a transaction is open at this point we have bigger problems.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

dsdb: check for dSHeuristics more carefully

This check would pass if the dSHeuristics was treated as always being
000000000 for searches which is not enough, we must check for a value
of 000000001 (userPassword enabled).

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13378

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

dsdb: Check for userPassword support after loading the databases

The net result of this is only that userPassword values (which were
world readable when set) would still be visible after userPassword
started setting the main DB password.

In AD, those values become hidden once the dSHeuristics bit is set,
but Samba lost that when fixing a performance issue with
f26a2845bd42e580ddeaf0eecc9b46b823a0c6bc

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13378

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

s3: tests: Regression test to ensure we can never return a DIRECTORY attribute on a stream.

Tests streams_xattr and also streams_depot.

Inspired from a real-world test case by Andrew Walker <awalker@ixsystems.com>.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13380

Signed-off-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Ralph Böhme <slow@samba.org>
Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Thu Apr 12 02:04:28 CEST 2018 on sn-devel-144

s3: smbd. Generic fix for incorrect reporting of stream dos attributes on a directory

According to MS-FSA a stream name does not have
separate DOS attribute metadata, so we must return
the DOS attribute from the base filename. With one caveat,
a non-default stream name can never be a directory.

As this is common to all streams data stores, we handle
it here instead of inside all stream VFS modules.

Otherwise identical logic would have to be added to
all streams modules in their [f]get_dos_attribute_fn()
VFS calls.

Found in real-world use case by Andrew Walker <awalker@ixsystems.com>.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13380

Signed-off-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Ralph Böhme <slow@samba.org>