VERSION: Disable GIT_SNAPSHOTS for the 4.7.0rc2 release

Signed-off-by: Stefan Metzmacher <metze@samba.org>

WHATSNEW: Add release notes for Samba 4.7.0rc2

Signed-off-by: Stefan Metzmacher <metze@samba.org>

CVE-2017-11103: Orpheus' Lyre KDC-REP service name validation

In _krb5_extract_ticket() the KDC-REP service name must be obtained from
encrypted version stored in 'enc_part' instead of the unencrypted version
stored in 'ticket'.  Use of the unecrypted version provides an
opportunity for successful server impersonation and other attacks.

Identified by Jeffrey Altman, Viktor Duchovni and Nico Williams.

Change-Id: I45ef61e8a46e0f6588d64b5bd572a24c7432547c

BUG: https://bugzilla.samba.org/show_bug.cgi?id=12894
(based on heimdal commit 6dd3eb836bbb80a00ffced4ad57077a1cdf227ea)

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

VERSION: Bump version up to 4.7.0rc2...

and re-enable GIT_SNAPSHOTS.

Signed-off-by: Stefan Metzmacher <metze@samba.org>

VERSION: Disable GIT_SNAPSHOTS for the 4.7.0rc1 release

Signed-off-by: Karolin Seeger <kseeger@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

VERSION: Bump version up to 4.7.0rc1

Signed-off-by: Karolin Seeger <kseeger@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

WHATSNEW: Prepare release notes for Samba 4.7.0rc1.

Signed-off-by: Karolin Seeger <kseeger@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

ctdb-daemon: Increase priority of logs when recovery happens

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-daemon: Increase priority of logs when shutting down

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-daemon: Increase priority of logs when ctdb starts up disabled/stopped

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-daemon: Increase priority of logs when node is stopped/continued

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-daemon: Increase priority of logs for recmaster changes

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-daemon: Increase priority of logs for node connect/disconnect

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

WHATSNEW: Fix typo

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

WHATSNEW: Add docs for ntlm auth changes

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

s3/tests: add a net cache samlogon test

Bug: https://bugzilla.samba.org/show_bug.cgi?id=12875

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

net: fix net cache samlogon list output

Don't print the table header for every entry.

Bug: https://bugzilla.samba.org/show_bug.cgi?id=12875

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

smbldap: expose bind callback via API and increase smbldap ABI version

Until we fully migrate to use gensec in smbldap, we need to continue
exposing bind callback to allow FreeIPA to integrate with smbldap.

Since smbldap API is now lacking direct access to 'struct
smbldap_state' and new API functions were added to give access to
individual members of this structure, it makes sense to increase ABI
version too.

Signed-off-by: Alexander Bokovoy <ab@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Tue Jul  4 11:14:49 CEST 2017 on sn-devel-144

samr: Disable NTLM-based password changes on the server if NTLM is disabled

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

selftest: Disable NTLM authentication in ktest environment

This allows us to prove that "ntlm auth = disabled" works

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=11923

param: Add new "disabled" value to "ntlm auth" to disable NTLM totally

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=11923
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

selftest: Add test to confirm NTLM authentication is enabled

(or later, that it is disabled)

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=11923

param: Disable LanMan authentication unless NTLMv1 is also enabled

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=11923

selftest: Use new ntlmv2-only and mschapv2-and-ntlmv2-only options

This will allow the py_credentials test to tell if these are in use

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

auth: Allow NTLMv1 if MSV1_0_ALLOW_MSVCHAPV2 is given and re-factor 'ntlm auth ='

The ntlm auth parameter is expanded to more clearly describe the
role of each option, and to allow the new mode that permits MSCHAPv2
(as declared by the client over the NETLOGON protocol) while
still banning NTLMv1.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=12252
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Based on a patch by Mantas Mikulėnas <mantas@utenos-kolegija.lt>:

Commit 0b500d413c5b ("Added MSV1_0_ALLOW_MSVCHAPV2 flag to ntlm_auth")
added the --allow-mschapv2 option, but didn't implement checking for it
server-side. This implements such checking.

Additionally, Samba now disables NTLMv1 authentication by default for
security reasons. To avoid having to re-enable it globally, 'ntlm auth'
becomes an enum and a new setting is added to allow only MSCHAPv2.

Signed-off-by: Mantas Mikulėnas <mantas@utenos-kolegija.lt>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

selftest: Add test for support for MSCHAPv2 and NTLMv1 on a server

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

s3-rpc_server: Disable the NETLOGON server by default

The NETLOGON server is only needed when the classic/NT4 DC is enabled
and has been the source of security issues in the past.  Therefore
reduce the attack surface.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

tests: Add simple check whether netlogon server is running

Netlogon only needs to run in DC environment. This is a simple test to
check whether the netlogon service is running. This will allow us to
disable the netlogon service on setups that don't require it.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

auth: Disable SChannel authentication if we are not a DC

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

dns_server: Only install common library if AD DC is enabled.

The library is used in selftest, so must still be built

This reverts commit d32b66b40c931fe8214faa2e1d40b34b86667d4c and
replaces the behaviour.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

net: add net cache samlogon list|show|ndrdump|delete

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
Autobuild-User(master): Ralph Böhme <slow@samba.org>
Autobuild-Date(master): Tue Jul  4 00:12:46 CEST 2017 on sn-devel-144

samlogon_cache: add netsamlog_cache_for_all()

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>

netlogon.idl: mark session keys with NDR_SECRET

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>

s4/torture: test fetching a resume key twice

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

s3/smbd: remove unneeded flags argument from SMB_VFS_OFFLOAD_WRITE_SEND

...and instead use the fsctl to infer required behaviour in the VFS
backends.

Note that this removes the check from vfs_default because there we only
handle FSCTL_SRV_COPYCHUNK(_WRITE) and must always perform the lock
checks.

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

s3/smbd: get rid of files_struct.aapl_copyfile_supported

A previous commit removed the special hook from the SMB layer, so we
don't need this anymore.

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

s4/torture: more tests for copy-chunk across shares

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

s3/vfs: make SMB_VFS_OFFLOAD_WRITE_SEND offload token based

Remove the source fsp argument and instead pass the offload token
generated with SMB_VFS_OFFLOAD_READ_SEND/RECV.

An actual offload fsctl is not implemented yet, neither in the VFS nor
at the SMB ioctl layer, and returns NT_STATUS_NOT_IMPLEMENTED

With these changes we now pass the copy-chunk-across-shares test.

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

s4/torture: add a test for copy-chunk across shares

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

s3/smbd: redesign macOS copyfile copy-chunk

The copy-chunk request chunk_count can be 0 and Windows server just
returns success saying number of copied chunks is 0.

macOS client overload this after negotiating AAPL via their SMB2
extensions, meaning it's a so called copyfile request (copy whole file
and all streams).

We previously checked this at the SMB layer, with this patch we just
send this down the VFS, if vfs_fruit is loaded it implements the macOS
copyile semantics, otherwise we get Windows behavour..

No change in behaviour.

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

s3/smbd: remove copy-chunk chunk merging optimisation

As we won't have the source fsp around with the coming token based
offload read/write based code, we can't merge chunks as that requires
checking against the source file size.

We could still merge chunks without checking, but getting the error
handling correct would require comlicated logic for the SMB2 ioctl
copy-chunk error reporting.

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

s3/smbd: remove unused arg smb1req from copychunk_check_handles()

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

s3/smbd: remove flags2 FLAGS2_READ_PERMIT_EXECUTE hack in the SMB2 code

By adding a SMB2 specific CHECK_READ_SMB2 macro called that always
grants read access if execute was granted, we can get rid of the flags2
hack.

All callers in the SMB2 code are converted to use the CHECK_READ_SMB2
macro.

Amongs other things, this later allows moving the handle checks in
copychunk_check_handles() down into the VFS layer where we don't have
access to the smbreq.

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

s3/smbd: remove ununsed req arg from CHECK_READ_IOCTL macro

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

s3/vfs: rename SMB_VFS_COPY_CHUNK_SEND/RECV to SMB_VFS_OFFLOAD_WRITE_SEND/RECV

No change in behaviour, just a rename in preperation of more changes to
SMB_VFS_OFFLOAD_WRITE_SEND. It helps keeping the diff of the actual
changes smaller.

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

s3/smbd: use SMB_VFS_OFFLOAD_READ_SEND/RECV

No change in behaviour, this just uses the new SMB_VFS_OFFLOAD_READ_SEND
in the duplicate extents and the resume key ioctls.

In the copy-chunk/resume-key case this means using
SMB_VFS_OFFLOAD_READ_SEND to create the resume-key token that is
returned to the client.

In the duplicate-extents case this ensures we can later call
offload-write, which requires a previous call to offload-read that
associates a token with a file-handle.

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

s3/vfs: add SMB_VFS_OFFLOAD_READ_SEND/RECV

Add SMB_VFS_OFFLOAD_READ_SEND an SMB_VFS_OFFLOAD_READ_RECV.

This paves the way for supporting server-side copy-chunk with source and
destination file-handles on different shares. It can be used to
implement copy offload fsctl in the future, but for now this will be
used as a mere copy-chunk replacement.

SMB_VFS_OFFLOAD_READ generates a token that associates an fsp with the
token and stores the fsp in a in-memory db.

Initially only a copy-chunk resume key fsctl is supported. In the future
this can be enhanced to support real offload fsctl.

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

s4/torture: pass destination tree to test_setup_copy_chunk

No change in behaviour, will be used in subsequent commits.

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

librpc/idl: make use storage_offload_token

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

librpc/idl: fix STORAGE_OFFLOAD_TOKEN_TYPE_ZERO_DATA definition

STORAGE_OFFLOAD_TOKEN_TYPE_ZERO_DATA is defined as 0xffff0001 in MS-FSCC
2.3.79.

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

librpc/idl: convert offload flags to a bitmap

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

lib/util: add more tfork tests

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

lib/util: adjust loglevel in tfork test with samba_runcmd_send()

No change in behaviour, this just ensures stdout and stderror are
logged with log level 0.

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

lib/util: make use of tfork in samba_runcmd_send()

This makes it possible to use samba_runcmd_send() in processes like smbd
that install a SIGCHLD handler that reaps all terminated children.

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

lib/util: enhanced tfork()

This function is a solution to the problem of fork() requiring special
preperations in the caller to handle SIGCHLD signals and to reap the
child by wait()ing for it.

Instead, tfork provides a pollable file descriptor. The caller gets the
file descriptor by calling tfork_event_fd() on the handle returned from
tfork_create() and the caller can then get the status of the child
with a call to tfork_status().

tfork avoids raising SIGCHLD signals in the caller by installing a
temporary SIGCHLD handler from inside tfork_create() and tfork_status().

The termination signal of other child processes not created with tfork()
is forwarded to the existing signal handler if any.

There's one thing this thing can't protect us against and that is if a
process installs a SIGCHLD handler from one thread while another thread
is running inside tfork_create() or tfork_status() and the signal
handler doesn't forward signals for exitted childs it didn't fork, ie
our childs.

Pair-Programmed-With: Stefan Metzmacher <metze@samba.org>

Signed-off-by: Ralph Boehme <slow@samba.org>
Signed-off-by: Stefan Metzmacher <metze@samba.org>

s3:rpc_server: wrap make_auth4_context() into {become,unbecome}_root()

This need to create a temporary messaging context in order to do
the auth logging. This can only be done as root.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=12850

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
Autobuild-User(master): Garming Sam <garming@samba.org>
Autobuild-Date(master): Mon Jul  3 08:15:29 CEST 2017 on sn-devel-144

WHATSNEW: Improved AD performance (particularly linked attributes)

Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

WHATSNEW: DNS at domain join improvements

Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

WHATSNEW: Additional hashes introduced with WDigest

Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

WHATSNEW: Improved RODC support

Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

selftest: Prime the netlogon cache during test_idmap_rfc2307

This ensures that the group memberships just created are reflected in the test
comparison.  Otherwise we are trusting that no caches are primed, which is
simply not safe in a test.

(The login will put a list of groups, as obtained by the login over NETLOGON or
via the PAC, into the samlogon cache).

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Autobuild-User(master): Stefan Metzmacher <metze@samba.org>
Autobuild-Date(master): Sun Jul  2 21:59:18 CEST 2017 on sn-devel-144

selftest: Bind rfc2307 tests to exactly one server

The tests make changes to the DC, do not wait for replication, then expect
those to be reflected in the client.  If they bind to another server this
will not hold true.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

selftest: Use tree_delete control in idmap_rfc2307 test

This control removes an entire subtree, which was the intention of the previouse code
but much more effectively.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

selftest: Do not force run of kcc at start of selftest

This should help to avoid clashes between periodic and manual runs of
the KCC during autobuild.

Pair-Programmed-With: Andrew Bartlett <abartlet@samba.org>

Signed-off-by: Bob Campbell <bobcampbell@catalyst.net.nz>
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

WHATSNEW: Add entry for Multi-process LDAP Server

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

WHATSNEW: Add an entry for the LDB whole DB locking issue

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

ldap: Run the LDAP server with the default (typically standard) process model

This allows one LDAP socket to proceed if another fails, and reduces the
impact of a crash becoming a DoS bug, as it only impacts one socket.

This may mean we have a lot of idle tasks, but this should not be a big
issue

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

dsdb: Improve debugging on start transacton failure

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

dsdb: Teach the Samba partition module how to lock all the DB backends

The metadata partition (sam.ldb) lock is not
enough to block another process in prepare_commit(),
because prepare_commit() is a no-op, if nothing
was changed in the specific backend.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

dsdb: Add tests showing that the CN=CONFIGURATION partition is also locked

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

dsdb: Add new test adding a record to the top level sam.ldb file

This shows that locks are made on this file as well

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

dsdb: Add more locking more tests, confirming blocking locks in both directions

These extended tests allow us to show that a search (read) blocks a
transaction commit (write), and that a transaction commit blocks a
search.

Pair-Programmed-With: Andrew Bartlett <abartlet@samba.org>

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Signed-off-by: Andrew Bartlett <abartlet@samba.org>

dsdb: Add test showing a search can't start while a transaction is already repared in a backend partition

Pair-programmed-with: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

dsdb: Add test showing a search can't start while a transaction is already repared

Pair-programmed-with: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

ldb: version 1.2.0

* handle one more LDB_FLAG_INTERNAL_DISABLE_SINGLE_VALUE_CHECK
  case in ldb_tdb
* fix ldb_tdb locking (performance) problems
* fix ldb_tdb search inconsistencies by adding
  read_[un]lock() hooks to the module stack
  (bug #12858)
* add cmocka based tests for the locking issues
* ldb_version.h provides LDB_VERSION_{MAJOR,MINOR,RELEASE} defines
* protect ldb_modules.h from being used by Samba < 4.7
  Note: that this release (as well as 1.1.30 and 1.1.31)
  may cause problems for older applications, e.g. Samba
  See https://bugzilla.samba.org/show_bug.cgi?id=12859

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

ldb:includes: protect ldb_modules.h from being used by Samba < 4.7

Samba versions before 4.7 are incompatible with the read_[un]lock()
behaviour introduced into ldb.

This makes sure older Samba versions fail to compile against
ldb >= 1.2.0.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=12859

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

ldb:wscript: define EXPECTED_SYSTEM_LDB_VERSION_{MAJOR,MINOR,RELEASE}

This indicates what feature set Samba assumes from the used
libldb from the system.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=12859

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

ldb:wscript: provide LDB_VERSION_{MAJOR,MINOR,RELEASE} in ldb_version.h

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

ldb:tests: Extend api.py testsuite to show transaction_commit() blocks against the whole-db read lock

The new ldb whole-db lock behaviour now allows this test

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

ldb:tests: Extend api.py testsuite to show transaction contents can not be seen outside the transaction

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

ldb:tests: Add test to show that locks are released on TALLOC_FREE(req)

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

ldb:tests: Correct comment about version numbers

(ldb releases have been made while this patch set was in train)

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

ldb: Lock the whole backend database for the duration of a search

We must hold locks not just for the duration of each search, but for the whole search
as our module stack may make multiple search requests to build up the whole result.

This is explains a number of replication and read corruption issues in Samba

Pair-Programmed-With: Stefan Metzmacher <metze@samba.org>

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Signed-off-by: Stefan Metzmacher <metze@samba.org>

ldb_tdb: Implement read_lock and read_unlock module operations

This allows Samba to provide a consistent view of the DB
despite the use of multiple databases via the partitions module
and over multiple callbacks via a module stack.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

ldb: Add read_lock and read_unlock to ldb_module_ops

This will be used to implement read locking in ldb_tdb

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

ldb:tests: Add test encoding current locking behaviour during ldb_search()

Currently, a lock is not held against modifications once the final
record is returned via a callback, so modifications can be made
during the DONE callback.  This makes it hard to write modules
that interpert an ldb search result and do further processing
so will change in the future to allow the full search to be
atomic.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

ldb:tests: Show that writes do not appear during an ldb_search()

A modify or rename during a search must not cause a search to change
output, and attributes having an index should in particular not see
any change in behaviour in this respect

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

ldb:tests: don't assert the results before doing the final search finished

This is required to pass the test in future, because
otherwise the clean up will fail because we hold locks.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

ldb:tdb: Ensure we correctly decrement ltdb->read_lock_count

If we do not do this, then we never take the all record lock, and instead do a lock
for every record as we go, which is very slow during a large search

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

ldb_pack: use ldb_dn_from_ldb_val() and avoid a duplicate strlen() call

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

tdb: version 1.3.14

* allow tdb_traverse_read before tdb_transaction[_prepare]_commit()
* Improve documentation for tdb_transaction_start()
* Add new function tdb_transaction_active()

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

tdb: Add new function tdb_transaction_active()

This will allow callers to avoid their own reference counting of transactions.

Additionally, this will always line up with the acutal transaction state, even
in the error cases where tdb can cancel the transaction

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

tdb: Improve documentation for tdb_transaction_start()

It now references the TDB_ALLOW_NESTING and TDB_DISALLOW_NESTING flags

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

tdb: Remove locking from tdb_traverse_read()

This restores the original intent of tdb_traverse_read() in
7dd31288a701d772e45b1960ac4ce4cc1be782ed

This is needed to avoid a deadlock with tdb_lockall() and the
transaction start, as ldb_tdb should take the allrecord lock during a
search (which calls tdb_traverse), and can otherwise deadlock against
a transaction starting in another process

We add a test to show that a transaction can now start while a read
traverse is in progress

This allows more operations to happen in parallel.  The blocking point
is moved to the prepare commit.

This in turn permits a roughly doubling of unindexed search
performance, because currently ldb_tdb omits to take the lock due to
an unrelated bug, but taking the allrecord lock triggers the
above-mentioned deadlock.

This behaviour was added in 251aaafe3a9213118ac3a92def9ab2104c40d12a for
Solaris 10 in 2005. But the run-fcntl-deadlock test works also on Solaris 10,
see https://lists.samba.org/archive/samba-technical/2017-April/119876.html.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

drsuapi: Improve debugging in DsAddEntry()

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

wafsamba: add maxversion and version_blacklist to CHECK_BUNDLED_SYSTEM[_PKG]()

BUG: https://bugzilla.samba.org/show_bug.cgi?id=12859

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

s3: VFS: Change SMB_VFS_CONNECTPATH to take const struct smb_filename * instead of const char *.

We need to migrate all pathname based VFS calls to use a struct
to finish modernising the VFS with extra timestamp and flags parameters.

Signed-off-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Richard Sharpe <realrichardsharpe@gmail.com>
Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Sat Jul  1 07:20:28 CEST 2017 on sn-devel-144

S3: smbd: Finish plumbing struct smb_filename * through the check_name() stack.

Signed-off-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Richard Sharpe <realrichardsharpe@gmail.com>

s3: smbd: Add missing out of memory check.

Signed-off-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Richard Sharpe <realrichardsharpe@gmail.com>

s3: VFS: Change SMB_VFS_REALPATH to take and return struct smb_filename * instead of char *.

We need to migrate all pathname based VFS calls to use a struct
to finish modernising the VFS with extra timestamp and flags parameters.

Signed-off-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Richard Sharpe <realrichardsharpe@gmail.com>