ctdb: Centralize ctdb_version_string

Whenever the current git hash changes, we recompile ctdb.c and
ctdb_daemon.c. As both have quite a few warnings with -Wall, this
makes it quite difficult to see the real warnings that pop up during
development. Centralize the ctdb_version_string to just a single file
without warnings.

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdbd: Fix a typo

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

replmd: RMD_VERSION incorrectly incremented for link conflicts

This problem was noticed when 2 DCs added the same linked attribute at
roughly the same time. One DC would have a later timestamp than the
other, so it would re-apply the same link information. However, when it
did this, replmd_update_la_val() would incorrectly increment the
RMD_VERSION for the attribute. We then end up with one DC having a
higher RMD_VERSION than the others (and it doesn't replicate the new
RMD_VERSION out).

During replication RMD_VERSION is used to determine whether a linked
attribute is old (and should be ignored), or whether the information is
new and should be applied to the DB. This RMD_VERSION discrepancy could
potentially cause a subsequent linked attribute update to be ignored.

Normally when a local DB operation is performed, we just pass in a
version of zero and get replmd_update_la_val() to increment what's
already in the DB. However, we *never* want this to happen during
replication - we should always use the version we receive from the peer
DC.

This patch fixes the problem by separating the API into two:
- replmd_update_la_val(): we're updating a linked attribute in the DB,
  and so as part of this operation we always want to increment the
  version number (the version no longer need to be passed in because
  we can work it out from the existing DB entry).
- replmd_set_la_val(): we want to set a linked attribute to use the
  exact values we're telling it, including the version. This is what
  replication needs to use.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13038
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Tue Sep 26 09:36:48 CEST 2017 on sn-devel-144

selftest: Add test for a re-animated object conflict

Added a test to simulate a user accidentally being deleted and 2
different admins trying to resolve the problem simultaneously - one by
re-animating the object and one by just creating a new object with
the same name.

Currently this test fails on Samba because it chooses the higher
version
number as the winner instead of the latest change.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13039

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

selftest: Windows resolves object conflicts differently to Samba

While testing link conflicts I noticed that Windows resolves conflicts
differently to Samba. Samba considers the version number first when
resolving the conflict, whereas Windows always takes the latest change.

The existing object conflict test cases didn't detect this problem
because they were both modifying the object the same number of times (so
they had the same version number).

I've added new tests that highlight the problem. They are basically the
same as the existing rename tests, except that only one DC does the
rename. Samba will always pick the renamed object as the winner, whereas
Windows picks the most recent change.

I've marked this test as a known fail for now.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13039

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

selftest: replica_sync did not fully cleanup if test failed

Normally the replica_sync tests do the cleanup at the end of the test
case, rather than in the tearDown(). However, if the tests don't run to
completion (because they fail), then the objects may not get cleaned up
properly, which causes the tests to fail on the 2nd test-env.

The problem is the object deletion only occurs on DC2 and it relies on
replication to propagate the deletion to DC1. Presumably this
propagation could be missed because the tests are repeatedly turning off
inbound replication on both DCs.

This patch changes the tearDown() so it tries to delete the objects off
both DCs, which appears to fix the problem.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

tfork: set waiter process title

Display a meaningful process title for the waiter process, now that smbd is
calling setproctitle_init() and this value will be displayed in ps
output.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Tue Sep 26 04:38:59 CEST 2017 on sn-devel-144

s4:smbd: set samba root process title

Set the process title in the samba root process to clearly identify it
in ps output.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>

s4:smbd: call setproctitle_init

Call setproctitle_init() in main which suppresses the
  "samba: setproctitle not initialized, please either call
   setproctitle_init() or link against libbsd-ctor."
messages, but more importantly it displays meaningful details in ps
output.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=9816

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

selftest: Use a unique hostname/IP for the no_nss testenv

The ad_dc_no_nss was re-using the ad_dc testenv but changing an
environment variable to disable the NSS wrapper module.

Presumably this would setup a second AD DC server with the same
hostname/IP as another DC (but with NSS disabled). This doesn't seem
like a good thing to be doing in the selftests. This patch changes
it so that the no_nss testenv uses a unique IP/hostname.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

selftest: Rename ntlmauth tests to ntlmdisabled

There are already some existing ntlm_auth tests, so the new tests I've
added make things a bit confusing. Also, ntlmdisabled probably better
reflects the specific case we're trying to test.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

selftest: Add new AD DC testenv with NTLM disabled

This is so that we test the source4 case as well. Currently the only
testenv with NTLM disabled is ktest, and that only exercises the source3
code.

I've tried to support the new test environment with minimal changes to the
Samba4.pm setup code.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

rpcclient: Fix "capabilities" command

This used to not properly store the chained credentials back into the
netlogon_creds_cli tdb. This by the way is the bug that all the
routines for the NT4 style sam replication had that just disappeared.

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Volker Lendecke <vl@samba.org>
Autobuild-Date(master): Mon Sep 25 13:42:19 CEST 2017 on sn-devel-144

netlogon_creds_cli: Pass "capabilities" up from creds_cli_check

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

winbindd: Use rpccli_connect_netlogon

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

cli_netlogon: rpccli_connect_netlogon

This is the one-stop shop to a working, schannel'ed connection to the
netlogon RPC interface. Jeremy tells me it needs more comments :-)

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

cli_netlogon: Return flags from rpccli_setup_netlogon_creds_locked

This will be used in a later commit in the rpcclient "capabilities"
command. Avoids another netlogon_creds_cli_get in the next commit.

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

rpcclient3: Factor out cli_rpc_pipe_open_bind_schannel()

This will be used for the "fast path" to netlogon when we already have
credentials.

This slightly widens the area of code covered by the netlogon_creds
lock: cli_rpc_pipe_open is now also covered by the lock.

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

cli_netlogon: Factor out rpccli_setup_netlogon_creds_locked

This does the reqchallenge/serverauth while assuming we have the
netlogon_creds_cli_lck already held. The _locked flavor will be called
from a routine that covers more under one single lock.

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

netlogon_creds_cli: Protect netlogon_creds_cli_auth by _lck

This widens the lock range to cover the check for established
credentials. Before this patch it could happen that more than one
winbind finds no credentials and does the auth3. This can pile up.

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

netlogon_creds_cli: Protect netlogon_creds_cli_check by _lck

netlogon_creds_cli_lck provides the locking around the operation

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

netlogon_creds_cli: Add netlogon_creds_cli_delete_lck

Like netlogon_creds_cli_delete, protected by netlogon_creds_cli_lck
instead of netlogon_creds_cli_lock.

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

netlogon_creds_cli: Add netlogon_creds_cli_lck

This adds an external locking scheme to protect our
netlogon_creds_CredentialState. This is needed because the routines
exposed by netlogon_creds_cli.h need a more flexible locking to
set up our credentials in a properly protected way.

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

rpc_client3: Avoid "cli_credentials" in cli_rpc_pipe_open_schannel_with_creds

This provides cleaner data dependencies. A netlogon_creds_ctx contains
everything required to open an schannel, there is no good reason to
require cli_credentials here.

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

netlogon_creds_cli: Create cli_credentials from netlogon creds ctx

A netlogon_creds_cli_context holds all information required to do an
schannel bind. Used in the next commit.

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

netlogon_creds_cli: Factor out netlogon_creds_cli_delete_internal

In a future commit we'll need a version that does not check for
context->db.locked_state

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

netlogon_creds_cli: Factor out netlogon_creds_cli_store_internal

In a future commit we'll need a version that does not check for
context->db.locked_state

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

netlogon_creds_cli: Print netlogon_creds_CredentialState

Add some debugging for the tdb records

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

netlogon_creds_cli: Simplify netlogon_creds_cli_get

netlogon_creds_cli_get_internal almost does everything needed, only
the invalidating for credential chain use is missing.

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

netlogon_creds_cli: Rename netlogon_creds_cli_lock_fetch->get_internal

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

netlogon_creds_cli: Transfer a comment

This part of from netlogon_creds_cli_get will go

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

netlogon_creds_cli: Remove tevent_req handling from netlogon_creds_cli_lock_fetch

Disentangle concerns, make netlogon_creds_cli_lock_fetch usable for
other callers

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

netlogon_creds_cli: Remove unused code

According to metze this was meant for test code that never materialized

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

netlogon_creds_cli: Simplify netlogon_creds_cli_delete

Don't implicitly TALLOC_FREE(creds) in the pure delete routine

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

netlogon_creds_cli: Simplify netlogon_creds_cli_store

Don't implicitly TALLOC_FREE(creds) in the pure store routine. This
mixes up responsibilities, and there's not enough callers to justify
the TALLOC_FREE to be centralized.

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

cli_netlogon: Remove an unnecessary if-condition

We don't need to check this here. rpccli_create_netlogon_creds_ctx via
netlogon_creds_cli_context_global returns NT_STATUS_INVALID_PARAMETER for an
unknown schannel type. Slightly different error code, but we could change the
one in netlogon_creds_cli_context_global if necessary.

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

cli_netlogon: Rename "netlogon_creds" to "creds_ctx"

Trying to understand this code it's important for me to name variables
indicating their use: A netlogon_creds_cli_context is a context with access to
credentials, it's not the credentials itself.

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

netlogon_creds_cli: Simplify netlogon_creds_cli_context_global

netlogon_creds_cli_open_global_db() already contains the NULL check. Use that.

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

netlogon_creds_cli: Fix talloc_stackframe leaks

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

scripting: Add script (backportable) to undo a GUID index

This script allows the DB to be read, and re-indexed, by an earlier Samba version,
most likely 4.7 with some backported patches.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Sat Sep 23 09:16:31 CEST 2017 on sn-devel-144

Do not re-use the attribute @IDXVERSION for SAMDB_INDEXING_VERSION

Confusing these two concepts is not a good idea, SAMDB_INDEXING_VERSION refers to
a change in a Samba rule to canonicalise one of our attributes, not the
in-DB index format.

As we already change @INDEXLIST in this version, this commit
is at no extra cost.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

dsdb: Set that Samba uses the GUID index in LDB

This is optional, but only to aid the downgrade script (and in case
there is some major issue found with it).  We don't support that mode,
as that would require us to test and maintain multiple code paths and
not optimise queries to be GUID centric.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

provision: Add a fixed GUID to the samba4top objectclass definition

This is only used in the OpenLDAP backend and will certainly be removed before this becomes production.

(a production backend will use the real AD top objectclass)

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

provision: Add a fixed objectGUID to the tmp DB used for LDAP backend schema work

This DB holds a copy of the schema, but now needs to have an objectGUID on each record.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

provision: make clear that the tmp ldb is running in @IDXGUID mode

This happended when the schema was set on the DB, forcing the full set of Samba behaviours

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

lib: gpo: Put enforced GPOs at the end of the list.

Enforced GPOs should be applied on top of all non-enforced GPOs,
so that they override policies set in non-enforced GPOs.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13046

Signed-off-by: Lutz Justen <ljusten@google.com>
Reviewed-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Guenther Deschner <gd@samba.org>
Autobuild-User(master): Günther Deschner <gd@samba.org>
Autobuild-Date(master): Sat Sep 23 05:25:19 CEST 2017 on sn-devel-144

lib: gpo: Fixes issue with GPOPTIONS_BLOCK_INHERITANCE.

GP links with the GPOPTIONS_BLOCK_INHERITANCE option set
were blocking GPOs from the same link (i.e. an OU with
the flag set would block its own GPOs). This patch makes
sure the GPOs from the link are added to the list.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13046

Signed-off-by: Lutz Justen <ljusten@google.com>
Reviewed-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Guenther Deschner <gd@samba.org>

lib: gpo: Changes order to match GPO application order.

The order of GPOs in a gpo_list generated by ads_get_gpo_list
did not match the order of application. Since GPOs are pushed
to the FRONT of gpo_list, GPOs have to be pushed in the opposite
order of application. (Pushing to front is useful to get
inheritance blocking right).

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13046

Signed-off-by: Lutz Justen <ljusten@google.com>
Reviewed-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Guenther Deschner <gd@samba.org>

ldb: Release ldb 1.3.0

* GUID Index support.

  NOTE: When activated by setting @IDXGUID in the @INDEXLIST dn, all
  entries in the DB are re-keyed in a way that is NOT visible to
  ldb 1.2.2 and earlier.  To re-key back to the previous format, remove
  the @IDXGUID attribute from @INDEXLIST using ldb 1.2.2 or later.

  (ldb 1.2.2 can re-key, but not otherwise read, the new DB format).

* Give LDB_ERR_CONSTRAINT_VIOLATION, not LDB_ERR_ENTRY_ALREADY_EXISTS
  when a duplicate value is detected in a unique index

* Print status information during a > 10,000 entry re-index
  (as this can be slow)

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Sat Sep 23 01:24:19 CEST 2017 on sn-devel-144

ldb_tdb: Treat distinguishedName and objectGUID (in Samba) as unique

This avoids loading any second index for these cases.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

ldb: Ensure we do not run out of File descriptors in autobuild

The python TestCase API will keep a reference to the test object until the end
of the tests, long after we need the actual LDB or the fd.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

ldb: Do not make an ldb file for API tests

An ldb context is valid without a backing file for tests of ldb.Message and ldb.MessageElement

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

ldb: Add test showing a search against the index is not possible

This is not actually a great test, as the filter would
fail to match these anyway, but it at least checks the
codepath is safe.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

ldb_tdb: Also ban a (indexed) search against like @IDXDN=foo

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

ldb_tdb: Update comments for the delete_index() pass of ltdb_reindex()

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

ldb_tdb: Print progress messages on re-index

A re-index of 10,000 entries is slow enough and rare enought that we can
justify the message being at LDB_DEBUG_WARNING as otherwise the administrator
will be sure the "lockup" was one.

The default for ldb is to print LDB_DEBUG_WARNING in comand-line tools
and the default for Samba is to log it at level 2.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

ldb_tdb: Remove incorrect early return from re-index

The ltdb->cache->attribute_indexes test is not correct with the GUID index mode
so for consistency remove it.  This will make re-index on a large un-indexed
database slower, but that is better than making the wrong choice on a large
GUID-indexed database.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

ldb: Add more tests covering in-transaction GUID index and unique index behaviour

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

ldb_tdb: Add UNIQUE_INDEX as a possible flag

This allows easy testing of our unique index code and behaivour from python

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

ldb_tdb: Remove LTDB_FLAG_HIDDEN and ignore "HIDDEN" in @ATTRIBUTES

This was (unintentionally) disabled by
6ef61825541131e16a03975cdbd344e2bbebf810 in 2006.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

ldb_tdb: Clean up index records on ltdb_index_add_new() failure.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

ldb_tdb: Describe index format and control points

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

ldb_tdb: Give a good error message on add without an objectGUID

(or whatever the @IDX_GUID value is)

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

ldb_tdb: Avoid canonicalise and base64 work for DN values, these are already OK

This is important with the GUID index, as a DN lookup is much more common now.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

ldb_tdb: Clean up list in after use in ltdb_key_dn_from_idx()

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

ldb_tdb: Avoid allocation of a DN between the GUID index and the DB lookup

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

ldb_tdb: Move constants into ldb_tdb.h

This helps ensure we keep these all in sync.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

ldb_tdb: Optimise ltdb_search_and_return_base() to re-use casefolding

The casefolding of a DN is one of the more expensive and pointless things in LDB
operation.  The ldb_dn abstraction works hard to avoid duplicating this work, but
we can work harder to save that information.

Here we copy in the DN, that has been casefolded already for the index,
and keep that as the returned DN, after stripping any extended components.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

ldb: Add tests for base DN of a different case

This ensures we cover the case where the DN does not match the DB exactly

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

ldb_tdb: Use the DN extracted from the DB to filter the message later

This should ensure that the upper or lower case the user chooses does not impact
on the filtering, at least for database that have checkBaseOnSearch set.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

ldb_tdb: Add tests for add/modify with the GUID index

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

ldb: Also test the new GUID index mode

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

ldb: Add an unused objectGUID to each record in SearchTests

This will then be used by the GUID index tests.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

ldb_tdb: Duplicate values are no longer permitted in the index

By removing the qsort() we avoid work.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

ldb_tdb: Reduce memory consumption in list_intersect()

We will never have more results than is in either list or list2.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

ldb_tdb: Use the binary search more efficiently in list_intersect()

This change ensures we walk the short list and look up into the longer of the two lists.

ltdb_dn_list_find_val() will do a binary search for the GUID case.

Before GUID indexes this was O(n*m), now it is O(n*log(m)).

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

ldb_tdb: Use a binary search to speed up ltdb_dn_list_find_val()

This only works if we have the GUID index format, as otherwise these are unsorted.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

ldb_tdb: Rework list_union to not return duplicates, and keep sort order

This allows the binary search to still operate on the list, even after
a or operator in the search expression

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

ldb_tdb: Sort inputs to list_union()

This allows us to merge the lists finding common values.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

ldb_tdb: sort GUID index list at add time by inserting in sorted order

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

binsearch.h: Re-licence under LGPLv3 per agreement of the copyright holders

Documented in mails to contributing@samba.org.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>

ldb_tdb: Change error code on unique index violation

The LDB_ERR_ENTRY_ALREADY_EXISTS error code is detected in repl_meta_data as indicating
that the DN exists, and that a conflict record should be created.

This is really a constraint violation, not a duplicate record.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

ldb_tdb: Re-add of both existing DN and GUID must gives LDB_ERR_ENTRY_ALREADY_EXISTS

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

ldb_tdb: Add unique index checking for @IDXDN

This will give us errors earlier if the index code becomes broken

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

ldb_tdb: Improve debugging in ltdb_modify_index_dn() on casefold failure

This is unlikely, but when it happens it will be really painful to debug.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

ldb_tdb: Add improved error strings on ltdb_key_dn_from_idx() failure

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

ldb_tdb: Read from @INDEXLIST or an override if we are using a GUID index

This allows all the previous patches to be enabled.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

ldb_tdb: Optionally use GUID index values a direct TDB keys

This connects the GUID based index records to GUID based TDB keys.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

ldb_tdb: Trust the BASE and ONELEVEL index

This avoids re-checking the fetched DN against the scope

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

ldb_tdb: Add ltdb_idx_to_key() and use it in ltdb_index_filter()

This will allow a common point to parse index records into a TDB key,
allowing them to be a GUID or DN in the future

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

ldb_tdb: Do not add an index for GUID_index_attribute

This would be pointless and we no longer query for it.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

ldb_tdb: Do not query an index on the GUID_index_attribute

The objectGUID (or similar) is already the record key, there is
no need to index it to itself.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

ldb_tdb: Optionally use GUID index in ltdb_search_dn1()

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

ldb_tdb: Use the objectGUID (or similar) as the TDB key in ltdb_key_msg()

When we have the full ldb_message we can read the objectGUID as the TDB key

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

ldb_tdb: Use ltdb_key_msg() in ltdb_delete_noindex()

This allows the optional use of GUID based TDB key.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

ldb_tdb: Add mem_ctx to ltdb_key_dn() and ltdb_key_msg()

This follows modern Samba coding style where memory
returned is allocated on a supplied memory context.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

ldb_tdb: Check version number on index

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

ldb_tdb: Add an index shortcut for a <GUID= DN

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

ldb_tdb: Add a function to get the GUID key for a DN

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>

ldb_tdb: Add a function to take a GUID and make the TDB_DATA key

This allows us to format the TDB key as DN=GUID=f7c953ee-cf9c-433f-b423-21ce04d09591
and so be compatible with an un-indexed search and a re-index with an old ldb.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>