samba.git
2 years agoctdb-tools: Use db_hash in ctdb_killtcp
Martin Schwenke [Thu, 29 Jun 2017 05:57:19 +0000 (15:57 +1000)]
ctdb-tools: Use db_hash in ctdb_killtcp

One less use of trbt_tree_t.  The code is easier to read and is
significantly smaller.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2 years agoctdb-tools: Use ctdb_connection and ctdb_connection_list structs
Martin Schwenke [Thu, 29 Jun 2017 04:46:31 +0000 (14:46 +1000)]
ctdb-tools: Use ctdb_connection and ctdb_connection_list structs

Also use new connection and sock addr utilities.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2 years agoctdb-protocol: Add ctdb_connection_list utilities
Martin Schwenke [Mon, 4 Sep 2017 07:01:05 +0000 (17:01 +1000)]
ctdb-protocol: Add ctdb_connection_list utilities

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2 years agoctdb-protocol: Add marshalling for ctdb_connection_list
Martin Schwenke [Tue, 5 Sep 2017 00:52:58 +0000 (10:52 +1000)]
ctdb-protocol: Add marshalling for ctdb_connection_list

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2 years agoctdb-protocol: Add new data structure ctdb_connection_list
Martin Schwenke [Mon, 4 Sep 2017 06:48:40 +0000 (16:48 +1000)]
ctdb-protocol: Add new data structure ctdb_connection_list

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2 years agoctdb-protocol: Add ctdb_connection utilities
Martin Schwenke [Mon, 4 Sep 2017 06:41:30 +0000 (16:41 +1000)]
ctdb-protocol: Add ctdb_connection utilities

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2 years agoctdb-protocol: Factor out static function ctdb_sock_addr_cmp_family()
Martin Schwenke [Wed, 13 Sep 2017 06:18:29 +0000 (16:18 +1000)]
ctdb-protocol: Factor out static function ctdb_sock_addr_cmp_family()

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2 years agoctdb-protocol: Add ctdb_sock_addr_from_string()
Martin Schwenke [Mon, 11 Sep 2017 06:39:38 +0000 (16:39 +1000)]
ctdb-protocol: Add ctdb_sock_addr_from_string()

This and the supporting functions duplicate functionality (parse_ip()
and parse_ip_port()) from common/system_util.c.  The old functions
will be removed at a later time.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2 years agoctdb-protocol: Optionally print port for address printing functions
Martin Schwenke [Mon, 11 Sep 2017 05:00:10 +0000 (15:00 +1000)]
ctdb-protocol: Optionally print port for address printing functions

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2 years agoctdb-protocol: Add utility function ctdb_sock_addr_to_buf()
Martin Schwenke [Fri, 30 Jun 2017 05:11:04 +0000 (15:11 +1000)]
ctdb-protocol: Add utility function ctdb_sock_addr_to_buf()

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2 years agoctdb-protocol: Add ctdb_sock_addr_port() and sock_addr_set_port()
Martin Schwenke [Fri, 2 Jun 2017 06:17:22 +0000 (16:17 +1000)]
ctdb-protocol: Add ctdb_sock_addr_port() and sock_addr_set_port()

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2 years agoctdb-protocol: Add server and client aliases in ctdb_connection
Martin Schwenke [Mon, 4 Sep 2017 06:20:55 +0000 (16:20 +1000)]
ctdb-protocol: Add server and client aliases in ctdb_connection

The current code is ambiguous in its use of src and dst.  This allows
new code to use server and client for clarity.

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
2 years agoctdb-common: Initialise socket addresses before reading into them
Martin Schwenke [Tue, 19 Sep 2017 04:54:26 +0000 (14:54 +1000)]
ctdb-common: Initialise socket addresses before reading into them

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2 years agoctdb-build: Split protocol-util as a separate subsystem
Martin Schwenke [Mon, 4 Sep 2017 06:00:48 +0000 (16:00 +1000)]
ctdb-build: Split protocol-util as a separate subsystem

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2 years agoctdb-build: Fix dependency for ctdbd
Martin Schwenke [Mon, 4 Sep 2017 06:00:27 +0000 (16:00 +1000)]
ctdb-build: Fix dependency for ctdbd

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2 years agos4-torture: move lease break handler outside the lease testsuite.
Günther Deschner [Thu, 29 Sep 2016 06:41:01 +0000 (08:41 +0200)]
s4-torture: move lease break handler outside the lease testsuite.

Guenther

Signed-off-by: Guenther Deschner <gd@samba.org>
Pair-Programmed-With: Jose A. Rivera <jarrpa@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
Autobuild-User(master): Ralph Böhme <slow@samba.org>
Autobuild-Date(master): Tue Sep 19 09:36:40 CEST 2017 on sn-devel-144

2 years agoselftest: Avoid a build started just before midnight failing
Martin Schwenke [Mon, 18 Sep 2017 22:33:11 +0000 (08:33 +1000)]
selftest: Avoid a build started just before midnight failing

It looks like commit 070f24bc9d2 was incomplete because it doesn't
match on the trailing fullstop and newline.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Martin Schwenke <martins@samba.org>
Autobuild-Date(master): Tue Sep 19 05:32:22 CEST 2017 on sn-devel-144

2 years agoRevert "ctdb-daemon: Don't explicitly stop monitoring during shutdown"
Martin Schwenke [Mon, 18 Sep 2017 06:36:32 +0000 (16:36 +1000)]
Revert "ctdb-daemon: Don't explicitly stop monitoring during shutdown"

This reverts commit 19318d28354190a9cfafa1696e41c90351f85af9.

With this commit, a shutdown that occurs while the startup event is
running can cause an abort because the startup callback will try to
decrease the run state from SHUTDOWN to RUNNING.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2 years agoRevert "ctdb-daemon: Remove unused function ctdb_stop_monitoring()"
Martin Schwenke [Mon, 18 Sep 2017 06:36:08 +0000 (16:36 +1000)]
Revert "ctdb-daemon: Remove unused function ctdb_stop_monitoring()"

This reverts commit b119104267635d96b1f99a2e98d35438841585bf.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2 years agoselftest: Add some tests for linked attribute conflicts
Tim Beale [Wed, 23 Aug 2017 00:45:09 +0000 (12:45 +1200)]
selftest: Add some tests for linked attribute conflicts

Currently we have tests that check we can resolve object conflicts, but
these don't test anything related to conflicting linked attributes.
This patch adds some basic tests that checks that Samba can resolve
conflicting linked attributes.

This highlights some problems with Samba, as the following tests
currently fail:
- test_conflict_single_valued_link: Samba currently can't resolve a
  conflicting targets for a single-valued linked attribute - the
  replication exits with an error.
- test_link_deletion_conflict: If 2 DCs add the same linked attribute,
  currently when they resolve this conflict the RMD_VERSION for the
  linked attribute incorrectly gets incremented. This means the version
  numbers get out of step and subsequent changes to the linked attribute
  can be dropped/ignored.
- test_full_sync_link_conflict: fails for the same reason as above.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
Autobuild-User(master): Garming Sam <garming@samba.org>
Autobuild-Date(master): Mon Sep 18 09:56:41 CEST 2017 on sn-devel-144

2 years agogetncchanges.c: max_links calculation didn't work well in some cases
Tim Beale [Tue, 15 Aug 2017 04:15:14 +0000 (16:15 +1200)]
getncchanges.c: max_links calculation didn't work well in some cases

The max_links calculation didn't work particularly well if max_links was
set to a value lower than max_objects.

As soon as repl_chunk->object_count exceeded repl_chunk->max_links, the
chunk would be deemed full, even if there was only one link to send (or
even worse, no links to send). For example, if max_objects=100 and
max_links=10, then it would send back chunks of 10 objects (or 9 objects
and 1 link).

I believe the historic reason this logic exists is to avoid overfilling
the response message. It's hard to tell what the appropriate limit would
be because the total message size would depend on how many attributes
each object has.

I couldn't think of logic that would be suitable for all cases. I toyed
with the idea of working out a percentage of how full the message is.
However, adjusting the max_links doesn't really make sense when the
settings are small enough, e.g. max_objects=100 and max_links=100 is
never going to overfill the message, so there's no reason to alter the
values.

In the end I went with:
- If the user is using non-default values, just use those.
- In the default value case, just use the historic calculation

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12972
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2 years agoreplmd: Avoid duplicated debug/warnings
Tim Beale [Fri, 11 Aug 2017 03:39:35 +0000 (15:39 +1200)]
replmd: Avoid duplicated debug/warnings

We display warnings if a target object is missing but it's still OK to
continue the replication. Currently we need to check the target twice -
once to verify it when we first receive it, and once when we actually
commit it (we can't skip the 2nd check altogether because in the join
case, they could occur quite far apart).

One annoying side-effect is we get the same warning message coming out
twice in these special cases.

In the cases where we're checking the dsdb_repl_flags, we can actually
just bypass the verification checks for the target object (if it doesn't
exist we still continue anyway). This may save us a tiny bit of
unnecessary work.

For cross-partition links, we can limit logging these warnings to when
the objects are actually being committed. This avoids spurious warnings
in the join case (i.e. we receive the link before we receive the target
object's partition, but we have received all partitions by the time we
actually commit the objects).

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12972
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2 years agoreplmd: Allow missing targets if GET_TGT has already been set
Tim Beale [Fri, 11 Aug 2017 01:53:31 +0000 (13:53 +1200)]
replmd: Allow missing targets if GET_TGT has already been set

While running the selftests, I noticed a case where DC replication
unexpectedly sends a linked attribute for a deleted object (created in
the drs.ridalloc_exop tests). The problem is due to the
msDS-NC-Replica-Locations attribute, which is a (known) one-way link.
Because it is a one-way link, when the test demotes the DC and deletes
the link target, there is no backlink to delete the link from the source
object.

After much debate and head-scratching, we decided that there wasn't an
ideal way to resolve this problem. Any automated intervention could
potentially do the wrong thing, especially if the link spans partitions.
Running dbcheck will find this problem and is able to fix it (providing
the deleted object is still a tombstone). So the recommendation is to
run dbcheck on your DCs every 6 months (or more frequently if using a
lower tombstone lifetime setting).

However, it does highlight a problem with the current GET_TGT
implementation. If the tombstone object had been expunged and you
upgraded to 4.8, then you would be stuck - replication would fail
because the target object can't be resolved, even with GET_TGT, and
dbcheck would not be able to fix the hanging link. The solution is to
not fail the replication for an unknown target if GET_TGT has already
been set (i.e. the dsdb_repl_flags contains
DSDB_REPL_FLAG_TARGETS_UPTODATE).

It's debatable whether we should add a hanging link in this case or
ignore/drop the link. Some cases to consider:
- If you're talking to a DC that still sends all the links last, you
  could still get object deletion between processing the source object's
  links and sending the target (GET_TGT just restarts the replication
  cycle from scratch). Adding a hanging link in this case would be
  incorrect and would add spurious information to the DB.
- Suppose there's a bug in Samba that incorrectly results in an object
  disappearing. If other DCs then remove any links that pointed to that
  object, it makes recovering from the problem harder. However, simply
  ignoring the link shouldn't result in data loss, i.e. replication won't
  remove the existing link information from other DCs. Data loss in this
  case would only occur if a new DC were brought online, or if it were a
  new link that was affected.
Based on this, I think ignoring the link does the least harm.

This problem also highlights that we should really be using the same
logic in both the unknown target and the deleted target cases.
Combining the logic and moving it into a common
replmd_allow_missing_target() function fixes the problem. (This also has
the side-effect of fixing another logic flaw - in the deleted object
case we would unnecessarily retry with GET_TGT if the target object was
in another partition. This is pointless work, because GET_TGT won't
resolve the target).

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12972
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2 years agogetncchanges.c: Support GET_TGT better with large numbers of links
Tim Beale [Fri, 11 Aug 2017 04:08:15 +0000 (16:08 +1200)]
getncchanges.c: Support GET_TGT better with large numbers of links

A source object can potentially link to thousands of target objects.
We have to be careful not to overfill the GetNCChanges response message
with more data than it's possible to send. We also don't want the client
to timeout while we're busy checking the linked attributes. The GET_TGT
support added so far is fairly dumb - this patch extends it to better
handle larger numbers of links.

To do so, this extends the repl_chunk usage so that it also works out if
the current chunk is full of links. Now as soon as the chunk is full of
either links or objects, we stop and send it back.

These changes now mean that we need to also check:
- that all the links for the last source object in the previous chunk
  have been sent, before we move on and send the next object. This only
  takes effect when immediate_link_sync is configured. It also means
  that a chunk in the middle of the replication cycle can now contain
  only links, and no objects.
- when GET_TGT is used, we only send back the links that we've verified
  the target object for. i.e. if we stop checking links because we timed
  out, we only send back the links whose targets were checked.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2 years agogetncchanges.c: Refactor to track more state using repl_chunk
Garming Sam [Tue, 8 Aug 2017 04:27:18 +0000 (16:27 +1200)]
getncchanges.c: Refactor to track more state using repl_chunk

To prepare GET_TGT to deal with a large number of links better, there
is now a 'repl_chunk' struct to help keep track of all the factors
relating to the current chunk of replication data (i.e. how many
objects/links we can send and how many we've already processed). This
means we can have a consistent way of working out whether the current
chunk is full (whether that be due to objects, links, or just too much
time taken).

This patch should not alter functionality. This is just a refactor to
add the basic framework, which will be used in the next patch.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2 years agogetncchanges.py: Add a multi-valued linked attribute test
Tim Beale [Tue, 15 Aug 2017 00:18:02 +0000 (12:18 +1200)]
getncchanges.py: Add a multi-valued linked attribute test

Add a test where a source object links to multiple different targets.
First we do the replication without GET_TGT and check that the server
can handle sending a chunk containing only links (in the middle of the
replication). Then we repeat the replication forcing GET_TGT to be used.

To avoid having to create 1500 objects/links, I've lowered the 'max
link sync' setting on the vampire_dc testenv to 250.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2 years agogetncchanges.py: Add a test for dropped cross-partition links
Tim Beale [Mon, 24 Jul 2017 02:43:54 +0000 (14:43 +1200)]
getncchanges.py: Add a test for dropped cross-partition links

Samba would drop linked attributes that span partitions if it didn't
know about the target object. This patch adds a test that exposes the
problem.

I've re-used the code from the previous re-animation test to do this.
I've also added a very basic DcConnection helper class that basically
stores the connection state information the drs_base.py uses for
replication. This allows us to switch the DC we want to replicate from
easily. This approach could potentially be retro-fitted to some of the
existing test cases, as it allows us to test both the DRS client code
and server code at the same time.

Note this test case relates to the code change for commit
fae5df891c11f642cb.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12972
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2 years agogetncchanges.py: Add test for replicating reanimated objects
Tim Beale [Thu, 20 Jul 2017 05:06:14 +0000 (17:06 +1200)]
getncchanges.py: Add test for replicating reanimated objects

Reading between the lines, this scenario seems to be the main reason
that Microsoft added the GET_TGT flag. MS AD can handle getting links
for unknown targets OK, but if it receives links for a deleted/recycled
target then it would tend to drop the received links. Samba client also
used to drop the links if talking to a Microsoft DC (or a Samba server
with GET_TGT support).

The specific scenario is the client side already knows about a deleted
object. That object is then re-animated and used as the target for a
linked attribute. *Then* the target object gets updated again so it gets
sent in a later replication chunk to the linked attribute, i.e. the
client receives the link before it learns that the target object has
been re-animated.

In this test we're interested in particular at how the client behaves
when it receives a linked attribute for a deleted object. (It *should*
retry with GET_TGT to make sure the target is up-to-date. However, it
was just dropping the linked attribute).

To exercise the client-side, we disable replication, setup the
links/objects on one DC the way we want them, then force a replication
to the second DC. We then check that when we query each DC, they both
tell us about the links/objects we're expecting (i.e. no links got
lost).

Note that this wasn't a problem with older versions of Samba-to-Samba
because sending the links last guaranteed that the target objects were
always up-to-date.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2 years agodrs: Add basic GET_TGT support
Tim Beale [Tue, 22 Aug 2017 22:23:10 +0000 (10:23 +1200)]
drs: Add basic GET_TGT support

This adds basic DRS_GET_TGT support. If the GET_TGT flag is specified
then the server will use the object cache to store the objects it sends
back. If the target object for a linked attribute is not in the cache
(i.e. it has not been sent already), then it is added to the response
message.

Note that large numbers of linked attributes will not be handled well
yet - the server could potentially try to send more than will fit in a
single repsonse message.

Also note that the client can sometimes set the GET_TGT flag even if the
server is still sending the links last. In this case, we know the client
supports GET_TGT so it's safe to send the links interleaved with the
source objects (the alternative of fetching the target objects but not
sending the links until last doesn't really make any sense).

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2 years agogetncchanges.py: Add tests for object deletion during replication
Tim Beale [Tue, 18 Jul 2017 23:38:55 +0000 (11:38 +1200)]
getncchanges.py: Add tests for object deletion during replication

Add tests that delete the source and target objects for linked
attributes in the middle of a replication cycle.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2 years agogetnc_exop.py: Extend EXOP_REPL_OBJ test case to use GET_TGT
Tim Beale [Mon, 17 Jul 2017 02:04:38 +0000 (14:04 +1200)]
getnc_exop.py: Extend EXOP_REPL_OBJ test case to use GET_TGT

We already check that when we use GET_ANC that we still only receive a
single object when EXOP_REPL_OBJ is used. This extends the test to also
check that only a single object is returned when GET_TGT is used.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2 years agogetncchanges.py: Add test for GET_ANC and GET_TGT combined
Tim Beale [Wed, 12 Jul 2017 23:47:16 +0000 (11:47 +1200)]
getncchanges.py: Add test for GET_ANC and GET_TGT combined

The code has to handle needing GET_ANC and GET_TGT in combination, i.e.
where we fetch the target object for the linked attribute and the target
object's parent is unknown as well. This patch adds a test case to
exercise this code path.

The second part of this test exercises GET_ANC/GET_TGT for an
incremental replication, where the objects are getting filtered by an
uptodateness-vector/HWM.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2 years agogetncchanges.py: Add test for adding links during replication
Tim Beale [Tue, 13 Jun 2017 00:14:45 +0000 (12:14 +1200)]
getncchanges.py: Add test for adding links during replication

We have identified a case where the Samba server can send linked
attributes but not the target object. In this case, the Samba DRS client
would hit the "Failed to re-resolve GUID" case in replmd and silently
discard the linked attribute.

However, Samba will resend the linked attribute in the next cycle
(because its USN is still higher than the committed HWM), so it should
recover OK. On older releases, this may have caused problems if the
first error resulting in a hanging link (which might mean the second
time it's processed it still fails to be added).

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2 years agogetncchanges.py: Add some GET_TGT test cases
Tim Beale [Wed, 12 Jul 2017 02:23:35 +0000 (14:23 +1200)]
getncchanges.py: Add some GET_TGT test cases

test_repl_get_tgt:
- Adds 2 sets of objects
- Links one set to the other
- Changes the order so the target object comes last in the
  replication (which means the client has to use GET_TGT)
- Checks that when GET_TGT is used that we have received all target
  objects we need to resolve the linked attibutes
- Checks that we expect to receive the linked attributes *before*
  the last chunk is sent (by default, Samba sends all the links at
  the end, so this fails)
- Checks that we eventually receive all expected objects, and all
  links we receive match what is expected

test_repl_get_tgt_chain:
  This adds the linked attributes in a more complicated chain. We add
  300 objects, but the links for 100 objects will point to a linked
  chain of 200 objects.
  This was mainly to determine whether or not Windows follows the
  target object (i.e. whether it sends all the links for the target
  object as well). It turns out Windows maintains its own linked
  attribute DB, so it sends the links based on USN.

Note that the 2 testenvs fail for different reasons. promoted_dc fails
because it is sending all the linked attributes last. vampire_dc fails
because it doesn't support GET_TGT yet, so it sends the link before the
peer knows about the target object.

Note that to test against vampire_dc (rather than the ad_dc_ntvfs DC),
we need to send the GetNCChanges requests to DC2 instead of DC1.
I've left the DC numbering scheme as is, but I've addeed a test_ldb_dc
handle to drs_base.py - it defaults to DC1, but tests can override it
easily and still have everything work.

While running the new tests through autobuild, I noticed an intermittent
LDAP_ENTRY_ALREADY_EXISTS failure in the test setup(). This appears to
be due to a timing issue in the background replication between the
multiple testenvs. Adding some randomness so that the test base OU is
unique seems to avoid the problem.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2 years agogetnc_exop.py: Fix GET_TGT behaviour in DRS tests
Tim Beale [Tue, 23 May 2017 02:37:56 +0000 (14:37 +1200)]
getnc_exop.py: Fix GET_TGT behaviour in DRS tests

The existing code never passed the more_flags parameter into the
actual getNCChanges request, i.e. _getnc_req10(). This meant the
existing GET_TGT tests effectively did nothing.

Passing the flag through properly means we have to now change the tests
as the DNs returned by Windows now include any target objects in the
linked attributes. These tests now fail against Samba (because it
doesn't support GET_TGT yet).

Also added comments to the tests to help explain what they are actually
doing.

Note that Samba and Windows can return the objects in different orders,
due to significant differences in their underlying DB implementations
(Windows stores links in a separate DB, so sends links ordered strictly
by USN, whereas Samba sends links based on the USN of the source
object). To make the test a fair comparison between Windows and Samba,
we need to use dn_ordered=False.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2 years agos4/smbd: set the process group.
Gary Lockyer [Mon, 21 Aug 2017 03:12:04 +0000 (15:12 +1200)]
s4/smbd: set the process group.

Set the process group in the samba daemon, the --no-process-group option
allows this to be disabled.  The no-process-group option needs to be
disabled in self test.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
Autobuild-User(master): Ralph Böhme <slow@samba.org>
Autobuild-Date(master): Mon Sep 18 04:39:50 CEST 2017 on sn-devel-144

2 years agowinbindd: Remove an obsolete comment
Volker Lendecke [Sun, 17 Sep 2017 17:40:00 +0000 (10:40 -0700)]
winbindd: Remove an obsolete comment

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Autobuild-User(master): Volker Lendecke <vl@samba.org>
Autobuild-Date(master): Sun Sep 17 23:35:51 CEST 2017 on sn-devel-144

2 years agoutil_runcmd: Free the fde in event handler.
Gary Lockyer [Fri, 8 Sep 2017 02:03:25 +0000 (14:03 +1200)]
util_runcmd: Free the fde in event handler.

Free the fde in the event handler to prevent the event triggering again
While not strictly necessary in this case, this code serves as an
example of the usage of tfork.

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13037

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
Autobuild-User(master): Ralph Böhme <slow@samba.org>
Autobuild-Date(master): Sat Sep 16 23:50:27 CEST 2017 on sn-devel-144

2 years agolib/util: only close the event_fd in tfork if the caller didn't call tfork_event_fd()
Ralph Boehme [Sat, 16 Sep 2017 08:22:31 +0000 (01:22 -0700)]
lib/util: only close the event_fd in tfork if the caller didn't call tfork_event_fd()

Make closing of the event_fd the global responsibility of the
parent process if it called tfork_event_fd().

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13037

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
2 years agoutil/tfork: Write to the status pipe
Gary Lockyer [Sun, 10 Sep 2017 22:25:49 +0000 (10:25 +1200)]
util/tfork: Write to the status pipe

The previous design relied on only calling close() of the status pipe.

We now write a single 0 byte to the status FD as well as closing it in the
parent process.  Both of these operations typically trigger a read
event on the other end of the FD, held in the waiter process (the child).

The child process blocks on the status FD, until it becomes readable.

However if there is a sibling process that was launched after the waiter
process they also will hold the status FD open and the status FD would,
until this change, never become readable to the waiter process (the child).

This caused the waiter process (child) not to exit and the parent process
to hang in tfork_status() while expecting the waitpid() to return.

That is, file descriptors are essentially global variables copied
to children in the process tree.  The last child that (unwittingly) holds
the file descriptor open is the one that needs to trigger the close() this
code previously depended on.

Without this change, there is no notification of process death until
all these unrelated children exit for their own reasons.

We can write up to 4K (PIPE_BUF) into this pipe before blocking,
but we only write one byte.  Additionally sys_write() refuses to block.

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13037

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
2 years agotests util/tfork: Tests for status and event fd
Gary Lockyer [Mon, 11 Sep 2017 02:48:21 +0000 (14:48 +1200)]
tests util/tfork: Tests for status and event fd

Add tests to ensure that:
- The event_fd becomes readable once the worker process has terminated
- That the event_fd is not closed by the tfork code.
  - If this is done in tevent code and the event fde has not been
    freed, "Bad talloc magic value - " errors can result.
- That the status call does not block if the parent process launches
  more than one child process.
  - The status file descriptor for a child is passed to the
    subsequent children.  These processes hold the FD open, so that
    closing the fd does not make the read end go readable, and the
    process calling status blocks.

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13037

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
2 years agocli_credentials: Apply some const
Volker Lendecke [Thu, 7 Sep 2017 10:34:34 +0000 (12:34 +0200)]
cli_credentials: Apply some const

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Sat Sep 16 12:28:17 CEST 2017 on sn-devel-144

2 years agolibcli: Apply some const
Volker Lendecke [Thu, 7 Sep 2017 10:34:03 +0000 (12:34 +0200)]
libcli: Apply some const

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2 years agonotifyd: Clarify a comment
Volker Lendecke [Wed, 6 Sep 2017 16:20:25 +0000 (18:20 +0200)]
notifyd: Clarify a comment

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2 years agonetlogon_creds_cli: Use data_blob_cmp in netlogon_creds_cli_validate
Volker Lendecke [Mon, 21 Aug 2017 10:00:23 +0000 (12:00 +0200)]
netlogon_creds_cli: Use data_blob_cmp in netlogon_creds_cli_validate

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2 years agonetlogon_creds_cli: Simplify netlogon_creds_cli_context_global
Volker Lendecke [Mon, 21 Aug 2017 09:54:29 +0000 (11:54 +0200)]
netlogon_creds_cli: Simplify netlogon_creds_cli_context_global

(require_sign_or_seal == false) looks odd :-)

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2 years agonetlogon_creds_cli: Simplify netlogon_creds_cli_context_common
Volker Lendecke [Mon, 21 Aug 2017 09:34:45 +0000 (11:34 +0200)]
netlogon_creds_cli: Simplify netlogon_creds_cli_context_common

IMHO a full talloc_stackframe is overkill for the one allocation that is left
here.

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2 years agonetlogon_creds_cli: Simplify netlogon_creds_cli_context_common
Volker Lendecke [Mon, 21 Aug 2017 09:34:45 +0000 (11:34 +0200)]
netlogon_creds_cli: Simplify netlogon_creds_cli_context_common

printf knows to only print part of a string. No need to talloc_strdup.

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2 years agorpc_client3: Fix a debug message
Volker Lendecke [Tue, 5 Sep 2017 12:08:41 +0000 (14:08 +0200)]
rpc_client3: Fix a debug message

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2 years agonetlogon_creds_cli: A netlogon_creds_cli_context needs a msg_ctx
Volker Lendecke [Tue, 5 Sep 2017 12:56:58 +0000 (14:56 +0200)]
netlogon_creds_cli: A netlogon_creds_cli_context needs a msg_ctx

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2 years agonetlogon_creds_cli: Remove an obsolete comment
Volker Lendecke [Fri, 25 Aug 2017 09:39:16 +0000 (11:39 +0200)]
netlogon_creds_cli: Remove an obsolete comment

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2 years agonetlogon_creds_cli: Avoid a static const struct
Volker Lendecke [Fri, 25 Aug 2017 09:27:30 +0000 (11:27 +0200)]
netlogon_creds_cli: Avoid a static const struct

Same number of .text bytes, but simpler code.

Yes, this is {{0}} instead of {0}, which I always promote. I've just read a
comment on stackoverflow (which I've unfortunately just closed the tab for :-()
that {{0}} might actually be the correct way to init a struct to zero if the
first struct element is again a struct. I'm lost. 25 years of C coding and I
have no clue of the language :-(

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2 years agocli_netlogon: Eliminate rpccli_setup_netlogon_creds_with_creds
Volker Lendecke [Wed, 6 Sep 2017 15:31:38 +0000 (17:31 +0200)]
cli_netlogon: Eliminate rpccli_setup_netlogon_creds_with_creds

Inlining the code from rpccli_setup_netlogon_creds

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2 years agocli_netlogon: Rename rpccli_create_netlogon_creds_with_creds
Volker Lendecke [Wed, 6 Sep 2017 15:23:47 +0000 (17:23 +0200)]
cli_netlogon: Rename rpccli_create_netlogon_creds_with_creds

This creates a context with access to a credentials, not credentials

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2 years agocli_netlogon: Make rpccli_setup_netlogon_creds static
Volker Lendecke [Wed, 6 Sep 2017 12:21:36 +0000 (14:21 +0200)]
cli_netlogon: Make rpccli_setup_netlogon_creds static

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2 years agolibnet: Use rpccli_setup_netlogon_creds_with_creds in join_unsecure
Volker Lendecke [Wed, 6 Sep 2017 12:20:32 +0000 (14:20 +0200)]
libnet: Use rpccli_setup_netlogon_creds_with_creds in join_unsecure

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2 years agocli_netlogon: Make rpccli_create_netlogon_creds static
Volker Lendecke [Wed, 6 Sep 2017 12:14:28 +0000 (14:14 +0200)]
cli_netlogon: Make rpccli_create_netlogon_creds static

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2 years agolibnet: Use rpccli_create_netlogon_creds_with_creds in join_unsecure
Volker Lendecke [Wed, 6 Sep 2017 12:12:27 +0000 (14:12 +0200)]
libnet: Use rpccli_create_netlogon_creds_with_creds in join_unsecure

rpccli_create_netlogon_creds_with_creds just extracts the values we set here
from cli_credentials, and the lower-level interface is supposed to go away.

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2 years agocli_netlogon: Pass server_dns_domain through rpccli_create_netlogon_creds
Volker Lendecke [Wed, 6 Sep 2017 11:48:18 +0000 (13:48 +0200)]
cli_netlogon: Pass server_dns_domain through rpccli_create_netlogon_creds

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2 years agonetlogon_creds_cli: Pass "server_dns_domain" through netlogon_creds_cli_context_global
Volker Lendecke [Wed, 6 Sep 2017 11:32:34 +0000 (13:32 +0200)]
netlogon_creds_cli: Pass "server_dns_domain" through netlogon_creds_cli_context_global

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2 years agonetlogon_creds_cli: Add "dns_domain" to netlogon_creds_cli_context
Volker Lendecke [Wed, 6 Sep 2017 11:29:07 +0000 (13:29 +0200)]
netlogon_creds_cli: Add "dns_domain" to netlogon_creds_cli_context

Used later for creating schannel cli_credentials

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2 years agos3: Avoid netlogon_creds_cli.h in includes.h
Volker Lendecke [Tue, 5 Sep 2017 11:37:41 +0000 (13:37 +0200)]
s3: Avoid netlogon_creds_cli.h in includes.h

There's no point recompiling all of source3 if netlogon_creds_cli.h is changed

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2 years agolib: util_tdb.h needs tdb.h
Volker Lendecke [Sun, 6 Aug 2017 13:42:08 +0000 (15:42 +0200)]
lib: util_tdb.h needs tdb.h

It uses TDB_DATA

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
2 years agolib: tevent: Remove select backend.
Jeremy Allison [Tue, 12 Sep 2017 19:08:38 +0000 (12:08 -0700)]
lib: tevent: Remove select backend.

select() is no longer useful on modern systems.

Signed-off-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>
Autobuild-User(master): Volker Lendecke <vl@samba.org>
Autobuild-Date(master): Sat Sep 16 08:35:39 CEST 2017 on sn-devel-144

2 years agowafsamba: We need to honor DESTDIR in INSTALL_DIR
Andreas Schneider [Tue, 12 Sep 2017 13:56:44 +0000 (15:56 +0200)]
wafsamba: We need to honor DESTDIR in INSTALL_DIR

BUG: https://bugzilla.samba.org/show_bug.cgi?id=12957

Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Sat Sep 16 04:47:29 CEST 2017 on sn-devel-144

2 years agosamba_upgradedns: When we setup the internal dns cleanup bind-dns dir
Andreas Schneider [Tue, 5 Sep 2017 09:47:27 +0000 (11:47 +0200)]
samba_upgradedns: When we setup the internal dns cleanup bind-dns dir

Make sure to remove everything from the bind-dns directory to avoid
possible security issues with the named group having write access to all
AD partions

BUG: https://bugzilla.samba.org/show_bug.cgi?id=12957

Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
2 years agosamba_upgradedns: Print better hints after we migrated the config
Andreas Schneider [Wed, 6 Sep 2017 05:25:40 +0000 (07:25 +0200)]
samba_upgradedns: Print better hints after we migrated the config

BUG: https://bugzilla.samba.org/show_bug.cgi?id=12957

Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
2 years agosamba_upgradedns: Change the group of the 'binddns dir' too
Andreas Schneider [Wed, 6 Sep 2017 08:06:40 +0000 (10:06 +0200)]
samba_upgradedns: Change the group of the 'binddns dir' too

BUG: https://bugzilla.samba.org/show_bug.cgi?id=12957

Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
2 years agopython:provision: Do not change the owner of the sam.ldb.d dir
Andreas Schneider [Wed, 6 Sep 2017 05:25:04 +0000 (07:25 +0200)]
python:provision: Do not change the owner of the sam.ldb.d dir

BUG: https://bugzilla.samba.org/show_bug.cgi?id=12957

Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
2 years agopython:provision: Change the group of the 'binddns dir' too
Andreas Schneider [Wed, 6 Sep 2017 05:23:57 +0000 (07:23 +0200)]
python:provision: Change the group of the 'binddns dir' too

BUG: https://bugzilla.samba.org/show_bug.cgi?id=12957

Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
2 years agos4:bind_dlz: Try the 'binddns dir' first
Andreas Schneider [Tue, 22 Aug 2017 15:10:01 +0000 (17:10 +0200)]
s4:bind_dlz: Try the 'binddns dir' first

The directory is normally empty if you did not provision or call
samba_upgradedns for the bind_dlz module.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=12957

Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
2 years agodynconfig: Fix location of the default 'binddns dir'
Andreas Schneider [Thu, 10 Aug 2017 13:04:08 +0000 (15:04 +0200)]
dynconfig: Fix location of the default 'binddns dir'

BUG: https://bugzilla.samba.org/show_bug.cgi?id=12957

Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
2 years agosamba:provision: Give a hint to copy the krb5.conf and not symlink it
Andreas Schneider [Tue, 5 Sep 2017 18:36:47 +0000 (20:36 +0200)]
samba:provision: Give a hint to copy the krb5.conf and not symlink it

BUG: https://bugzilla.samba.org/show_bug.cgi?id=12957

Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
2 years agowafsamba: Do not chmod already existing dirs on install
Andreas Schneider [Tue, 5 Sep 2017 12:18:44 +0000 (14:18 +0200)]
wafsamba: Do not chmod already existing dirs on install

This might break backward compatibility.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=12957

Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
2 years agogetncchanges.c: Send linked attributes in each chunk
Tim Beale [Tue, 6 Jun 2017 22:46:47 +0000 (10:46 +1200)]
getncchanges.c: Send linked attributes in each chunk

Instead of sending all the linked attributes at the end, add a
configurable option to send the links in each replication chunk.

The benefits of this approach are:
- it can reduce memory overhead, as we don't have to keep all the links
in memory over the entire replication cycle.
- the client should never end up knowing about objects but not their
links. (Although we're not sure that this has actually resulted in
replication problems, i.e. missing links).

Note that until we support GET_TGT, this approach can mean we now send
a link where the client doesn't know about the target object, causing
the client to siliently drop that linked attribute. Hence, this option
is switched off by default.

Implementation-wise, this code works fairly the same as before. Instead
of sorting the entire getnc_state->la_sorted array at the end and then
splitting it up over chunks, we now split the links up over chunks and
then sort them when we copy them into the message. This should be OK, as
I believe the MS-DRSR Doc says the links in the message should be sorted
(rather than sorting *all* the links overall). Windows behaviour seems
to chunk the links based on USN and then sort them.

getnc_state->la_idx now tracks which links in getnc_state->la_list[]
have already been sent (instead of tracking getnc_state->la_sorted).
This means the la_sorted array no longer needs to be stored in
getnc_state and we can free the array's memory once we've copied the
links into the message. Unfortunately, the link_given/link_total debug
no longer reports the correct information, so I've moved these into
getncchanges_state struct (and now free the struct a bit later so it's
safe to reference in the debug).

The vampire_dc testenv has been updated to use this new behaviour.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Autobuild-User(master): Douglas Bagnall <dbagnall@samba.org>
Autobuild-Date(master): Fri Sep 15 10:07:33 CEST 2017 on sn-devel-144

2 years agogetnchanges.c: Avoid unnecessary continue
Tim Beale [Tue, 22 Aug 2017 05:32:32 +0000 (17:32 +1200)]
getnchanges.c: Avoid unnecessary continue

There's not really much after the continue that we're skipping now. We
can just flip the logic and avoid the continue.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2 years agogetncchanges.c: Split out code to get an object for a response
Tim Beale [Tue, 22 Aug 2017 05:18:32 +0000 (17:18 +1200)]
getncchanges.c: Split out code to get an object for a response

Basically, everytime we try to add an object to the response, we want
to:
- Build it (i.e. pack it into an RPC message format)
- Add it to our object-cache if we're keeping one
- Add any ancestors needed for the client to resolve it (if GET_ANC)

GET_TGT is going to use the exact same code, so split this out into a
separate function, rather than duplicating it.

The GET_ANC case also uses almost identical code, but it differs in a
couple of minor aspects. I've left this as is for now, as I'm not sure
if this is by accident or by design.

Because all the memory was talloc'd off the 'obj' variable, we now need
to replace it with a tmp TALLOC_CTX.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2 years agogetncchanges.c: Reduce the parameters to get_nc_changes_build_object()
Tim Beale [Tue, 22 Aug 2017 04:50:38 +0000 (16:50 +1200)]
getncchanges.c: Reduce the parameters to get_nc_changes_build_object()

Fifteen parameters seems a bit excessive. Instead, pass it the structs
containing the information it cares about.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2 years agogetncchanges.c: Remove unused ncRoot_dn parameter
Tim Beale [Tue, 22 Aug 2017 04:29:17 +0000 (16:29 +1200)]
getncchanges.c: Remove unused ncRoot_dn parameter

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2 years agogetncchanges.c: Remove a really old TODO
Tim Beale [Tue, 22 Aug 2017 04:21:47 +0000 (16:21 +1200)]
getncchanges.c: Remove a really old TODO

This TODO was added in 2009 (before Samba supported linked_attributes
in getNCChanges())

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2 years agogetncchanges.c: Replace hard-coded numbers with a define
Tim Beale [Tue, 22 Aug 2017 04:19:54 +0000 (16:19 +1200)]
getncchanges.c: Replace hard-coded numbers with a define

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2 years agogetncchanges.c: Refactor how objects get added to the response
Tim Beale [Tue, 22 Aug 2017 04:17:10 +0000 (16:17 +1200)]
getncchanges.c: Refactor how objects get added to the response

Adding GET_TGT support is going to make things more complicated, and I
think we are going to struggle to do this without refactoring things a
bit.

This patch adds a helper struct to store state related to a single
GetNCChanges chunk. I plan to add to this with things like max_links,
max_objects, etc, which will cutdown on the number of variables/
parameters we pass around.

I found the double-pointer logic where we add objects to the response
confusing - hopefully this refactor simplifies things slightly, and it
allows us to reuse the code for the GET_TGT case.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2 years agogetncchanges.c: Refactor how we add ancestor links
Tim Beale [Tue, 22 Aug 2017 04:00:57 +0000 (16:00 +1200)]
getncchanges.c: Refactor how we add ancestor links

If the current object had already been sent as an ancestor, we were
duplicating the code that added its links and updated the HWM mark.
We want these to occur when we reach the place where the object's USN
naturally occurs.

Instead of duplicating this code, we can just skip the call to
get_nc_changes_build_object() if the object has already been sent.
There is already an existing 'nothing to send'/continue case after we've
updated the highwater mark.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2 years agogetncchanges.c: Add ancestor links when the object normally gets sent
Tim Beale [Tue, 22 Aug 2017 03:45:39 +0000 (15:45 +1200)]
getncchanges.c: Add ancestor links when the object normally gets sent

Currently we add links each time we send an object, but we don't
actually send these links until the end of the replication cycle.

In subsequent patches we want the links to be sent in the same chunk as
their source object, ideally in as close to USN order as possible.
Processing ancestors complicates this a bit, as the ancestor will have a
higher USN than what we're currently up to, and so potentially will the
ancestor's links.

This patch moves where the ancestor's links get added to the
getnc_state->la_list. The ancestor's links now get added when the object
would normally get sent based purely on its USN (we update the highwater
mark at this point too).

This should not affect functionality, i.e. because we send all the links
at the end, it should make no difference at what point they get added to
the list.

This duplicates a tiny bit of code, but this will be cleaned up in the
next patch.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2 years agogetncchanges.c: Split GET_ANC block out into its own function
Tim Beale [Tue, 22 Aug 2017 03:34:04 +0000 (15:34 +1200)]
getncchanges.c: Split GET_ANC block out into its own function

When we add GET_TGT support, it's going to need to reuse all this code
(i.e. to add any ancestors of the link target). This also trims down
the rather large dcesrv_drsuapi_DsGetNCChanges() function a bit.

Note also fixed a compiler warning in the WERR_DS_DRA_INCONSISTENT_DIT
error block which may have caused issues previously (statement was
terminated by a ',' rather than a ';').

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2 years agogetncchanges.c: Split sorting linked attributes into separate function
Tim Beale [Tue, 6 Jun 2017 03:03:33 +0000 (15:03 +1200)]
getncchanges.c: Split sorting linked attributes into separate function

Longer-term we want to split up the links so that they're sent over
multiple GetNCChanges response messages. So it makes sense to split this
code out into its own function. In the short-term, this removes some of
the complexity from dcesrv_drsuapi_DsGetNCChanges() so that the function
is not quite so big.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2 years agogetncchanges.c: Rename anc_cache to obj_cache
Tim Beale [Fri, 2 Jun 2017 02:42:34 +0000 (14:42 +1200)]
getncchanges.c: Rename anc_cache to obj_cache

When we add GET_TGT support we will reuse the ancestor cache and it
should work the same way - if we've already sent an object because it
was needed for resolving a child object or a link target, then there's
no point sending it again.

This just renames anc_cache --> obj_cache.

An extra is_get_anc flag has been added to getnc_state - once GET_TGT
support is added, we can't assume GET_ANC based solely on the existence
of the obj_cache.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
2 years agocharset: fix str[n]casecmp_m() by comparing lower case values
Stefan Metzmacher [Wed, 6 Sep 2017 07:47:20 +0000 (09:47 +0200)]
charset: fix str[n]casecmp_m() by comparing lower case values

The commits c615ebed6e3d273a682806b952d543e834e5630d^..f19ab5d334e3fb15761fb009e5de876dfc6ea785
replaced Str[n]CaseCmp() by str[n]casecmp_m().

The logic we had in str[n]casecmp_w() used to compare
the upper cased as well as the lower cased versions of the
characters and returned the difference between the lower cased versions.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13018

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
Autobuild-User(master): Ralph Böhme <slow@samba.org>
Autobuild-Date(master): Fri Sep 15 02:23:29 CEST 2017 on sn-devel-144

2 years agocharset/tests: also tests the system str[n]casecmp()
Stefan Metzmacher [Wed, 6 Sep 2017 09:24:28 +0000 (11:24 +0200)]
charset/tests: also tests the system str[n]casecmp()

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13018

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
2 years agocharset/tests: add more str[n]casecmp_m() tests to demonstrate the bug
Stefan Metzmacher [Wed, 6 Sep 2017 08:39:00 +0000 (10:39 +0200)]
charset/tests: add more str[n]casecmp_m() tests to demonstrate the bug

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13018

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
2 years agocharset/tests: assert the exact values of str[n]casecmp_m()
Stefan Metzmacher [Wed, 6 Sep 2017 08:38:37 +0000 (10:38 +0200)]
charset/tests: assert the exact values of str[n]casecmp_m()

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13018

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
2 years agoInstall dcerpc/__init__.py for all Python environments
Alexander Bokovoy [Wed, 13 Sep 2017 08:37:34 +0000 (11:37 +0300)]
Install dcerpc/__init__.py for all Python environments

Also fix whitespace. We use tabs, not spaces in Python/waf code.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13030

Signed-off-by: Alexander Bokovoy <ab@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Stefan Metzmacher <metze@samba.org>
Autobuild-Date(master): Thu Sep 14 22:29:39 CEST 2017 on sn-devel-144

2 years agos4-provision: Ensure the dummy main-domain DB used for DLZ has an @INDEXLIST
Andrew Bartlett [Wed, 30 Aug 2017 03:30:04 +0000 (15:30 +1200)]
s4-provision: Ensure the dummy main-domain DB used for DLZ has an @INDEXLIST

The other databases are created from copies of the main provision, but this one
is not, so did not previously get a valid @INDEXLIST.

This is important as otherwise we will not correctly notice support for
the GUID index or new DSDB features in @SAMBA_DSDB as this is gated
on seeing @SAMBA_FEATURES_SUPPORTED in @INDEXLIST.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2 years agorepl_meta_data: Show failing replicated entry in error code
Andrew Bartlett [Wed, 6 Sep 2017 04:24:35 +0000 (16:24 +1200)]
repl_meta_data: Show failing replicated entry in error code

This re-work of our LDIF printing avoids some of the privacy issue from
printing the full LDIF at level 4, while showing the entry that actually fails.

Instead, with e3988f8f74f4a11e8f26a548e0a33d20f4e863f7 we now print the DN
only at level 4, then the full message at 8.

With this patch on failure, we print the redacted failing message at 5.

While all of the DRS replication data is potentially sensitive
the passwords are most sensitive, and are now not printed unencrypted.

This discourages users from sending the full failing trace, as the
last entry is much more likely the issue.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2 years agoselftest: reindex in dbcheck-oldrelease after modifying the backend DB
Andrew Bartlett [Tue, 12 Sep 2017 02:17:35 +0000 (14:17 +1200)]
selftest: reindex in dbcheck-oldrelease after modifying the backend DB

Modifying the backend DB is not a supported operation, but helps us create test
situations.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2 years agoschema: Rework dsdb_schema_set_indices_and_attributes() db operations
Andrew Bartlett [Mon, 11 Sep 2017 03:22:23 +0000 (15:22 +1200)]
schema: Rework dsdb_schema_set_indices_and_attributes() db operations

Commit ec9b1e881c3eef503d6b4b311594113acf7d47d8 did not fully fix this.

There is no value in using dsdb_replace(), we are under the read lock
and replace just confuses things further.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13025

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2 years agoselftest: Check re-opening sam.ldb corrects the @ATTRIBUTES and @INDEXLIST
Andrew Bartlett [Wed, 13 Sep 2017 04:13:06 +0000 (16:13 +1200)]
selftest: Check re-opening sam.ldb corrects the @ATTRIBUTES and @INDEXLIST

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
2 years agoctdb-protocol: Drop marshalling for monitor controls
Martin Schwenke [Mon, 4 Sep 2017 04:54:47 +0000 (14:54 +1000)]
ctdb-protocol: Drop marshalling for monitor controls

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Thu Sep 14 18:42:28 CEST 2017 on sn-devel-144

2 years agoctdb-client: Drop client code for monitor controls
Martin Schwenke [Mon, 4 Sep 2017 04:51:38 +0000 (14:51 +1000)]
ctdb-client: Drop client code for monitor controls

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
2 years agoctdb-client: Drop old client code for monitor controls
Martin Schwenke [Mon, 4 Sep 2017 04:51:02 +0000 (14:51 +1000)]
ctdb-client: Drop old client code for monitor controls

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>