git.samba.org - samba.git/log

selftest: Run smb2.delete-on-close-perms also with "delete readonly = yes"

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13673

Signed-off-by: Christof Schmitt <cs@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Autobuild-User(master): Christof Schmitt <cs@samba.org>
Autobuild-Date(master): Sat Nov 3 05:55:45 CET 2018 on sn-devel-144

(cherry picked from commit 7dd3585f9c3ae04df45d98bfdc62663c7a69d3e0)

Autobuild-User(v4-9-test): Karolin Seeger <kseeger@samba.org>
Autobuild-Date(v4-9-test): Wed Nov 7 11:44:24 CET 2018 on sn-devel-144

selftest: Add share to test "delete readonly" option

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13673

Signed-off-by: Christof Schmitt <cs@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
(cherry picked from commit a8e79decbcfbae1b1a53ec81b942ee06db26bf8f)

smbd: Fix DELETE_ON_CLOSE behaviour on files with READ_ONLY attribute

MS-FSA states that a CREATE with FILE_DELETE_ON_CLOSE on an existing
file with READ_ONLY attribute has to return STATUS_CANNOT_DELETE. This
was missing in smbd as the check used the DOS attributes from the CREATE
instead of the DOS attributes on the existing file.

We need to handle the new file and existing file cases separately.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13673

Signed-off-by: Christof Schmitt <cs@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
(cherry picked from commit 162a5257c48f20d3752f644e86c9e626b46436c0)

smbtorture: Add test for DELETE_ON_CLOSE on files with READ_ONLY attribute

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13673

Signed-off-by: Christof Schmitt <cs@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
(cherry picked from commit dc9bbbe4141d8425e66fe9290ff611845f4bd1ce)

torture: Fix the clang build

It's used uninitialized if an early torture_assert fails

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13646

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
Autobuild-User(master): Volker Lendecke <vl@samba.org>
Autobuild-Date(master): Thu Nov 1 17:34:31 CET 2018 on sn-devel-144

(cherry picked from commit 9b28d47b0d86570be5a7c5628e460e01207afb00)

Autobuild-User(v4-9-test): Karolin Seeger <kseeger@samba.org>
Autobuild-Date(v4-9-test): Tue Nov 6 12:16:07 CET 2018 on sn-devel-144

vfs_fruit: let fruit_open_meta() with O_CREAT return a fake-fd

This is the final step in implementing the needed macOS semantics on the
FinderInfo stream: as long as the client hasn't written a non-zero
FinderInfo blob to the stream, there mustn't be a visible filesystem
entry for other openers.

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13646

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Thu Nov 1 01:14:23 CET 2018 on sn-devel-144

(cherry picked from commit 1b2de44ea8114cf2025e8b8c843131e2f2dbed27)

vfs_fruit: don't check for delete-on-close on the FinderInfo stream

macOS SMB server doesn't filter out the FinderInfo stream if it has
delete-on-close set.

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13646

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
(cherry picked from commit d8c1bb52018289878b9397d513ebbae63933a05f)

vfs_fruit: let fruit_pwrite_meta_stream also ftruncate empty FinderInfo

fruit_streaminfo currently filters out the FinderInfo stream is
delete-on-close is set. We set it here internally, but the client may
also set it over SMB. Turns out that the macOS SMB server does NOT
filter out FinderInfo stream with delete-on-close set, so we must change
the way filtering is done in fruit_streaminfo.

Filtering is now done based on the FinderInfo stream being 0-bytes large which
is why I'm adding the ftruncate here.

No idea why the tests that check the filtering passed the commits
leading up to this one, but if you revert this commit after applying the
whole patchset, the "delete AFP_AfpInfo by writing all 0" test will fail.

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13646

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
(cherry picked from commit 480695cd723cc4949e0b39ddb83560efac393412)

vfs_fruit: pass stream size to delete_invalid_meta_stream()

delete_invalid_meta_stream() is meant to guard against random data being
present in the FinderInfo stream. If the stream size is 0, it's likely a
freshly created stream where no data has been written to yet, so don't
delete it.

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13646

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
(cherry picked from commit 6e13dbddaec9f8118e11309297f85c3cdfd22e43)

vfs_fruit: let fruit handle all aio on the FinderInfo metadata stream

This will be required to support using fake fds for the FinderInfo
metadata stream.

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13646

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
(cherry picked from commit 6fd256afcd5fedc894036efbfba1fc6d2264fba9)

vfs_fruit: do ino calculation

As we'll start returning fake fds in open shortly, we can't rely on the
next module to calculat correct inode numbers for streams and must take
over that responsibility.

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13646

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
(cherry picked from commit 80afafe398566fd622f431966808d08ba9ec6473)

vfs_fruit: prepare fruit_pread_meta() for reading on fake-fd

If the read on the stream fails we may have hit a handle on a just
created stream (fio->created=true) with no data written yet.

If that's the case return an empty initialized FinderInfo blob.

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13646

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
(cherry picked from commit d7d92710711f6e555ed45c1dda528cd6a83e1bf5)

vfs_fruit: prepare fruit_pwrite_meta() for on-demand opening and writing

This avoid creating files or blobs in our streams backend when a client
creates a stream but hasn't written anything yet. This is the only sane
way to implement the following semantics:

* client 1: create stream "file:foo"

* client 2: open stream "file:foo"

The second operation of client 2 must fail with NT_STATUS_NOT_FOUND.

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13646

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
(cherry picked from commit 4a5c9a9e73230f640eb045a3c47af75b5be9f1d6)

vfs_fruit: prepare struct fio for fake-fd and on-demand opening

Not used for now, that comes in the subsequent commits.

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13646

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
(cherry picked from commit 96320eccc9a63b793ff8d05842bce54f120286f9)

vfs_fruit: add fio->created

fio->created tracks whether a create created a stream.

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13646

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
(cherry picked from commit 1e055a79541eb69eb2deeae897dde3665c5ffee2)

vfs_fruit: remove resource fork special casing

Directly unlinking a file with open handles is not good, don't do it.

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13646

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
(cherry picked from commit a26032c3f68028b01fb6a7d38851409db1858161)

vfs_fruit: add some debugging of dev/ino

Aids in debugging dev/ino mismatch failures in open_file_ntcreate.

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13646

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
(cherry picked from commit 993c7c4e711612225bd07b8fa5544c4bfef88a9e)

s4:torture/vfs/fruit: add test "empty_stream"

One to rule them all: consistently test critical operations on all
streams relevant to macOS clients: the FinderInfo stream, the Resource
Fork stream and an arbitrary stream that macOS maps to xattrs when
written to on a macOS SMB server.

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13646

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
(cherry picked from commit 9d8751db64de75a0de442365b317c1b9ce194170)

s4:torture/vfs/fruit: add check_stream_list_handle()

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13646

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
(cherry picked from commit aba502d440ae3478543d3be5c5bbcea39fb0a463)

s4:torture/util: add torture_smb2_open()

This seems to be missing: a simple wrapper to just open a file without
fancy options.

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13646

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
(cherry picked from commit 64b28e4a94365092f46052712a38e6fcfbb9f20c)

vfs_fruit: filter empty streams

First step in achieving macOS compliant behaviour wrt to empty streams:
- hide empty streams in streaminfo
- prevent opens of empty streams

This means that we may carry 0-byte sized streams in our streams
backend, but this shouldn't really hurt.

The previous attempt of deleting the streams when an SMB setinfo eof to
0 request came in, turned out be a road into desaster.

We could set delete-on-close on the stream, but that means we'd have to
check for it for every write on a stream and checking the
delete-on-close bits requires fetching the locking.tdb record, so this
is expensive and I'd like to avoid that overhead.

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13646

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
(cherry picked from commit ebfcf75e993b1a792db76b94aa898532e1c81eeb)

vfs_fruit: use check on global_fruit_config.nego_aapl for macOS specific behaviour

Ensure any non MS compliant protocol behaviour targetted at supporting
macOS clients are only effective if the client negotiated AAPL.

Currently this only guards the resource fork which only macOS client are
going to use, but subsequent commits add more this at this place.

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13646

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
(cherry picked from commit 046456fca60b9fef38a0cdf5a116883fb1c7dffe)

s4:torture/vfs/fruit: enable AAPL extensions in a bunch of tests

These tests check for macOS SMB server specific behaviour. They work
currently against Samba without enabling AAPL because in vfs_fruit we're
currently don't check whether AAPL has been negotiated in one place. A
subsequent commit will change that and this commit prepares for that
change.

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13646

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
(cherry picked from commit 7485007bb0a69dc8237c78be90f4bae06aad49b2)

vfs_fruit: don't unlink 0-byte size truncated streams

This caused all sort of havoc with subsequent SMB request that acted on
the handle of the then deleted backend storage (file or blob, depending
on the used streams module).

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13646

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
(cherry picked from commit 287082d739362ad068df37076712927ccd7ca90c)

s4:torture/vfs/fruit: write some data to a just created teststream

Doesn't currently make a difference, but this prepares for a later
change in vfs_fruit that will filter out empty streams (which is the
macOS behaviour).

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13646

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
(cherry picked from commit 3195ff9547824dd129e3fbcbbb8823fb10a0a5d4)

s4:torture/vfs/fruit: expand test "setinfo eof stream"

o Adds checks verifying that after setting eof to 0 on a stream, a
  subsequent open gets ENOENT, before and after closing the handle that
  had been used to set eof to 0.

o Verify that a write to a handle succeeds after that handle has been
  used to set eof to 0 on a stream.

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13646

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
(cherry picked from commit b35d9e74028727da24f349242cb0bc62403a9b09)

vfs_fruit: update handling of read-only creation of resource fork

macOS SMB server versions supports this since 10.12, so we adapt our
behaviour.

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13646

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
(cherry picked from commit 055ca448f1a2bd995291af1810c9ec94f90f32dd)

s4:torture/vfs/fruit: update test "creating rsrc with read-only access" for newer macOS versions

While this operation failed against older macOS versions, it passes
against versions 10.12 and newer. Update the test accordingly, a
subsequent commit will then update our implementation.

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13646

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
(cherry picked from commit c3cf09a0791ed4dfb7ca18d175396ac1aca5d2ed)

s4:torture/vfs/fruit: expand existing vfs_test "null afpinfo"

This adds a check that a read on a seperate handle also sees the
previously created AFP_AfpInfo stream.

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13646

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
(cherry picked from commit 6f428607e35055b9ce1a04a625d43740bf5c76b2)

s4:torture/vfs/fruit: expand existing test "setinfo delete-on-close AFP_AfpInfo" a little bit

Add a check that verifies a create on a stream gets
NT_STATUS_DELETE_PENDING after delete-on-close has been set.

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13646

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
(cherry picked from commit 0d9a80b6b2e3f51cad122ee0dc78c2f4dfdebcb5)

s4:torture/vfs/fruit: update test "read open rsrc after rename" to work with macOS

macOS SMB server seems to return NT_STATUS_SHARING_VIOLATION in this
case while Windows 2016 returns NT_STATUS_ACCESS_DENIED.

Lets stick with the Windows error code for now in the Samba fileserver,
but let the test pass against macOS.

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13646

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
(cherry picked from commit 9cd9859dc10d425d29774d221ec9ad697192b466)

s4:torture/vfs/fruit: ensure a directory handle is closed in all code paths

Otherwise we get a sharing violation when running against Samba and
opening the directory a second time.

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13646

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
(cherry picked from commit f7551d8fb3599a6a71dd0570bad02bdc48324107)

s4:torture/vfs/fruit: update test "stream names" to work with macOS

o create the basefile before trying to create a stream on it, otherwise
this fails on macOS

o write something to the stream, otherwise the stream is not listed as
macOS hides 0-byte sized streams

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13646

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
(cherry picked from commit 125498861abeeed5bde727ff98d88d9ce4af6c75)

s4:torture/vfs/fruit: update test "SMB2/CREATE context AAPL" to work against macOS

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13646

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
(cherry picked from commit dba9a4743fc42b57f03b6ace24753ce8091f629f)

s4:torture/vfs/fruit: set share_access to NTCREATEX_SHARE_ACCESS_MASK in check_stream_list

Avoid sharing conflicts with other opens on the basefile.

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13646

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
(cherry picked from commit 1d4e50649b02c7c2314aa04e08f9946e21b6b37d)

s4:torture/vfs/fruit: fix a few error checks in "delete AFP_AfpInfo by writing all 0"

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13646

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
(cherry picked from commit 394d1fc9ac8373ea7419ea9514750422bd6e504d)

s4:torture/vfs/fruit: skip a few tests when running against a macOS SMB server

These tests are designed to test specific vfs_fruit functionality.

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13646

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
(cherry picked from commit 242f42ef79e51e35ab33331e2f2d7c9feaac36f8)

vfs_streams_xattr: fix open implementation

Since a long time the modules's open function happily returned success
when opening a non existent stream without O_CREAT.

This change fixes it to return -1 and errno=ENOATTR if

o get_ea_value() returns NT_STATUS_NOT_FOUND (eg mapped from
getxattr() = -1, errno=ENOATTR) and

o flags doesn't contain O_CREAT

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13646

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
(cherry picked from commit 9f58997d7a1ab0358963a45cb8bf291e7afa8068)

ctdb-recovery: Ban a node that causes recovery failure

... instead of applying banning credits.

There have been a couple of cases where recovery repeatedly takes just
over 2 minutes to fail.  Therefore, banning credits expire between
failures and a continuously problematic node is never banned,
resulting in endless recoveries.  This is because it takes 2
applications of banning credits before a node is banned, which
generally involves 2 recovery failures.

The recovery helper makes up to 3 attempts to recover each database
during a single run.  If a node causes 3 failures then this is really
equivalent to 3 recovery failures in the model that existed before the
recovery helper added retries.  In that case the node would have been
banned after 2 failures.

So, instead of applying banning credits to the "most failing" node,
simply ban it directly from the recovery helper.

If multiple nodes are causing recovery failures then this can cause a
node to be banned more quickly than it might otherwise have been, even
pre-recovery-helper.  However, 90 seconds (i.e. 3 failures) is a long
time to be in recovery, so banning earlier seems like the best
approach.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13670

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Mon Nov  5 06:52:33 CET 2018 on sn-devel-144

(cherry picked from commit 27df4f002a594dbb2f2a38afaccf3e22f19818e1)

s3:smbd: remove now unused check if fsp is NULL

This was used internally to mark an aio request as cancelled. As the aio
cancellation functionality has been removed, we can now also remove this
check.

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13667

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
(cherry picked from commit 3fa45900c0ada6596daf09aa4c5054974f3394b0)

s3:smbd: fix SMB2 aio cancelling

As we currently don't attempt to cancel the internal aio request, we
must ignore the SMB2 cancel request and continue to process the SMB2
request, cf MS-SM2 3.3.5.16:

  If the target request is not successfully canceled, processing of the
  target request MUST continue and no response is sent to the cancel
  request.

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13667

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
(cherry picked from commit 07eb805cde4db6ac74740e3cfbc56ab6f8e2118a)

s4:torture/smb2/read: add test for cancelling SMB aio

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13667

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
(cherry picked from commit e37ff8c5fe18d400e378bf2591e209b30473d9f9)

vfs_delay_inject: implement pread_send and pwrite_send

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13667

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
(cherry picked from commit 35f9bc06722fb07143c832442d784beb204bd367)

s4:libcli/smb2: reapply request endtime

tevent_req_finish() removed a possible request timeout, make sure to
reinstall it. This happened when an interim SMB2 response was received.

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13667

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
(cherry picked from commit a6de555c51ca34ef24ac4b4cb672cd748d3197a1)

libcli: fill endtime if smbXcli_req_create() timeout is non-zero

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13667

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
(cherry picked from commit e7eec24d27b8694fd7626577a2aca05e99045964)

libcli: add smbXcli_req_endtime

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13667

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
(cherry picked from commit 94ad5ee662a5bfe1c79d8a9c3fcf51d0a1d652c7)

dsdb group audit tests: log_membership_changes extra tests

Add extra tests to ensure better test coverage of log_membership_changes

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
(cherry picked from commit aeef8b41fa03a32859f824f4a09560ad83bd2b50)

Autobuild-User(v4-9-test): Karolin Seeger <kseeger@samba.org>
Autobuild-Date(v4-9-test): Mon Nov 5 15:47:40 CET 2018 on sn-devel-144

dsdb group audit tests: check_version improve diagnostics

Change check_version to display the expected, actual along with the
line and name of the failing test, rather than the line in check_version

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
(cherry picked from commit c952fc1273397c04fddf177bcd809551d6324bdd)

dsdb group audit tests: check_timestamp improve diagnostics

Change check_timestamp to display the expected, actual along with the
line and name of the failing test, rather than the line in
check_timestamp.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
(cherry picked from commit e2970887140d558c6359fd9b3f8c2a4c26d2cf35)

dsdb group audit: align dn_compare with memcmp

Rename the parameter names and adjust the return codes from dn_compare
so that:
dn_compare(a, b) =>

LESS_THAN means a is less than b.
GREATER_THAN means a is greater than b.

Thanks to metze for suggesting the correct semantics for dn_compare

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13664

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
(cherry picked from commit 8420a4d0fddd71af608635a707ef20f37fa9b627)

dsdb group_audit: Test to replicate BUG 13664

The group audit code incorrectly logs member additions and deletions.

Thanks to metze for the debugging that isolated the issue, and for
suggesting the fix to dn_compare.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13664

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
(cherry picked from commit eeb4089dafc45277d8af19073ef9348451c1836a)

dsdb encrypted_secrets: Allow "ldb:// and "mdb://" in file path

Correctly handle "ldb://" and "mdb://" schemes in the file path when
determining the path for the encrypted secrets key file.

When creating a new user and specifying the local file path of the
sam.ldb DB, it was possible to create an account that you could not
login with. The path for the key file was incorrectly calculated
for the "ldb://" and "mdb://" schemes, the scheme was not stripped from
the path and the subsequent open of the key file failed.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13653

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Fri Oct 19 09:34:46 CEST 2018 on sn-devel-144

(cherry picked from commit 7b59cd74f9f75d85b91c6ca517d0243e7f6bd2e1)

dsdb encrypted_secrets tests: Allow "ldb://" in file path

When creating a new user and specifying the local file path of the
sam.ldb DB, it's possible to create an account that you can't actually
login with.

This commit contains tests to verify the bug.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13653

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
(cherry picked from commit e1eee614ca8a3f0f5609a3d9d8ce7ae926de1f9e)

python tests Blackbox: add random_password

Add the random_password method to the BlackboxTestCase class and remove
duplicated copies from other test cases. Also use SystemRandom so that
the generated passwords are more cryptographically sound.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
(cherry picked from commit b6e45fb479689cff028b1fe626533b035e313ce3)

ldb: Bump ldb version to 1.4.3

* Python: Ensure ldb.Dn can accept utf8 encoded unicode (bug 13616)

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13616

Signed-off-by: Noel Power <noel.power@suse.com>
Signed-off-by: Andrew Bartlett <abartlet@samba.org>

lib/ldb: Ensure ldb.Dn can accept utf8 encoded unicode

Additionally remove the associated known fail.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13616
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
(cherry picked from commit cddd54e8654c94dedd57c08af1987ce03212ce20)

lib/ldb/tests: add test for ldb.Dn passed utf8 unicode

object dn format should be a utf8 encoded string
Note: Currently this fails in python2 as the c python binding for
      the dn string param uses PyArg_ParseTupleAndKeywords() with 's'
      format, this will accept str *or* unicode in the default encoding.
      The default encoding in python2 is... ascii.

Also adding here a knownfail to squash the error produced by the test.

Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
(cherry picked from commit d1492ab919b19d1ca72f1d7c97ac0ca3bee13a2a)

lib/ldb: Test correct variable for no mem condition

Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
(cherry picked from commit d786e1fca95395e793867278bc0408e33c19908b)

dsdb: Add comments explaining the limitations of our current backlink behaviour

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13418

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Tim Beale <timbeale@catalyst.net.nz>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Tue Oct 30 10:32:51 CET 2018 on sn-devel-144

(cherry picked from commit 852e1db12b0afa04a738c03bb2609c084fe96a7f)

s4:samldb: internally use extended dns while changing the primaryGroupID field

This is important, otherwise we'll loose the <SID=> component of the
linked attribute.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13418

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
(cherry picked from commit 7a36cb30b716d56b84e894851c1a18e9eb3a0964)

s4:repl_meta_data: add support for DSDB_CONTROL_DBCHECK_FIX_LINK_DN_SID

This will be used by dbcheck in the next commits.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13418

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
(cherry picked from commit 0386307e34097f5d9233c970983c7306d1705a87)

s4:repl_meta_data: pass down struct replmd_replicated_request to replmd_modify_la_replace()

This will simplify further changes.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13418

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
(cherry picked from commit 1ef145d9d72d847055f6aba8a0070b3e1cfdabbc)

s4:repl_meta_data: pass down struct replmd_replicated_request to replmd_modify_la_delete()

This will simplify further changes.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13418

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
(cherry picked from commit 738b52eb0856c8fcdbb8589e8061bcc14700c23a)

s4:repl_meta_data: add missing \n to a DEBUG message in replmd_modify_la_add()

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13418

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
(cherry picked from commit 70a306d0bd6806d1fd00d45e3d8cc70c73d09f79)

s4:repl_meta_data: pass down struct replmd_replicated_request to replmd_modify_la_add()

This will simplify further changes.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13418

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
(cherry picked from commit 42e69a86ca583e3cb20c63b9c6930b4b3425485d)

s4:repl_meta_data: pass down struct replmd_replicated_request to replmd_modify_handle_linked_attribs()

This will simplify further changes.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13418

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
(cherry picked from commit 966c7febaf0245516481bde924ea6cd738eeb78b)

blackbox/dbcheck-links: Test broken links with missing <SID=...> on linked attributes

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13418

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
(cherry picked from commit f81771c8593327e058b9cb4330d7e77083df3ea9)

dbchecker: Fix missing <SID=...> on linked attributes

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13418

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
(cherry picked from commit a801799ebe26780653f4ed3fa3fc633e31871f7d)

dbchecker: improve verbose output of do_modify()

This makes it easier to debug dbcheck problems.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13418

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
(cherry picked from commit c5c99b569569ce36cac94e967ca53e3182abd6f7)

s4:dsdb: add DSDB_CONTROL_DBCHECK_FIX_LINK_DN_SID oid

This will be used to fix missing <SID=> components in future.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13418

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
(cherry picked from commit bb9c9e49a5e82f19626cb1b12ec9189fff5114e8)

testprogs/blackbox: add samba4.blackbox.test_primary_group test

This demonstrates the bug, that happens when the primaryGroupID
of a user is changed.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13418

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
(cherry picked from commit 364ed537e0bcb3a97cae0f2d1ff72de9423ce0e6)

s4:dsdb: fix comment on DSDB_CONTROL_DBCHECK_FIX_LINK_DN_NAME

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
(cherry picked from commit 60131b4452d43b3792e7f27a4190c88e7aabb1b4)

schema_samba4.ldif: add allocation of DSDB_CONTROL_DBCHECK_FIX_LINK_DN_NAME

This was already allocated in source4/dsdb/samdb/samdb.h with
commit 22208f52e6096fbe9413b8ff339d9446851e0874.

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
(cherry picked from commit 0189f23f5bda263c7462366ee16b2fe4bcda0119)

vfs_fruit: optionally delete AppleDouble files without Resourcefork data

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13642

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
(cherry picked from commit 3649f1a41a299b14609318ef52b44e2d53cba4b5)

vfs_fruit: add option "delete_empty_adfiles"

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13642

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
(cherry picked from commit e00e61345ccd88022cd24e62ac29e2c56a8f6117)

vfs_fruit: detect empty resource forks in ad_convert()

For some reason the macOS client often writes AppleDouble files with a
non-zero sized resource fork, but the resource fork data is just
boilerplate data with the following string close to the start

  This resource fork intentionally left blank

A dump with apple_dump looks like this:

Entry ID   : 00000002 : Resource Fork
Offset     : 00000052 : 82
Length     : 0000011E : 286

-RAW DUMP--:  0  1  2  3  4  5  6  7  8  9  A  B  C  D  E  F : (ASCII)
00000000   : 00 00 01 00 00 00 01 00 00 00 00 00 00 00 00 1E : ................
00000010   : 54 68 69 73 20 72 65 73 6F 75 72 63 65 20 66 6F : This resource fo
00000020   : 72 6B 20 69 6E 74 65 6E 74 69 6F 6E 61 6C 6C 79 : rk intentionally
00000030   : 20 6C 65 66 74 20 62 6C 61 6E 6B 20 20 20 00 00 :  left blank   ..
00000040   : 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 : ................
00000050   : 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 : ................
00000060   : 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 : ................
00000070   : 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 : ................
00000080   : 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 : ................
00000090   : 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 : ................
000000A0   : 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 : ................
000000B0   : 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 : ................
000000C0   : 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 : ................
000000D0   : 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 : ................
000000E0   : 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 : ................
000000F0   : 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 : ................
00000100   : 00 00 01 00 00 00 01 00 00 00 00 00 00 00 00 1E : ................
00000110   : 00 00 00 00 00 00 00 00 00 1C 00 1E FF FF       : ..............

We can safely discard this Resource Fork data.

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13642

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
(cherry picked from commit 7be979f958295474f0c0df6a4db0b5bca9a6676d)

vfs_fruit: add option "wipe_intentionally_left_blank_rfork"

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13642

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
(cherry picked from commit 2dbb2d287399e9c829a4fd4908a6dfba9fdfd7e8)

s4:torture: add test for AppleDouble ResourceFork conversion

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13642

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
(cherry picked from commit 6f022e61597994bc032e61876f24150d7acb3fc2)

s3:selftest: list vfs testssuites one per line

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13642

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
(cherry picked from commit c3a1f3ec9ba2402de2a876ca06086b2d53e122f0)

docs:vfs_fruit: add "delete_empty_adfiles" option

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13642

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
(cherry picked from commit 2a9574b138f620e0a65cd61a957b99653c8dcd78)

docs:vfs_fruit: add "wipe_intentionally_left_blank_rfork" option

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13642

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
(cherry picked from commit f7a98094f05dd4b9abf5dc9704222aa5a07584d4)

s3:winbind: Check return code of initialize_password_db()

See https://retrace.fedoraproject.org/faf/reports/1577174/

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13668

Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
(cherry picked from commit ba17cae4cab686b8d018c39d16706e621f9f93ac)

lib:socket: If returning early, set ifaces

Prevents a segfault in load_interfaces() when total interfaces == 1.
Fixes regression caused by da68a1b2f417ec82ea4ed3e7a4d867cef8ca8f93.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13665

Signed-off-by: David Mulder <dmulder@suse.com>
Reviewed-by: Ralph Böhme <slow@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Autobuild-User(master): Volker Lendecke <vl@samba.org>
Autobuild-Date(master): Sun Oct 28 00:35:35 CEST 2018 on sn-devel-144

(cherry picked from commit 5391e21dea3168691cee1e6ef6c5959e46d62d1e)

vfs_fruit: remove check for number of xattrs from ad_convert_xattr

Turns out that there exist AppleDouble files with an extended FinderInfo
entry that includes the xattr marshall buffer, but the count of xattrs
in the buffer is just zero.

We do want to discard this extended FinderInfo entry and convert it to a
simple fixed size FinderInfo entry, so remove the check.

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13649

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
(cherry picked from commit 5d565f636fcf49fc1bbbfbc24ef730d2f7cc2cf0)

dsdb: Ensure that a DN (now) pointing at a deleted object counts for objectclass-based MUST

Add the 'reveal_internals' controls when performing objectclass-based
checks of mandatory attributes. This prevents the extended_dn DSDB
module from suppressing attributes that point to deleted (i.e.
non-existent/expunged) objects.

This ensures that, when modifying an object (and often not even
touching the mandatory attribute) that the fact that an attribute is a
DN, and the DN target is deleted, that the schema check will still pass.

Otherwise a fromServer pointing at a dead server can cause failures,
i.e. you can't modify the affected object at all, because the DSDB
thinks a mandatory attribute is missing.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13621

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
(cherry picked from commit 4092b369aeeb7058d78b8d6f41dbbc6d69203ecc)

tests: Add corner-case test: fromServer points to dead server

The fromServer attribute is slightly unique, in that it's a DN (similar
to a one-way link), but it is also a mandatory attribute.

Currently, if fromServer gets a bad value (i.e. a dead server that has
been expunged), the DSDB rejects any attempts to modify the associated
nTDSConnection object (regardless of whether or not you're actually
changing the fromServer attribute).

This patch adds a test-case that demonstrates how the DB can get into
such a state.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13621

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
(cherry picked from commit dec3eda1f74f5bf7ea91c1be3d5dfd832e9672b9)

libcli: Add debug message if fail to negoatiate SMB protocol

Currently if the client and server can't negotiate an SMB protocol, you
just get the followiing error on the client-side, which doesn't tell you
much.
ERROR(runtime): uncaught exception - (3221225667, 'The network responded
incorrectly.')

This patch adds a debug message to help highlight what's actually going
wrong.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13621

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Fri Sep 28 11:25:29 CEST 2018 on sn-devel-144

(cherry picked from commit 34cbd89fec836f5de0cb5ba3f289b1f4ae00c5d7)

s3/smbd: Server responds incorrectly if no SMB protocol chosen

The SMBnegprot response from the server contains the DialectIndex of the
selected protocol from the client's request message. Currently, if no
protocol is selected, the server is responding with a DialectIndex=zero,
which is a valid index (PROTOCOL_CORE by default). The Windows spec, and
historically the code, should return DialectIndex=0xffff if no protocol
is chosen. The following commit changed it recently (presumably
inadvertently), so that it now returns DialectIndex=zero.

06940155f315529c5b5 s3:smbd: Fix size types in reply_negprot()

This results in somewhat confusing error messages on the client side:
ERROR(runtime): uncaught exception - (3221225997, 'The transport
connection has been reset.')

or, when signing is configured as mandatory:
smbXcli_negprot: SMB signing is mandatory and the selected protocol
level (1) doesn't support it.
ERROR(runtime): uncaught exception - (3221225506, '{Access Denied} A
process has requested access to an object but has not been granted those
access rights.')

This patch restores the old behaviour of returning 0xffff.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13621

Pair-Programmed-With: Ralph Boehme <slow@samba.org>
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
(cherry picked from commit 378706266496ce79c1887fe96ab3b15f56770244)

netcmd: Make sure SMB connection is signed when backing up sysvol

i.e. protect the client against man-in-the-middle attacks by default.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13621

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
(cherry picked from commit 0122f45f053ecc545950c31bf1fb33fba143478c)

python: Allow forced signing via smb.SMB()

Signed-off-by: David Mulder <dmulder@suse.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
(cherry picked from commit 4c7348e44d10ca519dd1322fd40b12c69e17a8e6)

Back-ported as a dependency required for:
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13621

selftest: Change backup testenvs to use non-default site

Previously (i.e. up until the last patch) the backup/restore commands
only worked if the Default-First-Site-Name site was present. If this
site didn't exist, then the various restore testenvs would fail to
start. This is now fixed, but this patch changes the backupfrom testenv
so that it uses a non-default site. This will detect the problem if it
is ever re-introduced.

To do this we need to change provision_ad_dc() so the
extra_provision_options can be specified as an argument. (Note that Perl
treats undef the same as an empty array).

By default, the restore will add the new DC into the
Default-First-Site-Name site. This means the backupfromdc and restored
testenvs will now have different sites, so we need to update the ldapcmp
filters to exclude site-specific attributes.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13621

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
(cherry picked from commit 696fa6a1e6c615a992a3016ff32405b864b62eec)

netcmd: Re-create default site for backup-restore (if missing)

Normally when a new DC joins a domain, samba-tool works out the new
DC's site automatically. However, it does this by querying the existing
DC using CLDAP. In the restore case, there is no DC running. We could
still query the DB on disk and work out the correct site based on the
new DC's IP, however:
- comparing between the CN=Subnet DNs and an IP-address string seems
like it'd be non-trivial to write, and
- in the lab-domain rename case, chances are the user will want a
completely different subnet to what's already in the DB.

The restore command now has a --site option so the user can specify an
appropriate site for the restored DC. This patch makes the restore
command work by default (i.e. without a --site option) even if the
default Default-First-Site-Name doesn't exist. Basically the solution is
to just check Default-First-Site-Name exists and create it if it
doesn't. As the recommended workflow is to use the restored DC as a
temporary seed that you'll later throw away, this approach seems
acceptable. Subsequent DCs will then be joined to the running restored
DC, so an appropriate site will be determined using CLDAP. The only
side-effect is potentially an extra Site object.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13621

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
(cherry picked from commit ce57a800c9bed7e6876cdc0baf3a2d5fdc879ecf)

tests: Add test-case for restore into non-default site

Add a test-case that exercises the new '--site' restore option and
ensures the restored DC gets added to the correct site.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13621

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
(cherry picked from commit ad69aaf7e13435111fc990954ff0bc81ed5325c5)

netcmd: Add --site option when restoring a domain

Restoring a backup only worked if the Default-First-Site-Name site was
still present. When the new restored DC account is created, it was
trying to add the new server's DN under CN=Default-First-Site-Name.
However, if the original domain was setup using a different site, then
the restore would fail because the DN didn't exist.

When running the restore command, you should be able to specify the
site that you want the new/restored DC to be in (same as during a
DC 'join'). Passing the correct --site argument is one way to avoid
this problem. (A subsequent patch will further improve the tool so it
can work around non-default sites automatically).

Note we also need to pass the site through to where the new DNS entries
get registered (in the rename case).

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13621

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
(cherry picked from commit e1f255a4d54b59924295ea875fdef62ccebb8811)

ctdb-daemon: Fix valgrind hit in event code

==25741== Syscall param write(buf) points to uninitialised byte(s)
==25741==    at 0x4939291: write (write.c:27)
==25741==    by 0x4868285: sys_write (sys_rw.c:68)
==25741==    by 0x13915D: sock_queue_trigger (sock_io.c:316)
==25741==    by 0x4DE6478: tevent_common_invoke_immediate_handler (in /usr/lib/x86_64-linux-gnu/libtevent.so.0.9.37)
==25741==    by 0x4DE64A2: tevent_common_loop_immediate (in /usr/lib/x86_64-linux-gnu/libtevent.so.0.9.37)
==25741==    by 0x4DEBE5A: ??? (in /usr/lib/x86_64-linux-gnu/libtevent.so.0.9.37)
==25741==    by 0x4DEA2D6: ??? (in /usr/lib/x86_64-linux-gnu/libtevent.so.0.9.37)
==25741==    by 0x4DE57E3: _tevent_loop_once (in /usr/lib/x86_64-linux-gnu/libtevent.so.0.9.37)
==25741==    by 0x15D1BA: ctdb_event_script_args (eventscript.c:821)
==25741==    by 0x13B437: ctdb_start_daemon (ctdb_daemon.c:1315)
==25741==    by 0x110642: main (ctdbd.c:393)
==25741==  Address 0x57888a4 is 100 bytes inside a block of size 144 alloc'd
==25741==    at 0x48357BF: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==25741==    by 0x4B9B7C0: talloc_named_const (in /usr/lib/x86_64-linux-gnu/libtalloc.so.2.1.14)
==25741==    by 0x15CCC6: eventd_client_write (eventscript.c:430)
==25741==    by 0x15CCC6: eventd_client_run (eventscript.c:556)
==25741==    by 0x15CCC6: ctdb_event_script_run (eventscript.c:649)
==25741==    by 0x15D198: ctdb_event_script_args (eventscript.c:812)
==25741==    by 0x13B437: ctdb_start_daemon (ctdb_daemon.c:1315)
==25741==    by 0x110642: main (ctdbd.c:393)
==25741==

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13659

Pair-programmed-with: Amitay Isaacs <amitay@gmail.com>
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Mon Oct 22 09:27:15 CEST 2018 on sn-devel-144

(cherry picked from commit fbea9d36996f248ba2b077f12ad16c199b853134)

Autobuild-User(v4-9-test): Karolin Seeger <kseeger@samba.org>
Autobuild-Date(v4-9-test): Thu Oct 25 11:05:48 CEST 2018 on sn-devel-144

ctdb-event: Check the return status of sock_daemon_set_startup_fd

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13659

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
(cherry picked from commit a1909603808b994b7822b697494e39e8da4aaa66)

ctdb-common: Set close-on-exec for startup fd

The startup_fd should not be propagated to the child processes created
from a daemon. It should only be used in the daemon code to return the
status of the startup. Another use of startup_fd is to notify the
parent if the daemon process has exited.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13659

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
(cherry picked from commit 80549927bc1741a4b8af8b8e830de4d37fa0c4a8)

ctdb-daemon: Exit if eventd goes away

ctdbd enters a broken state if eventd goes away.  A clean shutdown is
not possible because that involves running events.  Restarting eventd
is possible but this might mask a serious problem and it is possible
that eventd might keep on disappearing.  Just exit.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13659

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
(cherry picked from commit c9e1603a5d0c1a216439d4a2b0e7cdc05181e898)