Noel Power [Tue, 16 Oct 2018 18:39:41 +0000 (19:39 +0100)]
Add new CI job for new purepy3-none-env autobuild task
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Noel Power [Tue, 16 Oct 2018 18:25:26 +0000 (19:25 +0100)]
script: Add new (temporary) autobuild task for none-env tests.
Ideally we want all the tests to run under python3 by default (no
special task for this) and then convert the existing '-py3' tasks
to run the python tests with python3.
However at the moment the convertion process is not ready to do this,
for a while we need to run separate autobuild tasks for this.
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Noel Power [Wed, 17 Oct 2018 17:06:34 +0000 (18:06 +0100)]
python/samba/netcmd: PY3 fix CI error for samba.tests.samba_tool.help
Strangely the test was failing on CI only, looks like there is an
issue with order of elements returned from dict.items() with python3.4
(version of python in CI docker instance) and python3.6 (version on my
development machine). Changed code to sort the keys so order of help
printed out should be the same for each invocation.
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Noel Power [Wed, 10 Oct 2018 11:25:09 +0000 (12:25 +0100)]
python/samba/provision: PY3 PY3 port samba4.blackbox.provision-backend
Enclose filter with list as filter object has no 'len' method
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Noel Power [Wed, 10 Oct 2018 11:21:10 +0000 (12:21 +0100)]
python/samba: PY3 port samba4.blackbox.provision-backend
convert ldif content to str when necessary
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Noel Power [Wed, 10 Oct 2018 11:18:32 +0000 (12:18 +0100)]
python/samba/provision: PY3 port samba4.blackbox.provision-backend
Fix some attibrutes that need to be treated as str
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Noel Power [Wed, 10 Oct 2018 10:18:13 +0000 (11:18 +0100)]
python/samba/netcmd: PY3 make sure get_testparm_var returns 'str'
part of PY3 port samba4.blackbox.upgrade.samba3-upgrade*
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Noel Power [Wed, 10 Oct 2018 10:16:40 +0000 (11:16 +0100)]
python/samba/samba3: PY3 tdb.Tdb.get method expects bytes in PY3
part of PY3 port samba4.blackbox.upgrade.samba3-upgrade*
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Noel Power [Tue, 9 Oct 2018 20:16:08 +0000 (21:16 +0100)]
python/samba/netcmd: PY3 fix samba.tests.domain_backup_offline
Fix attributes that need to be treated as str not bytes
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Noel Power [Mon, 8 Oct 2018 14:21:45 +0000 (15:21 +0100)]
python/samba: PY3 port samba4.blackbox.functionalprep
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Noel Power [Mon, 8 Oct 2018 11:27:56 +0000 (12:27 +0100)]
python/samba/netcmd: PY3 Fix error in samba4.blackbox.schemaupgrade
Getting Exception: must be str, not ldb.bytes error in scheme_upgrade
phase of test
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Noel Power [Fri, 5 Oct 2018 15:49:45 +0000 (16:49 +0100)]
selftest/filter: PY3 Make filter-subunit forgiving of decoding errors
samba4.local.ndr for one is a test that outputs string in an encoding
that stdin.readline() guesses to be utf8 (but it isn't) filter subunit
can afford to be forgiving of some random text that can't be decoded as
utf8 so lets do that.
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Noel Power [Fri, 5 Oct 2018 14:10:52 +0000 (15:10 +0100)]
python/samba: PY3 fix samba4.blackbox.upgradeprovision.release-4-0-0
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Noel Power [Thu, 4 Oct 2018 17:29:37 +0000 (18:29 +0100)]
s4/scripting/bin: blobs needs to be bytes
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Noel Power [Wed, 3 Oct 2018 14:19:09 +0000 (15:19 +0100)]
s4/scripting/bin: PY3 fix samba4.blackbox.upgradeprovision.alpha13
Noel Power [Tue, 2 Oct 2018 13:45:53 +0000 (14:45 +0100)]
s4/scripting/bin: PY3 make sure GUID result of format is string
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Noel Power [Mon, 27 Aug 2018 19:29:53 +0000 (20:29 +0100)]
s4/scripting: Py2/Py3 for samba4.blackbox.upgradeprovision.alpha13
fixup source4/scripting/bin/samba_upgradedns
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Noel Power [Mon, 27 Aug 2018 12:08:26 +0000 (13:08 +0100)]
s4/scripting: PY2/PY3 port for samba4.blackbox.upgradeprovision.current
o Fix various ldb attribute that need to be converted to string
o dict has no 'has_key' method
o ndr_unpack needs bytes not string
o b64encode needs bytes (so open file with binary mode)
o StandardError was removed in python3 use Exception instead
o fix octal literals
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Noel Power [Sat, 25 Aug 2018 11:50:58 +0000 (12:50 +0100)]
python/samba: samba4.blackbox.dbcheck.release-4-1-0rc3
* Various string related fixed py2/py3
* Fix strange double decode followed by encode code (see comments
in change)
* Added dump_attr_values, simply printing attribute values (from
sequence) doesn't work (when using string '%s' format codes in
existing string). We need to print out string from bytes in PY3
and fallback to repr(which will print b'blah') if we get a decode error
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Noel Power [Sat, 25 Aug 2018 10:39:04 +0000 (11:39 +0100)]
samba_tool: Enclose iterator with list
Really strange bug caused by map being updated while being iterated.
This caused keys to be skipped and inconsistent and incorrect
results from ldapcmp.
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Noel Power [Thu, 23 Aug 2018 16:59:12 +0000 (17:59 +0100)]
python/samba: misc use of str for ldb.bytes
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Noel Power [Mon, 1 Oct 2018 13:47:06 +0000 (14:47 +0100)]
python/samba: ldb attribute string fix for wafsamba.tests
Noel Power [Mon, 6 Aug 2018 12:39:15 +0000 (13:39 +0100)]
PY3: ensure StringIO usage is py2/py3 compatible
Noel Power [Mon, 6 Aug 2018 14:02:18 +0000 (15:02 +0100)]
PY3: relative import fixes
Noel Power [Fri, 28 Sep 2018 15:49:27 +0000 (16:49 +0100)]
s4/scripting/bin: PY3 Fix exception tuple assignments.
In Python3 to access the exception arguments you need to now use
Exception.args, in Python2 you could access these direcly with the
'except' declaration.
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Noel Power [Sun, 5 Aug 2018 20:08:33 +0000 (21:08 +0100)]
PY3: port samba.tests.samba3sam
Noel Power [Wed, 5 Sep 2018 13:59:19 +0000 (14:59 +0100)]
python/samba: use PY3 version of ConfigParser
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Noel Power [Mon, 15 Oct 2018 09:58:23 +0000 (10:58 +0100)]
python/samba/gp_parse: PY2/PY3 compat porting for gp_init.py
Fixes
1) use compat versions of ConfigParser and StringIO
2) open file needs to be opened in binary mode as write_pretty_xml
routine uses BytesIO() object.
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Noel Power [Fri, 24 Aug 2018 08:55:08 +0000 (09:55 +0100)]
python/samba: add alias for ConfigParser for PY2/PY3 compatability
ConfigParser module changed name to configParser in PY3, additionally
the behaviour regarding interpolation has changed. ConfigParser now
has a default interpolation param whose behaviour demands that '%' is
escaped. To maintain behaviour with the python2 version this default
param needs to be changed. Add some alias(s) and 'shim' Configparser
symbol in samba.compat to cater for this.
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Noel Power [Sat, 4 Aug 2018 20:00:06 +0000 (21:00 +0100)]
PY3: convert samba.tests.strings to Py2/Py3
Previously the py2 api for strcasecmp_m/strstr_m required strings/unicode
but couldn't actually handle unicode with anything other than the default
encoding (e.g. ascii). The c-api as been fixed and the encoding steps
(which were unnecessary and causing errors in PY3) have been removed
Noel Power [Tue, 7 Aug 2018 15:21:35 +0000 (16:21 +0100)]
python: py_strcasecmp_m & py_strstr_m don't handle unicode properly
py_strcasecmp_m & py_strstr_m use PyArg_ParseTuple() with 's' which
in Py2 tries to decode string with the default (e.g. ascii) encoding
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Noel Power [Sat, 4 Aug 2018 16:01:24 +0000 (17:01 +0100)]
python/samba/tests: PY2/PY3 port samba.tests.dcerpc.integer
Python3 no longer has a long type so the 'L' postfix is no
longer valid. Additionally in python2 an int that exceeds will
be transparently converted into a long when necessary
Noel Power [Thu, 27 Sep 2018 17:16:49 +0000 (18:16 +0100)]
python/samba/netcmd: PY3 only possible to decode bytes
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Noel Power [Sat, 4 Aug 2018 13:23:28 +0000 (14:23 +0100)]
PY3: Only decode when necessary
Noel Power [Wed, 10 Oct 2018 10:14:59 +0000 (11:14 +0100)]
s4/setup/tests: PY3 samba-tool needs to be called with correct python ver.
Ensure samba-tool python version defined by $PYTHON
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Noel Power [Wed, 10 Oct 2018 09:15:31 +0000 (10:15 +0100)]
s4/scripting: PY3 Ensure python scripts are run with correct python ver.
As part of port samba4.blackbox.samba3dump to python2/3
make sure test_samba3dump.sh runs samba3dump with $PYTHON which should
define the correct python version.
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Noel Power [Mon, 27 Aug 2018 16:06:37 +0000 (17:06 +0100)]
testprogs/blackbox: Use PYTHON env variable for calling python scripts
Ensure samba-tool is called with correct python that is
defined by $PYTHON
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Noel Power [Thu, 27 Sep 2018 15:08:34 +0000 (16:08 +0100)]
python/samba/tests: make sure samba-tool is called with ${PYTHON}
Ensure python scripts are called with the python version that
is defined by $PYTHON
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Noel Power [Thu, 27 Sep 2018 08:30:40 +0000 (09:30 +0100)]
selftest/target: Make sure samba-tool is called with ${PYTHON}
Ensure python scripts are called with the python version that
is defined by $PYTHON
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Noel Power [Sat, 4 Aug 2018 14:38:40 +0000 (15:38 +0100)]
PY3: In a pure PY3 build filter-subunit was getting called without 'python'
tests were getting called with "| ${src}/selftest/filter-subunit" which
resulted in filter-subunit getting execve'd without a calling python. This
resulted in /usr/bin/python (default python) getting called and subsequent
imports failing.
Noel Power [Mon, 15 Oct 2018 09:36:19 +0000 (10:36 +0100)]
s4/scripting: PY3 need to convert cmp funct to key func for sort.
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Noel Power [Mon, 15 Oct 2018 15:04:25 +0000 (16:04 +0100)]
python/samba/gp_parse: PY3 fdeploy_sids needs to use key method for sort
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Noel Power [Mon, 15 Oct 2018 15:23:07 +0000 (16:23 +0100)]
samba-tool: PY3 dict view doesn't have sort method,
Can't sort a dict view, create a list from view then use list.sort
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Noel Power [Wed, 26 Sep 2018 16:22:16 +0000 (17:22 +0100)]
python/samba/tests: PY3 iterable has no sort method
map in python3 returns an iterable, in python2 it returned
a list. Iterable has no sort method, use sort function instead or
construct a list from the iterable so you can use list.sort
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Noel Power [Thu, 27 Sep 2018 17:15:49 +0000 (18:15 +0100)]
s4/scripting/bin: PY3 Make sure print statements are enclosed by '()'
Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Andrew Bartlett [Tue, 23 Oct 2018 03:27:29 +0000 (16:27 +1300)]
selftest: Add expected-value testing for userParameters
This does not means that bugs like https://bugzilla.samba.org/show_bug.cgi?id=11881
are fixed, however we do not wish to cause further issues
without noticing it, eg during python3 fixes for dbcheck.
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
Martin Schwenke [Wed, 17 Oct 2018 06:19:06 +0000 (17:19 +1100)]
ctdb-daemon: Fix valgrind hit in event code
==25741== Syscall param write(buf) points to uninitialised byte(s)
==25741== at 0x4939291: write (write.c:27)
==25741== by 0x4868285: sys_write (sys_rw.c:68)
==25741== by 0x13915D: sock_queue_trigger (sock_io.c:316)
==25741== by 0x4DE6478: tevent_common_invoke_immediate_handler (in /usr/lib/x86_64-linux-gnu/libtevent.so.0.9.37)
==25741== by 0x4DE64A2: tevent_common_loop_immediate (in /usr/lib/x86_64-linux-gnu/libtevent.so.0.9.37)
==25741== by 0x4DEBE5A: ??? (in /usr/lib/x86_64-linux-gnu/libtevent.so.0.9.37)
==25741== by 0x4DEA2D6: ??? (in /usr/lib/x86_64-linux-gnu/libtevent.so.0.9.37)
==25741== by 0x4DE57E3: _tevent_loop_once (in /usr/lib/x86_64-linux-gnu/libtevent.so.0.9.37)
==25741== by 0x15D1BA: ctdb_event_script_args (eventscript.c:821)
==25741== by 0x13B437: ctdb_start_daemon (ctdb_daemon.c:1315)
==25741== by 0x110642: main (ctdbd.c:393)
==25741== Address 0x57888a4 is 100 bytes inside a block of size 144 alloc'd
==25741== at 0x48357BF: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==25741== by 0x4B9B7C0: talloc_named_const (in /usr/lib/x86_64-linux-gnu/libtalloc.so.2.1.14)
==25741== by 0x15CCC6: eventd_client_write (eventscript.c:430)
==25741== by 0x15CCC6: eventd_client_run (eventscript.c:556)
==25741== by 0x15CCC6: ctdb_event_script_run (eventscript.c:649)
==25741== by 0x15D198: ctdb_event_script_args (eventscript.c:812)
==25741== by 0x13B437: ctdb_start_daemon (ctdb_daemon.c:1315)
==25741== by 0x110642: main (ctdbd.c:393)
==25741==
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13659
Pair-programmed-with: Amitay Isaacs <amitay@gmail.com>
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Mon Oct 22 09:27:15 CEST 2018 on sn-devel-144
Amitay Isaacs [Wed, 10 Oct 2018 07:19:32 +0000 (18:19 +1100)]
ctdb-event: Check the return status of sock_daemon_set_startup_fd
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13659
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Amitay Isaacs [Wed, 10 Oct 2018 07:16:33 +0000 (18:16 +1100)]
ctdb-common: Set close-on-exec for startup fd
The startup_fd should not be propagated to the child processes created
from a daemon. It should only be used in the daemon code to return the
status of the startup. Another use of startup_fd is to notify the
parent if the daemon process has exited.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13659
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Martin Schwenke [Thu, 11 Oct 2018 00:26:06 +0000 (11:26 +1100)]
ctdb-daemon: Exit if eventd goes away
ctdbd enters a broken state if eventd goes away. A clean shutdown is
not possible because that involves running events. Restarting eventd
is possible but this might mask a serious problem and it is possible
that eventd might keep on disappearing. Just exit.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13659
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Martin Schwenke [Wed, 10 Oct 2018 02:35:00 +0000 (13:35 +1100)]
ctdb-daemon: Return early when refusing to run an event script
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13659
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Andreas Schneider [Wed, 26 Sep 2018 12:47:20 +0000 (14:47 +0200)]
s3:smbcontrol: Simplify the return code check
Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Sat Oct 20 02:17:56 CEST 2018 on sn-devel-144
Andreas Schneider [Wed, 26 Sep 2018 12:34:07 +0000 (14:34 +0200)]
s4:torture: Fix the scope of the req variable in drsuapi test
Found by covscan.
Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Andreas Schneider [Wed, 26 Sep 2018 12:30:32 +0000 (14:30 +0200)]
ndr: Init variables of GUID_from_data_blob()
Found by covscan.
Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Andreas Schneider [Wed, 26 Sep 2018 12:29:50 +0000 (14:29 +0200)]
s3:registry: Avoid a double-free in reg_perfcount
Found by covscan.
Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
David Disseldorp [Tue, 16 Oct 2018 17:06:48 +0000 (19:06 +0200)]
talloc: deprecate talloc_set_memlimit()
The memlimit functionality was never utilized by Samba. It adds unneeded
complexity, so flag it as deprecated.
Signed-off-by: David Disseldorp <ddiss@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Volker Lendecke [Thu, 18 Oct 2018 19:53:36 +0000 (21:53 +0200)]
lib: Remove gencache.h from proto.h
It's a pain to recompile the world if gencache.h changes
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
Autobuild-User(master): Andreas Schneider <asn@cryptomilk.org>
Autobuild-Date(master): Fri Oct 19 18:52:50 CEST 2018 on sn-devel-144
Philipp Gesang [Thu, 4 Oct 2018 07:25:14 +0000 (09:25 +0200)]
s3:secrets: clean up sid before storing
SIDs may contain non-zero memory beyond SubAuthorityCount:
{
key(15) = "SECRETS/SID/FOO"
data(68) = "\01\04\00\00\00\00\00\05\15\00\00\00}u@\8C\08\A3\06nx\95\16\FE\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00`F\92\B7\03\00\00\00\18e\92\B7\03\00\00\00@H\92\B7\00\00\00\00"
}
These parts are lost when converting to ``string format syntax``
so a roundtrip conversion does not result in the same binary
representation.
Ensure that these never reach the tdb by using an initialized
copy. This allows bitwise comparisons of secrets.tdb after
dumping SIDs as text and reading them back.
Signed-off-by: Philipp Gesang <philipp.gesang@intra2net.com>
Reviewed-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Fri Oct 19 13:59:04 CEST 2018 on sn-devel-144
Gary Lockyer [Mon, 15 Oct 2018 03:02:40 +0000 (16:02 +1300)]
dsdb encrypted_secrets: Allow "ldb:// and "mdb://" in file path
Correctly handle "ldb://" and "mdb://" schemes in the file path when
determining the path for the encrypted secrets key file.
When creating a new user and specifying the local file path of the
sam.ldb DB, it was possible to create an account that you could not
login with. The path for the key file was incorrectly calculated
for the "ldb://" and "mdb://" schemes, the scheme was not stripped from
the path and the subsequent open of the key file failed.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13653
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Fri Oct 19 09:34:46 CEST 2018 on sn-devel-144
Gary Lockyer [Mon, 15 Oct 2018 03:01:47 +0000 (16:01 +1300)]
dsdb encrypted_secrets tests: Allow "ldb://" in file path
When creating a new user and specifying the local file path of the
sam.ldb DB, it's possible to create an account that you can't actually
login with.
This commit contains tests to verify the bug.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13653
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Gary Lockyer [Tue, 16 Oct 2018 20:10:10 +0000 (09:10 +1300)]
python tests Blackbox: add random_password
Add the random_password method to the BlackboxTestCase class and remove
duplicated copies from other test cases. Also use SystemRandom so that
the generated passwords are more cryptographically sound.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Douglas Bagnall [Thu, 18 Oct 2018 21:21:21 +0000 (10:21 +1300)]
ldb_ldif: avoid strlen(NULL)
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Autobuild-User(master): Martin Schwenke <martins@samba.org>
Autobuild-Date(master): Fri Oct 19 03:43:58 CEST 2018 on sn-devel-144
Douglas Bagnall [Wed, 17 Oct 2018 03:28:25 +0000 (16:28 +1300)]
ldb_ldif: be less horribly efficient in debugging
perf said all the time was in strlen.
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Douglas Bagnall <dbagnall@samba.org>
Autobuild-Date(master): Thu Oct 18 13:17:30 CEST 2018 on sn-devel-144
Douglas Bagnall [Wed, 17 Oct 2018 04:21:09 +0000 (17:21 +1300)]
py3_tests/kcc : test_verify can hit KCCError as well as GraphError
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Douglas Bagnall [Tue, 16 Oct 2018 20:50:41 +0000 (09:50 +1300)]
py3/tests/kcc: turn error into failure for flapping.d/kcc
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Joe Guo [Thu, 4 Oct 2018 02:37:49 +0000 (15:37 +1300)]
selftest: add tests for samba-tool drs uptodateness
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13658
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Thu Oct 18 10:02:19 CEST 2018 on sn-devel-144
Joe Guo [Wed, 3 Oct 2018 22:28:44 +0000 (11:28 +1300)]
netcmd/drs: add cmd_drs_uptodateness with json support
Add cmd to print uptodateness summary with json support.
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13658
Joe Guo [Wed, 3 Oct 2018 22:24:33 +0000 (11:24 +1300)]
uptodateness: add get_utdv_summary function
Get utdv summary from distances matrix and support attr filters.
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13658
Joe Guo [Wed, 3 Oct 2018 11:42:08 +0000 (00:42 +1300)]
uptodateness: migrate get_kcc_and_dsas as a function
We need to reuse it in drs cmd.
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13658
Joe Guo [Wed, 3 Oct 2018 10:45:12 +0000 (23:45 +1300)]
uptodateness: extract get_utdv_max_distance
To avoid returning 2 values from get_utdv_distances.
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13658
Joe Guo [Wed, 3 Oct 2018 10:21:11 +0000 (23:21 +1300)]
uptodateness: extract function get_utdv_distances
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13658
Joe Guo [Wed, 3 Oct 2018 10:09:56 +0000 (23:09 +1300)]
uptodateness: extract function get_utdv_edges
Extract function to reuse later.
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13658
Joe Guo [Wed, 3 Oct 2018 09:49:46 +0000 (22:49 +1300)]
netcmd/visualize: rm unused code line
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13658
Joe Guo [Wed, 3 Oct 2018 09:39:04 +0000 (22:39 +1300)]
uptodateness: migrate more methods from visualize
Move methods from cmd_uptodateness to new module.
Will reuse in drs cmd later.
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13658
Joe Guo [Wed, 3 Oct 2018 09:21:54 +0000 (22:21 +1300)]
uptodateness: add new module and migrate functions from visualize
Both visualize and drs cmd will have uptodateness functions.
Create a new module to reuse code.
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13658
Tim Beale [Thu, 18 Oct 2018 00:07:20 +0000 (13:07 +1300)]
join: Sanity-check LDB connection before failed join cleanup
Joining a large DB can take so long that the LDAP connection times out.
The previous patch fixed the 'happy case' where the join succeeds.
However, if the commit or replication fails (throwing an exception),
then the cleanup code can also fail when it tries to delete objects from
the remote DC. This then gives you an error pointing to
cleanup_old_accounts() rather than what actually went wrong.
This patch adds a sanity-check that if the join fails, that the LDB
connection to the remote DC is still alive, before we start deleting
objects.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13612
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Andrew Bartlett [Thu, 18 Oct 2018 03:50:19 +0000 (16:50 +1300)]
join: Avoid searching for more than strictly required during sanity check
We check for the default base DN as this does require authentication, but
we do not need to search for more than just that (so use SCOPE_BASE) and
we need no attributes, so ask for none
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Tim Beale [Wed, 17 Oct 2018 01:41:12 +0000 (14:41 +1300)]
join: LDAP connection to remote DC can timeout in large join
When joining a very large domain (e.g. 100K users), the replication can
take so long that the LDAP connection to the remote DC times out.
This patch avoids the problem by adding in a sanity-check after the
replication finishes that the LDB connection is still alive. If not,
then we reconnect.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13612
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Volker Lendecke [Tue, 9 Oct 2018 19:41:52 +0000 (21:41 +0200)]
gencache: Remove a redundant check
tdb_storev itself is robust against overflow due to multiple buffers
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Wed Oct 17 22:22:51 CEST 2018 on sn-devel-144
Volker Lendecke [Tue, 9 Oct 2018 12:04:50 +0000 (14:04 +0200)]
gencache: Remove a redundant check
gencache_pull_timeout checks for NULL ptr already
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Volker Lendecke [Tue, 9 Oct 2018 11:58:43 +0000 (13:58 +0200)]
gencache: Make gencache_pull_timeout return a payload DATA_BLOB
Both relevant callers created one anyway.
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Volker Lendecke [Tue, 9 Oct 2018 11:51:46 +0000 (13:51 +0200)]
gencache: Make gencache_pull_timeout a bit more robust
The previous version assumed a well-formed "val", we just handed it to
strtol without properly checking that it contains the delimiter. So
strtol could well run off the end of "val" in case of data corruption.
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Volker Lendecke [Tue, 9 Oct 2018 11:17:53 +0000 (13:17 +0200)]
gencache: Call string_term_tdb_data() only once
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Volker Lendecke [Tue, 9 Oct 2018 11:15:22 +0000 (13:15 +0200)]
gencache: Swap tests: Do cheapest first
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Volker Lendecke [Mon, 8 Oct 2018 07:07:59 +0000 (09:07 +0200)]
gencache: Avoid counting characters manually
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Volker Lendecke [Sat, 13 Oct 2018 08:41:22 +0000 (10:41 +0200)]
auth3: Avoid an explicit ZERO_STRUCT
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Volker Lendecke [Sat, 13 Oct 2018 08:58:32 +0000 (10:58 +0200)]
netsamlogon_cache: Improve a DBG message
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Volker Lendecke [Sat, 13 Oct 2018 08:57:13 +0000 (10:57 +0200)]
netsamlogon_cache: Add some error checks
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Volker Lendecke [Sat, 13 Oct 2018 08:55:00 +0000 (10:55 +0200)]
netsamlogon_cache: Use "goto fail", save some lines
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Volker Lendecke [Sat, 13 Oct 2018 08:10:52 +0000 (10:10 +0200)]
netsamlogon_cache: Fix talloc_stackframe error return leaks
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Tim Beale [Fri, 12 Oct 2018 00:54:34 +0000 (13:54 +1300)]
drs_util: Improve memory usage when joining large DB
drs_Replicate.replicate() could consume a large amount of memory when
replicating a large DB. This is not a leak - the memory gets freed when
the function returns (i.e. once the partition is fully replicated).
However, while the partition is in the process of being replicated, it
accumulates memory for each replication chunk it receives. This can have
considerable overhead with 1000s of objects/links in the partition.
This was exhausting memory when joining a VM with 1Gb RAM to a DC with
25K users (average ~15 group memberships per user).
It seems that by storing a reference to something that's on the ctr's
talloc tree, it doesn't free up the memory for each ctr message (until
the function actually returns and req is destroyed).
With 10K users (and average 15 group memberships per user), .replicate()
consumed 211Mb of memory, according to talloc.report_full(). With this
patch, it goes down to just the current ctr message (1-2Mb).
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Wed Oct 17 08:56:42 CEST 2018 on sn-devel-144
Tim Beale [Fri, 12 Oct 2018 00:29:32 +0000 (13:29 +1300)]
libnet/drs: Update replication debug to report link progress
Update the replication debug (for joins/backups) so that it's easier to
see how far through syncing the links we are. E.g. with 150,000 links,
you just get screeds of debug like this, with no real idea how far
through the replication is.
Partition[DC=addom,DC=samba,DC=example,DC=com] objects[11816/11720]
linked_values[1500/150024]
Partition[DC=addom,DC=samba,DC=example,DC=com] objects[11816/11720]
linked_values[1500/150024]
Partition[DC=addom,DC=samba,DC=example,DC=com] objects[11816/11720]
linked_values[1500/150024]
This patch now applies to links the same debug logic we use for objects,
and changes it to look like:
Partition[DC=addom,DC=samba,DC=example,DC=com] objects[11816/11720]
linked_values[57024/150024]
Partition[DC=addom,DC=samba,DC=example,DC=com] objects[11816/11720]
linked_values[58524/150024]
Partition[DC=addom,DC=samba,DC=example,DC=com] objects[11816/11720]
linked_values[60024/150024]
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Aaron Haslett [Mon, 15 Oct 2018 03:52:40 +0000 (16:52 +1300)]
dns: dlz_bind9 reference count logging
dlz_bind9 has to count the number of times the plugin is 'created' by bind's
plugin manager so it doesn't repeat setup. Logging doesn't reflect this
reference counting logic properly and so messages like "samba_dlz: shutdown"
can, confusingly, come up when the database connection has not actually been
severed. This patch adds the necessary logging.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13655
Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Reviewed-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Volker Lendecke [Sat, 13 Oct 2018 11:41:59 +0000 (13:41 +0200)]
lib: Move the "expired" for gencache_parse calculation into gencache.c
Make it more robust
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Tue Oct 16 21:20:19 CEST 2018 on sn-devel-144
Volker Lendecke [Sat, 13 Oct 2018 10:01:41 +0000 (12:01 +0200)]
namemap_cache: Absorb the expired calculation into namemap_cache.c
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Volker Lendecke [Sat, 13 Oct 2018 09:39:03 +0000 (11:39 +0200)]
winbindd_cache: Fix timeout calculation for sid<->name cache
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Andreas Schneider [Wed, 10 Oct 2018 14:09:32 +0000 (16:09 +0200)]
s3:lib:popt: Use memset_s() to burn password string
Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
Autobuild-User(master): Andreas Schneider <asn@cryptomilk.org>
Autobuild-Date(master): Tue Oct 16 11:38:40 CEST 2018 on sn-devel-144
Andreas Schneider [Wed, 10 Oct 2018 14:05:46 +0000 (16:05 +0200)]
replace: Add memset_s() if not available
See https://en.cppreference.com/w/c/string/byte/memset
Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
Douglas Bagnall [Wed, 10 Oct 2018 22:59:52 +0000 (11:59 +1300)]
samba-tool drs showrepl: do not crash if no dnsHostName found
This should not happen, but it does sometimes in an autobuild
environment. Rather than reporting this by crashing, we report it by
showing there is no DNS name.
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Autobuild-User(master): Douglas Bagnall <dbagnall@samba.org>
Autobuild-Date(master): Fri Oct 12 15:27:07 CEST 2018 on sn-devel-144
Tim Beale [Thu, 11 Oct 2018 04:50:52 +0000 (17:50 +1300)]
dsdb: Add dsdb_request_has_control() helper function
Most of the DSDB modules only want to check the existence of a control,
rather than access the control itself. Adding a helper function allows
the code to ask more natural-sounding yes/no questions, and tidies up
an ugly-looking long-line in extended_dn_out.c.
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
Autobuild-User(master): Douglas Bagnall <dbagnall@samba.org>
Autobuild-Date(master): Fri Oct 12 07:23:26 CEST 2018 on sn-devel-144