lib:util: Add a test for byteorder.h

Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Fri May 17 20:44:36 UTC 2019 on sn-devel-184

lib:util: Move VWV macro to smb_constants.h

Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

lib:util: Remove PPC big endian asm optimized code in byteorder.h

Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

lib:util: Remove unused ALIGN marcos from byteorder.h

Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

tdb: Release tdb 1.4.1

* Do not propogate any errors from tdb_repack() to to tdb_transaction_commit()

  The repack may fail due to lock ordering or memory limits, but the original data has
  been written so the call must succeed.  (bug 13952)

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Fri May 17 08:21:52 UTC 2019 on sn-devel-184

dsdb: Add tests for large LDAP responses

This behaviour is Samba-specific, we have not traditionally cut of responses at 1000
or so as Windows does, and we need to change that behaviour carefully.

This triggers this bug in TDB:
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13952

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

tdb: Do not return errors from tdb_repack() in the tail of tdb_transaction_commit()

The call to tdb_repack() inside tdb_transaction_commit()
is an optimization, not part of the transaction itself,
so failing due to lock or other errors isn't a fatal error
that should cause the caller to think the transaction was
a failure by returning -1.

The tdb transaction itself has finished and been committed
onto stable storage via fsync and all locks released at the
point tdb_repack() is called.

tdb_repack() is only called here as it's a convenient point
to attempt to reduce tdb fragmentation without having to add
a timer call to repack in all users of tdb.

This causes lock ordering issues in Samba, showing up as:

ldb: ltdb: tdb(../private/sam.ldb.d/DC=SAMBA2008R2,DC=EXAMPLE,DC=COM.ldb): tdb_transaction_prepare_commit: failed to upgrade hash locks: Locking error

This is because Samba has multiple tdb databases open, and the lock order between them
is important.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13952

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

smbd: Do oplock break messages in ndr

The previous scheme was overloaded, a idl definition is easier to
print, and it clarifies what data is actually needed

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Thu May 16 23:48:18 UTC 2019 on sn-devel-184

smbd: Use send_break_message() in send_break_to_none()

Centralize the marshalling code

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

smbd: Remove a pointless "continue" statement

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

smbd: Remove an obsolete comment from share_conflict()

delay_for_batch_oplocks() is no more. Also, open_mode_check (which
calls into this routine) is called before delay_for_oplock.

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

s3: smbd: Don't log at WARNING level when exiting the server on error.

Signed-off-by: Richard Sharpe <realrichardsharpe@gmail.com>
Reviewed-by: Jeremy Allison <jra@samba.org>

lib:util: Fix tfork return value if sigprocmask fails

Returning a non-zero value from a function with bool as return value is
the same as returning true. Change the return value to false if
sigprocmask or pthread_sigmask fails to indicate failure.

Detected with the help of cppcheck.

Signed-off-by: Rikard Falkeborn <rikard.falkeborn@gmail.com>
Reviewed-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
Autobuild-User(master): Andreas Schneider <asn@cryptomilk.org>
Autobuild-Date(master): Thu May 16 19:08:29 UTC 2019 on sn-devel-184

s3:loadparm: Ensure to truncate FS Volume Label at multibyte boundary

For FS_VOLUME_INFO/FS_INFO operation, a maximum of 32 characters are
sent back. However, since Samba chops off any share name with >32
bytes at 32, it is possible that a multi-byte share name can get chopped
off between a full character. This causes the string decoding for unicode
failure which sends back NT_STATUS_ILLEGAL_CHARACTER (EILSEQ) to the client
applications.

On Windows, Notepad doesn't like it, and refuses to open a file in this
case and fails with the following error:

  Invalid character. For multibyte character sets, only the leading byte is
  included without the trailing byte. For Unicode character sets, include
  the characters 0xFFFF and 0xFFFE.

Proposed fix:
- Find the last starting point of a multibyte codepoint if the character
  at 32nd byte is a subsequent byte of a MB codepoint.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13947

Signed-off-by: Shyamsunder Rathi <shyam.rathi@nutanix.com>
Reviewed-by: Hemanth Thummala <hemanth.thummala@nutanix.com>
Reviewed-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>

lib/crypto: squash 'cast between incompatible function types' warning

To avoid warning above produced by using
-Wcast-function-type we;

      + ensure PyCFunctions of type METH_VARARGS do not declare
        unused and problematic kargs param.

Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andreas Schneider <asn@samba.org>

lib/ldb-samba: squash 'cast between incompatible function types' warning

To avoid warning above produced by using
-Wcast-function-type we;

  + ensure PyCFunctions of type METH_NOARGS defined dummy arg

Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andreas Schneider <asn@samba.org>

lib/ldb: Fix incorrect return type for (setter) func type

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13948

Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andreas Schneider <asn@samba.org>

lib/ldb: squash 'cast between incompatible function types' warning

To avoid warning above produced by using
-Wcast-function-type we;

  + ensure PyCFunctions of type METH_NOARGS defined dummy arg
  + ensure PyCFunctions of type METH_KEYWORDS use PY_DISCARD_FUNC_SIG
    macro

Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andreas Schneider <asn@samba.org>

lib/talloc: squash 'cast between incompatible function types' warning

To avoid warning above produced by using
-Wcast-function-type we;

  + ensure PyCFunctions of type METH_NOARGS defined dummy arg

Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andreas Schneider <asn@samba.org>

lib/tevent: squash 'cast between incompatible function types' warning

squash 'cast between incompatible function types' warning

To avoid warning above produced by using
-Wcast-function-type we;

  + ensure PyCFunctions of type METH_NOARGS defined dummy arg
  + ensure PyCFunctions of type METH_KEYWORDS use PY_DISCARD_FUNC_SIG
    macro

Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andreas Schneider <asn@samba.org>

lib/tdb: squash 'cast between incompatible function types' warning

squash 'cast between incompatible function types' warning

To avoid warning above produced by using
-Wcast-function-type we;

  + ensure PyCFunctions of type METH_NOARGS defined dummy arg
  + ensure PyCFunctions of type METH_KEYWORDS use PY_DISCARD_FUNC_SIG
    macro

Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andreas Schneider <asn@samba.org>

squash 'cast between incompatible function types' warning

To avoid warning above produced by using
-Wcast-function-type we;

  + ensure PyCFunctions of type METH_NOARGS defined dummy arg
  + ensure PyCFunctions of type METH_KEYWORDS use PY_DISCARD_FUNC_SIG
    macro

Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andreas Schneider <asn@samba.org>

s4: squash 'cast between incompatible function types' warning

To avoid warning above produced by using
 -Wcast-function-type we;

  + ensure PyCFunctions of type METH_NOARGS defined dummy arg
  + ensure PyCFunctions of type METH_KEYWORDS use PY_DISCARD_FUNC_SIG
    macro
  + ensure PyCFunctions of type METH_KEYWORDS really actually use the
    problematic kargs param, if not remove it

Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andreas Schneider <asn@samba.org>

s3: squash 'cast between incompatible function types' warning

Some functions (e.g. py_smb_savefile) have an extra unecessary
*kwargs param in their signatures, these definitions are
causing 'cast between incompatible function types' warnings when
compiled with -Wcast-function-type. Some other functions have
the *kwargs which causes "cast between incompatible function types'
warnings which need to be squashed with use of the
PY_DISCARD_FUNC_SIG macro.

Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andreas Schneider <asn@samba.org>

s4/librpc: squash 'cast between incompatible function types' warning

Where possible make PyCFunction definition signature match. Sometimes
this is not possible (e.g. when the c-function is associated with a
python method definition with 'METH_VARARGS|METH_KEYWORDS' in this
case we use the PY_DISCARD_FUNC_SIG macro.

Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andreas Schneider <asn@samba.org>

s3/ntvfs: squash 'cast between incompatible function types' warning

Fix various PyCFunction definitions to avoid
'cast between incompatible function types' warnings when compiled
with -Wcast-function-type

Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andreas Schneider <asn@samba.org>

s3/smbd: squash 'cast between incompatible function types' warning

Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andreas Schneider <asn@samba.org>

pidl: Call PY_DISCARD_FUNC_SIG in generated code to avoid ugly warning

Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andreas Schneider <asn@samba.org>

python: Create macro to hide ugly function signature cast

-Wcast-function-type triggers various warnings related to bad api
decisions in python source. To avoid these warnings we hide them
by casting to void first. Macro here is to hide the uglyness and
provide an easy place to either/or
  + fix better in a single place in the future
  + identify the places involved should python api change in a way
    this can be handled better

Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andreas Schneider <asn@samba.org>

smbd: Slightly simplify delay_for_oplock()

We don't have to look at the leases.tdb record if it's our own lease.

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
Autobuild-User(master): Andreas Schneider <asn@cryptomilk.org>
Autobuild-Date(master): Thu May 16 07:59:52 UTC 2019 on sn-devel-184

smbd: Fix a typo

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>

smbd: Use a direct struct assignment in reply_setatr()

Saves .text bytes with -O3

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>

samba_autoconf: fix undefined behaviour sanitizer compile flags

Added the -g option, -fsaniize=null and -fsanitize=alignment, removed
the -fno-omit-frame-pointer option.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andreas Schneider <asn@samba.org>

s3: net: Test of fuzzer problems with net rpc registry import.

Found by Michael Hanselmann using fuzzing tools

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13842

Signed-off-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Wed May 15 23:08:58 UTC 2019 on sn-devel-184

s3: net: Rewrite of reg_parse_fd() to harden against buffer overwrites.

Remove unused handle_iconv_errno(). Fix leaks of iconv handles.

Found by Michael Hanselmann using fuzzing tools

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13842

Signed-off-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

s3: net: Harden srprs_str() against memcmp overread.

Found by Michael Hanselmann using fuzzing tools

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13842

Signed-off-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

s3: net: Harden act_val_hex() act_val_sz() against errors.

Found by Michael Hanselmann using fuzzing tools

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13842

Signed-off-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

s3: net: Harden guess_charset() against overflow errors.

Found by Michael Hanselmann using fuzzing tools

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13842

Signed-off-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

s4 dsdb/repl_meta_data: fix use after free in dsdb_audit_add_ldb_value

Fix use after free detected by AddressSanitizer

AddressSanitizer: heap-use-after-free on address 0x61400026a4a0
                  at pc 0x7fd555c52f12 bp 0x7ffed7231180 sp 0x7ffed7231170
                  READ of size 1 at 0x61400026a4a0 thread T0
    #0 0x7fd555c52f11 in ldb_should_b64_encode
       ../../lib/ldb/common/ldb_ldif.c:197
    #1 0x7fd539dc9417 in dsdb_audit_add_ldb_value
       ../../source4/dsdb/samdb/ldb_modules/audit_util.c:491
    #2 0x7fd539dc9417 in dsdb_audit_attributes_json
       ../../source4/dsdb/samdb/ldb_modules/audit_util.c:651
    #3 0x7fd539dc6a7e in operation_json
       ../../source4/dsdb/samdb/ldb_modules/audit_log.c:305

The problem is that at the successful end of these functions
el->values is overwritten with new_values.  However get_parsed_dns()
points p->v at the supplied el and it effectively gets used
as a working area by replmd_build_la_val().  So we must duplicate it
because our caller only called ldb_msg_copy_shallow().

The reason this matters is that the audit_log module is
above repl_meta_data in the stack, and tries to log the
ldb_message it saw after the reply (to include the error code).
If that ldb_message is changed it is not only misleading,
it can point to memory that has since gone away.

In this case the memory for the full extended DN in the
member attribute ended up on 'ac', a context lost by
the time repl_meta_data has finished processing.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13941

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Wed May 15 05:35:47 UTC 2019 on sn-devel-184

s4 dsdb/repl_meta_data: allocate new extended DNs during ADD on a better context

Lower down in this function new_values is assigned over el->values and is
filled in with the values of all the parsed DNs.  Therefore it is the natural
talloc parent.

This will allow el->values to be allocated on tmp_ctx in the next commit for
a working area during the function call.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

ldap: Add a database open after fork to speed-up prefork binds

Removing this (system) database open from the later auth stack results
in a 15-30% improvement in different types of binds and on different
hardware (presumably better with slower disk speeds).

Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

util/charset/torture: ensure each cp850 high bytes is 3 utf8 bytes

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

util/charset/convert: do not pretend to realloc

It seems very likely that our clever attempts to dynamically realloc
the output buffer were never triggered. Two lines of reasoning lead to
this conclusion:

1. We allocate 3 * srclen to start with, but no conversion we use will
   more than that. To be precise, from 8-bit charsets we will only deal
   with codepoints in the Unicode basic multilingual plane (up to 0xFFFF).

   These can all be expressed as 3 or fewer utf-8 bytes. In UTF16 they
   are naturally 2 bytes, while in the DOS codes they are 1 byte.

   We have checked the code tables, and can not find a plausible
   (e.g. not EBCDIC) DOS code page or unix charset that is outside
   this range.  Clients cannot chose the code page, the only code
   pages we will use come from 'unix charset' and 'dos charset'
   smb.conf parameters.

   Therefore the worst that can possibly happen is we expand 1 byte into 3
   (specifically, when converting some e.g. CP850 codepoints to UTF-8).

2. If the reallocation was ever used, the results would have been
   catastrophically wrong, as the input pointer was not reset.

Therefore we skip the complication of the goto loop and let E2BIG be
just another impossible error to report.

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

util/charset/convert: when retrying, retry from the start

iconv() advances the inbuf pointer; if we decide to realloc and re-iconv,
we need to reset inbuf to the source string

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

util/charset/convert: do not overflow dest len in corner case

Now, if destlen were SIZE_MAX - 1, destlen * 2 would wrap to SIZE_MAX - 3,
which makes (destlen * 2 + 2) == SIZE_MAX - 1, the same number again.
So we need the <= comparison in this case.

As things stand, it is not actually possible for destlen to be
SIZE_MAX (because it is always an even number after the first round,
and the first round is constrained to be < SIZE_MAX / 2, but *if*
destlen was SIZE_MAX, destlen * 2 + 2 would be 0, so that case is OK.
Similarly the SIZE_MAX - 2 and smaller cases were covered by the
original formula.

We add the comment for people who are wondering WTF is going on with
all this destlen manipulation.

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

util/charset/convert: do not overflow dest len

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

util/charset/convert_string: always set length

In failure cases the destination string pointer is set to NULL, but
the size is not changed. Some callers have not been checking the
return value and passing the destination pointer and uninitialised
length onto other functions. We can curse and blame those callers, but
let's also keep them safe.

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

test_regfio.c: include stdint.h before cmoka.h

This fix the following build failure on mips64el:
/usr/lfs/v0/rc-buildroot-test/scripts/instance-2/output/host/mips64el-buildroot-linux-uclibc/sysroot/usr/include/stdint.h:122:27: error: conflicting types for 'uintptr_t'
 typedef unsigned long int uintptr_t;
                           ^
In file included from ../source3/registry/tests/test_regfio.c:23:0:
../third_party/cmocka/cmocka.h:126:28: note: previous declaration of 'uintptr_t' was here
       typedef unsigned int uintptr_t;

Fixes:
 - http://autobuild.buildroot.org/results/bae0508e84c905dc23ad7cf1153cd1e9d8e4d734

Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

ldb: Release ldb 2.0.1

* Rework pack routines to remove unused features:
  - ldb_unpack_data_only_attr_list_flags() is removed
  - LDB_UNPACK_DATA_FLAG_NO_DATA_ALLOC is now always implied

  This improves the unpack performace significantly.

* Improve search performance via new internal flag
  LDB_UNPACK_DATA_FLAG_READ_LOCKED which removes a memdup

* Improve search performance during full scan by ignoring
  index records early.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>

ldb: remove unpack only attr list functionality

Unpack functions currently take an attribute list to restrict the set of
attributes to be returned in the constructed message. This
functionality is never used and complicates implementation of
forthcoming new pack format. This patch removes that functionality.
Using the unpack function then filtering the result turns
out not to be any slower.

NOTE: Configure with --abi-check-disable to build this commit. This
patch is part of a set of LDB ABI changes, and the version update is
done on the last commit.

Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>

ldb: move ldb_kv's filter into pack code

This patch moves ldb_kv's filter code into the pack code to replace
'only attr list' functionality which will be removed in forthcoming
commit. Unpacking data then filtering the result is not any slower
than the removed 'only attr list' approach.
'only attr list' test repurposed to test unpack -> filter flow.

NOTE: Configure with --abi-check-disable to build this commit. This
patch is part of a set of LDB ABI changes, and the version update is
done on the last commit.

Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>

ldb: removing msg and dn copying from filter attrs

Optimising filter_attrs by removing msg and dn allocation/copying. The
caller can construct the msg and possibly steal the dn.
Also giving the function an ldb for future use.

NOTE: Configure with --abi-check-disable to build this commit. This
patch is part of a set of LDB ABI changes, and the version update is
done on the last commit.

Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>

ldb: perf test for pack format

Performance test for pack function and unpack function run with flag
LDB_UNPACK_DATA_FLAG_NO_DATA_ALLOC. Run this test before and after
forthcoming pack format changes to test throughput improvement. On my
machine, the unpack improvement is around 50%.
The test doesn't really belong in LDB torture but it's the only place
where all the required functions are available.

Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>

ldb: removing alloc from unpack_data

Making unpack flag LDB_UNPACK_DATA_FLAG_NO_DATA_ALLOC required
behaviour, since allocating data during unpack is slow and unnecessary
in all current usages. In any future unpack usage, if editing of
returned memory is required, some function that duplicates the message
should be used, such as one of the filter_attrs functions, or msg_copy.

Signed-off-by: Aaron Haslett <aaronhaslett@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>

ldb_kv: Use ldb_msg_add_steal_value() in msg_add_distinguished_name()

This more modern routine allocates a nice talloc tree.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>

ldb_kv: Remove incorrect reference to LDB_UNPACK_DATA_FLAG_NO_DN

The DN is now exploded so as to improve other aspects of the search handling.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>

s3:libsmb: Add missing OOM check in fill_quota_buffer()

Found by covscan (csbuild).

Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Noel Power <nopower@suse.com>
Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Wed May 15 00:45:58 UTC 2019 on sn-devel-184

docs: add vfs_ceph_snapshots manpage

Signed-off-by: David Disseldorp <ddiss@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Autobuild-User(master): David Disseldorp <ddiss@samba.org>
Autobuild-Date(master): Tue May 14 23:31:27 UTC 2019 on sn-devel-184

vfs: add ceph_snapshots module

vfs_ceph_snapshots is a module for accessing CephFS snapshots as
Previous Versions. The module is separate from vfs_ceph, so that it can
also be used atop a CephFS kernel backed share with vfs_default.

Signed-off-by: David Disseldorp <ddiss@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

vfs_ceph: drop fdopendir handler

libcephfs doesn't currently offer an fdopendir equivalent, so the
existing implementation peeks at fsp->fsp_name->base_name, which can
break if vfs_ceph is used under a separate path-munging VFS module.

Return ENOSYS instead and rely on existing OpenDir_fsp() fallback.

Signed-off-by: David Disseldorp <ddiss@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

CVE-2018-16860 Heimdal KDC: Reject PA-S4U2Self with unkeyed checksum

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13685

Signed-off-by: Isaac Boukris <iboukris@gmail.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Karolin Seeger <kseeger@samba.org>
Autobuild-Date(master): Tue May 14 11:45:13 UTC 2019 on sn-devel-184

CVE-2018-16860 selftest: Add test for S4U2Self with unkeyed checksum

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13685

Signed-off-by: Isaac Boukris <iboukris@gmail.com>
Signed-off-by: Andrew Bartlett <abartlet@samba.org>

ctdb-common: Fix memory leak in run_proc

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13943

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Tue May 14 08:59:03 UTC 2019 on sn-devel-184

ctdb-common: Fix memory leak

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13943

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-recoverd: Fix memory leak

state is always freed before exiting this function, so allocate fde
off it instead of long-lived ctdb context.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13943

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-tests: Fix logic error in simple ctdb reloadips test

There is a chance that restoring IP addresses to the test node will
result in different IP addresses being assigned to that node.
Removing a single IP address may then fail (or be a no-op) if it is
done after the restore.

So, swap the single IP address removal to happen first, then restore,
then remove all IP addresses.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13924

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-tests: Make ctdb reloadips tests more reliable

ctdb reloadips will fail if it can't disable takover runs.  The most
likely reason for this is that there is already a takeover run in
progress.  We can't predict when this will happen, so retry if this
occurs.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13924

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-tests: Capture output in $out on failure as well

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13924

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

selftest: enable undefined behaviour sanitizer

Add a --undefined-sanitizer option to configure, this causes the tests
to be run with the undefined behaviout sanitizer enabled.

Errors can be suppressed by adding entries to selftest/ubsan.supp

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Tue May 14 07:20:28 UTC 2019 on sn-devel-184

s4 dsdb: fix use after free in samldb_rename_search_base_callback

Fix use after free detected by AddressSanitizer

AddressSanitizer: heap-use-after-free on address 0x60f0002b2738
                  at pc 0x7f89b1a213b5 bp 0x7ffce9528810 sp 0x7ffce9528800
                  READ of size 8 at 0x60f0002b2738 thread T0
    #0 0x7f89b1a213b4 in samldb_rename_search_base_callback
        ../../source4/dsdb/samdb/ldb_modules/samldb.c:4203
    #1 0x7f89d3a0db4a in ldb_module_send_entry
        ../../lib/ldb/common/ldb_modules.c:793
    #2 0x7f89b6f27356 in es_callback
        ../../source4/dsdb/samdb/ldb_modules/encrypted_secrets.c:1418

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13942

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

ctdb-tests: Remove old socket wrapper state directory during setup

Otherwise, when looping tests for a long time, nodes are unable to
connect to each other.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13924

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Mon May 13 08:42:44 UTC 2019 on sn-devel-184

ctdb-tests: Actually restart if cluster doesn't become healthy

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13924

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-tests: Add dump-logs command for local daemons

Dump a single merged log to stdout.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-tests: Add reqid wrapping test

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13930

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-daemon: Never use 0 as a client ID

ctdb_control_db_attach() and ctdb_control_db_detach() assume that any
control with client ID 0 comes from another daemon and treat it
specially.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13930

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

vfs_ceph: fix cephwrap_flistxattr() debug message

The @list buffer may be uninitialised prior to ceph_flistxattr()
invocation, so only log the address.

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13940

Signed-off-by: David Disseldorp <ddiss@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
Autobuild-User(master): David Disseldorp <ddiss@samba.org>
Autobuild-Date(master): Fri May 10 12:59:05 UTC 2019 on sn-devel-184

s3 rpc_client: Fix Asan stack use after scope

Fixes Asan error:

==1924==ERROR: AddressSanitizer: stack-use-after-scope on address
    0x7ffe63f873d0 at pc 0x7fb99dae1733 bp 0x7ffe63f86a00 sp 0x7ffe63f861a8
READ of size 24 at 0x7ffe63f873d0 thread T0
    #0 0x7fb99dae1732  (/usr/lib/x86_64-linux-gnu/libasan.so.4+0x79732)
    #1 0x7fb99cfe5549 in memcpy
        /usr/include/x86_64-linux-gnu/bits/string_fortified.h:34
    #2 0x7fb99cfe5549 in ndr_push_bytes
        ../../librpc/ndr/ndr_basic.c:729
    #3 0x7fb99cfe5646 in ndr_push_array_uint8
        ../../librpc/ndr/ndr_basic.c:754
    #4 0x7fb99a69dd1b in ndr_push_netr_ChallengeResponse
        librpc/gen_ndr/ndr_netlogon.c:462
    #5 0x7fb99a6c5fab in ndr_push_netr_NetworkInfo
        librpc/gen_ndr/ndr_netlogon.c:556
    #6 0x7fb99a6c749d in ndr_push_netr_LogonLevel
         librpc/gen_ndr/ndr_netlogon.c:783
    #7 0x7fb99a7222de in ndr_push_netr_LogonSamLogonEx
         librpc/gen_ndr/ndr_netlogon.c:16547
    #8 0x7fb99c982c97 in dcerpc_binding_handle_call_send
         ../../librpc/rpc/binding_handle.c:416

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13936

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Fri May 10 10:02:21 UTC 2019 on sn-devel-184

script/.gitlab-ci.yml: do not run o3 jobs if coverage enabled

Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

.gitlab-ci.yml: add pages job to generate coverage report and publish to gitlab pages

- add new stage "report"
- add pages job in "report" stage to collect .info files, generate html report and publish to gitlab pages.
- only trigger when SAMBA_CI_AUTOBUILD_ENABLE_COVERAGE == '--enable-coverage'

Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

.gitlab-ci.yml: add var SAMBA_CI_AUTOBUILD_ENABLE_COVERAGE

keep this var empty by default, since gcov will slow down the build process.
set to `--enable-coverage` to enable coverage on demand.

Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

script/autobuild.py: define LCOV_CMD and run it after each make test

Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

script/autobuild.py: add ENABLE_COVERAGE placeholder in configure cmd option

The value will be "--enable-coverage" or "".

Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

script/autobuild.py: replace more placeholders in cmds for coverage

Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

script/autobuild.py: add --enable-coverage option

Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

script/autobuild.py: mv optionparse to top

so the rest of the code can use the option values directly.

Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

script/autobuild.py: mv find_git_root and gitroot to top

so the reset of the code can use gitroot directly.

Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

script/autobuild.py: rename sdir to test_source_dir

Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

wscript: mv --enable-coverage option to global

We want to enable gcov for all tasks, move it to global wscript.

Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

selftest/target/Samba4.pm: increase max_wait from 60s to 120s to avoid timeout failure in samba-ad-dc-backup

Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

s4 librpc rpc pyrpc: Ensure tevent_context deleted last

Ensure that the tevent_context is deleted after the connection, to
prevent a use after free.

Note: Py_DECREF calls dcerpc_interface_dealloc so the
TALLOC_FREE(ret->mem_ctx) calls in the error paths of
py_dcerpc_interface_init_helper needed removal.

Bug: https://bugzilla.samba.org/show_bug.cgi?id=13932

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

samba_dnsupdate: flush dns update cache file after write

While running samba python tests with `coverage.py`, `make test` failed
to start ad_dc env since dns_update_cache stays empty.
Flush the file after write fix the issue.

Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>

s3/rpcclient: Fix bad (and illegal) func cast

../../source3/rpcclient/cmd_samr.c: In function ‘cmd_samr_get_usrdom_pwinfo’:
../../librpc/ndr/libndr.h:219:76: error: cast between incompatible function types from ‘void (*)(struct ndr_print *, const char *, uint32_t)’ {aka ‘void (*)(struct ndr_print *, const char *, unsigned int)’} to ‘void (*)(struct ndr_print *, const char *, const void *)’ [-Werror=cast-function-type]
 #define NDR_PRINT_STRUCT_STRING(ctx, type, p) ndr_print_struct_string(ctx, (ndr_print_fn_t)ndr_print_ ##type, #p, p)
                                                                            ^
../../source3/rpcclient/cmd_samr.c:2646:4: note: in expansion of macro ‘NDR_PRINT_STRUCT_STRING’
    NDR_PRINT_STRUCT_STRING(mem_ctx,

Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Fri May 10 02:26:02 UTC 2019 on sn-devel-184

s3/registry: fix various 'cast between incompatible function' warnings

[3395/3524] Compiling source3/registry/reg_import.c
../../source3/registry/reg_import.c: In function ‘reg_import_adapter’:
../../source3/registry/reg_import.c:237:16: error: cast between incompatible function types from ‘int (*)(void *)’ to ‘WERROR (*)(void *, void *, const char *, void **)’ {aka ‘struct <anonymous> (*)(void *, void *, const char *, void **)’} [-Werror=cast-function-type]
   cb.openkey = (reg_import_callback_openkey_t)&nop;
                ^
../../source3/registry/reg_import.c:240:17: error: cast between incompatible function types from ‘int (*)(void *)’ to ‘WERROR (*)(void *, void *)’ {aka ‘struct <anonymous> (*)(void *, void *)’} [-Werror=cast-function-type]
   cb.closekey = (reg_import_callback_closekey_t)&nop;
                 ^
../../source3/registry/reg_import.c:243:18: error: cast between incompatible function types from ‘int (*)(void *)’ to ‘WERROR (*)(void *, void *, const char *, void **, _Bool *)’ {aka ‘struct <anonymous> (*)(void *, void *, const char *, void **, _Bool *)’} [-Werror=cast-function-type]
   cb.createkey = (reg_import_callback_createkey_t)&nop;
                  ^
../../source3/registry/reg_import.c:246:18: error: cast between incompatible function types from ‘int (*)(void *)’ to ‘WERROR (*)(void *, void *, const char *)’ {aka ‘struct <anonymous> (*)(void *, void *, const char *)’} [-Werror=cast-function-type]
   cb.deletekey = (reg_import_callback_deletekey_t)&nop;
                  ^
../../source3/registry/reg_import.c:249:18: error: cast between incompatible function types from ‘int (*)(void *)’ to ‘WERROR (*)(void *, void *, const char *)’ {aka ‘struct <anonymous> (*)(void *, void *, const char *)’} [-Werror=cast-function-type]
   cb.deleteval = (reg_import_callback_deleteval_t)&nop;

Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

s3/registry: Fix incompatible func casts

[3425/3524] Compiling source3/registry/reg_parse.c
../../source3/registry/reg_parse.c: In function ‘reg_parse_new’:
../../source3/registry/reg_parse.c:223:12: error: cast between incompatible function types from ‘int (*)(void *)’ to ‘int (*)(void *, const char **, size_t,  _Bool)’ {aka ‘int (*)(void *, const char **, long unsigned int,  _Bool)’} [-Werror=cast-function-type]
   cb.key = (reg_parse_callback_key_t)&nop;
            ^
../../source3/registry/reg_parse.c:226:12: error: cast between incompatible function types from ‘int (*)(void *)’ to ‘int (*)(void *, const char *, uint32_t,  const uint8_t *, uint32_t)’ {aka ‘int (*)(void *, const char *, unsigned int,  const unsigned char *, unsigned int)’} [-Werror=cast-function-type]
   cb.val = (reg_parse_callback_val_t)&nop;
            ^
../../source3/registry/reg_parse.c:229:16: error: cast between incompatible function types from ‘int (*)(void *)’ to ‘int (*)(void *, const char *)’ [-Werror=cast-function-type]
   cb.val_del = (reg_parse_callback_val_del_t)&nop;
                ^
../../source3/registry/reg_parse.c:232:16: error: cast between incompatible function types from ‘int (*)(void *)’ to ‘int (*)(void *, const char *)’ [-Werror=cast-function-type]
   cb.comment = (reg_parse_callback_comment_t)&nop;

Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

s3/registry: Fix func cast error (diff in params size_t vs uint32)

reg_parse_callback_key_t alread has size_t for number of elemens in key
but
reg_parse_callback_val_t uses uint32_t for length of value data.

Many internal function seem to take size_t as data len (see reg_format_value
etc.) It seems changing the signature of the callback aligns it more closely
with the existing api.

../../source3/registry/reg_format.c:438:10: error: cast between incompatible function types from ‘int (*)(reg_format *, const char *, uint32_t,  const uint8_t *, size_t)’ {aka ‘int (*)(struct reg_format *, const char *, unsigned int,  const unsigned char *, long unsigned int)’} to ‘int (*)(void *, const char *, uint32_t,  const uint8_t *, uint32_t)’ {aka ‘int (*)(void *, const char *, unsigned int,  const unsigned char *, unsigned int)’} [-Werror=cast-function-type]
   .val = (reg_parse_callback_val_t)&reg_format_value,

Signed-off-by: Noel Power <noel.power@suse.com>
Reviewed-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

s4/lib/policy/gp_ldap: use ldb API to find messages

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

dsdb mod/linked_attributes: fix_link_slow(): clarify a comment.

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

dsdb/mod/extended_dn_out: use faster removal filters

When filtering out multiple elements, we end up memmove()ing the same
elements many times over. It is simpler to not do that by keeping track
of how many elements we are keeping.

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

ldb_msg: remove_element() checks element array bounds

Previously we half-heartedly checked one end.

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

ldb_kv: use ldb_msg_remove_element()

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>