ctdb-tests: Replace md5sum with posix cksum

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13520

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-tests: Use portable wc -c instead of stat -c "%s"

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13520

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-scripts: date "+%N" is non-portable

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13520

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-tests: Simplify pattern matching for ctime output

On freebsd, sed does not accept multiple pattern strings.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13520

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-tests: Do not try to match pstree output in eventd tests

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13520

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-common: Add fd argument to ctdb_connection_list_read()

This makes testing easier.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13520

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-protocol: Avoid fgets in ctdb_connection_list_read

C library buffering API can behave in unexpected fashion if underlying
fd for stdin, stdout or stderr is closed and re-opened.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13520

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-common: Add line based I/O

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13520

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-tests: Porting tests should ignore unsupported features

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13520

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-tests: Use sigcode to match signals

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13520

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-tests: Add signal code matching utility

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13520

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-tests: Add ps output filter for freebsd

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13520

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-client: Switch to ETIMEDOUT instead of ETIME

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13520

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-daemon: Switch to using ETIMEDOUT instead of ETIME

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13520

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-event: Switch to ETIMEDOUT instead of ETIME

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13520

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-common: Switch to ETIMEDOUT from ETIME

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13520

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-tests: Add required_error() to match on error codes

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13520

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-tests: Add errno matching utility

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13520

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-tests: Switch some test stubs to use /bin/sh

They don't use any bash features.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13520

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-tests: Improve portability by not using mktemp --tmpdir option

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13520

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-tests: Avoid use of non-portable getopt in stubs

getopt is being used with non-portable options.  In most cases use
simpler, POSIX-compliant getopts instead.

In the case of the ctdb test stub command, options can appear after
other arguments, so this requires an additional nested loop.

In the case of smnotify, there are no short options, so handle the
long options manually.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13520

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-tests: Avoid use of non-portable getopt in run_tests.sh

getopt is being used with non-portable options.  Use simpler,
POSIX-compliant getopts instead.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13520

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-tools: Avoid use of non-portable getopt in onnode

getopt is being used with non-portable options.  Use simpler,
POSIX-compliant getopts instead.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13520

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-tests: Improve portability by not using /bin/bash directly

FreeBSD and others do not have /bin/bash, so use "/usr/bin/env bash"
for better flexibility.

There are still many integration tests that use /bin/bash but this at
least lets FreeBSD start running tests.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13520

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-tools: Improve portability by not using /bin/bash directly

FreeBSD and others do not have /bin/bash, so use "/usr/bin/env bash"
for better flexibility.

There are still many integration tests that use /bin/bash but this at
least lets FreeBSD start running tests.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13520

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

autobuild: Test with and without building bundled popt

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Sat Jul 28 03:39:48 CEST 2018 on sn-devel-144

popt: Add check for iconv library

On glibc based systems, there is no separate iconv library.  Adding a
dependency without checking for the library breaks build on glibc based
systems.

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

popt: Check for headers only if building in-tree version

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

selftest: run smbtorture3 SMB2-BASIC tests against additional shares

This runs the smbtorture3 SMB2-BASIC and smb2.compound_find tests against shares
with "smbd:async dosmode" enabled.

On the vfs_aio_pthread_async_dosmode_force_sync* shares we
force a sync threadpool which ensures we test behaviour on systems that
don't support unshare(CLONE_FS) and also don't support
per-thread-credentials. This simulates the code path of non linux
systems. And makes sure that we don't regress there.

We also test with xattr_tdb and without.

Pair-Programmed-With: Stefan Metzmacher <metze@samba.org>

Signed-off-by: Ralph Boehme <slow@samba.org>
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Autobuild-User(master): Ralph Böhme <slow@samba.org>
Autobuild-Date(master): Fri Jul 27 16:04:02 CEST 2018 on sn-devel-144

selftest: set "smbd:async dosmode = no" in the vfs_aio_pthread share

This just explicitly sets the current default, to ensure the tests that
use this share always use the same "smbd:async dosmode" setting even if
the default changes in the future.

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

s4: torture: test closing dir handle with in-flight find

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

smbd: use async dos_mode_at_send in smbd_smb2_query_directory_send()

Finally: use the new dos_mode_at_send() in the directory enumeration
loop. This means that fetching the DOS attributes for directory entries
is done asynchronously with regard to the enumeration loop.

As the DOS attribute is typically read from an extended attribute in the
filesytem, this avoids sequentially blocking on IO. If the IO subsystem
is slow servicing these request, enabling async processing can result in
performance improvements.

A parametric option

  smbd:async dosmode = true | false (default: false)

can be used to enable the new async processing.

Simulating slow IO with usleep(5000) in the synchronous and asynchronous
versions of SMB_VFS_GET_DOS_ATTRIBUTES(), the results of enumerating a
directory with 10,000 files are:

    smbd:async dosmode = no:

        $ time bin/smbclient -U slow%x //localhost/test -c "ls dir\*" > /dev/null
        real    0m59.597s
        user    0m0.024s
        sys     0m0.012s

    smbd:async dosmode = yes:

        $ time bin/smbclient -U slow%x //localhost/test -c "ls dir\*" > /dev/null
        real    0m0.698s
        user    0m0.038s
        sys     0m0.025s

Performance gains in real world workloads depends on whether the actual
IO requests can be merged and parallelized by the kernel. Without such
wins at the IO layer, the async processing may even be slower then the
sync processing due to the additional overhead.

The following parameters can be used to adapt async processing behaviour
for specific workloads and systems:

        aio max threads = X (default: 100)
        smbd:max async dosmode = Y (default: "aio max threads" * 2)

By default we have at most twice the number of async requests in flight
as threads provided by the underlying threadpool. This ensures a worker
thread that finishes a job can directly pick up a new one without going
to sleep.

It may be advisable to reduce the number of threads to avoid scheduling
overhead while also increasing "smbd:max async dosmode".

Note that we disable async processing for certain VFS modules in the VFS
connect function to avoid the overhead of triggering the sync fallback
in dos_mode_at_send(). This is done for VFS modules that implement the
sync SMB_VFS_GET_DOS_ATTRIBUTES(), but not the async version (gpfs), and
for VFS modules that don't share a real filesystem where fchdir() can be
used (ceph, gluster). It is disabled for catia, because we realized that
the catia name translation macros used on
fsps (CATIA_FETCH_FSP_[PRE|POST]_NEXT) have a bug (#13547).

We use threadpool = smb_vfs_ev_glue_tp_chdir_safe() and then
pthreadpool_tevent_max_threads(threadpool) to get the number of maximum
worker threads which matches the pool used by the low level
SMB_VFS_GETXATTRAT_[SEND|RECV] implementation in vfs_default.

This is a terrible abstraction leak that should be removed in the future
by maybe making it possible to ask a VFS function which threadpool it
uses, internally suporting chaining so VFS function FOO that internally
uses BAR can forward the question to BAR.

On a hyphotetical system that had a getxattrat(dirfd, path, ...)
syscall and at the same time doesn't support per-thread current working
directories (eg FreeBSD doesn't have the latter) but has support for
per-thread-credentials, pthreadpool_tevent_max_threads() on the
tp_chdir_safe threadpool returns 1.

So when hooking the hyphotetical getxattrat() into the async
SMB_VFS_GETXATTRAT_[SEND|RECV] implementation in an VFS module, the
implementation could use the tp_path_safe threadpool, but the SMB2
layer would use the wrong threadpool in the call to
pthreadpool_tevent_max_threads(), resulting in no parallelism.

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

smbd: let smbd_dirptr_lanman2_entry return smb_fname

Note that smb_fname is relative to fsp, not conn!

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

smbd: deal with fsp->aio_requests in close_directory()

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

smbd: fix a long line in smb2_query_directory_next_entry()

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

smbd: factor out smb2_query_directory_next_entry() from smbd_smb2_query_directory_send()

This paves the way for adding async functions into the enumeration loop.

Best viewed with: git show -w

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

smbd: rework error exit in smbd_dirptr_lanman2_entry()

The next commit will add code that must be run if status is NT_STATUS_OK
or STATUS_MORE_ENTRIES.

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

smbd: pass get_dosmode to mode_fn in smbd_dirptr_get_entry()

This finally uses "get_dosmode" as passed in from the SMB2 layer, but
all callers still pass true, so no change in behaviour.

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

smbd: pass get_dosmode to smbd_dirptr_get_entry()

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

smbd: add "get_dosmode" argument to smbd_dirptr_lanman2_entry()

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

smbd: add dos_mode_at_send/recv()

Signed-off-by: Ralph Boehme <slow@samba.org>

smbd: factor out dosmode post processing

We apply some post processing to the dosmode returned from the VFS
function. Move this to a seperate function which will be reused in the
next commit in the async dos_mode_send/recv post processing.

Best viewed with: git show --histogram

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

vfs_default: implement SMB_VFS_GET_DOS_ATTRIBUTES_SEND/RECV

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

smbd: split out public parse_dos_attribute_blob() from get_ea_dos_attribute()

In preperation of adding an async version of get_ea_dos_attribute() that
will then call parse_dos_attribute_blob() too.

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

s3: vfs: add SMB_VFS_GET_DOS_ATTRIBUTES_SEND/RECV

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

vfs_xattr_tdb: implement SMB_VFS_GETXATTRAT_SEND/RECV

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

vfs_default: implement SMB_VFS_GETXATTRAT_SEND/RECV

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

s3: vfs: add SMB_VFS_GETXATTRAT_SEND/RECV

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

pthreadpool: we need to use pthreadpool_tevent_per_thread_cwd() on the callers pool

In pthreadpool_tevent_job_send() we remember if the job will be chdir
safe. It means we means we need to ask the callers pool when calling
pthreadpool_tevent_per_thread_cwd(), as the callers pool might
be a wrapper using pthreadpool_tevent_force_per_thread_cwd().

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>

ldb_mdb: #ifdef EBADE as it is not portable

E.g. FreeBSD 11.2 doesn't have it.

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>

ctdb-common: Drop unused function mkdir_p_or_die()

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Fri Jul 27 08:42:20 CEST 2018 on sn-devel-144

ctdb-common: Drop function parse_ip_mask() and supporting functions

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-daemon: Switch to using ctdb_sock_addr_mask_from_string()

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-tools: Switch to using ctdb_sock_addr_mask_from_string()

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-protocol: Add function ctdb_sock_addr_mask_from_string()

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-protocol: Fix compilation issue with strncpy()

When configured with --picky-developer and using -O3 with gcc 8.1:

../protocol/protocol_util.c: In function ‘ctdb_sock_addr_from_string’:
../protocol/protocol_util.c:282:2: error: ‘strncpy’ specified bound depends on the length of the source argument [-Werror=stringop-overflow=]
  strncpy(s, str, len+1);
  ^~~~~~~~~~~~~~~~~~~~~~
../protocol/protocol_util.c:277:8: note: length computed here
  len = strlen(str);
        ^~~~~~~~~~~

Use strlcpy() instead and check the result.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13545

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-common: Fix compilation issue with strncpy()

When configured with --picky-developer and using -O3 with gcc 8.1:

../common/system_socket.c: In function ‘parse_ip_mask’:
../common/system_socket.c:229:2: error: ‘strncpy’ specified bound depends on the length of the source argument [-Werror=stringop-overflow=]
  strncpy(s, str, len+1);
  ^~~~~~~~~~~~~~~~~~~~~~
../common/system_socket.c:223:8: note: length computed here
  len = strlen(str);
        ^~~~~~~~~~~

Use strlcpy() instead and check the result.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13545

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

smbd: Remove "share_mode_entry->lease"

smbstatus was the only user, and this could be solved by adapting
share_entry_forall.

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Fri Jul 27 01:42:31 CEST 2018 on sn-devel-144

smbstatus: Use share_mode_data->leases

This is the only user of share_mode_entry->lease

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

smbd: Pass "share_mode_data" to share_entry_forall callback

Quite a bit of the contents have been passed explicitly anyway.

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

popt: popt 1.16 needs -liconv

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

idmap: Make pointer initialization explicit

Took me a few seconds to find this NULL initializer

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

smbd: don't client->connections without checking client != NULL first in exit_server_common()

exit_server_common() can be called also in smbd processes without a
smbXsrv_client structure, e.g. the parent or some background tasks.

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Autobuild-User(master): Stefan Metzmacher <metze@samba.org>
Autobuild-Date(master): Thu Jul 26 01:29:38 CEST 2018 on sn-devel-144

examples/VFS/skel_transparent: make vfs_fn_pointers static

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
Autobuild-User(master): Stefan Metzmacher <metze@samba.org>
Autobuild-Date(master): Wed Jul 25 20:44:12 CEST 2018 on sn-devel-144

examples/VFS/skel_opaque: make vfs_fn_pointers static

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

examples/VFS/skel_opaque: fix a likely a copy/paste error

This line was probably copied over from skel_transparent.c, remove it.

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

s3:modules: add vfs_not_implemented module

This provides helper functions, which can be used by other modules,
if they don't implement a specific function.

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>

vfs_aio_pthread: use event context and threadpool from user_vfs_evg

Or the root glue in case we're already root.

Pair-Programmed-With: Stefan Metzmacher <metze@samba.org>

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Signed-off-by: Ralph Boehme <slow@samba.org>

s3: vfs: add user_vfs_evg to connection_struct

This will be used to in order to pass down the
impersonation magic from the SMB layer through
the SMB_VFS layer.

This includes the following options:

smbd:force sync user path safe threadpool
smbd:force sync user chdir safe threadpool
smbd:force sync root path safe threadpool
smbd:force sync root chdir safe threadpool

They can be used in order to test the non linux code
path on linux, once we get code that makes full use
of the new infrastructure.

Pair-Programmed-With: Stefan Metzmacher <metze@samba.org>

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Signed-off-by: Ralph Boehme <slow@samba.org>

s3: vfs: add smb_vfs_ev_glue

This adds VFS helper functions and that work on a struct smb_vfs_ev_glue
object which bundles two event contexts and a few threadpools.

This will be used to streamline the use of impersonating wrappers
in the SMB_VFS.

Notice the verbose comments in source3/smbd/vfs.c.

This will allow us to introduce path based async operations
to the SMB_VFS layer.

Pair-Programmed-With: Stefan Metzmacher <metze@samba.org>

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Signed-off-by: Ralph Boehme <slow@samba.org>

smbd: introduce sconn->sync_thread_pool

This just simulates a threadpool, but executes the
job functions inline (blocking) in the main thread.

This will be used to work arround some OS limitations,
e.g. if per thread credentials or per thread working directory
are not supported.

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>

smbd: rename sconn->pool to sconn->raw_thread_pool

This should in future not be used directly, we'll provide
wrapper pools, which will provide impersonation for
path based async calls.

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

smbd: add missing DO_PROFILE_INC(disconnect) to smbd_server_connection_terminate_ex()

For multi channel connections we should increment the disconnect count
also if we're not closing the last channel.

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>

smbd: disconnect/destroy all connections before calling smbXsrv_session_logoff_all()

This means the pending requests are destroyed before tevent_context
impersonation wrapper are destroyed.

Otherwise, with a pending struct tevent_req that is a child of
client->xconn like this in exit_server_common():

conn[ipv4:127.0.0.11:40745] reason[NT_STATUS_CONNECTION_RESET] at ../source3/smbd/smb2_server.c:4015
full talloc report on 'struct smbXsrv_connection' (total   6085 bytes in  43 blocks)
    struct smbd_smb2_request       contains    648 bytes in   1 blocks (ref 0) 0x55ed41634740
    struct smbd_smb2_request       contains   3438 bytes in  32 blocks (ref 0) 0x55ed416331e0
        struct tevent_req              contains   1824 bytes in  20 blocks (ref 0) 0x55ed41635860
            struct smb_filename            contains    206 bytes in   2 blocks (ref 0) 0x55ed41635560
                lease_v2_complex2.dat          contains     22 bytes in   1 blocks (ref 0) 0x55ed4161b950
            struct smbd_smb2_create_state  contains   1386 bytes in  16 blocks (ref 0) 0x55ed41635a10
                struct deferred_open_record    contains    804 bytes in  12 blocks (ref 0) 0x55ed41633090
                    struct defer_open_state        contains    764 bytes in  11 blocks (ref 0) 0x55ed41634f10
                        struct tevent_req              contains    748 bytes in  10 blocks (ref 0) 0x55ed41636390
                            struct tevent_timer            contains     96 bytes in   1 blocks (ref 0) 0x55ed41636ad0
                            struct dbwrap_watched_watch_state contains    420 bytes in   7 blocks (ref 0) 0x55ed41636540
                                struct tevent_req              contains    296 bytes in   5 blocks (ref 0) 0x55ed41636700
                                    struct messaging_filtered_read_state contains     64 bytes in   3 blocks (ref 0) 0x55ed416368b0
                                        struct messaging_dgm_fde       contains      8 bytes in   2 blocks (ref 0) 0x55ed41636950
                                            reference to: struct messaging_dgm_fde_ev

we crash when freeing the xconn as the the tevent_req child has a
cleanup function that tries to access a (wrapped) tevent context that
was already destroyed via smb1srv_tcon_disconnect_all() for SMB1 or
smbXsrv_session_logoff_all() for SMB2 a few lines above.

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>

smbd: only pass struct smbXsrv_client to smbXsrv_session_logoff_all()

That's all it needs.

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>

smbd: only pass struct smbXsrv_client to smb1srv_tcon_disconnect_all()

That's all it needs.

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>

pthreadpool: add a missing include

Reported-by: David Disseldorp <ddiss@samba.org>
Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: David Disseldorp <ddiss@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

lib audit_logging: add _WARN_UNUSED_RESULT_

Have the compiler issue a warning when the return code from the API is
ignored.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Gary Lockyer <gary@samba.org>
Autobuild-Date(master): Wed Jul 25 09:28:31 CEST 2018 on sn-devel-144

dsdb group_audit_test: Remove redundant mocking code

Remove a place holder test and unused mocking code.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

dsdb group auditing: remove HAVE_JANSSON from group_audit

This modules is ADDC only and JANSSON is required for the ADDC builds,
so the ifdef is no longer necessary.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

dsdb audit logging: remove HAVE_JANSSON from audit_log

This modules is ADDC only and JANSSON is required for the ADDC builds,
so the ifdef is no longer necessary.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

json: Add unit tests for error handling

Add cmocka unit tests to exercise the error handling in the JSON
routines.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

json: Modify API to use return codes

Modify the auditing JSON API to return a response code, as the consensus
was that the existing error handling was aesthetically displeasing.

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

docs/kerneloplocks: drop Irix references

Signed-off-by: David Disseldorp <ddiss@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Wed Jul 25 06:28:21 CEST 2018 on sn-devel-144

s3: smbd: SGI IRIX is officially dead. Remove the kernel oplock code for IRIX.

Signed-off-by: Jeremy Allison <jra@samba.org>
Reviewed-by: David Disseldorp <ddiss@samba.org>

s3: vfs: bump to version 40, Samba 4.10 will ship with that

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Autobuild-User(master): Stefan Metzmacher <metze@samba.org>
Autobuild-Date(master): Wed Jul 25 03:23:44 CEST 2018 on sn-devel-144

s3: vfs: bump to version 39, Samba 4.9 will ship with that

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

s3: vfs: add missing tevent_req_received() to SMB_VFS_FSYNC_RECV()

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

s3: vfs: add missing tevent_req_received() to SMB_VFS_PWRITE_RECV()

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

s3: vfs: add missing tevent_req_received() to SMB_VFS_PREAD_RECV()

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

vfs_default: fix async fsync idle/busy time profiling

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

s3: libsmb: use smb2cli_conn_max_trans_size() in cli_smb2_list()

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

s4: libcli/smb2: calculate correct credit charge for finds

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

s3: lib/xattr_tdb: fix listing xattrs

If there's no record in the xattr.tdb, dbwrap_fetch() will return
NT_STATUS_NOT_FOUND. That should not result in an error in callers of
xattr_tdb_load_attrs().

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

vfs_default: call smb_vfs_assert_all_fns()

This module needs to implement every call.

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>

examples/VFS/skel_transparent: call smb_vfs_assert_all_fns()

This template should always include all calls.

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>

examples/VFS/skel_transparent: add missing durable handle functions

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

examples/VFS/skel_transparent: add missing audit_file_fn

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

examples/VFS/skel_opaque: call smb_vfs_assert_all_fns()

This template should always implement all calls.

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>

examples/VFS/skel_opaque: add missing durable handle functions

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>