Amitay Isaacs [Fri, 27 Jul 2018 02:55:47 +0000 (12:55 +1000)]
popt: Check for headers only if building in-tree version
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Ralph Boehme [Thu, 26 Jul 2018 16:13:44 +0000 (18:13 +0200)]
selftest: run smbtorture3 SMB2-BASIC tests against additional shares
This runs the smbtorture3 SMB2-BASIC and smb2.compound_find tests against shares
with "smbd:async dosmode" enabled.
On the vfs_aio_pthread_async_dosmode_force_sync* shares we
force a sync threadpool which ensures we test behaviour on systems that
don't support unshare(CLONE_FS) and also don't support
per-thread-credentials. This simulates the code path of non linux
systems. And makes sure that we don't regress there.
We also test with xattr_tdb and without.
Pair-Programmed-With: Stefan Metzmacher <metze@samba.org>
Signed-off-by: Ralph Boehme <slow@samba.org>
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Autobuild-User(master): Ralph Böhme <slow@samba.org>
Autobuild-Date(master): Fri Jul 27 16:04:02 CEST 2018 on sn-devel-144
Ralph Boehme [Thu, 26 Jul 2018 16:11:22 +0000 (18:11 +0200)]
selftest: set "smbd:async dosmode = no" in the vfs_aio_pthread share
This just explicitly sets the current default, to ensure the tests that
use this share always use the same "smbd:async dosmode" setting even if
the default changes in the future.
Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Ralph Boehme [Thu, 22 Mar 2018 08:22:08 +0000 (09:22 +0100)]
s4: torture: test closing dir handle with in-flight find
Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Ralph Boehme [Wed, 25 Jul 2018 17:14:25 +0000 (19:14 +0200)]
smbd: use async dos_mode_at_send in smbd_smb2_query_directory_send()
Finally: use the new dos_mode_at_send() in the directory enumeration
loop. This means that fetching the DOS attributes for directory entries
is done asynchronously with regard to the enumeration loop.
As the DOS attribute is typically read from an extended attribute in the
filesytem, this avoids sequentially blocking on IO. If the IO subsystem
is slow servicing these request, enabling async processing can result in
performance improvements.
A parametric option
smbd:async dosmode = true | false (default: false)
can be used to enable the new async processing.
Simulating slow IO with usleep(5000) in the synchronous and asynchronous
versions of SMB_VFS_GET_DOS_ATTRIBUTES(), the results of enumerating a
directory with 10,000 files are:
smbd:async dosmode = no:
$ time bin/smbclient -U slow%x //localhost/test -c "ls dir\*" > /dev/null
real 0m59.597s
user 0m0.024s
sys 0m0.012s
smbd:async dosmode = yes:
$ time bin/smbclient -U slow%x //localhost/test -c "ls dir\*" > /dev/null
real 0m0.698s
user 0m0.038s
sys 0m0.025s
Performance gains in real world workloads depends on whether the actual
IO requests can be merged and parallelized by the kernel. Without such
wins at the IO layer, the async processing may even be slower then the
sync processing due to the additional overhead.
The following parameters can be used to adapt async processing behaviour
for specific workloads and systems:
aio max threads = X (default: 100)
smbd:max async dosmode = Y (default: "aio max threads" * 2)
By default we have at most twice the number of async requests in flight
as threads provided by the underlying threadpool. This ensures a worker
thread that finishes a job can directly pick up a new one without going
to sleep.
It may be advisable to reduce the number of threads to avoid scheduling
overhead while also increasing "smbd:max async dosmode".
Note that we disable async processing for certain VFS modules in the VFS
connect function to avoid the overhead of triggering the sync fallback
in dos_mode_at_send(). This is done for VFS modules that implement the
sync SMB_VFS_GET_DOS_ATTRIBUTES(), but not the async version (gpfs), and
for VFS modules that don't share a real filesystem where fchdir() can be
used (ceph, gluster). It is disabled for catia, because we realized that
the catia name translation macros used on
fsps (CATIA_FETCH_FSP_[PRE|POST]_NEXT) have a bug (#13547).
We use threadpool = smb_vfs_ev_glue_tp_chdir_safe() and then
pthreadpool_tevent_max_threads(threadpool) to get the number of maximum
worker threads which matches the pool used by the low level
SMB_VFS_GETXATTRAT_[SEND|RECV] implementation in vfs_default.
This is a terrible abstraction leak that should be removed in the future
by maybe making it possible to ask a VFS function which threadpool it
uses, internally suporting chaining so VFS function FOO that internally
uses BAR can forward the question to BAR.
On a hyphotetical system that had a getxattrat(dirfd, path, ...)
syscall and at the same time doesn't support per-thread current working
directories (eg FreeBSD doesn't have the latter) but has support for
per-thread-credentials, pthreadpool_tevent_max_threads() on the
tp_chdir_safe threadpool returns 1.
So when hooking the hyphotetical getxattrat() into the async
SMB_VFS_GETXATTRAT_[SEND|RECV] implementation in an VFS module, the
implementation could use the tp_path_safe threadpool, but the SMB2
layer would use the wrong threadpool in the call to
pthreadpool_tevent_max_threads(), resulting in no parallelism.
Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Ralph Boehme [Thu, 15 Mar 2018 18:03:59 +0000 (19:03 +0100)]
smbd: let smbd_dirptr_lanman2_entry return smb_fname
Note that smb_fname is relative to fsp, not conn!
Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Ralph Boehme [Wed, 21 Mar 2018 19:41:44 +0000 (20:41 +0100)]
smbd: deal with fsp->aio_requests in close_directory()
Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Ralph Boehme [Thu, 21 Jun 2018 10:44:42 +0000 (12:44 +0200)]
smbd: fix a long line in smb2_query_directory_next_entry()
Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Ralph Boehme [Tue, 3 Jul 2018 09:25:57 +0000 (11:25 +0200)]
smbd: factor out smb2_query_directory_next_entry() from smbd_smb2_query_directory_send()
This paves the way for adding async functions into the enumeration loop.
Best viewed with: git show -w
Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Ralph Boehme [Thu, 15 Mar 2018 17:57:50 +0000 (18:57 +0100)]
smbd: rework error exit in smbd_dirptr_lanman2_entry()
The next commit will add code that must be run if status is NT_STATUS_OK
or STATUS_MORE_ENTRIES.
Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Ralph Boehme [Thu, 15 Mar 2018 15:48:38 +0000 (16:48 +0100)]
smbd: pass get_dosmode to mode_fn in smbd_dirptr_get_entry()
This finally uses "get_dosmode" as passed in from the SMB2 layer, but
all callers still pass true, so no change in behaviour.
Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Ralph Boehme [Thu, 15 Mar 2018 15:08:11 +0000 (16:08 +0100)]
smbd: pass get_dosmode to smbd_dirptr_get_entry()
Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Ralph Boehme [Thu, 15 Mar 2018 14:50:41 +0000 (15:50 +0100)]
smbd: add "get_dosmode" argument to smbd_dirptr_lanman2_entry()
Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Ralph Boehme [Thu, 15 Mar 2018 14:21:53 +0000 (15:21 +0100)]
smbd: add dos_mode_at_send/recv()
Signed-off-by: Ralph Boehme <slow@samba.org>
Ralph Boehme [Wed, 25 Jul 2018 15:15:46 +0000 (17:15 +0200)]
smbd: factor out dosmode post processing
We apply some post processing to the dosmode returned from the VFS
function. Move this to a seperate function which will be reused in the
next commit in the async dos_mode_send/recv post processing.
Best viewed with: git show --histogram
Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Ralph Boehme [Thu, 15 Mar 2018 11:35:13 +0000 (12:35 +0100)]
vfs_default: implement SMB_VFS_GET_DOS_ATTRIBUTES_SEND/RECV
Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Ralph Boehme [Thu, 15 Mar 2018 09:56:28 +0000 (10:56 +0100)]
smbd: split out public parse_dos_attribute_blob() from get_ea_dos_attribute()
In preperation of adding an async version of get_ea_dos_attribute() that
will then call parse_dos_attribute_blob() too.
Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Ralph Boehme [Thu, 15 Mar 2018 12:08:55 +0000 (13:08 +0100)]
s3: vfs: add SMB_VFS_GET_DOS_ATTRIBUTES_SEND/RECV
Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Ralph Boehme [Thu, 21 Jun 2018 15:04:22 +0000 (17:04 +0200)]
vfs_xattr_tdb: implement SMB_VFS_GETXATTRAT_SEND/RECV
Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Ralph Boehme [Tue, 13 Mar 2018 15:17:27 +0000 (16:17 +0100)]
vfs_default: implement SMB_VFS_GETXATTRAT_SEND/RECV
Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Ralph Boehme [Tue, 13 Mar 2018 07:14:53 +0000 (08:14 +0100)]
s3: vfs: add SMB_VFS_GETXATTRAT_SEND/RECV
Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Stefan Metzmacher [Thu, 26 Jul 2018 07:41:22 +0000 (09:41 +0200)]
pthreadpool: we need to use pthreadpool_tevent_per_thread_cwd() on the callers pool
In pthreadpool_tevent_job_send() we remember if the job will be chdir
safe. It means we means we need to ask the callers pool when calling
pthreadpool_tevent_per_thread_cwd(), as the callers pool might
be a wrapper using pthreadpool_tevent_force_per_thread_cwd().
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
Stefan Metzmacher [Thu, 26 Jul 2018 00:29:10 +0000 (02:29 +0200)]
ldb_mdb: #ifdef EBADE as it is not portable
E.g. FreeBSD 11.2 doesn't have it.
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
Martin Schwenke [Fri, 8 Jun 2018 08:48:07 +0000 (18:48 +1000)]
ctdb-common: Drop unused function mkdir_p_or_die()
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Fri Jul 27 08:42:20 CEST 2018 on sn-devel-144
Martin Schwenke [Thu, 7 Jun 2018 21:33:32 +0000 (07:33 +1000)]
ctdb-common: Drop function parse_ip_mask() and supporting functions
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Martin Schwenke [Thu, 7 Jun 2018 21:27:07 +0000 (07:27 +1000)]
ctdb-daemon: Switch to using ctdb_sock_addr_mask_from_string()
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Martin Schwenke [Thu, 7 Jun 2018 21:26:51 +0000 (07:26 +1000)]
ctdb-tools: Switch to using ctdb_sock_addr_mask_from_string()
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Martin Schwenke [Thu, 7 Jun 2018 21:13:25 +0000 (07:13 +1000)]
ctdb-protocol: Add function ctdb_sock_addr_mask_from_string()
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Martin Schwenke [Thu, 26 Jul 2018 01:01:30 +0000 (11:01 +1000)]
ctdb-protocol: Fix compilation issue with strncpy()
When configured with --picky-developer and using -O3 with gcc 8.1:
../protocol/protocol_util.c: In function ‘ctdb_sock_addr_from_string’:
../protocol/protocol_util.c:282:2: error: ‘strncpy’ specified bound depends on the length of the source argument [-Werror=stringop-overflow=]
strncpy(s, str, len+1);
^~~~~~~~~~~~~~~~~~~~~~
../protocol/protocol_util.c:277:8: note: length computed here
len = strlen(str);
^~~~~~~~~~~
Use strlcpy() instead and check the result.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13545
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Martin Schwenke [Thu, 26 Jul 2018 01:00:28 +0000 (11:00 +1000)]
ctdb-common: Fix compilation issue with strncpy()
When configured with --picky-developer and using -O3 with gcc 8.1:
../common/system_socket.c: In function ‘parse_ip_mask’:
../common/system_socket.c:229:2: error: ‘strncpy’ specified bound depends on the length of the source argument [-Werror=stringop-overflow=]
strncpy(s, str, len+1);
^~~~~~~~~~~~~~~~~~~~~~
../common/system_socket.c:223:8: note: length computed here
len = strlen(str);
^~~~~~~~~~~
Use strlcpy() instead and check the result.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13545
Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Volker Lendecke [Wed, 25 Jul 2018 15:05:34 +0000 (17:05 +0200)]
smbd: Remove "share_mode_entry->lease"
smbstatus was the only user, and this could be solved by adapting
share_entry_forall.
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Fri Jul 27 01:42:31 CEST 2018 on sn-devel-144
Volker Lendecke [Wed, 25 Jul 2018 14:59:53 +0000 (16:59 +0200)]
smbstatus: Use share_mode_data->leases
This is the only user of share_mode_entry->lease
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Volker Lendecke [Wed, 25 Jul 2018 14:56:35 +0000 (16:56 +0200)]
smbd: Pass "share_mode_data" to share_entry_forall callback
Quite a bit of the contents have been passed explicitly anyway.
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Volker Lendecke [Wed, 25 Jul 2018 13:48:01 +0000 (15:48 +0200)]
popt: popt 1.16 needs -liconv
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Volker Lendecke [Wed, 25 Jul 2018 10:05:58 +0000 (12:05 +0200)]
idmap: Make pointer initialization explicit
Took me a few seconds to find this NULL initializer
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Ralph Boehme [Wed, 25 Jul 2018 18:02:23 +0000 (20:02 +0200)]
smbd: don't client->connections without checking client != NULL first in exit_server_common()
exit_server_common() can be called also in smbd processes without a
smbXsrv_client structure, e.g. the parent or some background tasks.
Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Autobuild-User(master): Stefan Metzmacher <metze@samba.org>
Autobuild-Date(master): Thu Jul 26 01:29:38 CEST 2018 on sn-devel-144
Stefan Metzmacher [Wed, 25 Jul 2018 10:30:37 +0000 (12:30 +0200)]
examples/VFS/skel_transparent: make vfs_fn_pointers static
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
Autobuild-User(master): Stefan Metzmacher <metze@samba.org>
Autobuild-Date(master): Wed Jul 25 20:44:12 CEST 2018 on sn-devel-144
Ralph Boehme [Wed, 25 Jul 2018 13:47:37 +0000 (15:47 +0200)]
examples/VFS/skel_opaque: make vfs_fn_pointers static
Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Ralph Boehme [Wed, 25 Jul 2018 13:45:44 +0000 (15:45 +0200)]
examples/VFS/skel_opaque: fix a likely a copy/paste error
This line was probably copied over from skel_transparent.c, remove it.
Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Stefan Metzmacher [Wed, 25 Jul 2018 10:26:05 +0000 (12:26 +0200)]
s3:modules: add vfs_not_implemented module
This provides helper functions, which can be used by other modules,
if they don't implement a specific function.
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
Ralph Boehme [Fri, 13 Jul 2018 14:48:19 +0000 (16:48 +0200)]
vfs_aio_pthread: use event context and threadpool from user_vfs_evg
Or the root glue in case we're already root.
Pair-Programmed-With: Stefan Metzmacher <metze@samba.org>
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Signed-off-by: Ralph Boehme <slow@samba.org>
Ralph Boehme [Thu, 5 Jul 2018 11:09:53 +0000 (13:09 +0200)]
s3: vfs: add user_vfs_evg to connection_struct
This will be used to in order to pass down the
impersonation magic from the SMB layer through
the SMB_VFS layer.
This includes the following options:
smbd:force sync user path safe threadpool
smbd:force sync user chdir safe threadpool
smbd:force sync root path safe threadpool
smbd:force sync root chdir safe threadpool
They can be used in order to test the non linux code
path on linux, once we get code that makes full use
of the new infrastructure.
Pair-Programmed-With: Stefan Metzmacher <metze@samba.org>
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Signed-off-by: Ralph Boehme <slow@samba.org>
Ralph Boehme [Sun, 8 Jul 2018 14:28:02 +0000 (16:28 +0200)]
s3: vfs: add smb_vfs_ev_glue
This adds VFS helper functions and that work on a struct smb_vfs_ev_glue
object which bundles two event contexts and a few threadpools.
This will be used to streamline the use of impersonating wrappers
in the SMB_VFS.
Notice the verbose comments in source3/smbd/vfs.c.
This will allow us to introduce path based async operations
to the SMB_VFS layer.
Pair-Programmed-With: Stefan Metzmacher <metze@samba.org>
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Signed-off-by: Ralph Boehme <slow@samba.org>
Stefan Metzmacher [Tue, 24 Jul 2018 08:56:34 +0000 (10:56 +0200)]
smbd: introduce sconn->sync_thread_pool
This just simulates a threadpool, but executes the
job functions inline (blocking) in the main thread.
This will be used to work arround some OS limitations,
e.g. if per thread credentials or per thread working directory
are not supported.
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
Ralph Boehme [Fri, 13 Jul 2018 15:17:50 +0000 (17:17 +0200)]
smbd: rename sconn->pool to sconn->raw_thread_pool
This should in future not be used directly, we'll provide
wrapper pools, which will provide impersonation for
path based async calls.
Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Stefan Metzmacher [Wed, 25 Jul 2018 14:44:18 +0000 (16:44 +0200)]
smbd: add missing DO_PROFILE_INC(disconnect) to smbd_server_connection_terminate_ex()
For multi channel connections we should increment the disconnect count
also if we're not closing the last channel.
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
Stefan Metzmacher [Tue, 24 Jul 2018 15:17:29 +0000 (17:17 +0200)]
smbd: disconnect/destroy all connections before calling smbXsrv_session_logoff_all()
This means the pending requests are destroyed before tevent_context
impersonation wrapper are destroyed.
Otherwise, with a pending struct tevent_req that is a child of
client->xconn like this in exit_server_common():
conn[ipv4:127.0.0.11:40745] reason[NT_STATUS_CONNECTION_RESET] at ../source3/smbd/smb2_server.c:4015
full talloc report on 'struct smbXsrv_connection' (total 6085 bytes in 43 blocks)
struct smbd_smb2_request contains 648 bytes in 1 blocks (ref 0) 0x55ed41634740
struct smbd_smb2_request contains 3438 bytes in 32 blocks (ref 0) 0x55ed416331e0
struct tevent_req contains 1824 bytes in 20 blocks (ref 0) 0x55ed41635860
struct smb_filename contains 206 bytes in 2 blocks (ref 0) 0x55ed41635560
lease_v2_complex2.dat contains 22 bytes in 1 blocks (ref 0) 0x55ed4161b950
struct smbd_smb2_create_state contains 1386 bytes in 16 blocks (ref 0) 0x55ed41635a10
struct deferred_open_record contains 804 bytes in 12 blocks (ref 0) 0x55ed41633090
struct defer_open_state contains 764 bytes in 11 blocks (ref 0) 0x55ed41634f10
struct tevent_req contains 748 bytes in 10 blocks (ref 0) 0x55ed41636390
struct tevent_timer contains 96 bytes in 1 blocks (ref 0) 0x55ed41636ad0
struct dbwrap_watched_watch_state contains 420 bytes in 7 blocks (ref 0) 0x55ed41636540
struct tevent_req contains 296 bytes in 5 blocks (ref 0) 0x55ed41636700
struct messaging_filtered_read_state contains 64 bytes in 3 blocks (ref 0) 0x55ed416368b0
struct messaging_dgm_fde contains 8 bytes in 2 blocks (ref 0) 0x55ed41636950
reference to: struct messaging_dgm_fde_ev
we crash when freeing the xconn as the the tevent_req child has a
cleanup function that tries to access a (wrapped) tevent context that
was already destroyed via smb1srv_tcon_disconnect_all() for SMB1 or
smbXsrv_session_logoff_all() for SMB2 a few lines above.
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
Stefan Metzmacher [Tue, 24 Jul 2018 15:13:39 +0000 (17:13 +0200)]
smbd: only pass struct smbXsrv_client to smbXsrv_session_logoff_all()
That's all it needs.
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
Stefan Metzmacher [Tue, 24 Jul 2018 15:13:39 +0000 (17:13 +0200)]
smbd: only pass struct smbXsrv_client to smb1srv_tcon_disconnect_all()
That's all it needs.
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
Ralph Boehme [Wed, 25 Jul 2018 10:03:52 +0000 (12:03 +0200)]
pthreadpool: add a missing include
Reported-by: David Disseldorp <ddiss@samba.org>
Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: David Disseldorp <ddiss@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Gary Lockyer [Tue, 24 Jul 2018 03:20:21 +0000 (15:20 +1200)]
lib audit_logging: add _WARN_UNUSED_RESULT_
Have the compiler issue a warning when the return code from the API is
ignored.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Gary Lockyer <gary@samba.org>
Autobuild-Date(master): Wed Jul 25 09:28:31 CEST 2018 on sn-devel-144
Gary Lockyer [Mon, 16 Jul 2018 02:38:12 +0000 (14:38 +1200)]
dsdb group_audit_test: Remove redundant mocking code
Remove a place holder test and unused mocking code.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Gary Lockyer [Fri, 13 Jul 2018 01:34:28 +0000 (13:34 +1200)]
dsdb group auditing: remove HAVE_JANSSON from group_audit
This modules is ADDC only and JANSSON is required for the ADDC builds,
so the ifdef is no longer necessary.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Gary Lockyer [Fri, 13 Jul 2018 01:33:59 +0000 (13:33 +1200)]
dsdb audit logging: remove HAVE_JANSSON from audit_log
This modules is ADDC only and JANSSON is required for the ADDC builds,
so the ifdef is no longer necessary.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Gary Lockyer [Thu, 12 Jul 2018 21:15:34 +0000 (09:15 +1200)]
json: Add unit tests for error handling
Add cmocka unit tests to exercise the error handling in the JSON
routines.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Gary Lockyer [Thu, 12 Jul 2018 21:14:09 +0000 (09:14 +1200)]
json: Modify API to use return codes
Modify the auditing JSON API to return a response code, as the consensus
was that the existing error handling was aesthetically displeasing.
Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
David Disseldorp [Tue, 24 Jul 2018 22:33:09 +0000 (00:33 +0200)]
docs/kerneloplocks: drop Irix references
Signed-off-by: David Disseldorp <ddiss@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Wed Jul 25 06:28:21 CEST 2018 on sn-devel-144
Jeremy Allison [Tue, 24 Jul 2018 17:56:30 +0000 (10:56 -0700)]
s3: smbd: SGI IRIX is officially dead. Remove the kernel oplock code for IRIX.
Signed-off-by: Jeremy Allison <jra@samba.org>
Reviewed-by: David Disseldorp <ddiss@samba.org>
Ralph Boehme [Tue, 13 Mar 2018 15:17:27 +0000 (16:17 +0100)]
s3: vfs: bump to version 40, Samba 4.10 will ship with that
Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Autobuild-User(master): Stefan Metzmacher <metze@samba.org>
Autobuild-Date(master): Wed Jul 25 03:23:44 CEST 2018 on sn-devel-144
Ralph Boehme [Tue, 13 Mar 2018 15:17:27 +0000 (16:17 +0100)]
s3: vfs: bump to version 39, Samba 4.9 will ship with that
Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Ralph Boehme [Thu, 12 Jul 2018 12:44:40 +0000 (14:44 +0200)]
s3: vfs: add missing tevent_req_received() to SMB_VFS_FSYNC_RECV()
Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Ralph Boehme [Thu, 12 Jul 2018 12:44:27 +0000 (14:44 +0200)]
s3: vfs: add missing tevent_req_received() to SMB_VFS_PWRITE_RECV()
Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Ralph Boehme [Thu, 12 Jul 2018 12:43:55 +0000 (14:43 +0200)]
s3: vfs: add missing tevent_req_received() to SMB_VFS_PREAD_RECV()
Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Ralph Boehme [Wed, 20 Jun 2018 08:54:04 +0000 (10:54 +0200)]
vfs_default: fix async fsync idle/busy time profiling
Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Ralph Boehme [Tue, 20 Mar 2018 14:27:44 +0000 (15:27 +0100)]
s3: libsmb: use smb2cli_conn_max_trans_size() in cli_smb2_list()
Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Ralph Boehme [Thu, 22 Mar 2018 09:07:49 +0000 (10:07 +0100)]
s4: libcli/smb2: calculate correct credit charge for finds
Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Ralph Boehme [Thu, 28 Jun 2018 19:47:54 +0000 (21:47 +0200)]
s3: lib/xattr_tdb: fix listing xattrs
If there's no record in the xattr.tdb, dbwrap_fetch() will return
NT_STATUS_NOT_FOUND. That should not result in an error in callers of
xattr_tdb_load_attrs().
Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Stefan Metzmacher [Mon, 23 Jul 2018 07:14:36 +0000 (09:14 +0200)]
vfs_default: call smb_vfs_assert_all_fns()
This module needs to implement every call.
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
Stefan Metzmacher [Mon, 23 Jul 2018 07:02:52 +0000 (09:02 +0200)]
examples/VFS/skel_transparent: call smb_vfs_assert_all_fns()
This template should always include all calls.
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
Ralph Boehme [Tue, 24 Jul 2018 20:03:01 +0000 (22:03 +0200)]
examples/VFS/skel_transparent: add missing durable handle functions
Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Ralph Boehme [Tue, 24 Jul 2018 16:58:59 +0000 (18:58 +0200)]
examples/VFS/skel_transparent: add missing audit_file_fn
Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Stefan Metzmacher [Mon, 23 Jul 2018 07:02:52 +0000 (09:02 +0200)]
examples/VFS/skel_opaque: call smb_vfs_assert_all_fns()
This template should always implement all calls.
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
Ralph Boehme [Tue, 24 Jul 2018 19:56:26 +0000 (21:56 +0200)]
examples/VFS/skel_opaque: add missing durable handle functions
Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Ralph Boehme [Tue, 24 Jul 2018 16:58:45 +0000 (18:58 +0200)]
examples/VFS/skel_opaque: add missing audit_file_fn
Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Ralph Boehme [Tue, 24 Jul 2018 10:30:33 +0000 (12:30 +0200)]
autobuild: add some basic tests for the all static build
This makes sure each module is at least loaded once
and registers itself as a module.
It means that the skel_opaque and skel_transparent vfs examples
are loaded.
Pair-Programmed-With: Stefan Metzmacher <metze@samba.org>
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Signed-off-by: Ralph Boehme <slow@samba.org>
Volker Lendecke [Thu, 3 May 2018 09:48:45 +0000 (11:48 +0200)]
winbind: Move variable declarations close to their use
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Tue Jul 24 23:31:43 CEST 2018 on sn-devel-144
Volker Lendecke [Fri, 4 May 2018 19:07:35 +0000 (21:07 +0200)]
winbind: Align integer types
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Volker Lendecke [Fri, 4 May 2018 13:23:18 +0000 (15:23 +0200)]
nsswitch: Correct users of "ctx->is_privileged"
winbindd_context->is_privileged is a bool
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Volker Lendecke [Fri, 4 May 2018 12:40:12 +0000 (14:40 +0200)]
nsswitch: Make two functions static
nss_irix was the only external user
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Volker Lendecke [Fri, 4 May 2018 12:33:55 +0000 (14:33 +0200)]
nsswitch: Remove IRIX support
According to wikipedia, IRIX has seen the last patch update in August 2006. As
of now, www.sgi.com is unreachable. Probably this code has not been built in
years. If someone wants to revive it, it can be found in the git history.
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Volker Lendecke [Fri, 4 May 2018 20:28:43 +0000 (22:28 +0200)]
lib: Fix prototype of srprs_str
Many callers use "-1" as the "len" argument. That's what ssize_t is for.
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Volker Lendecke [Wed, 25 Apr 2018 09:53:00 +0000 (11:53 +0200)]
lib: Remove an #include "includes.h"
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Volker Lendecke [Wed, 25 Apr 2018 09:51:36 +0000 (11:51 +0200)]
lib: Remove an #include "includes.h"
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Stefan Metzmacher [Thu, 3 May 2018 14:08:06 +0000 (16:08 +0200)]
s3:util_sec: add a cache to set_thread_credentials()
Calling set_thread_credentials() with the same values,
skips syscalls the 2nd time.
We only do this if '__thread' is supported to provide
thread local storage.
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
Autobuild-User(master): Ralph Böhme <slow@samba.org>
Autobuild-Date(master): Tue Jul 24 20:35:17 CEST 2018 on sn-devel-144
Ralph Boehme [Thu, 28 Jun 2018 12:28:34 +0000 (14:28 +0200)]
lib/util: rename USE_LINUX_THREAD_CREDENTIALS to HAVE_LINUX_THREAD_CREDENTIALS
The define reflects the results of a feature test, not a configure
option.
Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Ralph Boehme [Mon, 18 Jun 2018 14:57:18 +0000 (16:57 +0200)]
pthreadpool: test cancelling and freeing jobs of a wrapped pthreadpool_tevent
Pair-Programmed-With: Stefan Metzmacher <metze@samba.org>
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
Stefan Metzmacher [Fri, 20 Apr 2018 15:12:07 +0000 (17:12 +0200)]
pthreadpool: implement pthreadpool_tevent_wrapper_create() infrastructure
This can be used implement a generic per thread impersonation
for thread pools.
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
Stefan Metzmacher [Mon, 16 Jul 2018 12:43:01 +0000 (14:43 +0200)]
pthreadpool: add pthreadpool_restart_check[_monitor_{fd,drain}]()
This makes it possible to monitor the pthreadpool for exited worker
threads and may restart new threads from the main thread again.
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
Stefan Metzmacher [Wed, 18 Jul 2018 08:21:22 +0000 (10:21 +0200)]
pthreadpool: add tests for pthreadpool_tevent_[current_job_]per_thread_cwd()
Note this currently this doesn't enforce the support for
unshare(CLONE_FS) as some contraint container environment
(e.g. docker) reject the whole unshare() system call.
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
Stefan Metzmacher [Thu, 21 Jun 2018 23:02:41 +0000 (01:02 +0200)]
pthreadpool: add pthreadpool_tevent_[current_job_]per_thread_cwd()
This can be used to check if worker threads run with
unshare(CLONE_FS).
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
Ralph Boehme [Tue, 13 Mar 2018 15:59:32 +0000 (16:59 +0100)]
pthreadpool: call unshare(CLONE_FS) if available
This paves the way for pthreadpool jobs that are path based.
Callers can use pthreadpool_per_thread_cwd() to check if
the current pool supports it.
Pair-Programmed-With: Stefan Metzmacher <metze@samba.org>
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Signed-off-by: Ralph Boehme <slow@samba.org>
Ralph Boehme [Tue, 13 Mar 2018 15:58:49 +0000 (16:58 +0100)]
configure: check for Linux specific unshare() with CLONE_FS
Note we still need some kind of runtime detection as
it can fail in some constraint container setups, which
reject the whole unshare() syscall instead of just the
once used for container features.
In case unshare(CLONE_FS) works, we can have a per thread
current working directory and use [f]chdir() safely in
worker threads.
Pair-Programmed-With: Stefan Metzmacher <metze@samba.org>
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Signed-off-by: Ralph Boehme <slow@samba.org>
Ralph Boehme [Mon, 18 Jun 2018 13:32:30 +0000 (15:32 +0200)]
pthreadpool: test cancelling and freeing pending pthreadpool_tevent jobs/pools
Pair-Programmed-With: Stefan Metzmacher <metze@samba.org>
Signed-off-by: Ralph Boehme <slow@samba.org>
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Stefan Metzmacher [Fri, 22 Jun 2018 15:22:10 +0000 (17:22 +0200)]
pthreadpool: add a comment about a further optimization in pthreadpool_tevent_job_destructor()
This seems to be a really rare race, it's likely that the immediate
event will still trigger and cleanup.
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
Stefan Metzmacher [Fri, 22 Jun 2018 15:14:31 +0000 (17:14 +0200)]
pthreadpool: maintain a list of job_states on each pthreadpool_tevent_glue
We should avoid traversing a linked list within a thread without holding
a mutex!
Using a mutex would be very tricky as we'll likely deadlock with
the mutexes at the raw pthreadpool layer.
So we use somekind of spinlock using atomic_thread_fence in order to
protect the access to job->state->glue->{tctx,ev} in
pthreadpool_tevent_job_signal().
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
Stefan Metzmacher [Thu, 21 Jun 2018 10:43:08 +0000 (12:43 +0200)]
pthreadpool: add helgrind magic to PTHREAD_TEVENT_JOB_THREAD_FENCE_*()
This avoids the expected helgrind/drd warnings on the job states which
are protected by the thread fence.
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
Stefan Metzmacher [Thu, 21 Jun 2018 10:46:48 +0000 (12:46 +0200)]
lib/replace: also check for valgrind/helgrind.h
This will be used in lib/pthreadpool/pthreadpool_tevent.c
in order to avoid extected helgrind/drd warnings.
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
Stefan Metzmacher [Thu, 21 Jun 2018 10:46:06 +0000 (12:46 +0200)]
s3:wscript: don't check for valgrind related headers twice
We already check them in lib/replace/wscript.
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
Stefan Metzmacher [Wed, 20 Jun 2018 11:38:19 +0000 (13:38 +0200)]
pthreadpool: add some lockless coordination between the main and job threads
In the direction from the main process to the job thread, we have:
- 'maycancel', which is set when tevent_req_cancel() is called,
- 'orphaned' is the job request, tevent_context or pthreadpool_tevent
was talloc_free'ed.
The job function can consume these by using:
/*
* return true - if tevent_req_cancel() was called.
*/
bool pthreadpool_tevent_current_job_canceled(void);
/*
* return true - if talloc_free() was called on the job request,
* tevent_context or pthreadpool_tevent.
*/
bool pthreadpool_tevent_current_job_orphaned(void);
/*
* return true if canceled and orphaned are both false.
*/
bool pthreadpool_tevent_current_job_continue(void);
In the other direction we remember the following points
in the job execution:
- 'started' - set when the job is picked up by a worker thread
- 'executed' - set once the job function returned.
- 'finished' - set when pthreadpool_tevent_job_signal() is entered
- 'dropped' - set when pthreadpool_tevent_job_signal() leaves with orphaned
- 'signaled' - set when pthreadpool_tevent_job_signal() leaves normal
There're only one side writing each element,
either the main process or the job thread.
This means we can do the coordination with a full memory
barrier using atomic_thread_fence(memory_order_seq_cst).
lib/replace provides fallbacks if C11 stdatomic.h is not available.
A real pthreadpool requires pthread and atomic_thread_fence() (or an
replacement) to be available, otherwise we only have pthreadpool_sync.c.
But this should not make a real difference, as at least
__sync_synchronize() is availabe since 2005 in gcc.
We also require __thread which is available since 2002.
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
Stefan Metzmacher [Thu, 21 Jun 2018 12:17:35 +0000 (14:17 +0200)]
replace: add checks for atomic_thread_fence(memory_order_seq_cst) and add possible fallbacks
This implements a full memory barrier.
On ubuntu amd64 with results in an 'mfence' instruction.
This is required to syncronization between threads, where
there's typically only one write of a memory that should be
synced between all threads with the barrier.
Much more details can be found here:
https://gcc.gnu.org/onlinedocs/gcc-7.3.0/gcc/_005f_005fatomic-Builtins.html#g_t_005f_005fatomic-Builtins
https://gcc.gnu.org/onlinedocs/gcc-7.3.0/gcc/_005f_005fsync-Builtins.html#g_t_005f_005fsync-Builtins
The main one we use seems to be in C11 via stdatomic.h,
the oldest fallback is __sync_synchronize(), which is available
since 2005 in gcc.
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
git.samba.org / samba.git / log