selftest: Remove output directories to save disk space

Reviewed-by: Stefan Metzmacher <metze@samba.org>
(cherry picked from commit b9b637569960ae7eef5ee12436624af34a718a9a)

samba_upgradeprovision: Do not reset every DN when changing an SD

SD propogation is handled by an LDB module, we do not need to touch each
and every DN to make it happen.

Now that we do not need to put this via a hash, the dnToRecalculate
list is changed to be a list of Dn objects, not strings so that:

if dn in listWellknown

is handled using a schema comparison (avoiding different case forms
tripping it up).

Andrew Bartlett

Reviewed-by: Stefan Metzmacher <metze@samba.org>
(cherry picked from commit d5d88bd82b1cb51da09cf3b3dec40f180f5ed29f)

samba_upgradeprovision: do not maintain dnNotToRecalculate as a list

We only need a boolean indication, not the actual values.

Andrew Bartlett

Reviewed-by: Stefan Metzmacher <metze@samba.org>
(cherry picked from commit 0f247dce00fd26230cdb0566ce4f51a2ea8cfc2b)

samba_upgradeprovision: only run rebuild_sd in --full mode

This is a potentially destructive routine, and should not be run by default.

Andrew Bartlett

Reviewed-by: Stefan Metzmacher <metze@samba.org>
(cherry picked from commit 9bc32bfd65700c816ebb2a3004ad568327218f86)

samba_upgradeprovision: Remove alwaysRecalculate, this is too dangerous

I am unclear on why this was added, but the idea that we ever always reset data
in the directory is not reasonable to me, so I am removing it.

Andrew Bartlett

Reviewed-by: Stefan Metzmacher <metze@samba.org>
(cherry picked from commit 81cda856faf2a5efd38965fd4c3b1f5551ad94d9)

samba_upgradeprovision: Remove unused checkKeepAttributeOldMtd

lastProvisionUSNs is never None, instead the code requries the administrator to populate this
attribute in the directory.

Andrew Bartlett

Reviewed-by: Stefan Metzmacher <metze@samba.org>
(cherry picked from commit 09b82d5fdc05a1f440aa96a690c202d4b0df134b)

samba_upgradeprovision: Remove inherited ACEs before comparing the SDs

This avoids changing an SD when it is not really required.

Andrew Bartlett

Reviewed-by: Stefan Metzmacher <metze@samba.org>
(cherry picked from commit 9b8d5bba507615aee95a46fd9ae75aa782fd7e66)

scripting: Rework samba.upgradehelpers.get_diff_sddls to be get_diff_sds

This moves the SDDL conversion inside the get_diff_sds function and prepares
for removing inherited ACEs from the SD before comparison.

Andrew Bartlett

Reviewed-by: Stefan Metzmacher <metze@samba.org>
(cherry picked from commit 5074b98714c9e038cc31872111508c1d92562841)

samba_upgradeprovision: Remove auto-detection of pre-alpha9 databases

These are incredibly rare, and administrators running such databases
not only ask the Samba Team for help personally, they can read --help.

Andrew Bartlett

Reviewed-by: Stefan Metzmacher <metze@samba.org>
(cherry picked from commit 787a6aacc3003731784b29fd92c683036c8730a7)

selftest: Rename samba4.blackbox.upgradeprovision.py to samba4.blackbox.upgradeprovision.current

This name matches the other upgradeprovision tests for older saved provisions.

Andrew Bartlett

Reviewed-by: Stefan Metzmacher <metze@samba.org>
(cherry picked from commit 9d6af4938f7bc80b10202d7055c2c32a483bbb5f)

selftest: Run dbcheck and improved upgrdeprovision tests against release-4-0-0

The improved upgradeprovision tests now call ldapcmp to verify the
changes made do actually bring the database in line with a fresh
provision.

Andrew Bartlett

Reviewed-by: Stefan Metzmacher <metze@samba.org>
(cherry picked from commit 08f0562240155a871bd2a78d217db660e8ee3c91)

selftest: Add ldapcmp to ensure upgradeprovision of a fresh DB is a no-op

Reviewed-by: Stefan Metzmacher <metze@samba.org>
(cherry picked from commit d7936ee20c20635d62657cb821ff6dc4eb5fe33c)

selftest: Add in a provision from 4.0.0 to run tests against

Reviewed-by: Stefan Metzmacher <metze@samba.org>
(cherry picked from commit f1f36ad3517cd0e6bceb4b0cc37721a15be4d588)

selftest: Do an ldapcmp run against the upgraded domain

This checks (with a set of known issues marked in the --filter attribute) that the upgraded
domain matches a fresh provision.

Andrew Bartlett

Reviewed-by: Stefan Metzmacher <metze@samba.org>
(cherry picked from commit 72f73ebaff8d75fc39770ec785964b0d3c9738cc)

samba-tool ldapcmp: Add support for checking DNSDOMAIN and DNSFOREST by default

Reviewed-by: Stefan Metzmacher <metze@samba.org>
(cherry picked from commit 24c4d818d14c3931cf0cbff3070685fe409e66c6)

samba-tool dbcheck: fix msDS-HasInstantiatedNCs attributes to match instanceType on our ntdsDSA

This value is only a link to the local value of intanceType on our server, so only fix it for our server.

Andrew Bartlett

Reviewed-by: Stefan Metzmacher <metze@samba.org>
(cherry picked from commit f508435d23445a8b3076f89cbe042e2da1ac0701)

scripting: Correct parsing of binary DN

The DN is of the form B:8:01020304:DC=samba,DC=example,DC=com.  We need
to account for the case where the 8 is actually (say) 16, and so not just
one character.

Andrew Bartlett

Signed-off-by: Stefan Metzmacher <metze@samba.org>
(cherry picked from commit 97389c3ec24526837e91fcfcaf7439491fcdb214)

subunit: Add a sh macro for skipping a test

Reviewed-by: Stefan Metzmacher <metze@samba.org>
(cherry picked from commit 0180a027cbc9725ae13023ddfdb8079f147864c5)

samba-tool ldapcmp: Add --skip-missing-dn to not error on DNs present in one DB but not the other

This is needed to compare some parts of the database, particularly in --two mode, which
are just never going to have exactly the same DNs.

Andrew Bartlett

Reviewed-by: Stefan Metzmacher <metze@samba.org>
(cherry picked from commit 606f5d6cc6b018259ba0306fe3b55e21b4b70fdb)

samba-tool domain classicupgrade: Fix typo in error path for multiple account flags

Reviewed-by: Stefan Metzmacher <metze@samba.org>
(cherry picked from commit 161fa15697fab9effbe1db9640cece847dcf63cd)

samba-tool domain classicupgrade: Print a better error when the ldap backend PW was not found

Reviewed-by: Stefan Metzmacher <metze@samba.org>
(cherry picked from commit 669c302f2d78ab4bbd35982373ae079246c8979d)

samba-tool dbcheck: fix comment on err_wrong_sd

Reviewed-by: Stefan Metzmacher <metze@samba.org>
(cherry picked from commit 68f13f5d7e80f2041e140a12fc2f7139561c83ce)

Move python modules from source4/scripting/python/ to python/.

Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Sat Mar  2 03:57:34 CET 2013 on sn-devel-104
(cherry picked from commit 87afc3aee1ea593069322a49355dd8780d99e123)

build: Change bin/default/python -> bin/python symlink to bin/default/python_modules

This avoids a collision with the new top level python directory.

Andrew Bartlett
(cherry picked from commit 80fce353e740c793619005ac102ab07fb5e7d280)

build: Rename samba_python waf node to avoid duplicate name

This makes it clearer when debuging build issues.

Andrew Bartlett
(cherry picked from commit 2d13532cb3ad3a3deaee1f158408478284bc595a)

dsdb-descriptor: Avoid segfault copying an SD without an owner or group

This is an unusual SD, but it does exist is some very old upgraded databases.

Andrew Bartlett

Reviewed-by: Stefan Metzmacher <metze@samba.org>
Autobuild-User(master): Stefan Metzmacher <metze@samba.org>
Autobuild-Date(master): Fri Feb 22 11:06:17 CET 2013 on sn-devel-104
(cherry picked from commit e4d85fa73d3ce1f397fdd416af6b8c903a473824)

dsdb-descriptor: Spell out security descriptor flags as constants

Reviewed-by: Stefan Metzmacher <metze@samba.org>
(cherry picked from commit 87db2331fc855473d8b3cad1c4149807823aa3c4)

samba_upgradeprovision: Use tdb_util.tdb_copy not shutil.copy2

This is really important, because copying a file will both ignore
locks held by another process and break any locks we hold (due to
POSIX brain-damage regarding multiple fds on one file in a process).

By leaving this to tdbbackup in a child, both of these issues are avoided.

Andrew Bartlett

Reviewed-by: Matthieu Patou <mat@samba.org>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Tue Feb 19 07:48:18 CET 2013 on sn-devel-104
(cherry picked from commit 2cf83f7c645e4b216cf6f23857fd72ec0e6ca7a6)

samba_upgradeprovision: Do not update privileges.ldb any more (unchanged since 2009)

This update was only a total oblitoration of the existing database
and not a merge, and the shutil.copy would both disregard and break
locks on the database that are held at this point.

Andrew Bartlett

Reviewed-by: Matthieu Patou <mat@samba.org>
(cherry picked from commit 3c51e18a0cd1cb4b54cd29e312abd7cc2c0fbc98)

scripting: Make tdb_copy a common util function in samba.tdb_util

This will allow samba_upgradeprovision to also call it.

Andrew Bartlett

Reviewed-by: Matthieu Patou <mat@samba.org>
(cherry picked from commit 396df64ef6f2c66c35989ecda3e564d5578fe9f3)

scripting: Make tdb_copy use the python subprocess module

This makes the code more robust to spaces in the file names (etc).

Andrew Bartlett

Reviewed-by: Matthieu Patou <mat@samba.org>
(cherry picked from commit 2c2759e408d9c45c2aee0c2578f45edd246afec3)

samba_upgradeprovision: Remove options to fix FS ACLs

samba-tool ntacl sysvolreset handles this better, and makes this tool
much less confusing internally.

Andrew Bartlett

Reviewed-by: Matthieu Patou <mat@samba.org>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Tue Feb 19 06:06:41 CET 2013 on sn-devel-104
(cherry picked from commit 06780ae82281fb62a08d0c3604d2e679976756c2)

selftest: skip base.dir1 tests as they just spin on modern ext4

param: Remove incorrectly added defaults in AD DC allowing WORLD WRITABLE files

These defaults were incorrectly added in
fc5caffbc139d63cab1ec105884863f73772586f in what turns out to be an
incorrect fix for bug #9462, which was in turn introduced by the
swapping of security mask (default 0777) for create mask (0755) in
6adc7dad96b8c7366da042f0d93b28c1ecb092eb.

While the permissions on sysvol and netlogon (the default shares) were
fixed by provision, any additional shares that did not yet have an
explit ACL set would create world-writable files by default.

Administrators will need to manually correct the file permissions on
any additional shares that were created after installation of the AD
DC.

Andrew Bartlett

Reviewed-by: Michael Adam <obnox@samba.org>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Sun Mar 10 12:00:31 CET 2013 on sn-devel-104
(cherry picked from commit 287b5f6c0f40d3e3d09bc2ce80f5fee02cbae40f)

scripting/samba_upgradedns: Only look for IPv4/IPv6 addresses if we actually them

This allows the script to be used to create/remove the samba-specific dns-SERVER account
when we do not need to create the in-directory partition.

Andrew Bartlett

Reviewed-by: Stefan Metzmacher <metze@samba.org>
Autobuild-User(master): Stefan Metzmacher <metze@samba.org>
Autobuild-Date(master): Thu Jan 10 20:56:50 CET 2013 on sn-devel-104
(cherry picked from commit edbc26bca84ee77b5a9571ba8dc9416c0db25906)

samba-tool classicupgrade: Do not print the admin password during upgrade

This changes the code to only set and show a new password if no admin
user is found during the upgrade.

Andrew Bartlett

Reviewed-by: Stefan Metzmacher <metze@samba.org>
Autobuild-User(master): Stefan Metzmacher <metze@samba.org>
Autobuild-Date(master): Thu Jan 10 16:55:23 CET 2013 on sn-devel-104
(cherry picked from commit 051a1a9c6417c2cbffa7d091ae477a6c7922d363)

selftest: Add test for rfc2307 mapping handling

Reviewed-by: Stefan Metzmacher <metze@samba.org>
(cherry picked from commit c9d2ca585e198b1006bbf7f1a3c988c1188b66cb)

s4-idmap: Remove requirement that posixAccount or posixGroup be set for rfc2307

This change matches the source3/idmap/idmap_ad.c code, and allows this
feature to work with only the setting of the UID/GID in Active
Directory Users and Computers.

Andrew Bartlett

Reviewed-by: Stefan Metzmacher <metze@samba.org>
(cherry picked from commit 5e0fcb04a48d96669ed4376bfa17f679e3582236)

s4-dbcheck: Allow forcing an override of an old @MODULES record

Reviewed-by: Stefan Metzmacher <metze@samba.org>
(cherry picked from commit 99d872ee9261a299add4718c38234dfe9f7658fc)

build: Set LD_LIBRARY_PATH in install_with_python.sh

This ensures that the python install finishes correctly.

Andrew Bartlett

Reviewed-by: Stefan Metzmacher <metze@samba.org>
Autobuild-User(master): Stefan Metzmacher <metze@samba.org>
Autobuild-Date(master): Thu Jan 10 14:00:13 CET 2013 on sn-devel-104
(cherry picked from commit 213e7260a83d4349132e8c159798b476cec3f814)

smbd:posix_acls Remove incorrectly added lp_create_mask() and lp_dir_mask() calls

When 6adc7dad96b8c7366da042f0d93b28c1ecb092eb removed the calls to
lp_security_mask/lp_force_security_mode/lp_dir_security_mask/lp_force_dir_security_mode
these calls were replaced with lp_create_mask() and lp_dir_mask()

The issue is that while lp_security_mask() and lp_dir_security_mask defaulted to
0777, the replacement calls did not.  This changes behaviour, and incorrectly
prevents a posix mode being specified by the client from being applied to
the disk in the non-ACL enabled case.

Andrew Bartlett

Correct the name of the nss_winbind module for FreeBSD by creating a symlink from the FreeBSD required name to the built module.

Signed-off-by: Timur Bakeyev <timur@FreeBSD.org>
Reviewed-by: Andrew Bartlett <abartlett@samba.org>
Reviewed-by: Richard Sharpe <realrichardsharpe@gmail.com>
Autobuild-User(master): Richard Sharpe <sharpe@samba.org>
Autobuild-Date(master): Fri Mar  8 05:04:04 CET 2013 on sn-devel-104
(cherry picked from commit 11d128632357c9ae89d67aaf23c429fae83a1b29)

Fix bug #9704 - Fix nss_winbind name on FreeBSD.

Autobuild-User(v4-0-test): Karolin Seeger <kseeger@samba.org>
Autobuild-Date(v4-0-test): Tue Mar 12 10:43:32 CET 2013 on sn-devel-104

build: Do not force a specific perl from ${PERL} when running pod2man

pod2man should have the right #!/usr/bin/perl line already, and forcing it may
cause us to use the wrong perl.

Essentially treat this like any other system binary, rather than forcing
it to use the first perl we found.

This essentially reverts e80f576db66ad04592ac436ed74ceb9b96452060.
Current SLES11 does not seem to have the issue this was added to
address.

Andrew Bartlett

Reviewed-by: Michael Adam <obnox@samba.org>
Autobuild-User(master): Michael Adam <obnox@samba.org>
Autobuild-Date(master): Thu Feb 28 02:14:25 CET 2013 on sn-devel-104
(cherry picked from commit 1da22ab994822acacf9a0378c68b15cd32535390)

Fix bug #9703 - Fix build on solaris8: Do not force a specific perl on pod2man.

selftest: also skip raw.search as it also spins

Reviewed-by: Stefan Metzmacher <metze@samba.org>
(cherry picked from commit b40d134bc1866dd7e5b5e7dfc5bf01f6d55b1c1f)

The last 2 patches fix bug #9663 - make test hangs.

selftest: skip base.dir2 tests as they just spin on modern ext4

Reviewed-by: Stefan Metzmacher <metze@samba.org>
(cherry picked from commit a8c745a28c3278e9fbee6f802dc340fe660f27ca)

vfs_catia: add my copyright

Signed-off-by: Guenter Kukkukk <kukks@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
The last 5 patches address bug #9701 - vfs_catia is not working anymore (due to
a former regression).

Autobuild-User(v4-0-test): Karolin Seeger <kseeger@samba.org>
Autobuild-Date(v4-0-test): Mon Mar 11 10:55:48 CET 2013 on sn-devel-104

vfs_catia: add a sample for filename mapping to the manual page

Signed-off-by: Guenter Kukkukk <kukks@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

vfs_catia: fix the completely outdated manual page

The complete rework around 3.5.0 was not even mentioned somewhere.
(i was not able to find any info about that major change)

Signed-off-by: Guenter Kukkukk <kukks@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

vfs_catia: fix the translation to "vfs_translate_to_windows"

THANKS to an IRC user (Raimund ?) who asked for a char mapping possibility.
I suggested vfs_catia - but it did not work!
Hopefully now it will. :-)

Signed-off-by: Guenter Kukkukk <kukks@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

vfs_catia: add debug class for that module

Signed-off-by: Guenter Kukkukk <kukks@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

tsocket_bsd: Attempt to increase the SO_SNDBUF if we get EMSGSIZE in sendto()

This matches what was done for lib/socket/socket_unix.c in
c692bb02b039ae8fef6ba968fd13b36ad7d62a72.

(and is based on that patch by Landon Fuller <landonf@bikemonkey.org>)

Andrew Bartlett

Reviewed-by: Stefan Metzmacher <metze@samba.org>
Autobuild-User(master): Stefan Metzmacher <metze@samba.org>
Autobuild-Date(master): Mon Mar  4 11:15:35 CET 2013 on sn-devel-104
(cherry picked from commit 70e1b6185e3fb35fdc72eeb529ffb4b50122dc40)

The last 3 patches address bug #9697 - DsReplicaGetInfo fails due to sendto()
EMSGSIZE error on UNIX domain socket.

Autobuild-User(v4-0-test): Karolin Seeger <kseeger@samba.org>
Autobuild-Date(v4-0-test): Wed Mar  6 11:57:20 CET 2013 on sn-devel-104

s4-lib/socket: Return the original EMSGSIZE when sendto() and setsockopt() both fail

This ensures that should we be unable to increase the socket size, we return an
error that the application layer above might expect and be able to make
as reasonable response to (such as switching to a stream-based transport).

This fixes up c692bb02b039ae8fef6ba968fd13b36ad7d62a72.

As suggested by metze in https://bugzilla.samba.org/show_bug.cgi?id=9697#c4

Andrew Bartlett

Reviewed-by: Stefan Metzmacher <metze@samba.org>
(cherry picked from commit 50b42d1c5bb19e3a5050d7d23ac96e273d3974ee)

Handle EMSGSIZE on UNIX domain sockets.

On some systems (eg, FreeBSD) the default SO_SNDBUF for UNIX
domain sockets is to small, and EMSGSIZE is returned. Other
systems provide a larger default send buffer, but there is
still no guarantee that the buffer will be sized appropriately.

This patch modifies the sendto() path to attempt to resize
the SO_SNDBUF dynamically upon an EMSGSIZE failure, and then
retry the send.

This fixes local DCE/RPC errors on FreeBSD, eg:

https://lists.samba.org/archive/samba-technical/2013-January/089881.html

Signed-Off-By: Landon Fuller <landonf@bikemonkey.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Sat Mar  2 23:34:03 CET 2013 on sn-devel-104
(cherry picked from commit c692bb02b039ae8fef6ba968fd13b36ad7d62a72)

s4-socket: Make sure unix socket addresses are null terminated.

Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Günther Deschner <gd@samba.org>
(cherry picked from commit 7dd13503c0919766ecf3c8f5f48cda72aae8ac73)

Remove incomplete check for IPv6 link-local addresses.

This has been superceded by a check for link-local
addresses in get_interfaces()

Signed-Off-By: Landon Fuller <landonf@bikemonkey.org>
Reviewed-By: Richard Sharpe <realrichardsharpe@gmail.com>
Fix bug #9696 - Remove incomplete samba_dnsupdate IPv6 link-local address
check.

s3:lib/events: add missing TEVENT_TRACE_BEFORE/AFTER_WAIT handling

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Michael Adam <obnox@samba.org>
(cherry picked from commit 2fb69f51c47c8859485011879afb11650835ecbe)

The last 77 patches address bug #9695 - Backport tevent changes to bring
library to version 0.9.18.

Fix typo to make master/lib/tevent identical to 4.0.x/lib/tevent

This was missed in the patch that was applied to fix bug 9550.

Signed-off-by: Jeremy Allison <jra@samba.org>

tevent: change version to 0.9.18

This release contains a lot of fixes:

- Adding new timer events is now faster, if there's
  a large number of timer events.

- sigprocmask does not work on FreeBSD to stop further signals in a signal
  handler.

- TEVENT_NUM_SIGNALS is calculated by configure in order
  to support realtime signals on freebsd.

- ./configure --disable-python was fixed for the standalone build.

- Several crash bugs in the poll backend are fixed.

- The poll backend removes deleted events from the
  cached pollfd array now.

- The poll doesn't pass pollfd.events == 0 to poll()
  and maintains a list of disabled events,
  instead of consuming 100% cpu and/or triggering
  the callers handler.

- The poll backend detects POLLNVAL and reports EBADF
  instead of consuming 100% cpu.

- The select backend supports separate handlers
  for TEVENT_FD_READ and TEVENT_FD_WRITE.

- The poll and select backends are now doing fair
  queuing of fd events.

- The epoll has better error checking
  and supports separate handlers
  for TEVENT_FD_READ and TEVENT_FD_WRITE.

- The standard backend was rewritten to be a tiny
  wrapper on top of epoll with a fallback to poll,
  which means that it doesn't use select directly anymore.

- TEVENT_TRACE_BEFORE_LOOP_ONCE and TEVENT_TRACE_AFTER_LOOP_ONCE
  are added in order to allow the application to hook in
  before and after the loop_once() backend function is called.
  The TEVENT_HAS_LOOP_ONCE_TRACE_POINTS define can be used to
  detect the new feature.

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

tevent: optimize adding new zero timer events

Such events were used before we had immediate events.
It's likely that there're a lot of this events
and we need to add new ones in fifo order.

The tricky part is that tevent_common_add_timer()
should not use the optimization as it's used
by broken Samba versions, which don't use
tevent_common_loop_timer_delay() in source3/lib/events.c.

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

tevent: optimize adding new timer events

As new timestamps typically get higher:-)
it's better to traverse the existing list from
the tail.

This is not completely optimal, but it should be better
than before.

A second optimization could be done for zero timestamps,
we would just remember the last_zero_timer,
but that would change the internal ABI.
Normally thatshould not be a poblem, but the Samba's
source3/lib/events.c abuses tevent_internal.h
from the current source tree, even if an external tevent.h
is used. The other problem is that it makes use of
tevent_common_add_timer() without using
tevent_common_loop_timer_delay().

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

tevent: add a debug message in tevent_common_loop_timer_delay()

We should debug a message before and after running the handler.

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

tevent: add test_event_fd2()

This test fills the socket kernel buffers
and verifies that we don't report TEVENT_FD_WRITE
if the buffer is full.

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

tevent: add test_event_fd1()

This test verifies that TEVENT_FD_* flags are handled correctly.

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

tevent: use better names for the subtests

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

tevent: Regression test to ensure that a tevent backend can cope with separate read/write events on a single fd.

This tests the multiplex fd changes to the epoll backend to
ensure they work correctly.

Signed-off-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

tevent: use DEBUG_ERROR for the fallback message in epoll_panic()

A Samba autobuild passed without a fallback, so this is
really an error.

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

tevent: If epoll_ctl(..EPOLL_CTL_ADD,..) failes with EEXIST, merge the two fde's into one epoll event.

Signed-off-by: Jeremy Allison <jra@samba.org>
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

tevent: preferr the write handler if there're two possible handlers registered with epoll

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

tevent: In epoll_event_loop() ensure we trigger the right handler for a multiplexed fde event.

Signed-off-by: Jeremy Allison <jra@samba.org>
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

tevent: Add utility function epoll_handle_hup_or_err()

We'll use this to handle the EPOLL_ADDITIONAL_FD_FLAG_GOT_ERROR
and EPOLL_ADDITIONAL_FD_FLAG_REPORT_ERROR flags with multiplexed
events in the event loop.

Signed-off-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

tevent: handle EPOLL_ADDITIONAL_FD_FLAG_HAS_MPX in epoll_update_event()

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

tevent: handle multiplexed fde's in epoll_event_fd_destructor()

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

tevent: Fix epoll_mod_event() to cope with modifying a multiplexed fde event.

Signed-off-by: Jeremy Allison <jra@samba.org>
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

tevent: Fix up epoll_del_event to cope with deleting a multiplexed fde event.

Signed-off-by: Jeremy Allison <jra@samba.org>
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

tevent: handle multiplexed fde's in epoll_add_event()

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

tevent: Start to fix the epoll backend to support 2 fd events on the same fd correctly.

Add a utility function epoll_add_multiplex_fd() and
a new flag EPOLL_ADDITIONAL_FD_FLAG_HAS_MPX.

This will be called by epoll_add_event() to merge two
fde events with the same file descriptor.

Signed-off-by: Jeremy Allison <jra@samba.org>
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

tevent: ignore EBADF from epoll_ctl() and disable the event

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

tevent: unify handling of HAS_EVENT and REPORT_ERROR in epoll_{add,mod,del}_event()

epoll_{add,mod,del}_event() are only called via epoll_update_event()
and epoll_update_event() should not remove REPORT_ERROR itself.

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

tevent: always go through epoll_update_event()

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

tevent: s/epoll_change_event/epoll_update_event

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

tevent: the content of event is ignored by EPOLL_CTL_DEL

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

tevent: call epoll_panic() if EPOLL_CTL_DEL failed

Only ENOENT is ignored as it is no error.

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

tevent: use helper variables to hold the epoll_ctl() result

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

tevent: remove unused if (epoll_ev->epoll_fd == -1) return; checks

We'll never leave epoll_check_reopen() with epoll_fd == -1.

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

tevent: add std_event_loop_wait()

We also need to fallback in tevent_loop_wait()
otherwise we might miss events in the poll->fresh list.

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

tevent: don't call TALLOC_FREE(ev->additional_data) in std_fallback_to_poll()

The epoll backend has done the cleanup already.

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

tevent: avoid any operation on epoll_ev after a epoll_panic()

This calls TALLOC_FREE(ev->additional_data), which is epoll_ev
within epoll_panic() before calling the fallback handler.

In order to notice that a epoll_panic() happened, a caller can
register a pointer to a bool variable under epoll_ev->panic_state.

As epoll_check_reopen() can fail due to a epoll_panic(),
we need to force the replay flag if we have called any event handler.

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

tevent: add epoll_{create,ctl}_panic_fallback() for testing

This makes sure we only do random panics if a fallback handler
is registered.

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

tevent: use DLIST_DEMOTE() before calling on fde handler after select() and poll()

This makes sure we don't preferr events which are at the beginning
of the list.

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

tevent: debug a FATAL message on EBADF from the select backend

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

tevent: Fix multiple handler on the same fd bug in the tevent select backend.

When we're deciding what handlers to call in the select backend,
we didn't take into account the fact that the same fd may have
been added into the read FD_SET and the write FD_SET but with
different handlers.

We must match on both the file descriptor and the flags requested
before calling the handler.

Signed-off-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

tevent: revalidate fde->flags after poll()

This is important to avoid races between threads if the poll_mt
backend is used.

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

tevent: ignore POLLNVAL from poll() and disable the event

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

tevent: traverse the ev->fd_event list instead of the poll_ev->fds array

That has the same complexity but simplifies the code.

It also seems to fix a real problem, which was found
by the following test failure in Samba's autobuild,
while removing the epoll support from the "standard" backend.

    [402/1555 in 19m8s] samba3.raw.composite(plugin_s4_dc)
    smbtorture 4.1.0pre1-DEVELOPERBUILD
    Using seed 1361531675
    Testing savefile
    Testing parallel fetchfile with 10 ops
    waiting for completion
    maximum runtime exceeded for smbtorture - terminating
    UNEXPECTED(error): samba3.raw.composite.fetchfile (subunit.RemotedTestCase)(plugin_s4_dc)
    REASON: _StringException: _StringException: was started but never finished!
    UNEXPECTED(error): samba3.raw.composite.fetchfile(plugin_s4_dc) (subunit.RemotedTestCase)
    REASON: was started but never finished!

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

tevent: maintain a list of disabled events in the poll backend

We need to avoid passing pollfd.events == 0, to poll(),
as it will report POLLERR and POLLHUP events, but our caller
does not expect the event handler to be called.

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

tevent: merge poll_fresh_fde_destructor() into poll_event_fd_destructor()

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

tevent: use tevent_poll_event_add_fd_internal() in poll_event_add_fd()

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

tevent: make sure we cleanup the array passed to poll() after deleting an event

If we don't cleanup the array passed to poll after an
event was deleted, we may pass a bad file descriptor to poll().

This was found by the following test failure in Samba's
autobuild, while removing the epoll support from
the "standard" backend.

    [48/1555 in 4m37s] samba3.smbtorture_s3.plain(s3dc).LOCK4
    UNEXPECTED(failure): samba3.smbtorture_s3.plain(s3dc).LOCK4.smbtorture(s3dc)
    REASON: _StringException: _StringException: using seed 1361530718
    host=127.0.0.2 share=tmp user=metze myname=sn-devel-104
    Running LOCK4
    starting locktest4
    Failed to create file: NT_STATUS_INVALID_HANDLE
    finished locktest4
    TEST LOCK4 FAILED!
    LOCK4 took 190.492 secs

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

tevent: don't skip a fd event if the previous one was deleted during poll()

In a threaded environment it can happen that an tevent_fd is talloc_free'ed
while the main thread sleeps in the poll() syscall.

In such a case poll_event_fd_destructor() would set poll_ev->fdes[i] = NULL.

We then skip the removed event, but before we also skipped the one
that was located at the end of the array. We moved it to possition
'i', but the next loop uses 'i=i+1'.

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

tevent: remember the errno from select(), poll() and epoll_wait()

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>