s3/smbd: fix deferred open with streams and kernel oplocks

s3/smbd: fix deferred open with streams and kernel oplocks

I noticed smbd can get stuck in an open() call with kernel oplocks
enabled and named streams (provided by vfs_streams_xattr):

- client opens a file and with an exclusive oplock

- client starts writing to the file

- client opens an existing stream of the file

- the smbd process gets stuck in an open()

What happens is:

we had setup a locking.tdb record watch in defer_open(), the watch was
triggered, we reattempted the open and got stuck in a blocking open
because the oplock holder (ourselves) hadn't given up the oplock yet.

Cf e576bf5310bc9de9686a71539e9a1b60b4fba5cc for the commit that added
the kernel oplock retry logic. tldr: with kernel oplocks the first open
is non-blocking, but the second one is blocking.

Detailed analysis follows.

When opening a named stream of a file, Samba internally opens the
underlying "base" file first. This internal open of the basefile suceeds
and does *not* trigger an oplock break (because it is an internal open
that doesn't call open() at all) but it is added as an entry to the
locking.tdb record of the file.

Next, the stream open ends up in streams_xattr where a non-blocking
open() on the base file is called. This open fails with EWOULDBLOCK
because we have another fd with a kernel oplock on the file.

So we call defer_open() which sets up a watch on the locking.tdb record.

In the subsequent error unwinding code in open_file_ntcreate() and
callers we close the internal open file handle of the basefile which
also removes the entry from the locking.tdb record and so *changes the
record*.

This fires the record watch and in the callback defer_open_done() we
don't check whether the condition (oplock gone) we're interested in is
actually met. The callback blindly reschedules the open request with
schedule_deferred_open_message_smb().

schedule_deferred_open_message_smb() schedules an immediate tevent event
which has precedence over the IPC fd events in messaging, so the open is
always (!) reattempted before processing the oplock break message.

As explained above, this second open will be a blocking one so we get
stuck in a blocking open.

It doesn't help to make all opens non-blocking, that would just result
in a busy loop failing the open, as we never process the oplock break
message (remember, schedule_deferred_open_message_smb() used immediate
tevent events).

To fix this we must add some logic to the record watch callback to check
whether the record watch was done for a kernel oplock file and if yes,
check if the oplock state changed. If not, simply reschedule the
deferred open and keep waiting.

This logic is only needed for kernel oplocks, not for Samba-level
oplocks, because there's no risk of deadlocking, the worst that can
happen is a rescheduled open that fails again in the oplock checks and
gets deferred again.

Bug: https://bugzilla.samba.org/show_bug.cgi?id=7537

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>