selftest: knownfail updates after Heimdal Upgrade

The Heimdal upgrade brings the new feature of FAST, allowing more tests to pass.

However it causes a regression in FL2003 for the returned salt format in
the AS-REP, but FL 2003 has not been the default since Samba 4.2 as AES
keys are much stronger and should be preferred.

NOTE: THIS COMMIT WON'T COMPILE/WORK ON ITS OWN!

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

selftest: Update SimpleKerberosTests now that Samba supports FAST

Heimdal matches Windows in this respect

NOTE: THIS COMMIT WON'T COMPILE/WORK ON ITS OWN!

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

tests/krb5: Add option to check reply padata

So far we have only been checking padata in error replies and with FAST.
We should also check it in the general success case.

NOTE: THIS COMMIT WON'T COMPILE/WORK ON ITS OWN!

Signed-off-by: Joseph Sutton <josephsutton@catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

s4:kdc: Return PA-SUPPORTED-ENCTYPES

NOTE: THIS COMMIT WON'T COMPILE/WORK ON ITS OWN!

Signed-off-by: Joseph Sutton <josephsutton@catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

s4:kdc: Set supported enctypes in KDC entry

This allows us to return the supported enctypes to the client as
PA-SUPPORTED-ENCTYPES padata.

NOTE: THIS COMMIT WON'T COMPILE/WORK ON ITS OWN!

Signed-off-by: Joseph Sutton <josephsutton@catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

s4:kdc: Add PAC_ATTRIBUTES integration for Heimdal

NOTE: THIS COMMIT WON'T COMPILE/WORK ON ITS OWN!

Pair-Programmed-With: Stefan Metzmacher <metze@samba.org>

Signed-off-by: Joseph Sutton <josephsutton@catalyst.net.nz>
Signed-off-by: Stefan Metzmacher <metze@samba.org>

s4:kdc: Set require_pac and no-ENC_TS in FAST for new Heimdal import

This allows us to continue to avoid CVE-2020-25719 in particular
and pass our tests for expected FAST behaviour as the patches
we requested by upstream to be conditional, not hard-coded.

NOTE: THIS COMMIT WON'T COMPILE/WORK ON ITS OWN!

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

s4:kdc/heimdal: Always include the salt in the PA-ETYPE-INFO[2]

This matches Windows and is detected by our samba.tests.krb5.as_canonicalization_tests
test as this always expects the salt, which Windows always provides.

NOTE: THIS COMMIT WON'T COMPILE/WORK ON ITS OWN!

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

s4:kdc: cope with upstream rename of configuration parameters.

This copes with the upstream commit:

commit c757eb7fb04a9b0ca883ddb72c1bc75bf5d814f3
Author: Nicolas Williams <nico@cryptonector.com>
Date:   Fri Nov 25 17:21:04 2011 -0600

    Rename and fix as/tgs-use-strongest-key config parameters

        Different ticket session key enctype selection options should
        distinguish between target principal type (krbtgt vs. not), not
        between KDC request types.

NOTE: THIS COMMIT WON'T COMPILE/WORK ON ITS OWN!

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
[abartlet@samba.org Researched and updated the commit message]
Reviewed-by: Stefan Metzmacher <metze@samba.org>

s4:kdc: Move calls using the samba4 name to be right after each other

These all need to be in sync

NOTE: THIS COMMIT WON'T COMPILE/WORK ON ITS OWN!

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

s4:kdc: Adapt KDC to new Heimdal to load samba4 HDB plugin for keytab

NOTE: THIS COMMIT WON'T COMPILE/WORK ON ITS OWN!

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

s4:kdc/hdb: Store and retrieve a FX-COOKIE value

Note Windows uses the string "MICROSOFT" as cookie,
so it's wrong to have a per DC cookie, but we need to
adjust the Heimdal logic to support that.

NOTE: THIS COMMIT WON'T COMPILE/WORK ON ITS OWN!

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

s4:kdc: Set entry.flags.force_canonicalize to override the new Heimdal behaviour

This is needed to give hdb_samba4 the full control over the returned
principal, rather than the new code in the Heimdal KDC.

Including changes selected from code by Stefan Metzmacher <metze@samba.org>
in his Heimdal upgrade branch.

NOTE: THIS COMMIT WON'T COMPILE/WORK ON ITS OWN!

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

s4:kerberos: adapt the heimdal send_to_kdc hooks to the send_to_kdc/realm plugin interface

With the recent heimdal upgrade we better try to use the send_to_realm()
hooks as it allows us to handle the KDC lookup as well as only getting
each logical request just once in the testing code, which makes it
let dependend on the heimdal internal kdc lookup logic.

NOTE: THIS COMMIT WON'T COMPILE/WORK ON ITS OWN!

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Joseph Sutton <josephsutton@catalyst.net.nz>

s4:kerberos: adjust smb_krb5_debug_wrapper() to embedded heimdal

In future we need a real configure check for Heimdal 8.

NOTE: THIS COMMIT WON'T COMPILE/WORK ON ITS OWN!

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

tests/auth_log: adjust expected authDescription for test_smb_bad_user

With NO_SUCH_USER we don't know if any pre-authentication was requested,
so with the new Heimdal code we now used use "AS-REQ" instead of
assuming ENC-TS Pre-authentication.

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Joseph Sutton <josephsutton@catalyst.net.nz>

s4:kdc: Update to match updated Heimdal's new HDB version

Including updates to hook into the improved hdb_auth_status
by Stefan Metzmacher <metze@samba.org> from his Heimdal
upgrade branch.

NOTE: THIS COMMIT WON'T COMPILE/WORK ON ITS OWN!

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

s4:kdc: Adapt to use new combined windc interface in lorikeet-heimdal

This interface is as requested by Luke Howard towards possibly merging
this feature.

NOTE: THIS COMMIT WON'T COMPILE/WORK ON ITS OWN!

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

s4:kdc: Adapt wamba_wdc_check_client_access() to modern Heimdal

Modern Heimdal falls back to kdc_check_flags() internally
when KRB5_PLUGIN_NO_HANDLE is returned, avoiding the need
to call back into the internal KDC APIs.

Selected from patch by by Stefan Metzmacher <metze@samba.org>
from his Heimdal upgrade branch.

NOTE: THIS COMMIT WON'T COMPILE/WORK ON ITS OWN!

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

s4:kdc: Adapt samba_wdc_check_client_access() to upstream Heimdal

NOTE: THIS COMMIT WON'T COMPILE/WORK ON ITS OWN!

Signed-off-by: Joseph Sutton <josephsutton@catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

s4:kdc: Update samba_wdc_check_client_access() to match updated Heimdal

This based on a patch in Debian by Samuel Cabrero <scabrero@zentyal.com> in Debian.

NOTE: THIS COMMIT WON'T COMPILE/WORK ON ITS OWN!

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

s4:kdc: Do not encode the NTSTATUS error into a PA-DATA, just linearlise it

This allows another routine to do the wrapping.

NOTE: THIS COMMIT WON'T COMPILE/WORK ON ITS OWN!

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

s4:kdc: Fix build failure by including <heimbase.h>

NOTE: THIS COMMIT WON'T COMPILE/WORK ON ITS OWN!

Signed-off-by: Joseph Sutton <josephsutton@catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

tests: Update latin1 list and ignored file list for new Heimdal import

NOTE: THIS COMMIT WON'T COMPILE/WORK ON ITS OWN!

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

s4:heimdal_build: changes required to build after import

For libtommath we do this by using the list from makefile.commo
in in libtommath rather than trying to match the list by hand.

This will be easier to maintain over the long term.

Thanks to work over many years by:
 - Gary Lockyer <gary@catalyst.net.nz>
 - Stefan Metzmacher <metze@samba.org>
 - Andrew Bartlett <abartlet@samba.org>

NOTE: THIS COMMIT WON'T COMPILE/WORK ON ITS OWN!

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

s4:heimdal: import lorikeet-heimdal-202201172009 (commit 5a0b45cd723628b3690ea848548b05771c40f14e)

See
https://git.samba.org/?p=lorikeet-heimdal.git;a=shortlog;h=refs/heads/lorikeet-heimdal-202201172009
or
https://gitlab.com/samba-team/devel/lorikeet-heimdal/-/tree/lorikeet-heimdal-202201172009

NOTE: THIS COMMIT WON'T COMPILE/WORK ON ITS OWN!

Pair-Programmed-With: Joseph Sutton <josephsutton@catalyst.net.nz>

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Signed-off-by: Joseph Sutton <josephsutton@catalyst.net.nz>

s4:heimdal_build: include heimdal headers relative to heimdal_build

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Joseph Sutton <josephsutton@catalyst.net.nz>

netlogon.idl: Add FAST support bits

Signed-off-by: Joseph Sutton <josephsutton@catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

gitlab-ci: Use Fedora 34 for Coverity Scan

The Coverity Scan tools are not updated very often and miss support for the
latest gcc build. Lets use Fedora 34 for that and stay behind a bit.

Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
Autobuild-User(master): Andreas Schneider <asn@cryptomilk.org>
Autobuild-Date(master): Wed Jan 19 10:49:18 UTC 2022 on sn-devel-184

smbd: Remove a duplicate protoype

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Tue Jan 18 21:17:43 UTC 2022 on sn-devel-184

lib: Remove unused asprintf_strupper_m()

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

winbindd: Replace asprintf() with talloc_asprintf()

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

libads: Convert sitename_key() to talloc

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

net: Align a few integer types

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

libsmb: Avoid a cast

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

smbd: Align a few integer types

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

smbd: Fix a typo

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

torture3: Align two integer types

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

rpc_host: We have tevent_req_oom() for ENOMEM

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

lib: Remove unused tstream_npa_socketpair()

This was used in the pre samba-dcerpcd source3 rpc server.

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

lib: Save a few lines with str_list_add_printf()

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

lib: Save a few lines with str_list_add_printf()

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

smbd: Save a few lines with str_list_add_printf()

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

printing: Save a few lines with str_list_add_printf()

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

profile3: remove an unused include

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

s4:kdc: improve DEBUG messages in samba_wdc_reget_pac2()

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Joseph Sutton <josephsutton@catalyst.net.nz>
Autobuild-User(master): Joseph Sutton <jsutton@samba.org>
Autobuild-Date(master): Mon Jan 17 20:55:41 UTC 2022 on sn-devel-184

s4:auth: debug make_user_info_dc_pac() failures in kerberos_pac_to_user_info_dc()

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Joseph Sutton <josephsutton@catalyst.net.nz>

s4:torture: check for pac_blob==NULL in test_generate_session_info_pac() functions

We should return an error instead of crashing for tickets without a PAC.

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Joseph Sutton <josephsutton@catalyst.net.nz>

s4:heimdal_build: make version_script optional to HEIMDAL_LIBRARY()

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Joseph Sutton <josephsutton@catalyst.net.nz>

kdc: Fix leak

Signed-off-by: Joseph Sutton <josephsutton@catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

tests/krb5: Update supported enctype checking

We now do not expect the claims or compound ID bits to be set unless
explicitly specified, nor the DES bits.

Signed-off-by: Joseph Sutton <josephsutton@catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

tests/krb5: Add AS-REQ PAC tests

Signed-off-by: Joseph Sutton <josephsutton@catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

tests/krb5: Check encrypted-pa-data if present

Signed-off-by: Joseph Sutton <josephsutton@catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

tests/krb5: Add FAST enc-pa-rep tests

Signed-off-by: Joseph Sutton <josephsutton@catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

tests/krb5: Adjust expected error codes

Signed-off-by: Joseph Sutton <josephsutton@catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

tests/krb5: Generate unique UPNs for AS-REQ enterprise tests

This helps to avoid problems with account creation due to UPN uniqueness
constraints.

Signed-off-by: Joseph Sutton <josephsutton@catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

s4:torture: Remove netbios realm and lowercase realm tests

Tests for these are already present in
samba.tests.krb5.as_canonicalization_tests. These tests cause problems
with an upgraded Heimdal version, and we want to stop supporting
non-canonical realm names, so this commit removes them.

Signed-off-by: Joseph Sutton <josephsutton@catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

s4:torture: Make etype list variables static

If they are not made static, these variables end up being used by the
Kerberos libraries after they have gone out of scope.

Signed-off-by: Joseph Sutton <josephsutton@catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

build: reduce printf() calls in generated build_options.c

build_options.c is inefficient in multiple ways:
1) it's generated via one python fp.write() call per line
2) the generated code calls output() for each and every build option

This commit addresses (2), modifying write_build_options_header() and
write_build_options_footer(). write_build_options_section() could also
be collapsed into a single output() call, but this may lead to oversize
string literals, so has been left as is.

I observe no change in smbd --build-options output.

Signed-off-by: David Disseldorp <ddiss@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
Autobuild-User(master): David Disseldorp <ddiss@samba.org>
Autobuild-Date(master): Mon Jan 17 13:17:53 UTC 2022 on sn-devel-184

build: reduce fp.write calls for build_options.c generation

build_options.c is inefficient in multiple ways:
1) it's generated via one python fp.write() call per line
2) the generated code calls output() for each and every build option

This commit reduces fp.write() calls for (1). I observe no change in the
generated build_options.c .

Signed-off-by: David Disseldorp <ddiss@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>

s3:smbd: handle --build-options without parsing smb.conf

The smb.conf is parsed in post mode of a popt callback. The smbd
--build-options parameter should be handled when first encountered
to avoid requiring smb.conf presence.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=14945

Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: David Disseldorp <ddiss@samba.org>

WHATSNEW: Document CTDB leader and cluster lock changes

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Autobuild-User(master): Martin Schwenke <martins@samba.org>
Autobuild-Date(master): Mon Jan 17 11:16:14 UTC 2022 on sn-devel-184

ctdb-doc: Remove documentation for recovery process

This is many years out of date and recent changes make it worse.  It
is unlikely that anyone has the time to fix this in the near future,
so remove it because it is misleading.

Database recovery steps are well documented in comments in the
recovery helper.  Cluster monitoring documentation can be re-added
when things stop changing.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-doc: Update example configuration migration script

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-tests: Improve test coverage for leader role yield and elections

Rename test, clean up node selection.  Duplicate for for banning and
removing leader capability cases.  Repeat all 3 tests without cluster
lock.

All of the standard election triggers are now tested, with and without
cluster lock.  Due to test cluster configuration limitations, the
tests without cluster lock are skipped on a real cluster.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-tests: Support commenting out local daemons configuration options

Can be used to disable default options, such as cluster lock.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-config: Add configuration option [cluster] leader timeout

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-config: [legacy] recmaster capability -> [cluster] leader capability

Rename this configuration item and move it into the [cluster]
configuration section.

Update documentation to match.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-config: [cluster] recovery lock -> [cluster] cluster lock

Retain "recovery lock" and mark as deprecated for backward
compatibility.

Some documentation is still inconsistent.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-doc: Update documentation for leader and cluster lock

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-recoverd: Use race for cluster lock as election when lock is enabled

If the cluster is partitioned then nodes in one partition can not take
the lock anyway, so election is pointless.  It just introduces
unnecessary corner cases.

Instead just race for the lock.

When a node notices a lack of leader and notifies other nodes of an
election via an unknown leader broadcast, the cluster lock election is
hooked into this broadcast.

The test needs to be updated because losing the cluster lock can now
result in a leadership change.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-protocol: Mark {GET,SET}_RECMASTER controls obsolete

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-protocol: Drop marshalling for {GET,SET}_RECMASTER controls

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-daemon: Drop implementation of {GET,SET}_RECMASTER controls

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-protocol: Drop protocol client functions for recmaster controls

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-client: Drop unused recmaster functions

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-daemon: Drop unused old client recmaster functions

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-recoverd: Drop calls to ctdb_ctrl_setrecmaster()

Nothing fetches this value anymore.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-recoverd: Drop recovery master verification

This doesn't make sense if leader broadcasts are used.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-tools: recovery master -> leader

The following command names are changed:

  recmaster -> leader
  setrecmasterrole -> setleaderrole

Command output changed for the following commands:

  status
  getcapabilities

Documentation and tests are updated to reflect these changes.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-tools: Use leader broadcast in get_leader()

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-tools: Factor out get_leader()

This seems pointless but it localises a subsequent change and also
starts a terminology change in the tool code.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-tools: Handle leader broadcasts in ctdb tool

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-tools: Print "UNKNOWN" when leader PNN is unknown

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-client: Factor out function ctdb_client_wait_func_timeout()

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-tests: Factor out getting leader and waiting for leader change

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-tests: Add leader broadcasts to fake_ctdbd

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-tests: Implement srvid_handler for dispatching messages

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-recoverd: Simplify some stopped/banned checks to inactive checks

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-recoverd: No longer take cluster lock during recovery

Confirm instead that it is already held.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-recoverd: Add and use function cluster_lock_enabled()

Now all references to ctdb->recovery_lock are encapsulated in the
cluster lock code.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-recoverd: Terminology change: recovery lock -> cluster lock

No functional changes, just name changes for clarity.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-recoverd: Take cluster lock when election completes

It is no longer just a recovery lock but is always held by the cluster
leader.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-recoverd: Factor out function cluster_lock_take()

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-tests: Avoid a race

See the comment in the code for details.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-tests: Setup cluster with expected arguments

ctdb_test_init() doesn't actually pass arguments to local_daemons.sh.
This needs to be done using ctdb_nodes_start_custom().

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-recoverd: Drop leader validation

The introduction of the leader broadcast timeout provides an
alternative to the current leader validation.  Using the leader
broadcast may not be as fast but it is more correct.

When the leader node is stopped or banned, the only way of triggering
an election is currently to fetch the leader's node map to check
whether the it is still active.  This is because the leader will no
longer push the node map to other nodes.  However, having all nodes
fetch the node map from an inactive leader may be unreliable.

Most of the other cases are also handled more reliably by the leader
broadcast timeout.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-recoverd: Drop special case for elected-before-connected

This no longer occurs at startup due to the leader broadcast timeout.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-recoverd: Handle leader broadcast timeout

If no leader broadcasts have been received from the leader for more
than 5s then trigger an election.

Apart from being sane behaviour, this avoids elected-before-connected
bugs at startup, where a node elects itself leader before it is
connected to other nodes.

When a node processes a leader broadcast timeout it sends an unknown
leader broadcast to all nodes.  That causes cancellation of the leader
broadcast timeout across the cluster.  This is particular important at
startup, since nodes may be started in a staggered fashion.  Without
this cluster-wide cancellation, a node might notice the lack of
leader, win an election and complete a recovery before other nodes
notice the lack of leader.  When the leader broadcast timeout finally
occurs on the other nodes then they'll put the cluster back into an
unnecessary recovery.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb-recoverd: Send leader broadcasts

These are triggered on 1 second timer, but are only sent if the node
is the current leader and there is no election underway.

If this node can not be the leader then ensure it releases the
recovery lock.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>