Problem
=======
-CTDB currently only supports exclusive Read-Write locks for clients(samba) accessing the TDB databases.
-This works well mostly, but when very mny number of clients are accessing the same file, at the same time,
-this will cause the exclusive lock as well as the record itself to rapidly bounce between nodes, and acts as
-a scalability limitation.
+CTDB currently only supports exclusive Read-Write locks for clients(samba) accessing the
+TDB databases.
+This mostly works well but when very many clients are accessing the same file,
+at the same time, this causes the exclusive lock as well as the record itself to
+rapidly bounce between nodes and acts as a scalability limitation.
-This primarily affects locking.tdb and brlock.tdb, two databases where record access is mostly read, and where a read request is magnitudes more common than read-write requests.
+This primarily affects locking.tdb and brlock.tdb, two databases where record access is
+read-mostly and where writes are semi-rare.
-For the common case, if CTDB provided shared non-exlusive Read-Only lock semantincs, this would greatly improve scaling for these workloads.
+For the common case, if CTDB provided shared non-exclusive Read-Only lock semantics
+this would greatly improve scaling for these workloads.
Desired properties
==================
-We can not make backward incompatible changes the ctdb/ltdb header for the records.
+We can not make backward incompatible changes the ctdb_ltdb header for the records.
-A Read-Only lock enabled ctdb demon must be able to interoperate with a non-Read-Only lock enbled daemon.
+A Read-Only lock enabled ctdb demon must be able to interoperate with a non-Read-Only
+lock enbled daemon.
-Getting a Read-Only look should not be slower than getting a Read-Write lock.
+Getting a Read-Only lock should not be slower than getting a Read-Write lock.
-Requesting a Read-Only lock should never trigger a record migration.
+When revoking Read-Only locks for a record, this should involve only those nodes that
+currently hold a Read-Only lock and should avoid broadcasting opportunistic revocations.
+(must track which nodes are delegated to)
-When revoking Read-Only locks for a record, this should involve only those nodes that hold a Read-Only lock right now and should avoid broadcasting opportunistic revocations. (must track which nodes are delegated to)
+When a Read-Write lock is requested, if there are Read-Only locks delegated to other
+nodes, the DMASTER will defer the record migration until all read-only locks are first
+revoked (synchronous revoke).
-When a Read-Write lock is requested, if there are Read-Only locks delegated to other nodes, the DMASTER will defer the record migration until all read-only locks are first revoked (synchronous revoke).
+Due to the cost of revoking Read-Only locks has on getting a Read-Write lock, the
+implementation should try to avoid creating Read-Only locks unless it has indication
+that there is contention. This may mean that even if client requests a Read-Only lock
+we might still provide a full Read-Write lock in order to avoid the cost of revoking
+the locks in some cases.
-Due to the cost of revoking Read-Only locks has on getting a Read-Write lock, the implementation should try to avoid
-creating Read-Only locks, unless it has indication that there is contention. This may mean that even if client requests a Read-Only lock we may still provide a full Read-Write lock in order to avoid the cost of revoking the locks in some cases.
+Read-Only locks require additional state to be stored in a separate database, containing
+information about which nodes have have been delegated Read-Only locks.
+This database should be kept at minimal size.
-Read-Only locks require additional state to be stored in a separate database, containing information about which nodes have have been delegated Read-Only locks. This database should be kept at minimal size.
-
-Read-Only locks should not significantly complicate the normal record/create/migration/deletion cycle for normal records.
+Read-Only locks should not significantly complicate the normal record
+create/migration/deletion cycle for normal records.
Read-Only locks should not complicate the recovery process.
We should avoid forking new child processes as far as possible from the main daemon.
-Client-side implementation, samba, libctdb, others, should have minimal impact when Read-Only locks are implemented.
-Client-side implementation must be possible with only minor conditionals added to the existing lock-check-fetch-unlock loop that clients use today for Read-Write locks. So that clients only need one single loop that can handle both Read-Write locking as well as Read-Only locking. Clients should not need two nearly identical loops.
+Client-side implementation, samba, libctdb, others, should have minimal impact when
+Read-Only locks are implemented.
+Client-side implementation must be possible with only minor conditionals added to the
+existing lock-check-fetch-unlock loop that clients use today for Read-Write locks. So
+that clients only need one single loop that can handle both Read-Write locking as well
+as Read-Only locking. Clients should not need two nearly identical loops.
Implementation
==============
-Four new flags are allocated in the ctdb/ltdb record header.
+Four new flags are allocated in the ctdb_ltdb record header.
HAVE_DELEGATIONS, HAVE_READONLY_LOCK, REVOKING_READONLY and REVOKE_COMPLETE
-HAVE_DELEGATIONS is a flag that can only be set on the node that is currently the DMASTER for the record. When set, this
-flag indicates that there are Read-Only locks delegated to other nodes in the cluster for this record.
-
-HAVE_READONLY is a flag that is only set on nodes that are NOT the DMASTER for the record. If set this flag
-indicates that this record contains an up-to-date Read-Only version of this record.
-A client that only needs to read, but not to write, the record can safely use the content of this record as is regardless of the value of the DMASTER field of the record.
-
-REVOKING_READONLY is a flag that is used while a set of read only delegations are being revoked.
-This flag is only set when HAVE_DELEGATIONS is also set, and is cleared at the same time as HAVE_DELEGATIONS is cleared.
-Normal operations is that first the HAVE_DELEGATIONS flag is set when the first delegation is generated.
-When the delegations are about to be revoked, the REVOKING_READONLY flag is set too.
+HAVE_DELEGATIONS is a flag that can only be set on the node that is currently the
+DMASTER for the record. When set, this flag indicates that there are Read-Only locks
+delegated to other nodes in the cluster for this record.
+
+HAVE_READONLY is a flag that is only set on nodes that are NOT the DMASTER for the
+record. If set this flag indicates that this record contains an up-to-date Read-Only
+version of this record. A client that only needs to read, but not to write, the record
+can safely use the content of this record as is regardless of the value of the DMASTER
+field of the record.
+
+REVOKING_READONLY is a flag that is used while a set of read only delegations are being
+revoked.
+This flag is only set when HAVE_DELEGATIONS is also set, and is cleared at the same time
+as HAVE_DELEGATIONS is cleared.
+Normal operations is that first the HAVE_DELEGATIONS flag is set when the first
+delegation is generated. When the delegations are about to be revoked, the
+REVOKING_READONLY flag is set too.
Once all delegations are revoked, both flags are cleared at the same time.
-While REVOKING_READONLY is set, any requests for the record, either normal request or request for readonly will be deferred.
-Deferred requests are linked to a list of deferred requests for the hash of the record until the time that the revokation is completed.
+While REVOKING_READONLY is set, any requests for the record, either normal request or
+request for readonly will be deferred.
+Deferred requests are linked on a list for deferred requests until the time that the
+revokation is completed.
This flags is set by the main ctdb daemon when it starts revoking this record.
REVOKE_COMPLETE
-The actual revoke of records is done by a child process, spawned from the ctdb amin daemon when it starts the process to revoke the records.
-Once the child process has finished revoking all delegations, it will set the flag REVOKE_COMPLETE for this record to signal to the master daemon that the record has been successfully revoked.
-At this stage the child process will also trigger an event in the main daemon that revoke is complete, and that the main dameon should start re-processing all deferred calls.
+The actual revoke of records is done by a child process, spawned from the main ctdb
+daemon when it starts the process to revoke the records.
+Once the child process has finished revoking all delegations it will set the flag
+REVOKE_COMPLETE for this record to signal to the main daemon that the record has been
+successfully revoked.
+At this stage the child process will also trigger an event in the main daemon that
+revoke is complete and that the main daemon should start re-processing all deferred
+requests.
-Once the revoke process is completed. There will be at least one deferred call to access this record, the initical call to for an exclusive fetch_lock() that triggered the revoke process to be started.
-In addition to nthis deferred call there may also be additional requests that have also become deferred while the revoke was in process. These can be either exclusive fetch_locks() or they can be other calls to request a new readonly lock on the record.
-Once the revoke is completed, the main daemon will reprocess all exclusive fetch_lock() requests immediately and respond to the clients.
-But any requests for readadonly locks will be deferred for an additional period of time before they are re-processed.
-This is to allow the client that needs a fetch_lock() to update the record to get some time to access and work on the record without having to compete with the possibly very many requests to get new readonly delegations created.
+Once the revoke process is completed there will be at least one deferred request to
+access this record. That is the initical call to for an exclusive fetch_lock() that
+triggered the revoke process to be started.
+In addition to this deferred request there may also be additional requests that have
+also become deferred while the revoke was in process. These can be either exclusive
+fetch_locks() or they can be readonly lock requests.
+Once the revoke is completed the main daemon will reprocess all exclusive fetch_lock()
+requests immediately and respond to these clients.
+Any requests for readadonly lock requests will be deferred for an additional period of
+time before they are re-processed.
+This is to allow the client that needs a fetch_lock() to update the record to get some
+time to access and work on the record without having to compete with the possibly
+very many readonly requests.
+The ctdb_db structure is expanded so that it contains one extra TDB database for each
+normal, non-persistent database.
+This new database is used for tracking delegations for the records.
+A record in the normal database that has "HAVE_DELEGATION" set will always have a
+corresponding record at the same key. This record contains the set of all nodes that
+the record is delegated to.
+This tracking database is lockless, using TDB_NOLOCK, and is only ever accessed by
+the main ctdbd daemon.
+The lockless nature and the fact that no other process ever access this TDB means we
+are guaranteed non-blocking access to records in the tracking database.
-The ctdbdb structure is expanded so that it contains one extra TDB database for each normal, non-persistent datbase.
-This new database is used for tracking delegations for the records. A record in the normal database that has "HAVE_DELEGATION" set will always have a corresponding record at the same key. This record contains the set of all nodes that the record is delegated to.
-This tracking database is lockless, using TDB_NOLOCK, and is only ever accessed by the main ctdbd daemon.
-The lockless nature and the fact that no other process ever access this TDB means we are guranteed non-blocking access to records in the trcking database.
+The ctdb_call PDU is allocated with a new flag WANT_READONLY and possibly also a new
+callid: CTDB_FETCH_WITH_HEADER_FUNC.
+This new function returns not only the record, as CTDB_FETCH_FUNC does, but also
+returns the full ctdb_ltdb record HEADER prepended to the record.
+This function is optional, clients that do not care what the header is can continue
+using just CTDB_FETCH_FUNC
-The ctdb_call PDU is allocated with two new flags WANT_READONLY and WITH_HEADER.
-This first flag is used to explicitely requesting a read-only record from the DMASTER/LMASTER.
-The second flag is used to request that the fetch operation will return not only the data for the record but also
-the record header.
-If the record does not yet exist, this is a returned as an error to the client and the client will retry the request loop.
-A new control is added to make remote nodes remove the HAVE_READONLY_LOCK from a record.
+This flag is used to requesting a read-only record from the DMASTER/LMASTER.
+If the record does not yet exist, this is a returned as an error to the client and the
+client will retry the request loop.
+
+A new control is added to make remote nodes remove the HAVE_READONLY_LOCK from a record
+and to invalidate any deferred readonly copies from the databases.
finished:
-where we basically spin, until the record is migrated onto the node and we have managed pinning it down.
+where we basically spin, until the record is migrated onto the node and we have managed
+to pin it down.
-This will change to instead do
+This will change to instead to something like
try_again:
lock record in tdb
goto finished
else
unlock record
- ask ctdb for read-only copy (WANT_READONLY|WITH_HEADER)
+ ask ctdb for read-only copy (WANT_READONLY[|WITH_HEADER])
if failed to get read-only copy (*A)
ask ctdb to migrate the record onto the node
goto try_again
lock record in tdb
- if record fails RSN test
- unlock record
- ask ctdb to migrate the record onto the node
- goto try_again
- write the updated record from ctdb to tdb
goto finished
unlock record
finished:
-If the record does not yet exist in the local TDB, we always perform a full fetch for a Read-Write lock, even if only a Read-Only lock ws requested.
-This means that for first access, we use the the current grab a Read-Write lock, just like we do today.
-This creates the record and migrates it onto the node. The local node becomes DMASTER for the record.
-This did not yet trigger a Read-Only delegation to be created.
-Future reference to this record by the local samba daemons will still access/lock the record locally without triggereing a Read-Only delegation to be created since the record is already hosted on the local node as DMASTER.
-
-Only if the record is contended, i.e. it has been created but we are no longer the DMASTER for the record, only for this case will we create a Read-Only delegation for this record.
-
-This heuristics provide a mechanism where we will not create Read-Only delegations until we have some indication that the record may be contended.
-
-This avoids creating and revoking Read-Only delegations when only a single client is repeatedly accessing the same set of records.
+If the record does not yet exist in the local TDB, we always perform a full fetch for a
+Read-Write lock even if only a Read-Only lock was requested.
+This means that for first access we always grab a Read-Write lock and thus upgrade any
+requests for Read-Only locks into a Read-Write request.
+This creates the record, migrates it onto the node and makes the local node become
+the DMASTER for the record.
+
+Future reference to this same record by the local samba daemons will still access/lock
+the record locally without triggereing a Read-Only delegation to be created since the
+record is already hosted on the local node as DMASTER.
+
+Only if the record is contended, i.e. it has been created an migrated onto the node but
+we are no longer the DMASTER for this record, only for this case will we create a
+Read-Only delegation.
+This heuristics provide a mechanism where we will not create Read-Only delegations until
+we have some indication that the record may be contended.
+
+This avoids creating and revoking Read-Only delegations when only a single client is
+repeatedly accessing the same set of records.
This also aims to limit the size of the tracking tdb.
-Note that writing the copy of the Read-Only record to the TDB database is done by the client, not by ctdbd. This is to avoid a probable need for creating a child process for a likely contended record where locking the record would likely block.
-
Server implementation
=====================
When receiving a ctdb_call with the WANT_READONLY flag:
-If this is the LMASTER for the record and the record does not yet exist, LMASTER will return an error back to the client (*A above) and the client will try to recover. In particular, LMASTER will not create a new record for this case.
+If this is the LMASTER for the record and the record does not yet exist, LMASTER will
+return an error back to the client (*A above) and the client will try to recover.
+In particular, LMASTER will not create a new record for this case.
-If this is the LMASTER for the record and the record exists, the PDU will be forwrded to the DMASTER for the record.
+If this is the LMASTER for the record and the record exists, the PDU will be forwarded to
+the DMASTER for the record.
-If this node is not the DMASTER for this record, we forward the PDU back to the LMASTER. Just as we always do today.
+If this node is not the DMASTER for this record, we forward the PDU back to the
+LMASTER. Just as we always do today.
-If this is the DMASTER for the record, we need to create a Read-Only delegation. This is done by
+If this is the DMASTER for the record, we need to create a Read-Only delegation.
+This is done by
lock record
increase the RSN by one for this record
set the HAVE_DELEGATIONS flag for the record
When receiving a ctdb_call without the WANT_READONLY flag:
-If this is the DMASTER for the this might trigger a migration. So,
-IF the record has HAVE_DELEGATIONS set, we create a child process and defer processing of this PDU until the child process has completed.
+If this is the DMASTER for the this might trigger a migration. If there exists
+delegations we must first revoke these before allowing the Read-Write request from
+proceeding. So,
+IF the record has HAVE_DELEGATIONS set, we create a child process and defer processing
+of this PDU until the child process has completed.
-From the child process we will call out to all nodes that have delegations for this record and tell them to invalidate this record by clearing the HAVE_READONLY_LOCK from the record.
-Once all delegated nodes respond back, the child process signals back to the main daemon the revoke has completed.
-(child process may not access the tracking tdb since it is lockless)
+From the child process we will call out to all nodes that have delegations for this
+record and tell them to invalidate this record by clearing the HAVE_READONLY_LOCK from
+the record.
+Once all delegated nodes respond back, the child process signals back to the main daemon
+the revoke has completed. (child process may not access the tracking tdb since it is
+lockless)
Main process is triggered to re-process the PDU once the child process has finished.
-Main daemon deletes the corresponding record in the tracking database, clears the HAVE_DELEGATIONS flag for the record and then proceeds to perform the migration as usual.
+Main daemon deletes the corresponding record in the tracking database, clears the
+HAVE_DELEGATIONS flag for the record and then proceeds to perform the migration as usual.
-When receiving a ctdb_call without the flag we want all delegations to be revoked, so we must take care that the delegations are revoked unconditionally before we even check if we are already the DMASTER (in which case thie ctdb_call would normally just be no-op (*B below))
+When receiving a ctdb_call without the flag we want all delegations to be revoked,
+so we must take care that the delegations are revoked unconditionally before we even
+check if we are already the DMASTER (in which case the ctdb_call would normally just
+be no-op (*B below))
Recovery process changes
========================
-A recovery implicitely clears/revokes any read only records and delegations from all databases.
+A recovery implicitly clears/revokes any read only records and delegations from all
+databases.
-During delegations of Read-Only locks, this is done in such way that delegated records will have a RSN smaller than the DMASTER. This guarantees that read-only copies always have a RSN that is smaller than the DMASTER.
+During delegations of Read-Only locks, this is done in such way that delegated records
+will have a RSN smaller than the DMASTER. This guarantees that read-only copies always
+have a RSN that is smaller than the DMASTER.
-During recoveries we do not need to take any special action other than always picking the copy of the record that has the highest RSN, which is what we already do today.
+During recoveries we do not need to take any special action other than always picking
+the copy of the record that has the highest RSN, which is what we already do today.
-During the recovery process, we strip all flags off all records while writing the new contnent of the database during the PUSH_DB control.
+During the recovery process, we strip all flags off all records while writing the new
+content of the database during the PUSH_DB control.
-During processing of the PUSH_DB control and once the new database has been written we then also wipe the tracking database.
+During processing of the PUSH_DB control and once the new database has been written we
+then also wipe the tracking database.
This makes changes to the recovery process minimal and nonintrusive.
Vacuuming needs only minimal changes.
-When vacuuming runs, it will do a fetch_lock to migrate any remote records back onto the LMASTER before the record can be purged. This will automatically force all delegations for that record to be revoked before the migration is copied back onto the LMASTER.
-This handles the case where LMASTER is not the DMASTER for the record that will be purged.
-The migration here does force any delegations to be revoked before the migration takes place.
+When vacuuming runs, it will do a fetch_lock to migrate any remote records back onto the
+LMASTER before the record can be purged. This will automatically force all delegations
+for that record to be revoked before the migration is copied back onto the LMASTER.
+This handles the case where LMASTER is not the DMASTER for the record that will be
+purged.
+The migration in this case does force any delegations to be revoked before the
+vacuuming takes place.
Missing is the case when delegations exist and the LMASTER is also the DMASTER.
-For this case we need to change the vacuuming to unconditionally always try to do a fetch_lock when HAVE_DELEGATIONS is set, even if the record is already stored locally. (*B)
-This fetch lock will not cause any migrations by the ctdb daemon, but since it does not have the WANT_READONLY
-this will still force the delegations to be revoked but no migrations trigger.
+For this case we need to change the vacuuming to unconditionally always try to do a
+fetch_lock when HAVE_DELEGATIONS is set, even if the record is already stored locally.
+(*B)
+This fetch lock will not cause any migrations by the ctdb daemon, but since it does
+not have the WANT_READONLY this will still force the delegations to be revoked but no
+migration will trigger.
Traversal process
Non-readonly enabled daemons fetching records from Readonly enabled daemons:
Non-readonly enabled daemons do not know, and never set the WANT_READONLY flag so these daemons will always request a full migration for a full fetch-lock for all records. Thus a request from a non-readonly enabled daemon will always cause any existing delegations to be immediately revoked. Access will work but performance may be harmed since there will be a lot of revoking of delegations.
-Readonly enabled dameons fetching records with WANT_READONLY from non-readonly enabled daemons:
-Non-readonly enabled daemons ingore the WANT_READONLY flag and never return delegations. They always return a full record migration.
+Readonly enabled daemons fetching records with WANT_READONLY from non-readonly enabled daemons:
+Non-readonly enabled daemons ignore the WANT_READONLY flag and never return delegations. They always return a full record migration.
Full record migration is allowed by the protocol, even if the originator only requests the 'hint' WANT_READONLY,
so this access also interoperates between daemons with different capabilities.
git.samba.org / vlendec / samba-autobuild / .git / blobdiff