ctdb-recoverd: Add parallel database recovery helper

ctdb-recoverd: Add parallel database recovery helper

Parallel database recovery fixes the samba/ctdb deadlock during recovery.

Many times samba tries to grab multiple record locks in sequence.
Consider a case when samba is already holding a record lock on a database
and tries to get a record lock on second database.  If the second record
is not available on the local node, samba asks ctdb to migrate the record.
If recovery occurs at this time (e.g. node becoming inactive), ctdb
cannot freeze all the databases since samba is already holding a lock
and waiting for the second lock.  CTDB can process the second record
request only after the recovery is complete, thus causing a deadlock.

In parallel database recovery, each database is frozen and recovered
independent from each other.  So as soon as the second database is
recovered, CTDB will resend all the pending migration requests and Samba
can get the second lock.  Once samba releases both the locks, ctdb can
freeze the first database and recover it completing recovery process.

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>