RECOVERY - The cluster databases have all been frozen, pausing all services while the cluster awaits a recovery process to complete. A recovery process should finish within seconds. If a cluster is stuck in the RECOVERY state this would indicate a cluster malfunction which needs to be investigated.
</para>
<para>
- Once the recovery master detects an inconsistency, for example a node
+ Once the leader detects an inconsistency, for example a node
becomes disconnected/connected, the recovery daemon will trigger a
cluster recovery process, where all databases are remerged across the
- cluster. When this process starts, the recovery master will first
+ cluster. When this process starts, the leader will first
"freeze" all databases to prevent applications such as samba from
accessing the databases and it will also mark the recovery mode as
RECOVERY.
</para>
</refsect3>
<refsect3>
- <title>Recovery master</title>
+ <title>Leader</title>
<para>
- This is the cluster node that is currently designated as the recovery master. This node is responsible of monitoring the consistency of the cluster and to perform the actual recovery process when reqired.
+ This is the cluster node that is currently designated as the
+ leader. This node is responsible of monitoring the
+ consistency of the cluster and to perform the actual
+ recovery process when reqired.
</para>
<para>
- Only one node at a time can be the designated recovery master. Which
- node is designated the recovery master is decided by an election
+ Only one node at a time can be the designated leader. Which
+ node is designated the leader is decided by an election
process in the recovery daemons running on each node.
</para>
</refsect3>
hash:2 lmaster:2
hash:3 lmaster:3
Recovery mode:NORMAL (0)
-Recovery master:0
+Leader:0
</screen>
</refsect3>
</refsect2>
</refsect2>
<refsect2>
- <title>recmaster</title>
+ <title>leader</title>
<para>
- This command shows the pnn of the node which is currently the recmaster.
+ This command shows the pnn of the node which is currently the leader.
</para>
<para>
Example output:
</para>
<screen>
-RECMASTER: YES
+LEADER: YES
LMASTER: YES
</screen>
</refsect2>
<refsect2>
- <title>setrecmasterrole on|off</title>
+ <title>setleaderrole on|off</title>
<para>
- This command is used to enable/disable the RECMASTER capability for a node at runtime. This capability determines whether or not a node can be used as an RECMASTER for the cluster. A node that does not have the RECMASTER capability can not win a recmaster election. A node that already is the recmaster for the cluster when the capability is stripped off the node will remain the recmaster until the next cluster election.
+ This command is used to enable/disable the LEADER capability
+ for a node at runtime. This capability determines whether or
+ not a node can be elected leader of the cluster. A node that
+ does not have the LEADER capability can not be elected
+ leader. If the current leader has this capability removed then
+ an election will occur.
</para>
<para>
- Nodes will by default have this capability, but it can be stripped off nodes by the setting in the sysconfig file or by using this command.
+ Nodes have this capability enabled by default, but it can be
+ removed via the <command>cluster:leader capability</command>
+ configuration setting or by using this command.
</para>
<para>
See also "ctdb getcapabilities"
<refsect2>
<title>ipreallocate, sync</title>
<para>
- This command will force the recovery master to perform a full ip reallocation process and redistribute all ip addresses. This is useful to "reset" the allocations back to its default state if they have been changed using the "moveip" command. While a "recover" will also perform this reallocation, a recovery is much more hevyweight since it will also rebuild all the databases.
+ This command will force the leader to perform a full ip
+ reallocation process and redistribute all ip addresses. This
+ is useful to "reset" the allocations back to its default state
+ if they have been changed using the "moveip" command. While a
+ "recover" will also perform this reallocation, a recovery is
+ much more hevyweight since it will also rebuild all the
+ databases.
</para>
</refsect2>
# Recovery can under certain circumstances lead to old record copies
# resurrecting: Recovery selects the newest record copy purely by RSN. At
-# the end of the recovery, the recovery master is the dmaster for all
+# the end of the recovery, the leader is the dmaster for all
# records in all (non-persistent) databases. And the other nodes locally
# hold the complete copy of the databases. The bug is that the recovery
-# process does not increment the RSN on the recovery master at the end of
-# the recovery. Now clients acting directly on the Recovery master will
-# directly change a record's content on the recmaster without migration
+# process does not increment the RSN on the leader at the end of
+# the recovery. Now clients acting directly on the leader will
+# directly change a record's content on the leader without migration
# and hence without RSN bump. So a subsequent recovery can not tell that
-# the recmaster's copy is newer than the copies on the other nodes, since
-# their RSN is the same. Hence, if the recmaster is not node 0 (or more
+# the leader's copy is newer than the copies on the other nodes, since
+# their RSN is the same. Hence, if the leader is not node 0 (or more
# precisely not the active node with the lowest node number), the recovery
# will choose copies from nodes with lower number and stick to these.
# 1. Create a test database
-# 2. Add a record with value value1 on recovery master
+# 2. Add a record with value value1 on leader
# 3. Force a recovery
-# 4. Update the record with value value2 on recovery master
+# 4. Update the record with value value2 on leader
# 5. Force a recovery
# 6. Confirm that the value is value2
status=0
-# Make sure node 0 is not the recovery master
-echo "find out which node is recmaster"
-ctdb_onnode 0 recmaster
-recmaster="$out"
-if [ "$recmaster" = "0" ]; then
- echo "node 0 is recmaster, disable recmasterrole on node 0"
+# Make sure node 0 is not the leader
+echo "find out which node is leader"
+ctdb_onnode 0 leader
+leader="$out"
+if [ "$leader" = "0" ]; then
+ echo "node 0 is leader, disable leader role on node 0"
#
# Note:
- # It should be sufficient to run "ctdb setrecmasterrole off"
+ # It should be sufficient to run "ctdb setleaderrole off"
# on node 0 and wait for election and recovery to finish.
# But there were problems related to this in this automatic
# test, so for now use "ctdb stop" and "ctdb continue".
try_command_on_node 0 $CTDB continue
wait_until_node_has_status 0 notstopped
- ctdb_onnode 0 recmaster
- recmaster="$out"
- if [ "$recmaster" = "0" ]; then
- echo "failed to move recmaster to different node"
+ ctdb_onnode 0 leader
+ leader="$out"
+ if [ "$leader" = "0" ]; then
+ echo "failed to move leader to different node"
exit 1
fi
fi
-echo "Recmaster:$recmaster"
+echo "Leader:${leader}"
# Create a temporary non-persistent database to test with
echo "create test database $TESTDB"
-try_command_on_node $recmaster $CTDB attach $TESTDB
+ctdb_onnode "$leader" attach "$TESTDB"
# Wipe Test database
echo "wipe test database"
-try_command_on_node $recmaster $CTDB wipedb $TESTDB
+ctdb_onnode "$leader" wipedb "$TESTDB"
# Add a record key=test1 data=value1
echo "store key(test1) data(value1)"
-try_command_on_node $recmaster $CTDB writekey $TESTDB test1 value1
+ctdb_onnode "$leader" writekey "$TESTDB" test1 value1
# Fetch a record key=test1
echo "read key(test1)"
-try_command_on_node $recmaster $CTDB readkey $TESTDB test1
+ctdb_onnode "$leader" readkey "$TESTDB" test1
cat "$outfile"
# Do a recovery
echo "force recovery"
-try_command_on_node $recmaster $CTDB recover
+ctdb_onnode "$leader" recover
-wait_until_node_has_status $recmaster recovered
+wait_until_node_has_status "$leader" recovered
# Add a record key=test1 data=value2
echo "store key(test1) data(value2)"
-try_command_on_node $recmaster $CTDB writekey $TESTDB test1 value2
+ctdb_onnode "$leader" writekey "$TESTDB" test1 value2
# Fetch a record key=test1
echo "read key(test1)"
-try_command_on_node $recmaster $CTDB readkey $TESTDB test1
+ctdb_onnode "$leader" readkey "$TESTDB" test1
cat "$outfile"
# Do a recovery
echo "force recovery"
-try_command_on_node $recmaster $CTDB recover
+ctdb_onnode "$leader" recover
-wait_until_node_has_status $recmaster recovered
+wait_until_node_has_status "$leader" recovered
# Verify record key=test1
echo "read key(test1)"
-try_command_on_node $recmaster $CTDB readkey $TESTDB test1
+ctdb_onnode "$leader" readkey "$TESTDB" test1
cat "$outfile"
if [ "$out" = "Data: size:6 ptr:[value2]" ]; then
echo "GOOD: Recovery did not corrupt database"
if (ret == ETIMEDOUT) {
ret = 0;
} else if (ret != 0) {
- fprintf(stderr, "Error getting recovery master\n");
+ fprintf(stderr, "Error getting leader\n");
return ret;
}
}
}
-static void print_status(TALLOC_CTX *mem_ctx, struct ctdb_context *ctdb,
- struct ctdb_node_map *nodemap, uint32_t mypnn,
- struct ctdb_vnn_map *vnnmap, int recmode,
- uint32_t recmaster)
+static void print_status(TALLOC_CTX *mem_ctx,
+ struct ctdb_context *ctdb,
+ struct ctdb_node_map *nodemap,
+ uint32_t mypnn,
+ struct ctdb_vnn_map *vnnmap,
+ int recmode,
+ uint32_t leader)
{
unsigned int i;
printf("Recovery mode:%s (%d)\n",
recmode == CTDB_RECOVERY_NORMAL ? "NORMAL" : "RECOVERY",
recmode);
- printf("Recovery master:");
- print_pnn(recmaster);
+ printf("Leader:");
+ print_pnn(leader);
}
static int control_status(TALLOC_CTX *mem_ctx, struct ctdb_context *ctdb,
struct ctdb_node_map *nodemap;
struct ctdb_vnn_map *vnnmap;
int recmode;
- uint32_t recmaster;
+ uint32_t leader;
int ret;
if (argc != 0) {
return ret;
}
- ret = get_leader(mem_ctx, ctdb, &recmaster);
+ ret = get_leader(mem_ctx, ctdb, &leader);
if (ret != 0) {
return ret;
}
- print_status(mem_ctx, ctdb, nodemap, ctdb->cmd_pnn, vnnmap,
- recmode, recmaster);
+ print_status(mem_ctx,
+ ctdb,
+ nodemap,
+ ctdb->cmd_pnn,
+ vnnmap,
+ recmode,
+ leader);
return 0;
}
if (options.machinereadable == 1) {
printf("%s%s%s%s%s\n",
options.sep,
- "RECMASTER", options.sep,
+ "LEADER", options.sep,
"LMASTER", options.sep);
printf("%s%d%s%d%s\n", options.sep,
!! (caps & CTDB_CAP_RECMASTER), options.sep,
!! (caps & CTDB_CAP_LMASTER), options.sep);
} else {
- printf("RECMASTER: %s\n",
+ printf("LEADER: %s\n",
(caps & CTDB_CAP_RECMASTER) ? "YES" : "NO");
printf("LMASTER: %s\n",
(caps & CTDB_CAP_LMASTER) ? "YES" : "NO");
static int get_generation(TALLOC_CTX *mem_ctx, struct ctdb_context *ctdb,
uint32_t *generation)
{
- uint32_t recmaster;
+ uint32_t leader;
int recmode;
struct ctdb_vnn_map *vnnmap;
int ret;
again:
- ret = get_leader(mem_ctx, ctdb, &recmaster);
+ ret = get_leader(mem_ctx, ctdb, &leader);
if (ret != 0) {
- fprintf(stderr, "Failed to find recovery master\n");
+ fprintf(stderr, "Failed to find leader\n");
return ret;
}
- ret = ctdb_ctrl_get_recmode(mem_ctx, ctdb->ev, ctdb->client,
- recmaster, TIMEOUT(), &recmode);
+ ret = ctdb_ctrl_get_recmode(mem_ctx,
+ ctdb->ev,
+ ctdb->client,
+ leader,
+ TIMEOUT(),
+ &recmode);
if (ret != 0) {
- fprintf(stderr, "Failed to get recovery mode from node %u\n",
- recmaster);
+ fprintf(stderr,
+ "Failed to get recovery mode from node %u\n",
+ leader);
return ret;
}
goto again;
}
- ret = ctdb_ctrl_getvnnmap(mem_ctx, ctdb->ev, ctdb->client,
- recmaster, TIMEOUT(), &vnnmap);
+ ret = ctdb_ctrl_getvnnmap(mem_ctx,
+ ctdb->ev,
+ ctdb->client,
+ leader,
+ TIMEOUT(),
+ &vnnmap);
if (ret != 0) {
- fprintf(stderr, "Failed to get generation from node %u\n",
- recmaster);
+ fprintf(stderr,
+ "Failed to get generation from node %u\n",
+ leader);
return ret;
}
CTDB_BROADCAST_CONNECTED, pnn);
if (ret != 0) {
fprintf(stderr,
- "Failed to ask recovery master to distribute IPs\n");
+ "Failed to ask leader to distribute IPs\n");
return ret;
}
return ret;
}
-static int control_recmaster(TALLOC_CTX *mem_ctx, struct ctdb_context *ctdb,
- int argc, const char **argv)
+static int control_leader(TALLOC_CTX *mem_ctx,
+ struct ctdb_context *ctdb,
+ int argc,
+ const char **argv)
{
- uint32_t recmaster;
+ uint32_t leader;
int ret;
- ret = get_leader(mem_ctx, ctdb, &recmaster);
+ ret = get_leader(mem_ctx, ctdb, &leader);
if (ret != 0) {
return ret;
}
- print_pnn(recmaster);
+ print_pnn(leader);
return 0;
}
return 0;
}
-static int control_setrecmasterrole(TALLOC_CTX *mem_ctx,
- struct ctdb_context *ctdb,
- int argc, const char **argv)
+static int control_setleaderrole(TALLOC_CTX *mem_ctx,
+ struct ctdb_context *ctdb,
+ int argc,
+ const char **argv)
{
- uint32_t recmasterrole = 0;
+ uint32_t leaderrole = 0;
int ret;
if (argc != 1) {
- usage("setrecmasterrole");
+ usage("setleaderrole");
}
if (strcmp(argv[0], "on") == 0) {
- recmasterrole = 1;
+ leaderrole = 1;
} else if (strcmp(argv[0], "off") == 0) {
- recmasterrole = 0;
+ leaderrole = 0;
} else {
- usage("setrecmasterrole");
+ usage("setleaderrole");
}
- ret = ctdb_ctrl_set_recmasterrole(mem_ctx, ctdb->ev, ctdb->client,
- ctdb->cmd_pnn, TIMEOUT(),
- recmasterrole);
+ ret = ctdb_ctrl_set_recmasterrole(mem_ctx,
+ ctdb->ev,
+ ctdb->client,
+ ctdb->cmd_pnn,
+ TIMEOUT(),
+ leaderrole);
if (ret != 0) {
return ret;
}
"dump database from a backup file", "<file>" },
{ "wipedb", control_wipedb, false, false,
"wipe the contents of a database.", "<dbname|dbid>"},
- { "recmaster", control_recmaster, false, true,
- "show the pnn for the recovery master", NULL },
+ { "leader", control_leader, false, true,
+ "show the pnn of the leader", NULL },
{ "event", control_event, true, false,
"event and event script commands", NULL },
{ "scriptstatus", control_scriptstatus, true, false,
"get recovery lock file", NULL },
{ "setlmasterrole", control_setlmasterrole, false, true,
"set LMASTER role", "on|off" },
- { "setrecmasterrole", control_setrecmasterrole, false, true,
- "set RECMASTER role", "on|off"},
+ { "setleaderrole", control_setleaderrole, false, true,
+ "set LEADER role", "on|off"},
{ "setdbreadonly", control_setdbreadonly, false, true,
"enable readonly records", "<dbname|dbid>" },
{ "setdbsticky", control_setdbsticky, false, true,
git.samba.org / samba.git / commitdiff