</para>
</refsect1>
+ <refsect1>
+ <title>Recovery Lock</title>
+
+ <para>
+ CTDB uses a <emphasis>recovery lock</emphasis> to avoid a
+ <emphasis>split brain</emphasis>, where a cluster becomes
+ partitioned and each partition attempts to operate
+ independently. Issues that can result from a split brain
+ include file data corruption, because file locking metadata may
+ not be tracked correctly.
+ </para>
+
+ <para>
+ CTDB uses a <emphasis>cluster leader and follower</emphasis>
+ model of cluster management. All nodes in a cluster elect one
+ node to be the leader. The leader node coordinates privileged
+ operations such as database recovery and IP address failover.
+ CTDB refers to the leader node as the <emphasis>recovery
+ master</emphasis>. This node takes and holds the recovery lock
+ to assert its privileged role in the cluster.
+ </para>
+
+ <para>
+ By default, the recovery lock is implemented using a file
+ (specified by <parameter>CTDB_RECOVERY_LOCK</parameter>)
+ residing in shared storage (usually) on a cluster filesystem.
+ To support a recovery lock the cluster filesystem must support
+ lock coherence. See
+ <citerefentry><refentrytitle>ping_pong</refentrytitle>
+ <manvolnum>1</manvolnum></citerefentry> for more details.
+ </para>
+
+ <para>
+ The recovery lock can also be implemented using an arbitrary
+ cluster mutex call-out by using an exclamation point ('!') as
+ the first character of
+ <parameter>CTDB_RECOVERY_LOCK</parameter>. For example, a value
+ of <command>!/usr/local/bin/myhelper recovery</command> would
+ run the given helper with the specified arguments. See the
+ source code relating to cluster mutexes for clues about writing
+ call-outs.
+ </para>
+
+ <para>
+ If a cluster becomes partitioned (for example, due to a
+ communication failure) and a different recovery master is
+ elected by the nodes in each partition, then only one of these
+ recovery masters will be able to take the recovery lock. The
+ recovery master in the "losing" partition will not be able to
+ take the recovery lock and will be excluded from the cluster.
+ The nodes in the "losing" partition will elect each node in turn
+ as their recovery master so eventually all the nodes in that
+ partition will be excluded.
+ </para>
+
+ <para>
+ CTDB does sanity checks to ensure that the recovery lock is held
+ as expected.
+ </para>
+
+ <para>
+ CTDB can run without a recovery lock but this is not recommended
+ as there will be no protection from split brains.
+ </para>
+ </refsect1>
+
<refsect1>
<title>Private vs Public addresses</title>
<varname>CTDB_NODES</varname> configuration variable (see
<citerefentry><refentrytitle>ctdbd.conf</refentrytitle>
<manvolnum>5</manvolnum></citerefentry>, default
- <filename>/etc/ctdb/nodes</filename>). This file contains the
+ <filename>/usr/local/etc/ctdb/nodes</filename>). This file contains the
list of private addresses for all nodes in the cluster, one
per line. This file must be the same on all nodes in the
cluster.
<para>
It is strongly recommended that the private addresses are
configured on a private network that is separate from client
- networks.
+ networks. This is because the CTDB protocol is both
+ unauthenticated and unencrypted. If clients share the private
+ network then steps need to be taken to stop injection of
+ packets to relevant ports on the private addresses. It is
+ also likely that CTDB protocol traffic between nodes could
+ leak sensitive information if it can be intercepted.
</para>
<para>
- Example <filename>/etc/ctdb/nodes</filename> for a four node
+ Example <filename>/usr/local/etc/ctdb/nodes</filename> for a four node
cluster:
</para>
<screen format="linespecific">
configuration variable (see
<citerefentry><refentrytitle>ctdbd.conf</refentrytitle>
<manvolnum>5</manvolnum></citerefentry>, recommended
- <filename>/etc/ctdb/public_addresses</filename>). This file
+ <filename>/usr/local/etc/ctdb/public_addresses</filename>). This file
contains a list of the public addresses that the node is
capable of hosting, one per line. Each entry also contains
the netmask and the interface to which the address should be
</para>
<para>
- Example <filename>/etc/ctdb/public_addresses</filename> for a
+ Example <filename>/usr/local/etc/ctdb/public_addresses</filename> for a
node that can host 4 public addresses, on 2 different
interfaces:
</para>
Example: 4 nodes partitioned into two subgroups:
</para>
<screen format="linespecific">
-Node 0:/etc/ctdb/public_addresses
+Node 0:/usr/local/etc/ctdb/public_addresses
10.1.1.1/24 eth1
10.1.1.2/24 eth1
-Node 1:/etc/ctdb/public_addresses
+Node 1:/usr/local/etc/ctdb/public_addresses
10.1.1.1/24 eth1
10.1.1.2/24 eth1
-Node 2:/etc/ctdb/public_addresses
+Node 2:/usr/local/etc/ctdb/public_addresses
10.1.2.1/24 eth2
10.1.2.2/24 eth2
-Node 3:/etc/ctdb/public_addresses
+Node 3:/usr/local/etc/ctdb/public_addresses
10.1.2.1/24 eth2
10.1.2.2/24 eth2
</screen>
</listitem>
</varlistentry>
- <varlistentry>
- <term>LVS</term>
- <listitem>
- <para>
- Indicates that a node is configued in Linux Virtual Server
- (LVS) mode. In this mode the entire CTDB cluster uses one
- single public address for the entire cluster instead of
- using multiple public addresses in failover mode. This is
- an alternative to using a load-balancing layer-4 switch.
- See the <citetitle>LVS</citetitle> section for more
- details.
- </para>
- </listitem>
- </varlistentry>
-
- <varlistentry>
- <term>NATGW</term>
- <listitem>
- <para>
- Indicates that this node is configured to become the NAT
- gateway master in a NAT gateway group. See the
- <citetitle>NAT GATEWAY</citetitle> section for more
- details.
- </para>
- </listitem>
- </varlistentry>
-
</variablelist>
<para>
</para>
<para>
- In this mode the cluster selects a set of nodes in the cluster
- and loadbalance all client access to the LVS address across this
- set of nodes. This set of nodes are all LVS capable nodes that
- are HEALTHY, or if no HEALTHY nodes exists all LVS capable nodes
- regardless of health status. LVS will however never loadbalance
- traffic to nodes that are BANNED, STOPPED, DISABLED or
- DISCONNECTED. The <command>ctdb lvs</command> command is used to
- show which nodes are currently load-balanced across.
+ One extra LVS public address is assigned on the public network
+ to each LVS group. Each LVS group is a set of nodes in the
+ cluster that presents the same LVS address public address to the
+ outside world. Normally there would only be one LVS group
+ spanning an entire cluster, but in situations where one CTDB
+ cluster spans multiple physical sites it might be useful to have
+ one LVS group for each site. There can be multiple LVS groups
+ in a cluster but each node can only be member of one LVS group.
</para>
<para>
- One of the these nodes are elected as the LVSMASTER. This node
- receives all traffic from clients coming in to the LVS address
- and multiplexes it across the internal network to one of the
- nodes that LVS is using. When responding to the client, that
- node will send the data back directly to the client, bypassing
- the LVSMASTER node. The command <command>ctdb
- lvsmaster</command> will show which node is the current
- LVSMASTER.
+ Client access to the cluster is load-balanced across the HEALTHY
+ nodes in an LVS group. If no HEALTHY nodes exists then all
+ nodes in the group are used, regardless of health status. CTDB
+ will, however never load-balance LVS traffic to nodes that are
+ BANNED, STOPPED, DISABLED or DISCONNECTED. The <command>ctdb
+ lvs</command> command is used to show which nodes are currently
+ load-balanced across.
+ </para>
+
+ <para>
+ In each LVS group, one of the nodes is selected by CTDB to be
+ the LVS master. This node receives all traffic from clients
+ coming in to the LVS public address and multiplexes it across
+ the internal network to one of the nodes that LVS is using.
+ When responding to the client, that node will send the data back
+ directly to the client, bypassing the LVS master node. The
+ command <command>ctdb lvsmaster</command> will show which node
+ is the current LVS master.
</para>
<para>
</orderedlist>
</para>
- <para>
+ <para>
This means that all incoming traffic to the cluster will pass
through one physical node, which limits scalability. You can
send more data to the LVS address that one physical node can
requests are going through the LVSMASTER bottleneck and the
majority of the traffic volume (the data in the read replies)
goes straight from the processing node back to the clients. For
- read-intensive i/o patterns you can acheive very high throughput
+ read-intensive i/o patterns you can achieve very high throughput
rates in this mode.
</para>
<para>
To activate LVS on a CTDB node you must specify the
- <varname>CTDB_PUBLIC_INTERFACE</varname> and
- <varname>CTDB_LVS_PUBLIC_IP</varname> configuration variables.
- Setting the latter variable also enables the LVS capability on
- the node at startup.
+ <varname>CTDB_LVS_PUBLIC_IFACE</varname>,
+ <varname>CTDB_LVS_PUBLIC_IP</varname> and
+ <varname>CTDB_LVS_NODES</varname> configuration variables.
+ <varname>CTDB_LVS_NODES</varname> specifies a file containing
+ the private address of all nodes in the current node's LVS
+ group.
</para>
-
+
<para>
Example:
<screen format="linespecific">
-CTDB_PUBLIC_INTERFACE=eth1
+CTDB_LVS_PUBLIC_IFACE=eth1
CTDB_LVS_PUBLIC_IP=10.1.1.237
+CTDB_LVS_NODES=/usr/local/etc/ctdb/lvs_nodes
</screen>
</para>
+ <para>
+ Example <filename>/usr/local/etc/ctdb/lvs_nodes</filename>:
+ </para>
+ <screen format="linespecific">
+192.168.1.2
+192.168.1.3
+192.168.1.4
+ </screen>
+
+ <para>
+ Normally any node in an LVS group can act as the LVS master.
+ Nodes that are highly loaded due to other demands maybe
+ flagged with the "slave-only" option in the
+ <varname>CTDB_LVS_NODES</varname> file to limit the LVS
+ functionality of those nodes.
+ </para>
+
+ <para>
+ LVS nodes file that excludes 192.168.1.4 from being
+ the LVS master node:
+ </para>
+ <screen format="linespecific">
+192.168.1.2
+192.168.1.3
+192.168.1.4 slave-only
+ </screen>
+
</refsect2>
</refsect1>
-
+
+ <refsect1>
+ <title>TRACKING AND RESETTING TCP CONNECTIONS</title>
+
+ <para>
+ CTDB tracks TCP connections from clients to public IP addresses,
+ on known ports. When an IP address moves from one node to
+ another, all existing TCP connections to that IP address are
+ reset. The node taking over this IP address will also send
+ gratuitous ARPs (for IPv4, or neighbour advertisement, for
+ IPv6). This allows clients to reconnect quickly, rather than
+ waiting for TCP timeouts, which can be very long.
+ </para>
+
+ <para>
+ It is important that established TCP connections do not survive
+ a release and take of a public IP address on the same node.
+ Such connections can get out of sync with sequence and ACK
+ numbers, potentially causing a disruptive ACK storm.
+ </para>
+
+ </refsect1>
+
<refsect1>
<title>NAT GATEWAY</title>
NATGW is usually configured similar to the following example configuration:
</para>
<screen format="linespecific">
-CTDB_NATGW_NODES=/etc/ctdb/natgw_nodes
+CTDB_NATGW_NODES=/usr/local/etc/ctdb/natgw_nodes
CTDB_NATGW_PRIVATE_NETWORK=192.168.1.0/24
CTDB_NATGW_PUBLIC_IP=10.0.0.227/24
CTDB_NATGW_PUBLIC_IFACE=eth0
<para>
Normally any node in a NATGW group can act as the NATGW
master. Some configurations may have special nodes that lack
- connectivity to a public network. In such cases,
- <varname>CTDB_NATGW_SLAVE_ONLY</varname> can be used to limit the
- NATGW functionality of thos nodes.
+ connectivity to a public network. In such cases, those nodes
+ can be flagged with the "slave-only" option in the
+ <varname>CTDB_NATGW_NODES</varname> file to limit the NATGW
+ functionality of those nodes.
</para>
<para>
See the <citetitle>NAT GATEWAY</citetitle> section in
- <citerefentry><refentrytitle>ctdb.conf</refentrytitle>
+ <citerefentry><refentrytitle>ctdbd.conf</refentrytitle>
<manvolnum>5</manvolnum></citerefentry> for more details of
NATGW configuration.
</para>
</para>
<screen>
-/etc/ctdb/public_addresses:
+/usr/local/etc/ctdb/public_addresses:
192.168.1.98 eth2,eth3
192.168.1.99 eth2,eth3
-/etc/ctdb/policy_routing:
+/usr/local/etc/ctdb/policy_routing:
192.168.1.98 192.168.1.0/24
192.168.1.98 192.168.200.0/24 192.168.1.254
The specified script must be executable.
</para>
<para>
- Use of the provided <filename>/etc/ctdb/notify.sh</filename>
+ Use of the provided <filename>/usr/local/etc/ctdb/notify.sh</filename>
script is recommended. It executes files in
- <filename>/etc/ctdb/notify.d/</filename>.
+ <filename>/usr/local/etc/ctdb/notify.d/</filename>.
</para>
<para>
CTDB currently generates notifications after CTDB changes to
</para>
<simplelist>
- <member>EMERG (-3)</member>
- <member>ALERT (-2)</member>
- <member>CRIT (-1)</member>
- <member>ERR (0)</member>
- <member>WARNING (1)</member>
- <member>NOTICE (2)</member>
- <member>INFO (3)</member>
- <member>DEBUG (4)</member>
+ <member>ERR</member>
+ <member>WARNING</member>
+ <member>NOTICE</member>
+ <member>INFO</member>
+ <member>DEBUG</member>
</simplelist>
</refsect1>
<citerefentry><refentrytitle>ctdbd_wrapper</refentrytitle>
<manvolnum>1</manvolnum></citerefentry>,
+ <citerefentry><refentrytitle>ctdb_diagnostics</refentrytitle>
+ <manvolnum>1</manvolnum></citerefentry>,
+
<citerefentry><refentrytitle>ltdbtool</refentrytitle>
<manvolnum>1</manvolnum></citerefentry>,
<citerefentry><refentrytitle>ctdbd.conf</refentrytitle>
<manvolnum>5</manvolnum></citerefentry>,
+ <citerefentry><refentrytitle>ctdb-statistics</refentrytitle>
+ <manvolnum>7</manvolnum></citerefentry>,
+
<citerefentry><refentrytitle>ctdb-tunables</refentrytitle>
<manvolnum>7</manvolnum></citerefentry>,
git.samba.org / nivanova / samba-autobuild / .git / blobdiff