<para>
This file contains CTDB configuration variables that are affect
the operation of CTDB. The default location of this file is
- <filename>/etc/ctdb/ctdbd.conf</filename>.
+ <filename>/usr/local/etc/ctdb/ctdbd.conf</filename>.
</para>
<para>
<filename>/etc/default/ctdb</filename> (Debian). However, these
files should be reserved for variables used by the initscript.
A historical alternative is
- <filename>/etc/ctdb/sysconfig/ctdb</filename> - this is
+ <filename>/usr/local/etc/ctdb/sysconfig/ctdb</filename> - this is
deprecated.
</para>
</para>
<para>
- Default is <filename>/var/run/ctdb/ctdbd.pid</filename>.
+ Default is <filename>/usr/local/var/run/ctdb/ctdbd.pid</filename>.
Corresponds to <option>--pidfile</option>.
</para>
</listitem>
</listitem>
</varlistentry>
- <varlistentry>
- <term>CTDB_VARDIR=<parameter>DIRECTORY</parameter></term>
- <listitem>
- <para>
- DIRECTORY containing CTDB files that are modified at
- runtime.
- </para>
- <para>
- Defaults to <filename>/var/ctdb</filename>, unless
- <filename>/var/lib/ctdb</filename> already exists in which
- case it is used.
- </para>
- </listitem>
- </varlistentry>
-
-
</variablelist>
</refsect1>
<term>CTDB_DBDIR=<parameter>DIRECTORY</parameter></term>
<listitem>
<para>
- Defaults to <varname>CTDB_VARDIR</varname>. Corresponds to
- <option>--dbdir</option>.
+ Defaults to <filename>/usr/local/var/lib/ctdb</filename>.
+ </para>
+ <para>
+ Corresponds to <option>--dbdir</option>.
</para>
</listitem>
</varlistentry>
<term>CTDB_DBDIR_PERSISTENT=<parameter>DIRECTORY</parameter></term>
<listitem>
<para>
- Defaults to <varname>CTDB_VARDIR</varname>/persistent.
+ Defaults to <filename>/usr/local/var/lib/ctdb/persistent</filename>.
+ </para>
+ <para>
Corresponds to <option>--dbdir-persistent</option>.
</para>
</listitem>
<term>CTDB_DBDIR_STATE=<parameter>DIRECTORY</parameter></term>
<listitem>
<para>
- Defaults to <varname>CTDB_VARDIR</varname>/state.
+ Defaults to <filename>/usr/local/var/lib/ctdb/state</filename>.
+ </para>
+ <para>
Corresponds to <option>--dbdir-state</option>.
</para>
</listitem>
<term>CTDB_DEBUGLEVEL=<parameter>DEBUGLEVEL</parameter></term>
<listitem>
<para>
- Default is ERR (0). Corresponds to <option>-d</option> or
+ Default is NOTICE (2). Corresponds to <option>-d</option> or
<option>--debug</option>.
</para>
</listitem>
<listitem>
<para>
Default is <varname>CTDB_BASE</varname>/events.d, so usually
- <filename>/etc/ctdb/events.d</filename>. Corresponds to
+ <filename>/usr/local/etc/ctdb/events.d</filename>. Corresponds to
<option>--event-script-dir</option>.
</para>
</listitem>
<listitem>
<para>
STRING specifies where ctdbd will write its log. The
- default is file:<filename>/var/log/log.ctdb</filename> or
- similar - the prefix may differ depending on how CTDB was
- built. Corresponds to <option>--logging</option>.
+ default is
+ file:<filename>/usr/local/var/log/log.ctdb</filename>.
+ Corresponds to <option>--logging</option>.
</para>
<para>
Valid values are:
<listitem>
<para>
FILENAME where ctdbd will write its log. This is usually
- <filename>/var/log/log.ctdb</filename>.
+ <filename>/usr/local/var/log/log.ctdb</filename>.
</para>
</listitem>
</varlistentry>
<listitem>
<para>
Default is <varname>CTDB_BASE</varname>/nodes, so usually
- <filename>/etc/ctdb/nodes</filename>. Corresponds to
+ <filename>/usr/local/etc/ctdb/nodes</filename>. Corresponds to
<option>--nlist</option>.
</para>
</listitem>
<listitem>
<para>
No default, usually
- <filename>/etc/ctdb/notify.sh</filename>. Corresponds to
+ <filename>/usr/local/etc/ctdb/notify.sh</filename>. Corresponds to
<option>--notification-script</option>.
</para>
</listitem>
</listitem>
</varlistentry>
+ <varlistentry>
+ <term>CTDB_NODE_ADDRESS=<parameter>FILENAME</parameter></term>
+ <listitem>
+ <para>
+ IPADDR is the private IP address that ctdbd will bind to.
+ Corresponds to <option>--listen</option>.
+ </para>
+ <para>
+ By default ctdbd will select the first address from the
+ nodes list that in can bind to. See also
+ <citetitle>CTDB_NODES</citetitle>.
+ </para>
+ <para>
+ This option is only required when automatic address
+ detection can not be used. This can be the case when
+ running multiple ctdbd daemons/nodes on the same physical
+ host (usually for testing), using InfiniBand for the
+ private network or on Linux when sysctl
+ net.ipv4.ip_nonlocal_bind=1.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term>CTDB_PUBLIC_ADDRESSES=<parameter>FILENAME</parameter></term>
<listitem>
<para>
No default, usually
- <filename>/etc/ctdb/public_addresses</filename>.
+ <filename>/usr/local/etc/ctdb/public_addresses</filename>.
Corresponds to <option>--public-addresses</option>.
</para>
</listitem>
</para>
<para>
No default, usually
- <filename>/etc/ctdb/natgw_nodes</filename> when enabled.
+ <filename>/usr/local/etc/ctdb/natgw_nodes</filename> when enabled.
</para>
</listitem>
</varlistentry>
<refsect3>
<title>Example</title>
<screen>
-CTDB_NATGW_NODES=/etc/ctdb/natgw_nodes
+CTDB_NATGW_NODES=/usr/local/etc/ctdb/natgw_nodes
CTDB_NATGW_PRIVATE_NETWORK=192.168.1.0/24
CTDB_NATGW_DEFAULT_GATEWAY=10.0.0.1
CTDB_NATGW_PUBLIC_IP=10.0.0.227/24
always reachable would look like this:
</para>
<screen>
-CTDB_NATGW_NODES=/etc/ctdb/natgw_nodes
+CTDB_NATGW_NODES=/usr/local/etc/ctdb/natgw_nodes
CTDB_NATGW_PRIVATE_NETWORK=192.168.1.0/24
CTDB_NATGW_PUBLIC_IP=10.0.0.227/24
CTDB_NATGW_PUBLIC_IFACE=eth0
<para>
No default, usually
- <filename>/etc/ctdb/policy_routing</filename> when enabled.
+ <filename>/usr/local/etc/ctdb/policy_routing</filename> when enabled.
</para>
</listitem>
</varlistentry>
<refsect3>
<title>Example</title>
<screen>
-CTDB_PER_IP_ROUTING_CONF=/etc/ctdb/policy_routing
+CTDB_PER_IP_ROUTING_CONF=/usr/local/etc/ctdb/policy_routing
CTDB_PER_IP_ROUTING_RULE_PREF=100
CTDB_PER_IP_ROUTING_TABLE_ID_LOW=1000
CTDB_PER_IP_ROUTING_TABLE_ID_HIGH=9000
<title>NFS</title>
<para>
- This includes parameters for the kernel NFS server and the
- user-space
- <ulink url="https://github.com/nfs-ganesha/nfs-ganesha/wiki">NFS-Ganesha</ulink>
- server.
+ This includes parameters for the kernel NFS server.
+ Alternative NFS subsystems (such as <ulink
+ url="https://github.com/nfs-ganesha/nfs-ganesha/wiki">NFS-Ganesha</ulink>)
+ can be integrated using <varname>CTDB_NFS_CALLOUT</varname>.
</para>
<refsect3>
- <title>Eventscripts</title>
+ <title>Eventscript</title>
<simplelist>
<member><filename>60.nfs</filename></member>
- <member><filename>60.ganesha</filename></member>
</simplelist>
</refsect3>
</listitem>
</varlistentry>
- <varlistentry>
- <term>CTDB_GANESHA_REC_SUBDIR<parameter>SUBDIR</parameter></term>
- <listitem>
- <para>
- SUBDIR is the name of a top-level subdirectory in the
- first cluster filesystem. This subdirectory is used to
- allow communication between NFS-Ganesha and the
- <filename>60.ganesha</filename> script.
- </para>
- <para>
- Default is ".ganesha".
- </para>
- </listitem>
- </varlistentry>
-
<varlistentry>
<term>CTDB_MANAGES_NFS=yes|no</term>
<listitem>
</varlistentry>
<varlistentry>
- <term>CTDB_MONITOR_NFS_THREAD_COUNT=yes|no</term>
- <listitem>
- <para>
- Whether to monitor the NFS kernel server thread count.
- </para>
- <para>
- This works around a limitation in some NFS initscripts
- where some threads can be stuck in host filesystem calls
- (perhaps due to slow storage), a restart occurs, some
- threads don't exit, the start only adds the missing
- number of threads, the stuck threads exit, and the
- result is a lower than expected thread count. Note that
- if you must also set <varname>RPCNFSDCOUNT</varname>
- (RedHat/Debian) or <varname>USE_KERNEL_NFSD_NUMBER</varname>
- (SUSE) in your NFS configuration so the monitoring code
- knows how many threads there should be - if neither of
- these are set then this option will be ignored.
- </para>
- <para>
- Default is no.
- </para>
- </listitem>
- </varlistentry>
-
- <varlistentry>
- <term>CTDB_NFS_DUMP_STUCK_THREADS=<parameter>NUM</parameter></term>
- <listitem>
- <para>
- NUM is the number of NFS kernel server threads to dump
- stack traces for if some are still alive after stopping
- NFS during a restart.
- </para>
- <para>
- Default is 5. Set this to 0 to disable this feature.
- </para>
- </listitem>
- </varlistentry>
-
- <varlistentry>
- <term>CTDB_NFS_SERVER_MODE=kernel|ganesha</term>
+ <term>CTDB_NFS_CALLOUT=<parameter>COMMAND</parameter></term>
<listitem>
<para>
- Selects which NFS server to be managed.
- </para>
- <para>
- This replaces the deprecated variable
- <varname>NFS_SERVER_MODE</varname>.
+ COMMAND specifies the path to a callout to handle
+ interactions with the configured NFS system, including
+ startup, shutdown, monitoring.
</para>
<para>
- Default is "kernel".
- </para>
- </listitem>
- </varlistentry>
-
- <varlistentry>
- <term>CTDB_NFS_SKIP_KNFSD_ALIVE_CHECK=yes|no</term>
- <listitem>
- <para>
- During monitoring, should CTDB skip the
- <command>rpcinfo</command> check that is used to see if
- the NFS kernel server is functional.
- </para>
- <para>
- Default is no.
+ Default is the included
+ <command>nfs-linux-kernel-callout</command>.
</para>
</listitem>
</varlistentry>
<para>
IPADDR or HOSTNAME indicates the address that
<command>rpcinfo</command> should connect to when doing
- <command>rpcinfo</command> check on RPC service during
+ <command>rpcinfo</command> check on IPv4 RPC service during
monitoring. Optimally this would be "localhost".
However, this can add some performance overheads.
</para>
</varlistentry>
<varlistentry>
- <term>CTDB_SKIP_GANESHA_NFSD_CHECK=yes|no</term>
+ <term>CTDB_RPCINFO_LOCALHOST6=<parameter>IPADDR</parameter>|<parameter>HOSTNAME</parameter></term>
<listitem>
<para>
- As part of monitoring, should CTDB skip the check for
- the existence of each directory exported via
- NFS-Ganesha. This may be desirable if there is a large
- number of exports.
+ IPADDR or HOSTNAME indicates the address that
+ <command>rpcinfo</command> should connect to when doing
+ <command>rpcinfo</command> check on IPv6 RPC service
+ during monitoring. Optimally this would be "localhost6"
+ (or similar). However, this can add some performance
+ overheads.
</para>
<para>
- Default is no.
+ Default is "::1".
</para>
</listitem>
</varlistentry>
<para>
CTDB can experience seemingly random (performance and other)
- issues if system resources become too contrained. Options in
- this section can be enabled to allow certain system resources to
- be checked.
+ issues if system resources become too constrained. Options in
+ this section can be enabled to allow certain system resources
+ to be checked. They allows warnings to be logged and nodes to
+ be marked unhealthy when system resource usage reaches the
+ configured thresholds.
+ </para>
+
+ <para>
+ Some checks are enabled by default. It is recommended that
+ these checks remain enabled or are augmented by extra checks.
+ There is no supported way of completely disabling the checks.
</para>
<refsect3>
<title>Eventscripts</title>
<simplelist>
- <member><filename>00.ctdb</filename></member>
- <member><filename>40.fs_use</filename></member>
+ <member><filename>05.system</filename></member>
</simplelist>
<para>
- Filesystem usage monitoring is in
- <filename>40.fs_use</filename>. This eventscript is not
- enabled by default. Use <command>ctdb
- enablescript</command> to enable it.
+ Filesystem and memory usage monitoring is in
+ <filename>05.system</filename>.
</para>
</refsect3>
<variablelist>
<varlistentry>
- <term>CTDB_CHECK_FS_USE=<parameter>FS-LIMIT-LIST</parameter></term>
+ <term>CTDB_MONITOR_FILESYSTEM_USAGE=<parameter>FS-LIMIT-LIST</parameter></term>
<listitem>
<para>
FS-LIMIT-LIST is a space-separated list of
- <parameter>FILESYSTEM</parameter>:<parameter>LIMIT</parameter>
- pairs indicating that a node should be flagged unhealthy
- if the space used on FILESYSTEM reaches LIMIT%.
+ <parameter>FILESYSTEM</parameter>:<parameter>WARN_LIMIT</parameter><optional>:<parameter>UNHEALTHY_LIMIT</parameter></optional>
+ triples indicating that warnings should be logged if the
+ space used on FILESYSTEM reaches WARN_LIMIT%. If usage
+ reaches UNHEALTHY_LIMIT then the node should be flagged
+ unhealthy. Either WARN_LIMIT or UNHEALTHY_LIMIT may be
+ left blank, meaning that check will be omitted.
</para>
<para>
- No default.
- </para>
-
- <para>
- Note that this feature uses the
- <filename>40.fs_use</filename> eventscript, which is not
- enabled by default. Use <command>ctdb
- enablescript</command> to enable it.
+ Default is to warn for each filesystem containing a
+ database directory (<envar>CTDB_DBDIR</envar>,
+ <envar>CTDB_DBDIR_PERSISTENT</envar>,
+ <envar>CTDB_DBDIR_STATE</envar>) with a threshold of
+ 90%.
</para>
</listitem>
</varlistentry>
<varlistentry>
- <term>CTDB_CHECK_SWAP_IS_NOT_USED=yes|no</term>
+ <term>CTDB_MONITOR_MEMORY_USAGE=<parameter>MEM-LIMITS</parameter></term>
<listitem>
<para>
- Should a warning be logged if swap space is in use.
+ MEM-LIMITS takes the form
+ <parameter>WARN_LIMIT</parameter><optional>:<parameter>UNHEALTHY_LIMIT</parameter></optional>
+ indicating that warnings should be logged if memory
+ usage reaches WARN_LIMIT%. If usage reaches
+ UNHEALTHY_LIMIT then the node should be flagged
+ unhealthy. Either WARN_LIMIT or UNHEALTHY_LIMIT may be
+ left blank, meaning that check will be omitted.
</para>
<para>
- Default is no.
+ Default is 80, so warnings will be logged when memory
+ usage reaches 80%.
</para>
</listitem>
</varlistentry>
<varlistentry>
- <term>CTDB_MONITOR_FREE_MEMORY=<parameter>NUM</parameter></term>
+ <term>CTDB_MONITOR_SWAP_USAGE=<parameter>SWAP-LIMITS</parameter></term>
<listitem>
<para>
- NUM is a lower limit on available system memory, expressed
- in megabytes. If this is set and the amount of available
- memory falls below this limit then some debug information
- will be logged, the node will be disabled and then CTDB
- will be shut down.
+ SWAP-LIMITS takes the form
+ <parameter>WARN_LIMIT</parameter><optional>:<parameter>UNHEALTHY_LIMIT</parameter></optional>
+ indicating that warnings should be logged if
+ swap usage reaches WARN_LIMIT%. If usage reaches
+ UNHEALTHY_LIMIT then the node should be flagged
+ unhealthy. Either WARN_LIMIT or UNHEALTHY_LIMIT may be
+ left blank, meaning that check will be omitted.
</para>
<para>
- No default.
- </para>
- </listitem>
- </varlistentry>
-
- <varlistentry>
- <term>CTDB_MONITOR_FREE_MEMORY_WARN=<parameter>NUM</parameter></term>
- <listitem>
- <para>
- NUM is a lower limit on available system memory, expressed
- in megabytes. If this is set and the amount of available
- memory falls below this limit then a warning will be
- logged.
- </para>
- <para>
- No default.
+ Default is 25, so warnings will be logged when swap
+ usage reaches 25%.
</para>
</listitem>
</varlistentry>
</varlistentry>
<varlistentry>
- <term>CTDB_ETCDIR=<parameter>DIRECTORY</parameter></term>
+ <term>CTDB_SYS_ETCDIR=<parameter>DIRECTORY</parameter></term>
<listitem>
<para>
DIRECTORY containing system configuration files. This is
</listitem>
</varlistentry>
+ <varlistentry>
+ <term>CTDB_MAX_OPEN_FILES=<parameter>NUM</parameter></term>
+ <listitem>
+ <para>
+ NUM is the maximum number of open files.
+ </para>
+ <para>
+ There is no default.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term>CTDB_RC_LOCAL=<parameter>FILENAME</parameter></term>
<listitem>
<manvolnum>1</manvolnum></citerefentry> to be run under
<citerefentry><refentrytitle>valgrind</refentrytitle>
<manvolnum>1</manvolnum></citerefentry> with logs going to
- <filename>/var/log/ctdb_valgrind</filename>. If neither
+ <filename>/usr/local/var/log/ctdb_valgrind</filename>. If neither
"yes" nor "no" then the value is assumed to be a COMMAND
(e.g. a <command>valgrind</command> variation, a
<citerefentry><refentrytitle>gdb</refentrytitle>
</listitem>
</varlistentry>
+ <varlistentry>
+ <term>CTDB_VARDIR=<parameter>DIRECTORY</parameter></term>
+ <listitem>
+ <para>
+ DIRECTORY containing CTDB files that are modified at
+ runtime.
+ </para>
+ <para>
+ Defaults to <filename>/usr/local/var/lib/ctdb</filename>.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</refsect1>
<title>FILES</title>
<simplelist>
- <member><filename>/etc/ctdb/ctdbd.conf</filename></member>
+ <member><filename>/usr/local/etc/ctdb/ctdbd.conf</filename></member>
<member><filename>/etc/sysconfig/ctdb</filename></member>
<member><filename>/etc/default/ctdb</filename></member>
- <member><filename>/etc/ctdb/sysconfig/ctdb</filename></member>
+ <member><filename>/usr/local/etc/ctdb/sysconfig/ctdb</filename></member>
</simplelist>
</refsect1>