ctdbd

Name

ctdbd — The CTDB cluster daemon

Synopsis

ctdbd

ctdbd [-? --help] [-d --debug=<INTEGER>] {--dbdir=<directory>} {--dbdir-persistent=<directory>} [--event-script-dir=<directory>] [-i --interactive] [--listen=<address>] [--logfile=<filename>] [--lvs] {--nlist=<filename>} [--no-lmaster] [--no-recmaster] [--nosetsched] [--public-addresses=<filename>] [--public-interface=<interface>] {--reclock=<filename>} [--single-public-ip=<address>] [--socket=<filename>] [--start-as-disabled] [--start-as-stopped] [--syslog] [--torture] [--transport=<STRING>] [--usage]

DESCRIPTION

+ctdbd

Name

ctdbd — The CTDB cluster daemon

Synopsis

ctdbd

DESCRIPTION

ctdbd is the main ctdb daemon.

ctdbd provides a clustered version of the TDB database with automatic rebuild/recovery of the databases upon nodefailures. @@ -8,7 +8,7 @@ ctdbd provides monitoring of all nodes in the cluster and automatically reconfigures the cluster and recovers upon node failures.

ctdbd is the main component in clustered Samba that provides a high-availability load-sharing CIFS server cluster. -

OPTIONS

-? --help: +

OPTIONS

-? --help: Print some help text to the screen.
-d --debug=<DEBUGLEVEL>: This option sets the debuglevel on the ctdbd daemon which controls what will be written to the logfile. The default is 0 which will only log important events and errors. A larger number will provide additional logging. @@ -129,10 +129,10 @@ implemented in the future.
--usage: Print useage information to the screen. -

Private vs Public addresses

When used for ip takeover in a HA environment, each node in a ctdb cluster has multiple ip addresses assigned to it. One private and one or more public. -

Private address

This is the physical ip address of the node which is configured in linux and attached to a physical interface. This address uniquely identifies a physical node in the cluster and is the ip addresses @@ -162,7 +162,7 @@ 10.1.1.2 10.1.1.3 10.1.1.4 -

Public address

A public address on the other hand is not attached to an interface. This address is managed by ctdbd itself and is attached/detached to a physical node at runtime. @@ -223,7 +223,7 @@ unavailable. 10.1.1.1 can not be failed over to node 2 or node 3 since these nodes do not have this ip address listed in their public addresses file. -

Node status

The current status of each node in the cluster can be viewed by the 'ctdb status' command.

@@ -260,50 +260,50 @@ RECMASTER or NATGW. This node does not perticipate in the CTDB cluster but can still be communicated with. I.e. ctdb commands can be sent to it. -

PUBLIC TUNABLES

These are the public tuneables that can be used to control how ctdb behaves. -

KeepaliveInterval

Default: 1

KeepaliveInterval

Default: 1

How often should the nodes send keepalives to eachother. -

KeepaliveLimit

Default: 5

KeepaliveLimit

Default: 5

After how many keepalive intervals without any traffic should a node wait until marking the peer as DISCONNECTED. -

MonitorInterval

Default: 15

MonitorInterval

Default: 15

How often should ctdb run the event scripts to check for a nodes health. -

TickleUpdateInterval

Default: 20

TickleUpdateInterval

Default: 20

How often will ctdb record and store the "tickle" information used to kickstart stalled tcp connections after a recovery. -

EventScriptTimeout

Default: 20

EventScriptTimeout

Default: 20

How long should ctdb let an event script run before aborting it and marking the node unhealthy. -

RecoveryBanPeriod

Default: 300

RecoveryBanPeriod

Default: 300

If a node becomes banned causing repetitive recovery failures. The node will eventually become banned from the cluster. This controls how long the culprit node will be banned from the cluster before it is allowed to try to join the cluster again. Dont set to small. A node gets banned for a reason and it is usually due to real problems with the node. -

DatabaseHashSize

Default: 100000

DatabaseHashSize

Default: 100000

Size of the hash chains for the local store of the tdbs that ctdb manages. -

RerecoveryTimeout

Default: 10

RerecoveryTimeout

Default: 10

Once a recovery has completed, no additional recoveries are permitted until this timeout has expired. -

EnableBans

Default: 1

EnableBans

Default: 1

When set to 0, this disables BANNING completely in the cluster and thus nodes can not get banned, even it they break. Dont set to 0. -

DeterministicIPs

Default: 1

DeterministicIPs

Default: 1

When enabled, this tunable makes ctdb try to keep public IP addresses locked to specific nodes as far as possible. This makes it easier for debugging since you can know that as long as all nodes are healthy public IP X will always be hosted by node Y.

The cost of using deterministic IP address assignment is that it disables part of the logic where ctdb tries to reduce the number of public IP assignment changes in the cluster. This tunable may increase the number of IP failover/failbacks that are performed on the cluster by a small margin. -

DisableWhenUnhealthy

Default: 0

DisableWhenUnhealthy

Default: 0

When set, As soon as a node becomes unhealthy, that node will also automatically become permanently DISABLED. Once a node is DISABLED, the only way to make it participate in the cluster again and host services is by manually enabling the node again using 'ctdb enable'.

This disables parts of the resilience and robustness of the cluster and should ONLY be used when the system administrator is actively monitoring the cluster, so that nodes can be enabled again. -

NoIPFailback

Default: 0

NoIPFailback

Default: 0

When set to 1, ctdb will not perform failback of IP addresses when a node becomes healthy. Ctdb WILL perform failover of public IP addresses when a node becomes UNHEALTHY, but when the node becomes HEALTHY again, ctdb will not fail the addresses back.

Use with caution! Normally when a node becomes available to the cluster ctdb will try to reassign public IP addresses onto the new node as a way to distribute the workload evenly across the clusternode. Ctdb tries to make sure that all running nodes have approximately the same number of public addresses it hosts.

When you enable this tunable, CTDB will no longer attempt to rebalance the cluster by failing IP addresses back to the new nodes. An unbalanced cluster will therefore remain unbalanced until there is manual intervention from the administrator. When this parameter is set, you can manually fail public IP addresses over to the new node(s) using the 'ctdb moveip' command. -

LVS

LVS is a mode where CTDB presents one single IP address for the entire cluster. This is an alternative to using public IP addresses and round-robin DNS to loadbalance clients across the cluster. @@ -344,7 +344,7 @@ ctdb will try to reassign public IP addresses onto the new node as a way to dist the processing node back to the clients. For read-intensive i/o patterns you can acheive very high throughput rates in this mode.

Note: you can use LVS and public addresses at the same time. -

Configuration

To activate LVS on a CTDB node you must specify CTDB_PUBLIC_INTERFACE and CTDB_LVS_PUBLIC_ADDRESS in /etc/sysconfig/ctdb.

@@ -367,7 +367,7 @@ You must also specify the "--lvs" command line argument to ctdbd to activete LVS all of the clients from the node BEFORE you enable LVS. Also make sure that when you ping these hosts that the traffic is routed out through the eth0 interface. -

REMOTE CLUSTER NODES

It is possible to have a CTDB cluster that spans across a WAN link. For example where you have a CTDB cluster in your datacentre but you also want to have one additional CTDB node located at a remote branch site. @@ -396,7 +396,7 @@ CTDB_CAPABILITY_RECMASTER=no

Verify with the command "ctdb getcapabilities" that that node no longer has the recmaster or the lmaster capabilities. -

NAT-GW

Sometimes it is desireable to run services on the CTDB node which will need to originate outgoing traffic to external servers. This might be contacting NIS servers, LDAP servers etc. etc. @@ -419,7 +419,7 @@ CTDB_CAPABILITY_RECMASTER=no if there are no public addresses assigned to the node. This is the simplest way but it uses up a lot of ip addresses since you have to assign both static and also public addresses to each node. -

NAT-GW

A second way is to use the built in NAT-GW feature in CTDB. With NAT-GW you assign one public NATGW address for each natgw group. Each NATGW group is a set of nodes in the cluster that shares the same @@ -434,7 +434,7 @@ CTDB_CAPABILITY_RECMASTER=no In each NATGW group, one of the nodes is designated the NAT Gateway through which all traffic that is originated by nodes in this group will be routed through if a public addresses are not available. -

Configuration

NAT-GW is configured in /etc/sysconfigctdb by setting the following variables:

@@ -473,31 +473,31 @@ CTDB_CAPABILITY_RECMASTER=no
 # CTDB_NATGW_DEFAULT_GATEWAY=10.0.0.1
 # CTDB_NATGW_PRIVATE_NETWORK=10.1.1.0/24
 # CTDB_NATGW_NODES=/etc/ctdb/natgw_nodes
-

CTDB_NATGW_PUBLIC_IP

This is an ip address in the public network that is used for all outgoing traffic when the public addresses are not assigned. This address will be assigned to one of the nodes in the cluster which will masquerade all traffic for the other nodes.

Format of this parameter is IPADDRESS/NETMASK -

CTDB_NATGW_PUBLIC_IFACE

This is the physical interface where the CTDB_NATGW_PUBLIC_IP will be assigned to. This should be an interface connected to the public network.

Format of this parameter is INTERFACE -

CTDB_NATGW_DEFAULT_GATEWAY

This is the default gateway to use on the node that is elected to host the CTDB_NATGW_PUBLIC_IP. This is the default gateway on the public network.

Format of this parameter is IPADDRESS -

CTDB_NATGW_PRIVATE_NETWORK

This is the network/netmask used for the interal private network.

Format of this parameter is IPADDRESS/NETMASK -

CTDB_NATGW_NODES

This is the list of all nodes that belong to the same NATGW group as this node. The default is /etc/ctdb/natgw_nodes. -

Operation

When the NAT-GW functionality is used, one of the nodes is elected to act as a NAT router for all the other nodes in the group when they need to originate traffic to the external public network. @@ -512,23 +512,11 @@ CTDB_CAPABILITY_RECMASTER=no

This is implemented in the 11.natgw eventscript. Please see the eventscript for further information. -

Activating NATGW without forcing a cluster restart

- Normally you would update the configuration and restart the nodes to activate NATGW. It is however also possible to manually start NATGW without restarting CTDB, but it does require a recovery. -

- To activate NATGW functionality on a cluster without restarting the actual nodes you can use the following process : -

Activating NATGW on a running system

- Configure NATGW as described above on all nodes. -

- Run "echo 3 > /proc/sys/net/ipv4/conf/all/arp_ignore" on all nodes. -

- Run "echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce" on all nodes. -

- Force a recovery by running "ctdb recover". -

ClamAV Daemon

CTDB has support to manage the popular anti-virus daemon ClamAV. This support is implemented through the eventscript : /etc/ctdb/events.d/31.clamd. -

Configuration

Start by configuring CLAMAV normally and test that it works. Once this is done, copy the configuration files over to all the nodes so that all nodes share identical CLAMAV configurations. @@ -557,10 +545,10 @@ Once you have restarted CTDBD, use ctdb scriptstatus

and verify that the 31.clamd eventscript is listed and that it was executed successfully. -

COPYRIGHT/LICENSE

Copyright (C) Andrew Tridgell 2007
Copyright (C) Ronnie sahlberg 2007

diff --git a/ctdb/doc/ctdbd.1.xml b/ctdb/doc/ctdbd.1.xml index 40e6c82f53d..992500c54b4 100644 --- a/ctdb/doc/ctdbd.1.xml +++ b/ctdb/doc/ctdbd.1.xml @@ -924,39 +924,6 @@ CTDB_CAPABILITY_RECMASTER=no This is implemented in the 11.natgw eventscript. Please see the eventscript for further information. - - - Activating NATGW without forcing a cluster restart - - Normally you would update the configuration and restart the nodes to activate NATGW. It is however also possible to manually start NATGW without restarting CTDB, but it does require a recovery. - - - To activate NATGW functionality on a cluster without restarting the actual nodes you can use the following process : - - - Activating NATGW on a running system - - - Configure NATGW as described above on all nodes. - - - - - Run "echo 3 > /proc/sys/net/ipv4/conf/all/arp_ignore" on all nodes. - - - - - Run "echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce" on all nodes. - - - - - Force a recovery by running "ctdb recover". - - - - diff --git a/ctdb/include/ctdb.h b/ctdb/include/ctdb.h index ebe40c302c9..0c0a9c8fbb2 100644 --- a/ctdb/include/ctdb.h +++ b/ctdb/include/ctdb.h @@ -661,4 +661,6 @@ int32_t get_debug_by_desc(const char *desc); int ctdb_ctrl_stop_node(struct ctdb_context *ctdb, struct timeval timeout, uint32_t destnode); int ctdb_ctrl_continue_node(struct ctdb_context *ctdb, struct timeval timeout, uint32_t destnode); +int ctdb_ctrl_setnatgwstate(struct ctdb_context *ctdb, struct timeval timeout, uint32_t destnode, uint32_t natgwstate); + #endif diff --git a/ctdb/include/ctdb_private.h b/ctdb/include/ctdb_private.h index 9750552c1f2..d00fb86043b 100644 --- a/ctdb/include/ctdb_private.h +++ b/ctdb/include/ctdb_private.h @@ -359,6 +359,8 @@ enum ctdb_freeze_mode {CTDB_FREEZE_NONE, CTDB_FREEZE_PENDING, CTDB_FREEZE_FROZEN #define CTDB_CAP_LMASTER 0x00000002 /* This capability is set if CTDB_LVS_PUBLIC_IP is set */ #define CTDB_CAP_LVS 0x00000004 +/* This capability is set if NATGW is enabled */ +#define CTDB_CAP_NATGW 0x00000008 /* main state of the ctdb daemon */ struct ctdb_context { @@ -573,6 +575,7 @@ enum ctdb_controls {CTDB_CONTROL_PROCESS_EXISTS = 0, CTDB_CONTROL_SET_RECLOCK_FILE = 100, CTDB_CONTROL_STOP_NODE = 101, CTDB_CONTROL_CONTINUE_NODE = 102, + CTDB_CONTROL_SET_NATGWSTATE = 103, }; /* diff --git a/ctdb/server/ctdb_control.c b/ctdb/server/ctdb_control.c index a18ef0ca6fe..5b43f8b2180 100644 --- a/ctdb/server/ctdb_control.c +++ b/ctdb/server/ctdb_control.c @@ -470,6 +470,19 @@ static int32_t ctdb_control_dispatch(struct ctdb_context *ctdb, CHECK_CONTROL_DATA_SIZE(0); return ctdb_control_continue_node(ctdb); + case CTDB_CONTROL_SET_NATGWSTATE: { + uint32_t natgwstate; + + CHECK_CONTROL_DATA_SIZE(sizeof(uint32_t)); + natgwstate = *(uint32_t *)indata.dptr; + if (natgwstate == 0) { + ctdb->capabilities &= ~CTDB_CAP_NATGW; + } else { + ctdb->capabilities |= CTDB_CAP_NATGW; + } + return 0; + } + default: DEBUG(DEBUG_CRIT,(__location__ " Unknown CTDB control opcode %u\n", opcode)); return -1; diff --git a/ctdb/tools/ctdb.c b/ctdb/tools/ctdb.c index e0e728f4f99..e2528e6d045 100644 --- a/ctdb/tools/ctdb.c +++ b/ctdb/tools/ctdb.c @@ -1976,12 +1976,14 @@ static int control_getcapabilities(struct ctdb_context *ctdb, int argc, const ch printf("RECMASTER: %s\n", (capabilities&CTDB_CAP_RECMASTER)?"YES":"NO"); printf("LMASTER: %s\n", (capabilities&CTDB_CAP_LMASTER)?"YES":"NO"); printf("LVS: %s\n", (capabilities&CTDB_CAP_LVS)?"YES":"NO"); + printf("NATGW: %s\n", (capabilities&CTDB_CAP_NATGW)?"YES":"NO"); } else { - printf(":RECMASTER:LMASTER:LVS:\n"); - printf(":%d:%d:%d:\n", + printf(":RECMASTER:LMASTER:LVS:NATGW:\n"); + printf(":%d:%d:%d:%d:\n", !!(capabilities&CTDB_CAP_RECMASTER), !!(capabilities&CTDB_CAP_LMASTER), - !!(capabilities&CTDB_CAP_LVS)); + !!(capabilities&CTDB_CAP_LVS), + !!(capabilities&CTDB_CAP_NATGW)); } return 0; } @@ -2425,6 +2427,35 @@ static int control_setreclock(struct ctdb_context *ctdb, int argc, const char ** return 0; } +/* + set the natgw state ot on/off + */ +static int control_setnatgwstate(struct ctdb_context *ctdb, int argc, const char **argv) +{ + int ret; + uint32_t natgwstate; + + if (argc == 0) { + usage(); + } + + if (!strcmp(argv[0], "on")) { + natgwstate = 1; + } else if (!strcmp(argv[0], "off")) { + natgwstate = 0; + } else { + usage(); + } + + ret = ctdb_ctrl_setnatgwstate(ctdb, TIMELIMIT(), options.pnn, natgwstate); + if (ret != 0) { + DEBUG(DEBUG_ERR, ("Unable to set the natgw state for node %u\n", options.pnn)); + return ret; + } + + return 0; +} + /* set debug level on a node or all nodes */ @@ -3190,6 +3221,7 @@ static const struct { { "xpnn", control_xpnn, true, true, "find the pnn of the local node without talking to the daemon (unreliable)" }, { "getreclock", control_getreclock, false, false, "Show the reclock file of a node"}, { "setreclock", control_setreclock, false, false, "Set/clear the reclock file of a node", "[filename]"}, + { "setnatgwstate", control_setnatgwstate, false, false, "Set NATGW state to on/off", "{on|off}"}, }; /* -- 2.34.1

Name

Synopsis

DESCRIPTION

Name

Synopsis

DESCRIPTION

OPTIONS

OPTIONS

Private vs Public addresses

Private vs Public addresses

Private address

Private address

Public address

Public address

Node status

Node status

PUBLIC TUNABLES

PUBLIC TUNABLES

KeepaliveInterval

KeepaliveInterval

KeepaliveLimit

KeepaliveLimit

MonitorInterval

MonitorInterval

TickleUpdateInterval

TickleUpdateInterval

EventScriptTimeout

EventScriptTimeout

RecoveryBanPeriod

RecoveryBanPeriod

DatabaseHashSize

DatabaseHashSize

RerecoveryTimeout

RerecoveryTimeout

EnableBans

EnableBans

DeterministicIPs

DeterministicIPs

DisableWhenUnhealthy

DisableWhenUnhealthy

NoIPFailback

NoIPFailback

LVS

LVS

Configuration

Configuration

REMOTE CLUSTER NODES

REMOTE CLUSTER NODES

NAT-GW

NAT-GW

NAT-GW

NAT-GW

Configuration

Configuration

CTDB_NATGW_PUBLIC_IP

CTDB_NATGW_PUBLIC_IP

CTDB_NATGW_PUBLIC_IFACE

CTDB_NATGW_PUBLIC_IFACE

CTDB_NATGW_DEFAULT_GATEWAY

CTDB_NATGW_DEFAULT_GATEWAY

CTDB_NATGW_PRIVATE_NETWORK

CTDB_NATGW_PRIVATE_NETWORK

CTDB_NATGW_NODES

CTDB_NATGW_NODES

Operation

Operation

Activating NATGW without forcing a cluster restart

ClamAV Daemon

ClamAV Daemon

Configuration

Configuration

SEE ALSO

SEE ALSO

COPYRIGHT/LICENSE

COPYRIGHT/LICENSE