Ronnie Sahlberg [Wed, 14 Oct 2009 04:51:57 +0000 (15:51 +1100)]
Merge commit 'martins/onnode_options'
Ronnie Sahlberg [Wed, 14 Oct 2009 03:52:24 +0000 (14:52 +1100)]
version 1.0.96
Ronnie Sahlberg [Wed, 14 Oct 2009 03:14:28 +0000 (14:14 +1100)]
add more debugging output to eventscripts and when a script has timed out,
print a full "pstree -p" to the log.
Example :
|-ctdbd(29826)-+-ctdbd(29862)
| `-ctdbd(31897)-+-00.ctdb(31898)---sleep(31908)
change the default timeout to 60 seconds for eventscripts
Martin Schwenke [Wed, 14 Oct 2009 02:49:30 +0000 (13:49 +1100)]
Merge commit 'origin/master' into onnode_options
Martin Schwenke [Wed, 14 Oct 2009 02:44:57 +0000 (13:44 +1100)]
New onnode options: -f to specify nodes file, -n to allow use of hostnames.
The -f option allows an alternate nodes file to be specified,
overriding the CTDB_NODES_FILE environment variable.
The -n option allows hostnames to be used instead of node numbers.
Using a range of hostnames is invalid, so hostnames can't contain
hyphens ('-') - sorry! You can use this option without a nodes file
by specifying "-f /dev/null".
Signed-off-by: Martin Schwenke <martin@meltin.net>
Ronnie Sahlberg [Wed, 14 Oct 2009 01:12:04 +0000 (12:12 +1100)]
move the logging of the warning "No reclock file used" to the startup case so we only print this warning on "service ctdb start" and not for "service ctdb *"
Ronnie Sahlberg [Wed, 14 Oct 2009 00:59:16 +0000 (11:59 +1100)]
when we change state between healthy/unhealthy, make sure we ask the recovery
master to perform an explicit ip reallocation.
This is more reliable and faster than having the recovery dameon track these
changes, and since we now have an explicit method to ask the recovery daemon
to perform an explicit ip reallocation, we should use this.
Ronnie Sahlberg [Tue, 13 Oct 2009 23:14:03 +0000 (10:14 +1100)]
allow a pre .95 version of a recovery master to freeze databases on a post .95 node by remapping priority numbers and log this to log.ctdb
Ronnie Sahlberg [Tue, 13 Oct 2009 22:15:24 +0000 (09:15 +1100)]
always create the nfs state directories during the monitor event.
this allows us to configure and enable nfs at runtime without having to restart ctdbd
Ronnie Sahlberg [Tue, 13 Oct 2009 21:17:49 +0000 (08:17 +1100)]
Port Volkers deadlock avoidance patch to HEAD.
This patch ensures that we lock all non-notify related databases first and
then the notify databases to avoiud a deadlock where samba needs to lock records on two databases at once (and notify being the second database).
Newer versions of samba would instead use the set-db-prio control to set this explicitely on a database per database basis instead of relying on hardcoded database names. This patch will be reverted in the future when all updated versions of samba has been pushed out.
Ronnie Sahlberg [Mon, 12 Oct 2009 22:49:05 +0000 (09:49 +1100)]
we must break the loop as soon as we find a suitable recmaster does exist
otherwise "tdb ipreallocate" will silently fail to update the addresses.
Ronnie Sahlberg [Mon, 12 Oct 2009 07:53:20 +0000 (18:53 +1100)]
new version 1.0.95
Ronnie Sahlberg [Mon, 12 Oct 2009 07:41:57 +0000 (18:41 +1100)]
use the correct expected size for thew _cancel control
Ronnie Sahlberg [Mon, 12 Oct 2009 07:31:59 +0000 (18:31 +1100)]
add a dispatch to the recovery transaction cancel call
Ronnie Sahlberg [Mon, 12 Oct 2009 05:51:36 +0000 (16:51 +1100)]
Merge commit 'martins/master'
Ronnie Sahlberg [Mon, 12 Oct 2009 05:48:05 +0000 (16:48 +1100)]
add a new control for explicitely cancelling recovery transactions, i.e. the
transactions we start across all tdb databased during the recovery.
this allows us to properly clean up and delete these tdb transactions on a
recovery failure.
Martin Schwenke [Mon, 12 Oct 2009 05:32:49 +0000 (16:32 +1100)]
Clean up ctdb_check_directories* eventscript functions.
There are 2 problems with this code:
* The loop in ctdb_check_directories_probe() breaks on filenames
containing whitespace.
The fix to protect them is to pass "$@" to this function and have it
operate on "$@".
Note that there's still a problem with whitespace in filenames in
the 50.samba eventscript. To fix this ctdb_check_directories_probe
should read the filenames from stdin. Another time...
* The check for '%' in filenames in ctdb_check_directories_probe()
ends up involving several forks. On a modern machine this can cost
a couple of minutes when checking a large number of directories.
The fix is to use a case statement.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Martin Schwenke [Mon, 12 Oct 2009 05:17:37 +0000 (16:17 +1100)]
40.vsftpd: reset the fail counter in the "recovered" event.
Each recovery that involves IP reassignments results in a restart of
vsftpd in the "recovered" event. Currently, we can have several
recoveries in quick succession and the "monitor" event following each
can fail because vsftpd isn't ready yet. This results in cumulative
failures, so the node is marked unhealthy, even though vsftpd has
never had a proper opportunity to become ready.
This resets the fail count after each recovery.
While we're here, also move the delete of the restart flag file into
the body of the conditional.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Ronnie Sahlberg [Mon, 12 Oct 2009 02:06:16 +0000 (13:06 +1100)]
allow setting the recmode even when not completely frozen.
we sometimes have to do this when we want to trigger a recovery
Ronnie Sahlberg [Mon, 12 Oct 2009 01:08:39 +0000 (12:08 +1100)]
initial attempt at freezing databases in priority order
Ronnie Sahlberg [Sun, 11 Oct 2009 22:22:17 +0000 (09:22 +1100)]
uptade the freeze/thaw commands to be able to send the requested database priority to freeze/thaw to the daemon.
this is encoded in the srvid field of the request header
Ronnie Sahlberg [Sat, 10 Oct 2009 05:28:20 +0000 (16:28 +1100)]
during recovery, update all remote nodes so they use the same priorities
for the databases as this node.
Ronnie Sahlberg [Sat, 10 Oct 2009 04:04:18 +0000 (15:04 +1100)]
add a control to read the db priority from a database
Ronnie Sahlberg [Sat, 10 Oct 2009 03:26:09 +0000 (14:26 +1100)]
add a control to set a database priority. Let newly created databases default to priority 1.
database priorities will be used to control in which order databases are locked during recovery in.
Ronnie Sahlberg [Sat, 10 Oct 2009 02:55:11 +0000 (13:55 +1100)]
verify the DISABLED flag and compare with the previous flag we have registered for that node and not what the node says is the difference.
this prevents a situation where the remove node may cause spurious ip reallocations.
Ronnie Sahlberg [Fri, 9 Oct 2009 11:22:11 +0000 (22:22 +1100)]
Fix bug spotted by Metze,
the argument to ctdb_control_event_Script_disabled() is a string not a uint32
Ronnie Sahlberg [Thu, 8 Oct 2009 08:17:57 +0000 (19:17 +1100)]
version 1.0.94
Ronnie Sahlberg [Thu, 8 Oct 2009 05:45:25 +0000 (16:45 +1100)]
if a node fails to become frozen during recovery, mark it up with as a culprit so it will soon get banned
Ronnie Sahlberg [Tue, 6 Oct 2009 06:05:14 +0000 (17:05 +1100)]
version 1.0.93
Ronnie Sahlberg [Tue, 6 Oct 2009 05:09:24 +0000 (16:09 +1100)]
update natgw eventscript to allow you to fore it to update and / or to remove the configuration at runtime
Martin Schwenke [Tue, 6 Oct 2009 02:39:31 +0000 (13:39 +1100)]
Merge commit 'origin/master'
Martin Schwenke [Tue, 6 Oct 2009 02:38:00 +0000 (13:38 +1100)]
Document CTDB_NODES_FILE environment variable used by onnode.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Ronnie Sahlberg [Tue, 6 Oct 2009 01:25:44 +0000 (12:25 +1100)]
always send the release/take ip controls to make sure all nodes are updated
Ronnie Sahlberg [Tue, 6 Oct 2009 01:11:32 +0000 (12:11 +1100)]
add a new message to ask the recovery daemon to temporarily disable checking ip address consistency.
This is useful when we are moving addresses using moveip in the cluster since otherwise if we collide with the recovery daemons own check we could cause a recovery
Ronnie Sahlberg [Tue, 6 Oct 2009 00:41:18 +0000 (11:41 +1100)]
update addip/moveip/delip to make it less likely to trigger an accidental recovery
Ronnie Sahlberg [Tue, 6 Oct 2009 00:40:38 +0000 (11:40 +1100)]
change some loglevels and also pront the pnn of the ip for takeip/releaseip logging
Ronnie Sahlberg [Mon, 5 Oct 2009 23:52:31 +0000 (10:52 +1100)]
add a new function to collect a list of all active nodes EXCEPT a certain node
Ronnie Sahlberg [Mon, 5 Oct 2009 22:35:15 +0000 (09:35 +1100)]
allocate takeoverip state as a child of vnn and also make the takeocerip context a child of vnn
Ronnie Sahlberg [Mon, 5 Oct 2009 21:19:25 +0000 (08:19 +1100)]
When adding a public ip to a node, make sure to push the assignment of ip addresses out to all nodes so all nodes become aware who currently holds the ip.
Ronnie Sahlberg [Fri, 2 Oct 2009 04:38:16 +0000 (14:38 +1000)]
version 1.0.92
Ronnie Sahlberg [Fri, 2 Oct 2009 03:41:54 +0000 (13:41 +1000)]
we should close this file on exec
Ronnie Sahlberg [Thu, 1 Oct 2009 05:46:01 +0000 (15:46 +1000)]
Merge commit 'martins/master'
Martin Schwenke [Thu, 1 Oct 2009 05:39:09 +0000 (15:39 +1000)]
Test suite: The ctdb ping test should allow time to go backwards.
Time can actually go backwards during this test if ntpd happens to
adjust it little bit. So we should cope...
Signed-off-by: Martin Schwenke <martin@meltin.net>
Ronnie Sahlberg [Thu, 1 Oct 2009 04:53:35 +0000 (14:53 +1000)]
dont exit on a commit failure
Ronnie Sahlberg [Thu, 1 Oct 2009 04:51:32 +0000 (14:51 +1000)]
Revert "Revert "allow the transaction commit to fail""
This reverts commit
74e416108df6934f45ca646d709785dd76ab3c35.
Ronnie Sahlberg [Thu, 1 Oct 2009 04:31:55 +0000 (14:31 +1000)]
document how to use the notification script
Ronnie Sahlberg [Thu, 1 Oct 2009 04:05:30 +0000 (14:05 +1000)]
add a new notification to trigger on when ctdb has started
Martin Schwenke [Wed, 30 Sep 2009 11:21:56 +0000 (21:21 +1000)]
Minor fixes to 01.reclock eventscript.
test -z really needs its argument to be quoted. Simplified a status
test.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Martin Schwenke [Wed, 30 Sep 2009 11:05:16 +0000 (21:05 +1000)]
40.vsftpd monitor event only fails after 2 failures to connect to port 21.
Change the monitor event in 40.vsftpd so it only fails if there are 2
successive failures connecting to port 21. This reduces the
likelihood of unhealthy nodes due to vsftpd being restarted for
reconfiguration due to node failover or system reconfiguration.
New eventscript functions ctdb_counter_init, ctdb_counter_incr,
ctdb_counter_limit. These are used to count arbitrary things in
eventscripts, depending on the eventscript name and a tag that is
passed, and determine if a specified limit has been hit. They're good
for counting failures!
These functions are used in 40.vsftpd and also in 01.reclock - the
latter used to do the counting without these functions.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Martin Schwenke [Wed, 30 Sep 2009 09:22:59 +0000 (19:22 +1000)]
Merge commit 'origin/master'
Ronnie Sahlberg [Tue, 29 Sep 2009 03:31:41 +0000 (13:31 +1000)]
New version 1.0.91
Ronnie Sahlberg [Tue, 29 Sep 2009 03:20:18 +0000 (13:20 +1000)]
From Wolfgang Mueller-Friedt
Remove the explicit vacuum/repack commands from the 00.ctdb eventscript
and implement this in the ctdb daemon.
Combine vacuuming and repacking into one
cheap read traverse to enumerate all candidate records
and one write traverse that both repacks the database and also deletes the record locally where we are lmaster and where the records have already been deleted remotely.
this code also adds initial autotuning heuristics for the vacuum intervals and how many records to delete in each iteration.
minor stylish changes made by ronnie s
Martin Schwenke [Tue, 29 Sep 2009 02:59:10 +0000 (12:59 +1000)]
Merge commit 'origin/master'
Ronnie Sahlberg [Mon, 28 Sep 2009 04:12:59 +0000 (14:12 +1000)]
change the reclock fail count to 19 monitor intervals before we shut down ctdbd
Ronnie Sahlberg [Mon, 28 Sep 2009 04:06:40 +0000 (14:06 +1000)]
add a new eventscript 01.reclock
if the reclock file has been set, then this script will test that the
reclock file can actually be accessed.
if the file does not exist, or if the attempts to stat the file hangs,
the node will be marked unhealthy after the third failed monitoring event
and after the tenth failure, ctdb itself will shutdown.
Ronnie Sahlberg [Mon, 28 Sep 2009 03:39:54 +0000 (13:39 +1000)]
add machinereadable output for the ctdb getreclock command
Martin Schwenke [Fri, 25 Sep 2009 08:00:17 +0000 (18:00 +1000)]
Test suite: Print debug info on node status timeouts.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Ronnie Sahlberg [Fri, 25 Sep 2009 07:34:59 +0000 (17:34 +1000)]
Merge commit 'obnox/master-rebase'
Ronnie Sahlberg [Fri, 25 Sep 2009 03:18:18 +0000 (13:18 +1000)]
Merge root@10.1.1.27:/shared/ctdb/ctdb-git
Ronnie Sahlberg [Fri, 25 Sep 2009 03:14:53 +0000 (13:14 +1000)]
with the new banning logic with one struct for each node we no longer "forget" the other culprits as often as we used to do, which means that things like "ctdb recover" can now actually lead to a node becomming banned if we perform too many recoveries too frequently.
change this to provide absolution to all nodes once they have participated in a recovery session.
Michael Adam [Thu, 10 Sep 2009 14:21:01 +0000 (16:21 +0200)]
Revert "dont check if commit failed, we do allow the commit to fail sometimes"
This reverts commit
affa6f47432507e84b7e76b88a2c27fff8e6e2e4.
Transaction commit should not be allowed to fail.
This is a fatal error.
Michael
Michael Adam [Thu, 10 Sep 2009 14:20:26 +0000 (16:20 +0200)]
Revert "allow the transaction commit to fail"
This reverts commit
7a6134e684c9ac4763bf198ef1410867b6082c94.
Transaction commit should not be allowed to fail.
This is a fatal error.
Michael
Michael Adam [Tue, 4 Aug 2009 07:45:50 +0000 (09:45 +0200)]
ctdb_client: fix race in starting concurrent transactions on a single node
There are two races in concurrent transactions on a single node.
One in starting a transaction, and one with committing (replaying).
This commit closes the first race by storing the pid in the
transaction-lock record and comparing the own pid against it
as a measure to prevent starting a second transaction when
a second node has come inbetween and changed the pid in the lock
record.
Michael
Ronnie Sahlberg [Fri, 18 Sep 2009 04:23:37 +0000 (14:23 +1000)]
Merge commit 'martins/master'
Ronnie Sahlberg [Fri, 18 Sep 2009 02:58:30 +0000 (12:58 +1000)]
dont mark the recovery daemon as a ban culprit just because a node in the cluster was set to recvoery mode == ACTIVE.
This happens normally when someone explicitely triggers a recovery using "ctdb recover"
Ronnie Sahlberg [Tue, 15 Sep 2009 09:33:53 +0000 (19:33 +1000)]
try restarting ststd indefinitely not just once
Ronnie Sahlberg [Tue, 15 Sep 2009 09:33:35 +0000 (19:33 +1000)]
Revert "try to restart statd everytime it fails, not just the first time"
This reverts commit
4f7b39a4871af28df1c4545ec37db179fa47a7da.
Ronnie Sahlberg [Tue, 15 Sep 2009 03:35:58 +0000 (13:35 +1000)]
try to restart statd everytime it fails, not just the first time
Ronnie Sahlberg [Mon, 14 Sep 2009 22:05:33 +0000 (08:05 +1000)]
Merge commit 'obnox/master-rebase'
Ronnie Sahlberg [Fri, 11 Sep 2009 21:05:21 +0000 (07:05 +1000)]
Merge root@10.1.1.27:/shared/ctdb/ctdb-git
Ronnie Sahlberg [Fri, 11 Sep 2009 21:30:18 +0000 (07:30 +1000)]
new version 1.0.90
Martin Schwenke [Fri, 11 Sep 2009 06:15:31 +0000 (16:15 +1000)]
Test suite: Update "complex" tests for wait_until_node_has_status() change.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Martin Schwenke [Fri, 11 Sep 2009 05:55:53 +0000 (15:55 +1000)]
Test suite: wait_until_node_has_status() now uses "onnode any".
Many tests currently do this sort of thing:
onnode 0 $CTDB_TEST_WRAPPER wait_until_node_has_status 1 disconnected
In fact, they all use exactly the same "onnode 0 $CTDB_TEST_WRAPPER"
idiom. This is both repetitious and dangerous, since node 0 might be
shutdown during a test. Instead, we push "onnode any
$CTDB_TEST_WRAPPER" (which selects a connected node) into
wait_until_node_has_status() and just call that function directly in
tests, like this:
wait_until_node_has_status 1 disconnected
Signed-off-by: Martin Schwenke <martin@meltin.net>
Martin Schwenke [Fri, 11 Sep 2009 04:06:12 +0000 (14:06 +1000)]
Test suite: Rework the cluster (re)start code.
Make it possible to start on only 1 node - for tests that need to
restart a particular node.
_ctdb_hack_options() attempts to see what options are being passed to
a daemon that is being run via the initscript. It then sets a
corresponding environment variable that the initscript knows about.
Currently only the --start-as-stopped option is supported. This is
extremely ugly but it seems like the only way... :-(
Signed-off-by: Martin Schwenke <martin@meltin.net>
Michael Adam [Fri, 28 Aug 2009 11:01:27 +0000 (13:01 +0200)]
Introduce sysconfig variable CTDB_SYSLOG=yes/no (default "no").
This allows for controlling start of ctdbd with or without the option "--syslog"
from the sysconfig/ctdb file.
Michael
Michael Adam [Fri, 28 Aug 2009 10:45:43 +0000 (12:45 +0200)]
ctdb_logging: fix a comment typo.
Michael
Michael Adam [Thu, 27 Aug 2009 23:04:47 +0000 (01:04 +0200)]
Rename the CTDB_INIT_STYLE "ubuntu" to "debian" - this is where it comes from.
Micheal
Mathieu Parent [Thu, 27 Aug 2009 22:58:52 +0000 (00:58 +0200)]
Update outdated autotools helper files.
This fixes https://bugzilla.samba.org/show_bug.cgi?id=6370
and http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=536256
Signed-off-by: Michael Adam <obnox@samba.org>
Mathieu Parent [Thu, 27 Aug 2009 21:44:39 +0000 (23:44 +0200)]
Fix bashism in nfstickle event script.
Signed-off-by: Michael Adam <obnox@samba.org>
Mathieu Parent [Thu, 27 Aug 2009 21:36:07 +0000 (23:36 +0200)]
Fix bashisms in samba event script.
Signed-off-by: Michael Adam <obnox@samba.org>
Mathieu Parent [Thu, 27 Aug 2009 21:35:41 +0000 (23:35 +0200)]
Fix bashisms in multipathd event script.
Signed-off-by: Michael Adam <obnox@samba.org>
Mathieu Parent [Thu, 27 Aug 2009 21:35:03 +0000 (23:35 +0200)]
Fix bashism in natgw eventscript.
Signed-off-by: Michael Adam <obnox@samba.org>
Ronnie Sahlberg [Wed, 9 Sep 2009 02:50:55 +0000 (12:50 +1000)]
allow the transaction commit to fail
Ronnie Sahlberg [Wed, 9 Sep 2009 02:50:21 +0000 (12:50 +1000)]
Merge commit 'martins/master'
Martin Schwenke [Wed, 9 Sep 2009 02:48:40 +0000 (12:48 +1000)]
Merge commit 'origin/master'
Ronnie Sahlberg [Wed, 9 Sep 2009 02:48:21 +0000 (12:48 +1000)]
dont check if commit failed, we do allow the commit to fail sometimes
Ronnie Sahlberg [Wed, 9 Sep 2009 00:57:39 +0000 (10:57 +1000)]
dont force an election just because the ban flag differs across the cluster.
a simple push to resync this flag is sufficient
Martin Schwenke [Tue, 8 Sep 2009 05:19:24 +0000 (15:19 +1000)]
Document onnode "onnode any".
Signed-off-by: Martin Schwenke <martin@meltin.net>
Martin Schwenke [Tue, 8 Sep 2009 05:10:20 +0000 (15:10 +1000)]
onnode: add "any" nodespec to select any node with running CTDB.
In testing and other situations (e.g. eventscripts) it is necessary to
select a node where a ctdb command can be run. The whole idea here is
to avoid nodes where ctdbd is not running and where most ctdb commands
would fail. This implements a standard way of doing this involving a
recursive onnode command.
There is still a small window for a race, where the selected node is
suddenly shutdown, but this is unavoidable.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Martin Schwenke [Mon, 7 Sep 2009 05:29:34 +0000 (15:29 +1000)]
Merge commit 'origin/master'
Ronnie Sahlberg [Thu, 3 Sep 2009 18:09:30 +0000 (04:09 +1000)]
lower the loglevel for the info messages that a public ip is not hosted locally for takeip/releaseip
Ronnie Sahlberg [Thu, 3 Sep 2009 17:05:37 +0000 (03:05 +1000)]
new version 1.0.89
Ronnie Sahlberg [Thu, 3 Sep 2009 16:59:24 +0000 (02:59 +1000)]
make it possible to have ctdb manage (start/stop/monitor) winbind without having samba
Ronnie Sahlberg [Thu, 3 Sep 2009 16:00:14 +0000 (02:00 +1000)]
Merge root@10.1.1.27:/shared/ctdb/ctdb-git
Ronnie Sahlberg [Thu, 3 Sep 2009 16:20:39 +0000 (02:20 +1000)]
new prototype banning code
Ronnie Sahlberg [Tue, 1 Sep 2009 18:39:17 +0000 (04:39 +1000)]
overwrite the state file, dont append to it.
dont log errors is trying to delete a nonexisting state file
this eliminates some annoying log entries in the ctdb log
Ronnie Sahlberg [Tue, 1 Sep 2009 17:12:27 +0000 (03:12 +1000)]
redirect stderr to dev null since the rule might not exist when we try to unconditionally delete it
Michael Adam [Thu, 27 Aug 2009 20:09:42 +0000 (22:09 +0200)]
set broadcast addresses in the takeip event.
Michael
Ronnie Sahlberg [Thu, 27 Aug 2009 19:19:44 +0000 (05:19 +1000)]
remove a check for the reclock file we dont need
Martin Schwenke [Thu, 27 Aug 2009 02:35:52 +0000 (12:35 +1000)]
Test suite: fix minor typo in complex/32_cifs_tickle.sh
Signed-off-by: Martin Schwenke <martin@meltin.net>