metze/ctdb/wip.git
9 years agotdb: kill last bits from swig
Kirill Smelkov [Wed, 21 Oct 2009 17:18:54 +0000 (21:18 +0400)]
tdb: kill last bits from swig

We no longer use swig for pytdb, so there is no need for swig make
rules. Also pytdb.c header should be updated.

Signed-off-by: Kirill Smelkov <kirr@mns.spb.ru>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
(cherry picked from samba commit ecbe5ebd8d1c44a478c613eb1cace0521142d0d3)

Signed-off-by: Stefan Metzmacher <metze@samba.org>
9 years agolib/tdb: sync build files from samba master
Stefan Metzmacher [Thu, 19 Nov 2009 14:44:10 +0000 (15:44 +0100)]
lib/tdb: sync build files from samba master

metze

Signed-off-by: Stefan Metzmacher <metze@samba.org>
9 years agos3 build: Remove unused fstat check to fix a bunch of HAVE_FSTAT warnings (cherry...
Stefan Metzmacher [Thu, 19 Nov 2009 15:52:46 +0000 (16:52 +0100)]
s3 build: Remove unused fstat check to fix a bunch of HAVE_FSTAT warnings (cherry picked from samba commit 2c2545d45af9c7479bf032fff1263e102926a495)

Signed-off-by: Stefan Metzmacher <metze@samba.org>
9 years agolib/tdb: include replace.h and system/filesys.h in pytdb.c
Stefan Metzmacher [Fri, 30 Jan 2009 18:38:59 +0000 (19:38 +0100)]
lib/tdb: include replace.h and system/filesys.h in pytdb.c

This fixes the build on Tru64.

metze
(cherry picked from samba commit 3718cf294ad1f3e00178cb34b2c914c9aecf2016)

Signed-off-by: Stefan Metzmacher <metze@samba.org>
9 years agoAvoid using a utility header for Python replacements included in Samba, since this...
Stefan Metzmacher [Thu, 19 Nov 2009 15:50:50 +0000 (16:50 +0100)]
Avoid using a utility header for Python replacements included in Samba, since this will not be shipped with talloc/tdb/tevent/etc. (cherry picked from samba commit ba5d6e6d706ebf07640ee24d701255cbb4803525)

Signed-off-by: Stefan Metzmacher <metze@samba.org>
9 years agos3/s4 build: Fix Py_RETURN_NONE to work with python versions < 2.4 (cherry picked...
Stefan Metzmacher [Thu, 19 Nov 2009 15:49:13 +0000 (16:49 +0100)]
s3/s4 build: Fix Py_RETURN_NONE to work with python versions < 2.4 (cherry picked from samba commit 61a23c5eea2203721a70a4ad41bfdc6cd1d6a2a7)

Signed-off-by: Stefan Metzmacher <metze@samba.org>
9 years agopy: Properly increase the reference counter of Py_None. (cherry picked from samba...
Stefan Metzmacher [Thu, 19 Nov 2009 15:31:29 +0000 (16:31 +0100)]
py: Properly increase the reference counter of Py_None. (cherry picked from samba commit d2c70d24e12293d9b4272eb310a6a4c4582b2d92)

Signed-off-by: Stefan Metzmacher <metze@samba.org>
9 years agoMake sure to not close tdb database more than once. (cherry picked from samba commit...
Jelmer Vernooij [Thu, 18 Dec 2008 20:42:50 +0000 (20:42 +0000)]
Make sure to not close tdb database more than once. (cherry picked from samba commit 6fe6983e4c960abc69d5fa80cbef534ae515209a)

Signed-off-by: Stefan Metzmacher <metze@samba.org>
9 years agoImplement missing functions in pytdb. (cherry picked from samba commit 2da551bbcc6cab...
Jelmer Vernooij [Thu, 18 Dec 2008 19:41:02 +0000 (19:41 +0000)]
Implement missing functions in pytdb. (cherry picked from samba commit 2da551bbcc6cab296769c193b0b82aaa6256cece)

Signed-off-by: Stefan Metzmacher <metze@samba.org>
9 years agoAdd simple manually written replacement for the tdb module. (cherry picked from samba...
Stefan Metzmacher [Thu, 19 Nov 2009 15:25:49 +0000 (16:25 +0100)]
Add simple manually written replacement for the tdb module. (cherry picked from samba commit 2a61fd41e9d4d2e3d7b47a816544c88bb369bc03)

Signed-off-by: Stefan Metzmacher <metze@samba.org>
9 years agotdb: Add simple reimplementation of tdbdump in Python as an example of the tdb Python...
Jelmer Vernooij [Thu, 10 Jan 2008 20:44:38 +0000 (21:44 +0100)]
tdb: Add simple reimplementation of tdbdump in Python as an example of the tdb Python bindings. (This used to be commit 47d797f7885b1e7bcff724496ecb1990e8440eea) (cherry picked from samba commit 6bdd1425b75c8931965f0e5627f5a63dc6820a7c)

Signed-off-by: Stefan Metzmacher <metze@samba.org>
9 years agoRemove unecessary msync. Jeremy. (cherry picked from samba commit 0bae1ef3de8fda5e1e2...
Jeremy Allison [Fri, 5 Sep 2008 22:32:22 +0000 (15:32 -0700)]
Remove unecessary msync. Jeremy. (cherry picked from samba commit 0bae1ef3de8fda5e1e2d641b14a408e627396912) (This used to be commit db2acaf46fdc38078b6b28b68909e289f6c9e0ec) (cherry picked from samba commit a1cf3ad5d655cf5a847df6d6299b3af1a30ec1e3)

Signed-off-by: Stefan Metzmacher <metze@samba.org>
9 years agoThe msync manpage reports that msync *must* be called before munmap. Failure to do...
Stefan Metzmacher [Thu, 19 Nov 2009 14:35:38 +0000 (15:35 +0100)]
The msync manpage reports that msync *must* be called before munmap. Failure to do so may result in lost data. Fix an ifdef check, I really think we meant to check HAVE_MMAP here. (cherry picked from samba commit 74c8575b3f3b90ea21ae6aa7ccd95947838af956) (This used to be commit 8fd54bb55f0c23bd025d1719abcbe75c6a2ea8ac) (cherry picked from samba commit b39e332bd7803dd4ce8267f750c497b5deb837ee)

Signed-off-by: Stefan Metzmacher <metze@samba.org>
9 years agoAttempt to fix bug 5684
Volker Lendecke [Tue, 12 Aug 2008 20:31:52 +0000 (22:31 +0200)]
Attempt to fix bug 5684

With the ctdb checkin dde9f3f006 tdb optimized out write lock checks for
write-enabled transaction. Sadly, this also removed the possibility to ever
remove dead records left over from tdb_delete calls within a transaction.

Tridge, please check this! Did dde9f3f006 have any reason beyond performance
optimizations?

Thanks,

Volker
(cherry picked from samba commit 3f884c4ae36f3260e63626bdd4989d9258ae6497)
(This used to be commit 1d85e0647e287d269b3f6b534da88f497d6f76c3)
(cherry picked from samba commit 8c88209c6f4b57b0dbe1459bd5bc583c5d321758)

Signed-off-by: Stefan Metzmacher <metze@samba.org>
9 years agolib/tdb/tools/tdbtorture.c: fixed memory leak.
Slava Semushin [Sat, 6 Jun 2009 17:06:04 +0000 (00:06 +0700)]
lib/tdb/tools/tdbtorture.c: fixed memory leak.

Found by cppcheck:
[lib/tdb/tools/tdbtorture.c:326]: (error) Memory leak: pids
(cherry picked from samba commit 497b9e460b5defd60de89b9c5bed890f83ca86a1)

Signed-off-by: Stefan Metzmacher <metze@samba.org>
9 years agoadded basic testing of tdb_transaction_prepare_commit() in tdbtorture (cherry picked...
Andrew Tridgell [Tue, 31 Mar 2009 02:59:03 +0000 (13:59 +1100)]
added basic testing of tdb_transaction_prepare_commit() in tdbtorture (cherry picked from samba commit 84547b8dba3c0cf4e20b3c50d9386081d475df6b)

Signed-off-by: Stefan Metzmacher <metze@samba.org>
9 years agomake tdbbackup use transactions
Andrew Tridgell [Tue, 16 Dec 2008 03:36:56 +0000 (14:36 +1100)]
make tdbbackup use transactions

tdbbackup was originally written before we had transactions, and it
attempted to use its own fsync() calls to make it safe. Now that we
have transactions we can do it in a much safer (and faster!) fashion
(cherry picked from samba commit 2e4247782bd5812bc8e7ea24194c8436748bb2fa)

Signed-off-by: Stefan Metzmacher <metze@samba.org>
9 years agofixed tdbbackup to give tdb error messages (cherry picked from samba commit 08be1420b...
Andrew Tridgell [Thu, 28 May 2009 07:35:12 +0000 (17:35 +1000)]
fixed tdbbackup to give tdb error messages (cherry picked from samba commit 08be1420ba52ef9bba90d0f811c7810841ee8568)

Signed-off-by: Stefan Metzmacher <metze@samba.org>
9 years agolib/tdb: add -t (always use transactions) option to tdbtorture
Rusty Russell [Wed, 21 Oct 2009 13:40:54 +0000 (00:10 +1030)]
lib/tdb: add -t (always use transactions) option to tdbtorture

This means you can kill it at any time and expect no corruption.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
(cherry picked from samba commit 0fc6800005ffb532a5e5699c97f13f1de138d51f)

Signed-off-by: Stefan Metzmacher <metze@samba.org>
9 years agolib/tdb: wean off TDB_ERRCODE.
Rusty Russell [Wed, 21 Oct 2009 13:39:43 +0000 (00:09 +1030)]
lib/tdb: wean off TDB_ERRCODE.

It was a regrettable hack which I used to reduce line count in tdb; in fact it caused confusion as can be seen in this patch.
In particular, ecode now needs to be set before TDB_LOG anyway, and having it exposed in
the header is useless (the struct tdb_context isn't defined, so it's doubly useless).
Also, we should never set errno, as io.c was doing.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
(cherry picked from samba commit b77f41d58b05101e02d8ac0e54cb0e30807d89c2)

Signed-off-by: Stefan Metzmacher <metze@samba.org>
9 years agolib/tdb: TDB_TRACE support (for developers)
Rusty Russell [Tue, 20 Oct 2009 01:49:41 +0000 (12:19 +1030)]
lib/tdb: TDB_TRACE support (for developers)

When TDB_TRACE is defined (in tdb_private.h), verbose tracing of tdb operations is enabled.
This can be replayed using "replay_trace" from http://ccan.ozlabs.org/info/tdb.

The majority of this patch comes from moving internal functions to _<funcname> to
avoid double-tracing.  There should be no additional overhead for the normal (!TDB_TRACE)
case.

Note that the verbose traces compress really well with rzip.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
(cherry picked from samba commit 703004340c3e0f43f741bd368d2525cfd187d590)

Signed-off-by: Stefan Metzmacher <metze@samba.org>
9 years agotdb: fixed the intermittent failure of tdbtorture in the build farm
Andrew Tridgell [Mon, 19 Oct 2009 23:59:40 +0000 (10:59 +1100)]
tdb: fixed the intermittent failure of tdbtorture in the build farm

There was a race condition that caused the torture.tdb to be left in a
state that needed recovery. The torture code thought that any message
from the tdb code was an error, so the "recovered" message, which is a
TDB_DEBUG_TRACE message, marked the run as being an error when it
isn't.
(cherry picked from samba commit 5dcf0069b68149575fe6166019dd90b73885926b)

Signed-off-by: Stefan Metzmacher <metze@samba.org>
9 years agotdb:tdbtool: fix indentation.
Michael Adam [Sun, 20 Sep 2009 22:08:34 +0000 (00:08 +0200)]
tdb:tdbtool: fix indentation.

Michael
(cherry picked from samba commit e440a2e11e78a562f97971c0dfe0cf3f694996ff)

Signed-off-by: Stefan Metzmacher <metze@samba.org>
9 years agoFix all warnings in source3 with gcc4.3. Jeremy. (cherry picked from samba commit...
Stefan Metzmacher [Thu, 19 Nov 2009 13:31:29 +0000 (14:31 +0100)]
Fix all warnings in source3 with gcc4.3. Jeremy. (cherry picked from samba commit 07e0094365e8dc360a83eec2e7cf9b1d5d8d6d00)

Signed-off-by: Stefan Metzmacher <metze@samba.org>
9 years agos3/s4: Fix "shadows a global declaration" warning (cherry picked from samba commit...
Tim Prouty [Tue, 16 Dec 2008 16:33:58 +0000 (08:33 -0800)]
s3/s4: Fix "shadows a global declaration" warning (cherry picked from samba commit e48a5cd5d4fc2626b09af16a0ac68c638d081437)

Signed-off-by: Stefan Metzmacher <metze@samba.org>
9 years agotdb: Fix some recently introduced warnings in tdbtool (cherry picked from samba commi...
Tim Prouty [Wed, 27 May 2009 19:52:37 +0000 (12:52 -0700)]
tdb: Fix some recently introduced warnings in tdbtool (cherry picked from samba commit c299833bf8e6506c793d6e8283743949aaac9ef4)

Signed-off-by: Stefan Metzmacher <metze@samba.org>
9 years agoadded some more speed tests to tdbtool
Andrew Tridgell [Mon, 25 May 2009 23:24:37 +0000 (09:24 +1000)]
added some more speed tests to tdbtool

This adds 3 simple speed tests to tdbtool, for transaction store,
store and fetch.

On my laptop this shows transactions costing about 10ms
(cherry picked from samba commit e15027155d3d880abde83124e252b3dd10a9aae4)

Signed-off-by: Stefan Metzmacher <metze@samba.org>
9 years agotdb:tdbtool: add transaction_start/_commit/_cancel commands.
Michael Adam [Sun, 20 Sep 2009 21:58:27 +0000 (23:58 +0200)]
tdb:tdbtool: add transaction_start/_commit/_cancel commands.

So one can perform tdbtool operations protected by transactions.

Michael
(cherry picked from samba commit 91e1bab2e9a3f33151061554f2dcf05bcb728312)

Signed-off-by: Stefan Metzmacher <metze@samba.org>
9 years agotdb:tdbtool: add the "speed" command to the help text.
Michael Adam [Sun, 20 Sep 2009 21:58:05 +0000 (23:58 +0200)]
tdb:tdbtool: add the "speed" command to the help text.

Michael
(cherry picked from samba commit 817383d88d820e7be5b5dd3d2da350d4a03a94de)

Signed-off-by: Stefan Metzmacher <metze@samba.org>
9 years agoAdded a simple tdb integrity check to tdbtool. The command "check" runs traverse...
Holger Hetterich [Sat, 1 Nov 2008 23:12:32 +0000 (00:12 +0100)]
Added a simple tdb integrity check to tdbtool. The command "check" runs traverse on the currently open tdb, and returns the number of entries if the integrity check is successful. (cherry picked from samba commit 42366bcbbdd42bb9d5821dfcc9dbe71a1eafa330)

Signed-off-by: Stefan Metzmacher <metze@samba.org>
9 years agotdb: allow reads after prepare commit
Andrew Tridgell [Tue, 15 Sep 2009 21:04:22 +0000 (14:04 -0700)]
tdb: allow reads after prepare commit

We previously only allowed a commit to happen after a prepare
commit. It is in fact safe to allow reads between a prepare and a
commit, and the s4 replication code can make use of that, so allow it.
(cherry picked from samba commit 46c99ec2a3781d53138245182345c6c2ddd9a258)

Signed-off-by: Stefan Metzmacher <metze@samba.org>
9 years agotdb:mksigs: allow PRINTF_ATTRIBUTE(..) macros function types as funcion args
Michael Adam [Fri, 11 Sep 2009 13:20:34 +0000 (15:20 +0200)]
tdb:mksigs: allow PRINTF_ATTRIBUTE(..) macros function types as funcion args

Michael
(cherry picked from samba commit 55dcf928eb6ce603c5e95a9a80856a4deb33d0c6)

Signed-off-by: Stefan Metzmacher <metze@samba.org>
9 years agotdb:mksigs: normalize bool -> _Bool
Michael Adam [Fri, 11 Sep 2009 13:16:03 +0000 (15:16 +0200)]
tdb:mksigs: normalize bool -> _Bool

Michael
(cherry picked from samba commit cfa4e7ec7540d1100649839a10968303189fe929)

Signed-off-by: Stefan Metzmacher <metze@samba.org>
9 years agotdb:mksigs: ignore symbols (like _DEPRECATED_) after closing function parentheses
Michael Adam [Fri, 11 Sep 2009 13:11:16 +0000 (15:11 +0200)]
tdb:mksigs: ignore symbols (like _DEPRECATED_) after closing function parentheses

Michael
(cherry picked from samba commit 25939a627f15b7a21110767d47be0f50f32d3943)

Signed-off-by: Stefan Metzmacher <metze@samba.org>
9 years agotdb:mksigs: correctly ignode multiline function typedefs
Michael Adam [Fri, 11 Sep 2009 13:01:46 +0000 (15:01 +0200)]
tdb:mksigs: correctly ignode multiline function typedefs

by first concatenating multilint parentheses and removing typefes afterwards.

Michael
(cherry picked from samba commit 13bfcd5a93c47c9db8b644560a1bcc398facb136)

Signed-off-by: Stefan Metzmacher <metze@samba.org>
9 years agotdb:mksigs: ignore struct forward declarations.
Michael Adam [Fri, 11 Sep 2009 12:55:52 +0000 (14:55 +0200)]
tdb:mksigs: ignore struct forward declarations.

Michael
(cherry picked from samba commit ecd12bfb382da072595391d5bf11a893d39a0479)

Signed-off-by: Stefan Metzmacher <metze@samba.org>
9 years agotdb:mksyms: allow characters after closing functions parenthesis.
Michael Adam [Fri, 11 Sep 2009 12:54:30 +0000 (14:54 +0200)]
tdb:mksyms: allow characters after closing functions parenthesis.

Michael
(cherry picked from samba commit 400f08450b26f38a7dafd1d458542b4d9a8cb19e)

Signed-off-by: Stefan Metzmacher <metze@samba.org>
9 years agotdb:mksyms: allow double pointer return value of functions.
Michael Adam [Fri, 11 Sep 2009 12:54:02 +0000 (14:54 +0200)]
tdb:mksyms: allow double pointer return value of functions.

Michael
(cherry picked from samba commit 907e05595fdcc4ef77ad627bc0f3732faa59de68)

Signed-off-by: Stefan Metzmacher <metze@samba.org>
9 years agotdb: fix c++ build warning.
Günther Deschner [Mon, 7 Sep 2009 09:56:10 +0000 (11:56 +0200)]
tdb: fix c++ build warning.

Guenther
(cherry picked from samba commit 1c2f4919abe99d07540b7380ad16d475d62aa50b)

Signed-off-by: Stefan Metzmacher <metze@samba.org>
9 years agoOne would expect I could spell my name... (cherry picked from samba commit 0d120be36b...
Michael Adam [Sat, 29 Aug 2009 22:55:28 +0000 (00:55 +0200)]
One would expect I could spell my name... (cherry picked from samba commit 0d120be36bfc561e3f679d081993ccc6bea2a401)

Signed-off-by: Stefan Metzmacher <metze@samba.org>
9 years agotdb: add script/abi_checks.sh. check for abi changes without gcc magic.
Michael Adam [Fri, 28 Aug 2009 13:53:12 +0000 (15:53 +0200)]
tdb: add script/abi_checks.sh. check for abi changes without gcc magic.

USAGE: abi_checks.sh LIBRARY_NAME header1 [header2 ...]

This creates symbol signature lists using the mksyms and mksigs scripts
and compares them with the checked in lists.

Michael
(cherry picked from samba commit 9636e0d373e75cce7063b710227eaaf53f447c1b)

Signed-off-by: Stefan Metzmacher <metze@samba.org>
9 years agotdb: add script to extract signatures from header files.
Michael Adam [Fri, 28 Aug 2009 13:08:19 +0000 (15:08 +0200)]
tdb: add script to extract signatures from header files.

This produces output like the output gcc produces when
invoked with the -aux-info switch.

Run like this: cat include/tdb.h | ./script/mksigs.pl

This simple parser is probably too coarse to handle all
possible header files, but it treats tdb.h correctly...

Michael
(cherry picked from samba commit 0760a04ef9f7d2f3d966017295712769d02b8b9f)

Signed-off-by: Stefan Metzmacher <metze@samba.org>
9 years agotdb: add scripts to extract library symbols (exports file) from headers
Michael Adam [Fri, 28 Aug 2009 13:01:17 +0000 (15:01 +0200)]
tdb: add scripts to extract library symbols (exports file) from headers

Michael
(cherry picked from samba commit 006fd0c43c7c403b8671dfc46e5ee31e92480e1f)

Signed-off-by: Stefan Metzmacher <metze@samba.org>
9 years agolib/tdb: don't overwrite TDBs with different version numbers.
Rusty Russell [Fri, 28 Aug 2009 02:26:34 +0000 (11:56 +0930)]
lib/tdb: don't overwrite TDBs with different version numbers.

In future, this may happen, and we don't want to clobber them.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
(cherry picked from samba commit 398d0c2929026fccb3409316720a4dcad225ab05)

Signed-off-by: Stefan Metzmacher <metze@samba.org>
9 years agoAdd define guards around otherwise unused variable. Jeremy. (cherry picked from samba...
Jeremy Allison [Thu, 6 Aug 2009 18:47:08 +0000 (11:47 -0700)]
Add define guards around otherwise unused variable. Jeremy. (cherry picked from samba commit 4fc9f9c3f943cdeb27e37f0ee068cdd0da7cb00c)

Signed-off-by: Stefan Metzmacher <metze@samba.org>
9 years agoThere is one signedness issue in tdb which prevents traverses of TDB records over...
Rusty Russell [Thu, 6 Aug 2009 03:13:42 +0000 (13:13 +1000)]
There is one signedness issue in tdb which prevents traverses of TDB records over the 2G offset on systems which support 64 bit file offsets. This fixes that case.

On systems with 32 bit offsets, expansion and fcntl locking on these records
will fail anyway.  SAMBA already does '#define _FILE_OFFSET_BITS 64' in
config.h (on my 32-bit x86 Linux system at least) to get 64 bit file offsets.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
(cherry picked from samba commit 252f7da702fd0409f7bfff05ef594911ededa32f)

Signed-off-by: Stefan Metzmacher <metze@samba.org>
9 years agoRevert "cleanup: remove a tunable we no longer use in the eventscripts any more :"
Ronnie Sahlberg [Tue, 15 Dec 2009 22:51:17 +0000 (09:51 +1100)]
Revert "cleanup:   remove a tunable we no longer use in the eventscripts any more :"

This reverts commit 401f421fa003d9515df15e759b50b56e0c67d69c.

Conflicts:

include/ctdb_private.h
server/ctdb_tunables.c

9 years agoMerge branch 'trans3'
Ronnie Sahlberg [Tue, 15 Dec 2009 10:00:22 +0000 (21:00 +1100)]
Merge branch 'trans3'

9 years agoAuthor: Rusty Russell <rusty@rustcorp.com.au>
Ronnie Sahlberg [Tue, 15 Dec 2009 09:56:16 +0000 (20:56 +1100)]
Author: Rusty Russell <rusty@rustcorp.com.au>
Date:   Tue Dec 15 15:53:30 2009 +1030

    eventscript: hack to avoid overloading valgrind

    Now we fork one child per script, when running under valgrind the
load
    gets quite high.  This is because valgrind does a lot of work after
exit,
    and we don't wait for the children to finish; we start the next one
when
    the child reports status via the pipe.

    This fix is ugly, but simple.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
9 years agoThis is a dodgy patch.
Ronnie Sahlberg [Tue, 15 Dec 2009 08:04:52 +0000 (19:04 +1100)]
This is a dodgy patch.

I saw once where the master ctdbd logging structure was talloc freed
which caused issues.
So only free the structure if it is NOT the master structure.

This needs to be looked into in more detail.

9 years agoRevert "Make fetch_locked more scalable"
Ronnie Sahlberg [Tue, 15 Dec 2009 03:26:28 +0000 (14:26 +1100)]
Revert "Make fetch_locked more scalable"

This reverts commit 5736e17c139c9a8049e235429aeae0c6c9d0e93d.

9 years agoMerge commit 'obnox/ctdb-wip-trans3' into trans3
Ronnie Sahlberg [Tue, 15 Dec 2009 03:25:55 +0000 (14:25 +1100)]
Merge commit 'obnox/ctdb-wip-trans3' into trans3

9 years agoadd a new test tool that just locks and releases the same record over and over
Ronnie Sahlberg [Tue, 15 Dec 2009 01:14:49 +0000 (12:14 +1100)]
add a new test tool that just locks and releases the same record over and over

9 years agoctdb_fetch requires the number of nodes being specified.
Ronnie Sahlberg [Tue, 15 Dec 2009 00:29:16 +0000 (11:29 +1100)]
ctdb_fetch requires the number of nodes being specified.
Have it log an error and terminate if thie parameter was omitted

9 years agoWhen setting up the logging, set the event to trigger a read of a log message from...
Ronnie Sahlberg [Mon, 14 Dec 2009 23:45:18 +0000 (10:45 +1100)]
When setting up the logging, set the event to trigger a read of a log message from a child process as a child of the "log" structure and not the ctdb structure,
or else we can crash if we receive log messages from a child but the log structure has been freed()

9 years agoFrom rusty:
Ronnie Sahlberg [Mon, 14 Dec 2009 23:23:58 +0000 (10:23 +1100)]
From rusty:

Subject: eventscript: fix spinning at 100% cpu when child exits.

ctdbd was spinning reading 0 from a pipe, as soon as the first
eventscript finishes.

This was caused by the intersection between a78b8ea7168e "Run only one
event for each epoll_wait/select call" and 32cfdc3aec34 "eventscript:
ctdb_fork_with_logging()".  Unavoidable mid-air collision, since both
worked fine and both were developed simultaneously.

When the script exits, we have two pipes open to it: one for any
stdout/stderr for logging (ctdb_log_handler), and one for the result
(ctdb_event_script_handler).  The latter frees everything, including
the log fd and event structure.

We used to get one callback to ctdb_log_handler, which got a harmless
0-length read, then one to ctdb_event_script_handler which cleaned up.
Now we only do one callback per poll, we need the logging function to
clean itself up so we can make process.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
9 years agoRename the tunable EventScriptBanCount to EventScriptTimeoutCount
Ronnie Sahlberg [Mon, 14 Dec 2009 04:53:23 +0000 (15:53 +1100)]
Rename the tunable EventScriptBanCount to EventScriptTimeoutCount
since we no longer ban nodes when dodgy scripts continue to hang.

We now only mark nodes as unhealthy if monitor events fail or timeout. Never ban.

9 years agocleanup: remove a tunable we no longer use in the eventscripts any more :
Ronnie Sahlberg [Mon, 14 Dec 2009 04:48:47 +0000 (15:48 +1100)]
cleanup:   remove a tunable we no longer use in the eventscripts any more :
EventScriptUnhealthyOnTimeout

9 years agoctdb: don't print OUTPUT: for DISABLED scripts
Rusty Russell [Thu, 10 Dec 2009 10:05:54 +0000 (20:35 +1030)]
ctdb: don't print OUTPUT: for DISABLED scripts

In other news, did you know ctime() returns a \n-terminated string?

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
9 years agoeventscript: fix monitoring when killed by another script command
Rusty Russell [Thu, 10 Dec 2009 09:55:33 +0000 (20:25 +1030)]
eventscript: fix monitoring when killed by another script command

Commit c1ba1392fe "eventscript: get rid of ctdb_control_event_script_finished
altogether" was wrong: there is one case where we want to free the script
without transferring their status to last_status.  This happens because we
always kill an running monitor command when we run any other command.

This still isn't quite right (and never was): the callback will be called
with status value 0, which might flip us to HEALTHY if we were unhealthy.
This is conveniently fixed in my next set of patches :)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
9 years agoremove the variable "disable when unhealthy"
Ronnie Sahlberg [Mon, 14 Dec 2009 04:40:54 +0000 (15:40 +1100)]
remove the variable "disable when unhealthy"
there is no rational need for a setting where we permanently mark nodes as disabled everytime an eventscript fails

9 years agoRevert "recovery: add special pull-logic for persistent databases"
Michael Adam [Fri, 11 Dec 2009 16:05:30 +0000 (17:05 +0100)]
Revert "recovery: add special pull-logic for persistent databases"

This reverts commit 8aef46d2aab3efb322dda51eaa202653cefd5222.

This special recovery logic is wrong now with the transaction rewrite.
The treatment of persistent databases will later be rewritten to use the
database sequence number.

Michael

9 years agoMake fetch_locked more scalable
Volker Lendecke [Wed, 9 Dec 2009 14:11:45 +0000 (15:11 +0100)]
Make fetch_locked more scalable

This patch improves the handling of the fetch_lock operation on non-persistent
databases that ctdb clients have to do very frequently.

The normal flow how this goes is the following:

1. Client does a local fetch_lock on the database

2. Client looks if the local node is dmaster.
   If yes, everything is fine
   If no, continue here

3. Client unlocks the local record

4. Client issues a "get me the record" call to ctdbd

5. ctdbd goes out and fetches the dmaster role

6. ctdbd tells the client to retry

7. Client starts over again

The problem is between step 6 and 7: Before the client has had the chance to
retry (i.e. catch the record with a fetch_locked), another node might have come
asking ctdbd to migrate away the record again. This is a real problem, I've
seen >20 loops of this kind in real workloads.

This patch does the following: Whenever ctdb receives a record as result of
step 5, it puts the key on a "holdback list". As long as a key is on this list,
a request to migrate away the dmaster is put on hold. It is the client's duty
to issue the "CTDB_CONTROL_GOTIT" control when it has successfully done step 2
after having asked ctdb to fetch the record. This will release the key from the
"holdback list" and re-issue all dmaster migration requests.

As a safeguard against malicious clients, once a second (default 1000msecs,
tunable "HoldbackCleanupInterval" in milliseconds) ctdbd goes over the list of
held back keys, deletes them and releases all held back migration requests.

9 years agoImport "talloc_array_length" from upstream talloc
Volker Lendecke [Thu, 10 Dec 2009 12:02:29 +0000 (13:02 +0100)]
Import "talloc_array_length" from upstream talloc

9 years agotests: temporarily disable the transaction test tool.
Michael Adam [Fri, 11 Dec 2009 15:39:58 +0000 (16:39 +0100)]
tests: temporarily disable the transaction test tool.

Make it return success for make test.
This is temporarily disabled until the rewrite of the
transaction code (in samba and the daemon) using the global
lock feature has been ported to the ctdb client code.

Michael

9 years agoAdd a new control CTDB_GET_DB_SEQNUM - fetch a persistent db's sequence number.
Michael Adam [Fri, 11 Dec 2009 14:31:02 +0000 (15:31 +0100)]
Add a new control CTDB_GET_DB_SEQNUM - fetch a persistent db's sequence number.

Michael

9 years agodefine CTDB_DB_SEQNUM_KEY - used with the new implementation of transactions.
Michael Adam [Fri, 11 Dec 2009 13:19:55 +0000 (14:19 +0100)]
define CTDB_DB_SEQNUM_KEY - used with the new implementation of transactions.

Michael

9 years agoTiny simplification of ctdb_queue_packet()
Volker Lendecke [Wed, 9 Dec 2009 16:20:23 +0000 (17:20 +0100)]
Tiny simplification of ctdb_queue_packet()

9 years agoRename a struct member for clarity
Volker Lendecke [Tue, 8 Dec 2009 16:00:55 +0000 (17:00 +0100)]
Rename a struct member for clarity

9 years agoserver: add a new control CTDB_CONTROL_TRANS3_COMMIT
Michael Adam [Thu, 3 Dec 2009 16:59:49 +0000 (17:59 +0100)]
server: add a new control CTDB_CONTROL_TRANS3_COMMIT

This is a simplified version of the trans2 commit control:
It just rolls out the marshall buffer to all active nodes.

It is the main ctdbd part of the re-implementation of the
persistent transactions. The client code is changed to
take a global lock to start a transactions and store into
the marshal buffer instead of writing to the local tdb
under a local transaction.

The old transaction implementation is going to be
removed in a later commit.

Michael

9 years agoFrom: Volker Lendecke <vl@samba.org>
Ronnie Sahlberg [Wed, 9 Dec 2009 21:53:55 +0000 (08:53 +1100)]
From: Volker Lendecke <vl@samba.org>
Date: Wed, 9 Dec 2009 22:45:12 +0100
Subject: [PATCH] Revert an accidential commit

9 years agotests: remove the no_trans mode from ctdb_transaction.
Michael Adam [Wed, 9 Dec 2009 21:04:48 +0000 (22:04 +0100)]
tests: remove the no_trans mode from ctdb_transaction.

Writes without transaction are not possible any more on
persistent databases.

Michael

9 years agotests: remove the persistent_unsafe writes test.
Michael Adam [Thu, 30 Jul 2009 09:59:59 +0000 (11:59 +0200)]
tests: remove the persistent_unsafe writes test.

This is useless now that persistent write operations without
transaction are forbidden.

Michael

9 years agotests: remove persistent_safe write test.
Michael Adam [Thu, 30 Jul 2009 09:59:02 +0000 (11:59 +0200)]
tests: remove persistent_safe write test.

This is useless now that persistent writes without transactions are forbidden.

Michael

9 years agotest: add test 54_ctdb_transaction_recovery.sh
Michael Adam [Wed, 9 Dec 2009 20:38:44 +0000 (21:38 +0100)]
test: add test 54_ctdb_transaction_recovery.sh

This is like the 53_ctdb_transaction test, but it additionally
runs a loop with recoveries while the transactions are running.

When called like this, the transaction loops run for 10 minutes:

CTDB_TEST_TIMELIMIT=600 tests/scripts/run_tests tests/simple/54_ctdb_transaction_recovery.sh

The default timelimit is 30 seconds.

Michael

9 years agotest: get value for --timelimit from environment var CTDB_TEST_TIMELIMIT in transacti...
Michael Adam [Wed, 9 Dec 2009 20:36:42 +0000 (21:36 +0100)]
test: get value for --timelimit from environment var CTDB_TEST_TIMELIMIT in transaction test

Michael

9 years agoclient: lower level of commit retry message WARNING->DEBUG
Michael Adam [Wed, 9 Dec 2009 14:05:20 +0000 (15:05 +0100)]
client: lower level of commit retry message WARNING->DEBUG

This can happen frequently when recoveries intercept transactions.

Michael

9 years agoclient: lower debug level of transaction-active-retry message to DEBUG
Michael Adam [Wed, 9 Dec 2009 12:48:49 +0000 (13:48 +0100)]
client: lower debug level of transaction-active-retry message to DEBUG

This reduces some noise.

Michael

9 years agocall: lower the debug message "refusing migration while transction" to lvl INFO
Michael Adam [Wed, 9 Dec 2009 12:43:38 +0000 (13:43 +0100)]
call: lower the debug message "refusing migration while transction" to lvl INFO

This gets just too noisy on a busy system.
And it is purley informational anyways...

Michael

9 years agoRun only one event for each epoll_wait/select call
Volker Lendecke [Wed, 9 Dec 2009 16:14:16 +0000 (17:14 +0100)]
Run only one event for each epoll_wait/select call

This might be a bit less efficient, but experience in winbind has shown that
event callbacks can trigger changes in the socket state in very hard to
diagnose ways.

9 years agoreduce vacuuming lognoise
Christian Ambach [Tue, 8 Dec 2009 18:23:19 +0000 (19:23 +0100)]
reduce vacuuming lognoise

syslog.h says:

LOG_NOTICE      5    normal but significant condition
LOG_INFO        6    informational

several vacuuming related logs logged at NOTICE level although I don't see
any real significance, these are just informational messages for me

Signed-off-by: Christian Ambach <christian.ambach@de.ibm.com>
9 years agoimprove time jump logging
Christian Ambach [Tue, 8 Dec 2009 18:08:37 +0000 (19:08 +0100)]
improve time jump logging

add the __location__ macro to the logs to get a better idea
in which loop the problem occured

Signed-off-by: Christian Ambach <christian.ambach@de.ibm.com>
9 years agoMerge commit 'rusty/script-report'
Ronnie Sahlberg [Wed, 9 Dec 2009 03:26:42 +0000 (14:26 +1100)]
Merge commit 'rusty/script-report'

9 years agoBond devices can have any name the user configures, so
Ronnie Sahlberg [Wed, 9 Dec 2009 00:33:04 +0000 (11:33 +1100)]
Bond devices can have any name the user configures, so
when checking link status for an interface, first
check if this interface is in fact a bond device
(by the precense of a /proc/net/bonding/IFACE file)
and use that file for checking status.

Othervise assume ib* is an infiniband interface which we donnt know how
to check, or otherwise it is an ethernet interface and ethtool should
hopefully work.

9 years agomake sure to also check that interfaces used for NATGW are ok
Ronnie Sahlberg [Wed, 9 Dec 2009 00:13:29 +0000 (11:13 +1100)]
make sure to also check that interfaces used for NATGW are ok
and have a link.
if not the node should become unhealthy

9 years agoevents/50.samba: only use wbinfo --ping-dc if available
Stefan Metzmacher [Mon, 7 Dec 2009 13:37:21 +0000 (14:37 +0100)]
events/50.samba: only use wbinfo --ping-dc if available

metze

9 years agoctdb: scriptstatus can now query non-monitor events
Rusty Russell [Mon, 7 Dec 2009 15:20:55 +0000 (01:50 +1030)]
ctdb: scriptstatus can now query non-monitor events

We also no longer return an error before scripts have been run; a special
zero-length data means we have never run the scripts.

"ctdb scriptstatus all" returns all event script results.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
9 years agoeventscript: expost call names and enum
Rusty Russell [Mon, 7 Dec 2009 15:17:13 +0000 (01:47 +1030)]
eventscript: expost call names and enum

We're going to need this so ctdb can query non-monitor status.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
9 years agoeventscript: lock logging on timeout.
Rusty Russell [Mon, 7 Dec 2009 15:02:36 +0000 (01:32 +1030)]
eventscript: lock logging on timeout.

Ronnie suggested this; seems like a very good idea.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
9 years agoctdb: support --machinereadable (-Y) for scriptstatus
Rusty Russell [Mon, 7 Dec 2009 15:01:53 +0000 (01:31 +1030)]
ctdb: support --machinereadable (-Y) for scriptstatus

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
9 years agoeventscript: get rid of ctdb_control_event_script_finished altogether
Rusty Russell [Tue, 8 Dec 2009 01:59:10 +0000 (12:29 +1030)]
eventscript: get rid of ctdb_control_event_script_finished altogether

We always have to call it before freeing the state; we should just do
this work in the destructor itself.

Unfortunately, the script state would already be freed by the time
the state destructor is called, so we make the script state a child of
ctdb, and talloc_free() it manually on the one path which doesn't use
the destructor.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
9 years agoeventscript: save state for all script invocations
Rusty Russell [Tue, 8 Dec 2009 01:57:48 +0000 (12:27 +1030)]
eventscript: save state for all script invocations

Rather than only tranferring to last_status for monitor events, do
it for every event (ctdb->last_status is now an array).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
9 years agoeventscript: cleanup finished to take state arg
Rusty Russell [Tue, 8 Dec 2009 01:54:56 +0000 (12:24 +1030)]
eventscript: cleanup finished to take state arg

We only need ctdb->current_monitor so we can kill it when we want to run
something else; we don't need to use it here as we always know what script
we are running.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
9 years agoeventscript: use wire format internally for script status.
Rusty Russell [Tue, 8 Dec 2009 02:18:17 +0000 (12:48 +1030)]
eventscript: use wire format internally for script status.

The only difference between the exposed an internal structure now is
that the name and output fields were pointers.  Switch to using
ctdb_scripts_wire/ctdb_script_wire internally as well so marshalling
is a noop.

We now reject scripts which are too long and truncate logging to the
511 characters we have space for (the entire output will be in the
normal ctdbd log).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
9 years agoeventscript: rename ctdb_monitoring_wire to ctdb_scripts_wire
Rusty Russell [Mon, 7 Dec 2009 14:21:24 +0000 (00:51 +1030)]
eventscript: rename ctdb_monitoring_wire to ctdb_scripts_wire

We're going to allow fetching status of all script runs, so this
name is no longer appropriate.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
9 years agoeventscript: get_current_script() helper
Rusty Russell [Tue, 8 Dec 2009 02:17:24 +0000 (12:47 +1030)]
eventscript: get_current_script() helper

This neatens the code slightly.  We also use the name 'current' in
ctdb_event_script_handler() for uniformity.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
9 years agoeventscript: use an array rather than a linked list of scripts
Rusty Russell [Tue, 8 Dec 2009 02:17:05 +0000 (12:47 +1030)]
eventscript: use an array rather than a linked list of scripts

This brings us closer to the wire format, by using a simple array
and a 'current' iterator.

The downside is that a 'struct ctdb_script' is no longer a talloc
object: the state must be passed to our log fn, and the current
script extracted with &state->scripts->scripts[state->current].

The wackiness of marshalling is simplified, and as a bonus, we can
distinguish between an empty event directory
(state->scripts->num_scripts == 0) and and error (state->scripts ==
NULL).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
9 years agoeventscript: record script status for all events
Rusty Russell [Tue, 8 Dec 2009 02:16:18 +0000 (12:46 +1030)]
eventscript: record script status for all events

This unifies almost everything: the state->current pointer points to
the struct ctdb_script where we record start, finish, status and
output.

We still only marshall up the monitor events; the rest disappear when
the state structure is freed.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
9 years agoeventscript: use scripts array directly, rather than separate list
Rusty Russell [Tue, 8 Dec 2009 02:15:17 +0000 (12:45 +1030)]
eventscript: use scripts array directly, rather than separate list

We rename ctdb_monitor_script_status to ctdb_script, and instead of
allocating them as the scripts are executed, we allocate them up front
and keep a "current" interator.

This slightly simplifies the code, though it means we only marshall up
to the last successfully run script.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
9 years agoeventscript: ctdb_fork_with_logging()
Rusty Russell [Tue, 8 Dec 2009 02:14:30 +0000 (12:44 +1030)]
eventscript: ctdb_fork_with_logging()

A new helper functions which sets up an event attached to the child's
stdout/stderr which gets routed to the logging callback after being
placed in the normal logs.

This is a generalization of the previous code which was hardcoded to
call ctdb_log_event_script_output.

The only subtlety is that we hang the child fds off the output buffer;
the destructor for that will flush, which means it has to be destroyed
before the output buffer is.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>