kai/samba-autobuild/.git
7 years agoscript: really sleep in "random-sleep.sh"
Michael Adam [Wed, 20 Jun 2012 22:39:21 +0000 (00:39 +0200)]
script: really sleep in "random-sleep.sh"

Signed-off-by: Stefan Metzmacher <metze@samba.org>
7 years agosamba-tool: gpo: Fix creation of filesystem ACL from directory ACL
Amitay Isaacs [Tue, 19 Jun 2012 06:49:33 +0000 (16:49 +1000)]
samba-tool: gpo: Fix creation of filesystem ACL from directory ACL

Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Thu Jun 21 03:25:57 CEST 2012 on sn-devel-104

7 years agos4-pysmb: Parse security info as an unsigned integer
Amitay Isaacs [Wed, 20 Jun 2012 07:29:55 +0000 (17:29 +1000)]
s4-pysmb: Parse security info as an unsigned integer

7 years agos4-torture: Expand whoami test to confirm the user token.
Andrew Bartlett [Wed, 20 Jun 2012 08:23:18 +0000 (18:23 +1000)]
s4-torture: Expand whoami test to confirm the user token.

This uses the tokenGroups attribute on LDAP and the posix whoami call
to confirm that user token matches between LDAP and CIFS.

I have a seperate patch for the anonymous case, because this isn't
consistent at this stage, and we need to study and fix that.

Andrew Bartlett

Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Wed Jun 20 18:43:43 CEST 2012 on sn-devel-104

7 years agos4-torture: Change the unix.whoami test to use torture_assert()
Andrew Bartlett [Wed, 20 Jun 2012 07:26:48 +0000 (17:26 +1000)]
s4-torture: Change the unix.whoami test to use torture_assert()

7 years agontdb: fix occasional abort in testing.
Rusty Russell [Wed, 20 Jun 2012 12:01:21 +0000 (21:31 +0930)]
ntdb: fix occasional abort in testing.

Occasionally, the capability test inserts multiple used records and they
clash, but our primitive test layout engine doesn't handle hash clashes
and aborts.

Force a seed value which we know doesn't clash.

Reported-by: Andrew Bartlett <abartlet@samba.org>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Autobuild-User(master): Rusty Russell <rusty@rustcorp.com.au>
Autobuild-Date(master): Wed Jun 20 16:50:20 CEST 2012 on sn-devel-104

7 years agoscript/autobuild: delay start of small projects by 60 to 600 seconds
Stefan Metzmacher [Wed, 20 Jun 2012 09:03:48 +0000 (11:03 +0200)]
script/autobuild: delay start of small projects by 60 to 600 seconds

This should reduce the startup load.

metze

Autobuild-User(master): Stefan Metzmacher <metze@samba.org>
Autobuild-Date(master): Wed Jun 20 14:56:51 CEST 2012 on sn-devel-104

7 years agoscript/autobuild: don't use 'make -j 4' for bin/smbtorture4
Stefan Metzmacher [Wed, 20 Jun 2012 09:14:04 +0000 (11:14 +0200)]
script/autobuild: don't use 'make -j 4' for bin/smbtorture4

The '4' has no effect there, which results in NUM_JOBS == NUM_CORES.

metze

7 years agoscript/autobuild: don't use make -j for small projects
Stefan Metzmacher [Wed, 20 Jun 2012 08:36:23 +0000 (10:36 +0200)]
script/autobuild: don't use make -j for small projects

The top level and source3 builds take much longer than the standalone
projects, so there's no need to use 'make -j'.

metze

7 years agos3:smbd: fix the build with the --with-profiling-data option
Stefan Metzmacher [Wed, 20 Jun 2012 06:53:10 +0000 (08:53 +0200)]
s3:smbd: fix the build with the --with-profiling-data option

metze

7 years agos3:include: add START/END_PROFILE_STAMP()
Stefan Metzmacher [Wed, 20 Jun 2012 06:44:30 +0000 (08:44 +0200)]
s3:include: add START/END_PROFILE_STAMP()

This is needed when the profiling spans multiple functions.

metze

7 years agowafsamba/irixcc: next try to inject '-c99' the the build
Stefan Metzmacher [Wed, 20 Jun 2012 06:45:42 +0000 (08:45 +0200)]
wafsamba/irixcc: next try to inject '-c99' the the build

metze

7 years agoRevert "wafsamba/irixcc: add '-c99' option to cc"
Stefan Metzmacher [Tue, 19 Jun 2012 22:14:58 +0000 (00:14 +0200)]
Revert "wafsamba/irixcc: add '-c99' option to cc"

This reverts commit 59daf91f397c2ebef283ab9e6ecaabbe9dc58fc5.

Doesn't seem to work :-(

metze

7 years agoidmap-hash: Attempt to fix Coverity ID 709116 Overflowed array index write
Volker Lendecke [Wed, 20 Jun 2012 08:26:58 +0000 (10:26 +0200)]
idmap-hash: Attempt to fix Coverity ID 709116 Overflowed array index write

Signed-off-by: Michael Adam <obnox@samba.org>
Autobuild-User(master): Michael Adam <obnox@samba.org>
Autobuild-Date(master): Wed Jun 20 12:53:59 CEST 2012 on sn-devel-104

7 years agoidmap-hash: Fix Coverity ID 709117 Dereference before null check
Volker Lendecke [Wed, 20 Jun 2012 06:43:36 +0000 (08:43 +0200)]
idmap-hash: Fix Coverity ID 709117 Dereference before null check

Signed-off-by: Michael Adam <obnox@samba.org>
7 years agoidmap-hash: Fix Coverity 709118 Dereference before null check
Volker Lendecke [Wed, 20 Jun 2012 06:42:20 +0000 (08:42 +0200)]
idmap-hash: Fix Coverity 709118 Dereference before null check

Signed-off-by: Michael Adam <obnox@samba.org>
7 years agoidmap-hash: Adapt mapfile_read_line to README.Coding
Volker Lendecke [Wed, 20 Jun 2012 08:43:29 +0000 (10:43 +0200)]
idmap-hash: Adapt mapfile_read_line to README.Coding

Signed-off-by: Michael Adam <obnox@samba.org>
7 years agoidmap-hash: Fix Coverity ID 709119 Unused pointer value
Volker Lendecke [Wed, 20 Jun 2012 06:40:22 +0000 (08:40 +0200)]
idmap-hash: Fix Coverity ID 709119 Unused pointer value

"p" is overwritten further down again before it's first use

Signed-off-by: Michael Adam <obnox@samba.org>
7 years agoVERSION: Move on to beta3!
Andrew Bartlett [Wed, 20 Jun 2012 03:25:02 +0000 (13:25 +1000)]
VERSION: Move on to beta3!

Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Wed Jun 20 10:14:37 CEST 2012 on sn-devel-104

7 years agoVERSION: Mark as the beta2 release samba-4.0.0beta2
Andrew Bartlett [Wed, 20 Jun 2012 03:24:30 +0000 (13:24 +1000)]
VERSION: Mark as the beta2 release

7 years agoWHATNEW: Update for the beta2 release
Andrew Bartlett [Wed, 20 Jun 2012 03:23:56 +0000 (13:23 +1000)]
WHATNEW: Update for the beta2 release

7 years agos4-idmap: Add parameter 'idmap_ldb:use rfc2307' and correct implementation errors
Andrew Bartlett [Wed, 20 Jun 2012 02:51:43 +0000 (12:51 +1000)]
s4-idmap: Add parameter 'idmap_ldb:use rfc2307' and correct implementation errors

7 years agos4-provision: Remove --slapd-path option
Andrew Bartlett [Wed, 20 Jun 2012 02:41:06 +0000 (12:41 +1000)]
s4-provision: Remove --slapd-path option

This just leaves a default enough for the test code to still check the start
of the provision.  This may well be removed in future, and we wish to reduce
the extra options to provision.

Andrew Bartlett

7 years agos4-provision: Remove last unused remenants of the 'sid generator' configuration
Andrew Bartlett [Wed, 20 Jun 2012 01:48:11 +0000 (11:48 +1000)]
s4-provision: Remove last unused remenants of the 'sid generator' configuration

This was part of the now-abandoned S4 AD LDAP backend project.

Andrew Bartlett

7 years agoRevert "s4-libcli: Remove unused finddcs_nbt"
Andrew Bartlett [Fri, 8 Jun 2012 10:43:18 +0000 (20:43 +1000)]
Revert "s4-libcli: Remove unused finddcs_nbt"

This reverts commit 06c90cb6f55701effa4cbafaf189a4de8471949b.

There is genuine interest in using this currently unused code, so put
it back into the tree to avoid folks having to rewrite it.

It should be carefully hooked back into libnet at some point, and
possibly told how to talk to the s3 nmbd socket if nbt_server isn't
running.

The wscript patches are skipped, due to the way the extra
dep interacted with the build system.  When used, this will be resolved.

Andrew Bartlett

7 years agoFix unused variable warnings.
Jeremy Allison [Tue, 19 Jun 2012 17:26:58 +0000 (10:26 -0700)]
Fix unused variable warnings.

Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Tue Jun 19 21:17:51 CEST 2012 on sn-devel-104

7 years agoFix unused variable warnings.
Jeremy Allison [Tue, 19 Jun 2012 17:23:02 +0000 (10:23 -0700)]
Fix unused variable warnings.

7 years agoRemove unused variables.
Jeremy Allison [Tue, 19 Jun 2012 17:18:28 +0000 (10:18 -0700)]
Remove unused variables.

7 years agoRemove unused variables and code.
Jeremy Allison [Tue, 19 Jun 2012 17:16:32 +0000 (10:16 -0700)]
Remove unused variables and code.

7 years agoFix more "set but not used" warnings.
Jeremy Allison [Tue, 19 Jun 2012 17:09:45 +0000 (10:09 -0700)]
Fix more "set but not used" warnings.

7 years agoFix a bunch of "set but not used" warnings.
Jeremy Allison [Tue, 19 Jun 2012 17:06:34 +0000 (10:06 -0700)]
Fix a bunch of "set but not used" warnings.

7 years agos3: We can depend on pthreadpool now
Volker Lendecke [Wed, 13 Jun 2012 11:06:16 +0000 (13:06 +0200)]
s3: We can depend on pthreadpool now

Signed-off-by: Jeremy Allison <jra@samba.org>
7 years agos3: Add a dummy implementation for the pthreadpool API
Volker Lendecke [Wed, 13 Jun 2012 11:05:17 +0000 (13:05 +0200)]
s3: Add a dummy implementation for the pthreadpool API

Signed-off-by: Jeremy Allison <jra@samba.org>
7 years agoRemove unused code now we set the correct command codes.
Jeremy Allison [Mon, 18 Jun 2012 23:36:30 +0000 (16:36 -0700)]
Remove unused code now we set the correct command codes.

7 years agoFix warning: variable ‘status’ set but not used.
Jeremy Allison [Mon, 18 Jun 2012 21:37:15 +0000 (14:37 -0700)]
Fix warning: variable ‘status’ set but not used.

7 years agoFix warnings using %lu to print a 64-bit value (should by %llu).
Jeremy Allison [Mon, 18 Jun 2012 21:35:02 +0000 (14:35 -0700)]
Fix warnings using %lu to print a 64-bit value (should by %llu).

7 years agoFix warning: variable ‘in_start_time’ set but not used.
Jeremy Allison [Mon, 18 Jun 2012 21:28:48 +0000 (14:28 -0700)]
Fix  warning: variable ‘in_start_time’ set but not used.

7 years agoFix warning: variable ‘XX’ set but not used.
Jeremy Allison [Mon, 18 Jun 2012 21:25:27 +0000 (14:25 -0700)]
Fix warning: variable ‘XX’ set but not used.

7 years agowaf: Fix make etags
Volker Lendecke [Tue, 19 Jun 2012 14:33:39 +0000 (16:33 +0200)]
waf: Fix make etags

vlendec@host:~/git/s3> make etags
WAF_MAKE=1 ./buildtools/bin/waf etags
Running: etags $(find /home/vlendec/git/s3 -name "*.[ch]" | egrep -v \.inst\.)
sh: /usr/bin/etags: Argument list too long
'etags' finished successfully (0.368s)

Autobuild-User(master): Stefan Metzmacher <metze@samba.org>
Autobuild-Date(master): Tue Jun 19 18:56:53 CEST 2012 on sn-devel-104

7 years agoselftest/flapping: mark samba4.nss.test using winbind as flakey
Stefan Metzmacher [Tue, 19 Jun 2012 13:19:44 +0000 (15:19 +0200)]
selftest/flapping: mark samba4.nss.test using winbind as flakey

[1427/1518 in 1h12m39s] samba4.nss.test using winbind(member)
UNEXPECTED(failure): samba4.nss.test using winbind(member).run nsstest(member)
REASON: _StringException: _StringException: ERROR setpwent: NSS_STATUS=-1  1 (nss_errno=0)
ERROR getpwent: NSS_STATUS=-1  1 (nss_errno=0)
ERROR endpwent: NSS_STATUS=-1  1 (nss_errno=0)
ERROR setgrent: NSS_STATUS=-1  1 (nss_errno=0)
ERROR getgrent: NSS_STATUS=-1  1 (nss_errno=0)
ERROR endgrent: NSS_STATUS=-1  1 (nss_errno=0)
ERROR Non existent user gave error -1
ERROR Non existent uid gave error -1
ERROR Non existent group gave error -1
ERROR Non existent gid gave error -1
total_errors=10

This seems to happen in all environments

metze

7 years agoRevert "s3-lib Remove unused sys_fcntl_long()"
Stefan Metzmacher [Tue, 19 Jun 2012 09:34:04 +0000 (11:34 +0200)]
Revert "s3-lib Remove unused sys_fcntl_long()"

This reverts commit 846a697e20478798288afb43cdb7a9f389a15c69.

This is still used in source3/smbd/oplock_irix.c

metze

7 years agos3:rpc_server/lsasd: remove dependency to libgen.h and basename()
Stefan Metzmacher [Tue, 19 Jun 2012 09:33:12 +0000 (11:33 +0200)]
s3:rpc_server/lsasd: remove dependency to libgen.h and basename()

metze

7 years agos3-auth: make_new_system_info_session() now does not query nss
Andrew Bartlett [Wed, 13 Jun 2012 23:41:05 +0000 (09:41 +1000)]
s3-auth: make_new_system_info_session() now does not query nss

This is important in some situations where these queries might be
costly LDAP queries and is just not required for the system token.
This is because the system token should be just the NT
AUTHORITY\SYSTEM user and just enough unix info to allow the token to
be used.

Andrew Bartlett

Signed-off-by: Andreas Schneider <asn@samba.org>
Autobuild-User(master): Andreas Schneider <asn@cryptomilk.org>
Autobuild-Date(master): Tue Jun 19 13:34:28 CEST 2012 on sn-devel-104

7 years agos3-auth: make_new_system_info_session() not query passdb
Andrew Bartlett [Tue, 12 Jun 2012 07:15:19 +0000 (17:15 +1000)]
s3-auth: make_new_system_info_session() not query passdb

This is important in some situations where these queries might be
costly LDAP queries and is just not required for the system token.
This is because the system token should be just the NT
AUTHORITY\SYSTEM user and just enough unix info to allow the token to
be used.

We query only NSS to get the name of sec_initial_uid()

Signed-off-by: Andreas Schneider <asn@samba.org>
7 years agos3-auth: inline make_session_info functions into only callers
Andrew Bartlett [Sun, 3 Jun 2012 13:12:22 +0000 (23:12 +1000)]
s3-auth: inline make_session_info functions into only callers

With the split up to handle system specially, there is no need for these
static helper functions any more.

Andrew Bartlett

Signed-off-by: Andreas Schneider <asn@samba.org>
7 years agoauth: Use only security_token_is_system to determine that a user is SYSTEM
Andrew Bartlett [Sun, 3 Jun 2012 12:34:13 +0000 (22:34 +1000)]
auth: Use only security_token_is_system to determine that a user is SYSTEM

This removes the duplication on how to detect that a user is system in Samba
now that the smbd system account is also only SID_NT_SYSTEM we can use the same
check everywhere.

Andrew Bartlett

Signed-off-by: Andreas Schneider <asn@samba.org>
7 years agos3-auth: Fix system token as just being SID_NT_SYSTEM
Andrew Bartlett [Sun, 3 Jun 2012 12:21:42 +0000 (22:21 +1000)]
s3-auth: Fix system token as just being SID_NT_SYSTEM

This removes the SID matching the sec_initial_uid().

Andrew Bartlett

Signed-off-by: Andreas Schneider <asn@samba.org>
7 years agos3-spoolss: delete_drivers should be called as the connecting user.
Andreas Schneider [Tue, 19 Jun 2012 07:48:38 +0000 (09:48 +0200)]
s3-spoolss: delete_drivers should be called as the connecting user.

Signed-off-by: Andreas Schneider <asn@samba.org>
7 years agontdb: add autoconf support.
Rusty Russell [Tue, 19 Jun 2012 03:13:17 +0000 (12:43 +0930)]
ntdb: add autoconf support.

This is copied from tdb; we build the utilities, but as nothing else
links against it, we shouldn't be adding anything to the normal samba
binary sizes.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Autobuild-User(master): Rusty Russell <rusty@rustcorp.com.au>
Autobuild-Date(master): Tue Jun 19 07:31:06 CEST 2012 on sn-devel-104

7 years agontdb: put it back into the build.
Rusty Russell [Tue, 19 Jun 2012 03:13:11 +0000 (12:43 +0930)]
ntdb: put it back into the build.

This doesn't do anything with it yet, just wires it back into the build.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
7 years agolibcli: use tdb directly, not tdb_compat.
Rusty Russell [Tue, 19 Jun 2012 03:13:10 +0000 (12:43 +0930)]
libcli: use tdb directly, not tdb_compat.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
7 years agolib/tdb_wrap: use tdb directly, not tdb_compat.
Rusty Russell [Tue, 19 Jun 2012 03:13:10 +0000 (12:43 +0930)]
lib/tdb_wrap: use tdb directly, not tdb_compat.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
7 years agoldb: use tdb directly, not tdb_compat.
Rusty Russell [Tue, 19 Jun 2012 03:13:10 +0000 (12:43 +0930)]
ldb: use tdb directly, not tdb_compat.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
7 years agolib/dbwrap: depend directly on tdb, not tdb_compat.
Rusty Russell [Tue, 19 Jun 2012 03:13:10 +0000 (12:43 +0930)]
lib/dbwrap: depend directly on tdb, not tdb_compat.

Simple change, as we get rid of tdb_compat in favour of either ntdb directly
or dbwrap.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
7 years agolib/util_tdb: depend directly on tdb, not tdb_compat.
Rusty Russell [Tue, 19 Jun 2012 03:13:10 +0000 (12:43 +0930)]
lib/util_tdb: depend directly on tdb, not tdb_compat.

Simple change, as we get rid of tdb_compat in favour of tdb directly.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
7 years agontdb: update documentation.
Rusty Russell [Tue, 19 Jun 2012 03:13:09 +0000 (12:43 +0930)]
ntdb: update documentation.

Update the design.lyx file with the latest status and the change in hashing.
Also, refresh and add examples to the TDB_porting.txt file.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
7 years agontdb: optimize ntdb_fetch.
Rusty Russell [Tue, 19 Jun 2012 03:13:09 +0000 (12:43 +0930)]
ntdb: optimize ntdb_fetch.

We access the key on lookup, then access the data in the caller.  It
makes more sense to access both at once.  We also put in a likely()
for the case where the hash is not chained.

Before:
Adding 1000 records: 3644-3724(3675) ns (129656 bytes)
Finding 1000 records: 1596-1696(1622) ns (129656 bytes)
Missing 1000 records: 1409-1525(1452) ns (129656 bytes)
Traversing 1000 records: 1636-1747(1668) ns (129656 bytes)
Deleting 1000 records: 3138-3223(3175) ns (129656 bytes)
Re-adding 1000 records: 3278-3414(3329) ns (129656 bytes)
Appending 1000 records: 5396-5529(5426) ns (253312 bytes)
Churning 1000 records: 9451-10095(9584) ns (253312 bytes)
smbtorture results (--entries=1000)
ntdb speed 183881-191112(188223) ops/sec

After:
Adding 1000 records: 3590-3701(3640) ns (129656 bytes)
Finding 1000 records: 1539-1605(1566) ns (129656 bytes)
Missing 1000 records: 1398-1440(1413) ns (129656 bytes)
Traversing 1000 records: 1629-2015(1710) ns (129656 bytes)
Deleting 1000 records: 3118-3236(3163) ns (129656 bytes)
Re-adding 1000 records: 3235-3355(3275) ns (129656 bytes)
Appending 1000 records: 5335-5444(5385) ns (253312 bytes)
Churning 1000 records: 9350-9955(9494) ns (253312 bytes)
smbtorture results (--entries=1000)
ntdb speed 180559-199981(195106) ops/sec

7 years agontdb: add -h arg to ntdbrestore
Rusty Russell [Tue, 19 Jun 2012 03:13:09 +0000 (12:43 +0930)]
ntdb: add -h arg to ntdbrestore

Since our default hashsize is 8192 not 131, we look fat when we convert
near-empty TDBs.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
7 years agontdb: reduce default hashsize on ntdbtorture.
Rusty Russell [Tue, 19 Jun 2012 03:13:08 +0000 (12:43 +0930)]
ntdb: reduce default hashsize on ntdbtorture.

Just like tdbtorture, having a hashsize of 2 stresses us much more!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
7 years agontdb: add NTDB_ATTRIBUTE_HASHSIZE
Rusty Russell [Tue, 19 Jun 2012 03:13:08 +0000 (12:43 +0930)]
ntdb: add NTDB_ATTRIBUTE_HASHSIZE

Since we've given up on expansion, let them frob the hashsize again.
We have attributes, so we should use them for optional stuff like
this.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
7 years agontdb: remove hash table trees.
Rusty Russell [Tue, 19 Jun 2012 03:13:04 +0000 (12:43 +0930)]
ntdb: remove hash table trees.

TDB2 started with a top-level hash of 1024 entries, divided into 128
groups of 8 buckets.  When a bucket filled, the 8 bucket group
expanded into pointers into 8 new 64-entry hash tables.  When these
filled, they expanded in turn, etc.

It's a nice idea to automatically expand the hash tables, but it
doesn't pay off.  Remove it for NTDB.

1) It only beats TDB performance when the database is huge and the
   TDB hashsize is small.  We are about 20% slower on medium-size
   databases (1000 to 10000 records), worse on really small ones.
2) Since we're 64 bits, our hash tables are already twice as expensive
   as TDB.
3) Since our hash function is good, it means that all groups tend to
   fill at the same time, meaning the hash enlarges by a factor of 128
   all at once, leading to a very large database at that point.
4) Our efficiency would improve if we enlarged the top level, but
   that makes our minimum db size even worse: it's already over 8k,
   and jumps to 1M after about 1000 entries!
5) Making the sub group size larger gives a shallower tree, which
   performs better, but makes the "hash explosion" problem worse.
6) The code is complicated, having to handle delete and reshuffling
   groups of hash buckets, and expansion of buckets.
7) We have to handle the case where all the records somehow end up with
   the same hash value, which requires special code to chain records for
   that case.

On the other hand, it would be nice if we didn't degrade as badly as
TDB does when the hash chains get long.

This patch removes the hash-growing code, but instead of chaining like
TDB does when a bucket fills, we point the bucket to an array of
record pointers.  Since each on-disk NTDB pointer contains some hash
bits from the record (we steal the upper 8 bits of the offset), 99.5%
of the time we don't need to load the record to determine if it
matches.  This makes an array of offsets much more cache-friendly than
a linked list.

Here are the times (in ns) for tdb_store of N records, tdb_store of N
records the second time, and a fetch of all N records.  I've also
included the final database size and the smbtorture local.[n]tdb_speed
results.

Benchmark details:
1) Compiled with -O2.
2) assert() was disabled in TDB2 and NTDB.
3) The "optimize fetch" patch was applied to NTDB.

10 runs, using tmpfs (otherwise massive swapping as db hits ~30M,
despite plenty of RAM).

Insert Re-ins Fetch Size dbspeed
(nsec) (nsec) (nsec) (Kb) (ops/sec)
TDB (10000 hashsize):
100 records:  3882  3320 1609    53 203204
1000 records:  3651  3281 1571   115 218021
10000 records:  3404  3326 1595   880 202874
100000 records:  4317  3825 2097  8262 126811
1000000 records: 11568 11578 9320 77005  25046

TDB2 (1024 hashsize, expandable):
100 records:  3867  3329 1699    17 187100
1000 records:  4040  3249 1639   154 186255
10000 records:  4143  3300 1695  1226 185110
100000 records:  4481  3425 1800 17848 163483
1000000 records:  4055  3534 1878   106386 160774

NTDB (8192 hashsize)
100 records:  4259  3376 1692    82 190852
1000 records:  3640  3275 1566   130 195106
10000 records:  4337  3438 1614   773 188362
100000 records:  4750  5165 1746  9001 169197
1000000 records:  4897  5180 2341 83838 121901

Analysis:
1) TDB wins on small databases, beating TDB2 by ~15%, NTDB by ~10%.
2) TDB starts to lose when hash chains get 10 long (fetch 10% slower
   than TDB2/NTDB).
3) TDB does horribly when hash chains get 100 long (fetch 4x slower
   than NTDB, 5x slower than TDB2, insert about 2-3x slower).
4) TDB2 databases are 40% larger than TDB1.  NTDB is about 15% larger
   than TDB1

7 years agontdb: special accessor functions for read/write of an offset.
Rusty Russell [Tue, 19 Jun 2012 03:12:13 +0000 (12:42 +0930)]
ntdb: special accessor functions for read/write of an offset.

We also split off the NTDB_CONVERT case (where the ntdb is of a
different endian) into its own io function.

NTDB speed:
Adding 10000 records: 3894-9951(8553) ns (815528 bytes)
Finding 10000 records: 1644-4294(3580) ns (815528 bytes)
Missing 10000 records: 1497-4018(3303) ns (815528 bytes)
Traversing 10000 records: 1585-4225(3505) ns (815528 bytes)
Deleting 10000 records: 3088-8154(6927) ns (815528 bytes)
Re-adding 10000 records: 3192-8308(7089) ns (815528 bytes)
Appending 10000 records: 5187-13307(11365) ns (1274312 bytes)
Churning 10000 records: 6772-17567(15078) ns (1274312 bytes)
NTDB speed in transaction:
Adding 10000 records: 1602-2404(2214) ns (815528 bytes)
Finding 10000 records: 456-871(778) ns (815528 bytes)
Missing 10000 records: 393-522(503) ns (815528 bytes)
Traversing 10000 records: 729-1015(945) ns (815528 bytes)
Deleting 10000 records: 1065-1476(1374) ns (815528 bytes)
Re-adding 10000 records: 1397-1930(1819) ns (815528 bytes)
Appending 10000 records: 2927-3351(3184) ns (1274312 bytes)
Churning 10000 records: 3921-4697(4378) ns (1274312 bytes)
smbtorture results:
ntdb speed 86581-191518(175666) ops/sec
Applying patch..increase-top-level.patch

7 years agontdb: inline oob check
Rusty Russell [Tue, 19 Jun 2012 03:12:12 +0000 (12:42 +0930)]
ntdb: inline oob check

The simple "is it in range" check can be inline; complex cases can be
handed through to the normal or transaction handler.

NTDB speed:
Adding 10000 records: 4111-9983(9149) ns (815528 bytes)
Finding 10000 records: 1667-4464(3810) ns (815528 bytes)
Missing 10000 records: 1511-3992(3546) ns (815528 bytes)
Traversing 10000 records: 1698-4254(3724) ns (815528 bytes)
Deleting 10000 records: 3608-7998(7358) ns (815528 bytes)
Re-adding 10000 records: 3259-8504(7805) ns (815528 bytes)
Appending 10000 records: 5393-13579(12356) ns (1274312 bytes)
Churning 10000 records: 6966-17813(16136) ns (1274312 bytes)
NTDB speed in transaction:
Adding 10000 records: 916-2230(2004) ns (815528 bytes)
Finding 10000 records: 330-866(770) ns (815528 bytes)
Missing 10000 records: 196-520(471) ns (815528 bytes)
Traversing 10000 records: 356-879(800) ns (815528 bytes)
Deleting 10000 records: 505-1267(1108) ns (815528 bytes)
Re-adding 10000 records: 658-1681(1477) ns (815528 bytes)
Appending 10000 records: 1088-2827(2498) ns (1274312 bytes)
Churning 10000 records: 1636-4267(3785) ns (1274312 bytes)
smbtorture results:
ntdb speed 85588-189430(157110) ops/sec

7 years agontdb: allocator attribute.
Rusty Russell [Tue, 19 Jun 2012 03:12:08 +0000 (12:42 +0930)]
ntdb: allocator attribute.

This is designed to allow us to make ntdb_context (and NTDB_DATA returned
from ntdb_fetch) a talloc pointer.  But it can also be used for any other
alternate allocator.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
7 years agontdb: still prepare recovery area with NTDB_NOSYNC.
Rusty Russell [Tue, 19 Jun 2012 03:09:33 +0000 (12:39 +0930)]
ntdb: still prepare recovery area with NTDB_NOSYNC.

NTDB_NOSYNC now just prevents the fsync/msync calls, which speeds
testing while still providing full coverage.  It also provides safety
against processes dying during transaction commit (though obviously,
not against the machine dying).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
7 years agontdb: simply disallow NULL names.
Rusty Russell [Tue, 19 Jun 2012 03:08:33 +0000 (12:38 +0930)]
ntdb: simply disallow NULL names.

TDB allows this for internal databases, but it's a bad idea, since the
name is useful for logging.

They're a hassle to deal with, and we'd just end up putting "unnamed"
in there, so let the user deal with it.  If they don't, they get an
informative core dump.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
7 years agontdb: reduce transaction pagesize from 64k to 16k.
Rusty Russell [Mon, 18 Jun 2012 13:00:30 +0000 (22:30 +0930)]
ntdb: reduce transaction pagesize from 64k to 16k.

The performance numbers for transaction pagesize are indeterminate:
larger pagesizes means a smaller transaction array, and a better
chance of having a contiguous record (more efficient for
ntdb_parse_record and some internal operations inside a transaction).

On the other hand, large pagesize means more I/O even if we change a
few bytes.

But it also controls the multiple by which we will enlarge the file,
and hence the minimum db size.  It's 4k for tdb1, but 16k seems
reasonable in these modern times.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
7 years agontdb: remove last block transactoin logic.
Rusty Russell [Mon, 18 Jun 2012 13:00:30 +0000 (22:30 +0930)]
ntdb: remove last block transactoin logic.

Now our database is always a multiple of NTDB_PGSIZE, we can remove the
special handling for the last block.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
7 years agontdb: create initial database to be multiple of NTDB_PGSIZE.
Rusty Russell [Mon, 18 Jun 2012 13:00:29 +0000 (22:30 +0930)]
ntdb: create initial database to be multiple of NTDB_PGSIZE.

As copied from tdb1, there is logic in the transaction code to handle
a non-PGSIZE multiple db, but in fact this only happens for a
completely unused database: as soon as we add anything to it, it is
expanded to a NTDB_PGSIZE multiple.

If we create the database with a free record which pads it out to
NTDB_PGSIZE, we can remove this last-page-is-different logic.

Of course, the fake ntdbs we create in our tests now also need to be
multiples of NTDB_PGSIZE, so we change some numbers there too.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
7 years agontdb: make sure file is always a multiple of PAGESIZE (now NTDB_PGSIZE)
Rusty Russell [Mon, 18 Jun 2012 13:00:29 +0000 (22:30 +0930)]
ntdb: make sure file is always a multiple of PAGESIZE (now NTDB_PGSIZE)

ntdb uses tdb's transaction code, and it has an undocumented but implicit
assumption: that the transaction recovery area is always aligned to the
transaction pagesize.  This means that no block will overlap the recovery
area.

This is maintained by rounding the size of the database up, so do the same
for ntdb.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
7 years agontdb: fix recovery data write.
Rusty Russell [Mon, 18 Jun 2012 13:00:29 +0000 (22:30 +0930)]
ntdb: fix recovery data write.

We were missing the last few bytes.  Found by 100 runs of ntdbtorture
-t -k.

The transaction test code didn't catch this, because usually those
last few bytes are irrelevant to the actual contents of the database.
We fix the test.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
7 years agontdb: enhance external-helper test code.
Rusty Russell [Mon, 18 Jun 2012 13:00:29 +0000 (22:30 +0930)]
ntdb: enhance external-helper test code.

Our external test helper is a bit primitive when it comes to doing STORE or
FETCH commands: let us specify the data we expect, instead of assuming it's
the same as the key.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
7 years agontdb: use NTDB_LOG_WARNING level for failed open() without O_CREAT.
Rusty Russell [Mon, 18 Jun 2012 13:00:28 +0000 (22:30 +0930)]
ntdb: use NTDB_LOG_WARNING level for failed open() without O_CREAT.

This is a fairly common pattern in Samba, and if we log an error on
every open it spams the logs.  On the other hand, other errors are
potentially more serious, so we still use NTDB_LOG_ERROR on them.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
7 years agoccan: remove bogus debug print.
Rusty Russell [Mon, 18 Jun 2012 13:00:28 +0000 (22:30 +0930)]
ccan: remove bogus debug print.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
7 years agontdb: make fork test more thorough.
Rusty Russell [Mon, 18 Jun 2012 13:00:28 +0000 (22:30 +0930)]
ntdb: make fork test more thorough.

We document that the child of a fork() can do a brunlock() if the parent
does a brlock: we should not log an error when they do this.

Also, test the case where we fork() and return inside a parse function
(which is allowed).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
7 years agontdb: print \n at end of log messages in tests.
Rusty Russell [Mon, 18 Jun 2012 13:00:28 +0000 (22:30 +0930)]
ntdb: print \n at end of log messages in tests.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
7 years agontdb: reduce race between creating file and getting open lock.
Rusty Russell [Mon, 18 Jun 2012 13:00:27 +0000 (22:30 +0930)]
ntdb: reduce race between creating file and getting open lock.

In tdb, we grab the open lock immediately after we open the file.  In
ntdb, we usually did some work first.  tdbtorture managed to get in
before the creator grabbed the lock:

testing with 3 processes, 5000 loops, seed=1338246020
ntdb:torture.ntdb:IO Error:ntdb_open: torture.ntdb is not a ntdb file
29023:torture.ntdb:db open failed

At cost of a little duplicated code, we can reduce the race.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
7 years agontdb: catch any valgrind errors in test
Rusty Russell [Mon, 18 Jun 2012 13:00:27 +0000 (22:30 +0930)]
ntdb: catch any valgrind errors in test

Make --valgrind and --valgrind-log options work!

Amitay figured this out!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
7 years agontdb: catch any valgrind errors in test
Rusty Russell [Mon, 18 Jun 2012 13:00:27 +0000 (22:30 +0930)]
ntdb: catch any valgrind errors in test

We need --error-exitcode=, otherwise valgrind errors don't cause the
test to fail.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
7 years agontdb: remove ntdb_error()
Rusty Russell [Mon, 18 Jun 2012 13:00:27 +0000 (22:30 +0930)]
ntdb: remove ntdb_error()

It was a hack to make compatibility easier.  Since we're not doing that,
it can go away: all callers must use the return value now.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
7 years agoTDB2: Goodbye TDB2, Hello NTDB.
Rusty Russell [Mon, 18 Jun 2012 13:00:26 +0000 (22:30 +0930)]
TDB2: Goodbye TDB2, Hello NTDB.

This renames everything from tdb2 to ntdb: importantly, we no longer
use the tdb_ namespace, so you can link against both ntdb and tdb if
you want to.

This also enables building of standalone ntdb by the autobuild script.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
7 years agotdb2: Fix typo in TDB1_porting.txt
Kirill Smelkov [Mon, 18 Jun 2012 13:00:26 +0000 (22:30 +0930)]
tdb2: Fix typo in TDB1_porting.txt

Judging by code it's tdb1, where you needed to free old key's dptr
manually.

Signed-off-by: Kirill Smelkov <kirr@mns.spb.ru>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
7 years agoTDB2: more internal cleanups after TDB1 compatibility removal.
Rusty Russell [Mon, 18 Jun 2012 13:00:26 +0000 (22:30 +0930)]
TDB2: more internal cleanups after TDB1 compatibility removal.

This eliminates the separate tdb2 substructure, and makes some
tdb1-required functions static.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
7 years agoTDB2: remove TDB1 compatibility.
Rusty Russell [Mon, 18 Jun 2012 13:00:25 +0000 (22:30 +0930)]
TDB2: remove TDB1 compatibility.

This rips out all the TDB1 compatibility from tdb2.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
7 years agoTDB2: make SAMBA use tdb1 again for the moment.
Rusty Russell [Mon, 18 Jun 2012 13:00:04 +0000 (22:30 +0930)]
TDB2: make SAMBA use tdb1 again for the moment.

Otherwise the following surgery will break the SAMBA build and testsuite.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
7 years agoccan: check for err.h ourselves
Rusty Russell [Mon, 18 Jun 2012 10:36:45 +0000 (20:06 +0930)]
ccan: check for err.h ourselves

Heimdal does this, but that doesn't help the autoconf build or the standalone
libntdb build.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
7 years agoWHATSNEW: Fix typo.
Jelmer Vernooij [Mon, 18 Jun 2012 23:01:17 +0000 (01:01 +0200)]
WHATSNEW: Fix typo.

"dcerpc endpoint services" -> "dcerpc endpoint servers"

Autobuild-User(master): Jelmer Vernooij <jelmer@samba.org>
Autobuild-Date(master): Tue Jun 19 04:40:12 CEST 2012 on sn-devel-104

7 years agotdb/wscript: Remove unecessary semicolons.
Jelmer Vernooij [Mon, 28 May 2012 12:16:44 +0000 (14:16 +0200)]
tdb/wscript: Remove unecessary semicolons.

7 years agowafsamba/irixcc: add '-c99' option to cc
Stefan Metzmacher [Mon, 18 Jun 2012 22:46:51 +0000 (00:46 +0200)]
wafsamba/irixcc: add '-c99' option to cc

Lets see if this fixes the build on IRIX.

metze

Autobuild-User(master): Stefan Metzmacher <metze@samba.org>
Autobuild-Date(master): Tue Jun 19 02:42:21 CEST 2012 on sn-devel-104

7 years agoRevert "s3: temporary hack to make the waf build work withouth autotools being required"
Björn Jacke [Mon, 18 Jun 2012 12:13:26 +0000 (14:13 +0200)]
Revert "s3: temporary hack to make the waf build work withouth autotools being required"

This reverts commit f1becfa27b6b4e35541e6df0cafdec0ad47d2e3f. The hack was
actually only required due to a configuration issue in our buildfarm scripts.

Autobuild-User(master): Björn Jacke <bj@sernet.de>
Autobuild-Date(master): Mon Jun 18 20:07:08 CEST 2012 on sn-devel-104

7 years agoselftest/flapping: samba4.nss.test is also flakey for s3member
Stefan Metzmacher [Mon, 18 Jun 2012 13:25:25 +0000 (15:25 +0200)]
selftest/flapping: samba4.nss.test is also flakey for s3member

[1426/1518 in 1h24m58s] samba4.nss.test using winbind(s3member)
UNEXPECTED(failure): samba4.nss.test using winbind(s3member).run nsstest(s3member)
REASON: _StringException: _StringException: ERROR setpwent: NSS_STATUS=-1  1 (nss_errno=0)
ERROR getpwent: NSS_STATUS=-1  1 (nss_errno=0)
ERROR endpwent: NSS_STATUS=-1  1 (nss_errno=0)
ERROR setgrent: NSS_STATUS=-1  1 (nss_errno=0)
ERROR getgrent: NSS_STATUS=-1  1 (nss_errno=0)
ERROR endgrent: NSS_STATUS=-1  1 (nss_errno=0)
ERROR Non existent user gave error -1
ERROR Non existent uid gave error -1
ERROR Non existent group gave error -1
ERROR Non existent gid gave error -1
total_errors=10

metze

Autobuild-User(master): Stefan Metzmacher <metze@samba.org>
Autobuild-Date(master): Mon Jun 18 17:59:25 CEST 2012 on sn-devel-104

7 years agos3:autoconf: add -Iautoconf -Iautoconf/source3 at configure stage
Stefan Metzmacher [Mon, 18 Jun 2012 07:14:55 +0000 (09:14 +0200)]
s3:autoconf: add -Iautoconf -Iautoconf/source3 at configure stage

There're some configure tests which require this.

metze

7 years agos3:Makefile.in: remove pidl generated files with 'make realdistclean'
Stefan Metzmacher [Mon, 18 Jun 2012 07:33:48 +0000 (09:33 +0200)]
s3:Makefile.in: remove pidl generated files with 'make realdistclean'

metze

7 years agos3:Makefile.in: fix 'make realdistclean' after moving generated files to autoconf/
Stefan Metzmacher [Mon, 18 Jun 2012 07:32:17 +0000 (09:32 +0200)]
s3:Makefile.in: fix 'make realdistclean' after moving generated files to autoconf/

metze

7 years agos3:Makefile.in: fix 'make clean' after moving generated files to autoconf/
Stefan Metzmacher [Mon, 18 Jun 2012 07:32:17 +0000 (09:32 +0200)]
s3:Makefile.in: fix 'make clean' after moving generated files to autoconf/

metze

7 years agos3:autogen.sh: fix autoconf/lib/param/param_proto.h location
Stefan Metzmacher [Mon, 18 Jun 2012 08:31:09 +0000 (10:31 +0200)]
s3:autogen.sh: fix autoconf/lib/param/param_proto.h location

metze

7 years agolib/param: add missing prototype of lpcfg_parm_long()
Stefan Metzmacher [Mon, 18 Jun 2012 08:02:47 +0000 (10:02 +0200)]
lib/param: add missing prototype of lpcfg_parm_long()

metze

7 years agos3:autoconf-build: build the idmap backends tdb2, rid, and hash by default (shared)
Michael Adam [Mon, 18 Jun 2012 09:04:43 +0000 (11:04 +0200)]
s3:autoconf-build: build the idmap backends tdb2, rid, and hash by default (shared)

Autobuild-User(master): Michael Adam <obnox@samba.org>
Autobuild-Date(master): Mon Jun 18 13:38:50 CEST 2012 on sn-devel-104

7 years agos3:waf-build: build the idmap backends tdb2, rid, and hash by default (shared)
Michael Adam [Mon, 18 Jun 2012 08:53:53 +0000 (10:53 +0200)]
s3:waf-build: build the idmap backends tdb2, rid, and hash by default (shared)

7 years agos3:idmap_tdb: fix miss in rename of dbwrap_trans_store_uint32() to dbwrap_trans_store...
Michael Adam [Mon, 18 Jun 2012 08:27:54 +0000 (10:27 +0200)]
s3:idmap_tdb: fix miss in rename of dbwrap_trans_store_uint32() to dbwrap_trans_store_uint32_bystring()