ira/wip.git
10 years agos4-dns: use a loadparm list for samba_runcmd() commands
Andrew Tridgell [Fri, 26 Feb 2010 06:09:35 +0000 (17:09 +1100)]
s4-dns: use a loadparm list for samba_runcmd() commands

This allows commands with multiple arguments and quoting to be used,
while still avoiding running a shell (and this having shell expansion
problems)

10 years agos4-provision: added dns_update_list
Andrew Tridgell [Fri, 26 Feb 2010 03:27:28 +0000 (14:27 +1100)]
s4-provision: added dns_update_list

This contains the list of DNS names we should have as a DC

10 years agos4-provision: setup the dns_update_list at provision time
Andrew Tridgell [Fri, 26 Feb 2010 03:25:07 +0000 (14:25 +1100)]
s4-provision: setup the dns_update_list at provision time

This file is substituted at runtime by samba_dnsupdate

10 years agos4-provision: if we aren't doing variable substitution then don't check for vars
Andrew Tridgell [Fri, 26 Feb 2010 03:24:38 +0000 (14:24 +1100)]
s4-provision: if we aren't doing variable substitution then don't check for vars

10 years agodns: auto-delete incorrect SRV entries for our hostname
Andrew Tridgell [Fri, 26 Feb 2010 02:58:32 +0000 (13:58 +1100)]
dns: auto-delete incorrect SRV entries for our hostname

10 years agos4-dns: add automatic dynamic DNS updating script
Andrew Tridgell [Fri, 26 Feb 2010 02:30:44 +0000 (13:30 +1100)]
s4-dns: add automatic dynamic DNS updating script

This script checks a list of DNS names that we should have, and does
dynamic DNS updates using our machine account credentials to add any
missing DNS entries.

This allows us to correctly add all the DNS entries we need when we
join an existing domain as a DC

Pair-Programmed-With: Andrew Bartlett <abartlet@samba.org>

10 years agos4-provision: fixed port number for gc ldap DNS SRV entry
Andrew Tridgell [Fri, 26 Feb 2010 02:27:26 +0000 (13:27 +1100)]
s4-provision: fixed port number for gc ldap DNS SRV entry

Pair-Programmed-With: Andrew Bartlett <abartlet@samba.org>

10 years agos4-param: added 'nsupdate command' option, default to /usr/bin/nsupdate -g
Andrew Tridgell [Fri, 26 Feb 2010 02:25:42 +0000 (13:25 +1100)]
s4-param: added 'nsupdate command' option, default to /usr/bin/nsupdate -g

10 years agosamdb: added get_ntds_GUID() method
Andrew Tridgell [Thu, 25 Feb 2010 23:15:43 +0000 (10:15 +1100)]
samdb: added get_ntds_GUID() method

10 years agopy-samdb: added server_site_name method
Andrew Tridgell [Thu, 25 Feb 2010 07:35:14 +0000 (18:35 +1100)]
py-samdb: added server_site_name method

10 years agopyglue: added py_samdb_server_site_name()
Andrew Tridgell [Thu, 25 Feb 2010 07:35:01 +0000 (18:35 +1100)]
pyglue: added py_samdb_server_site_name()

10 years agopy-samdb: added get_invocation_id() method
Andrew Tridgell [Thu, 25 Feb 2010 07:15:47 +0000 (18:15 +1100)]
py-samdb: added get_invocation_id() method

10 years agopyglue: added py_samdb_ntds_invocation_id()
Andrew Tridgell [Thu, 25 Feb 2010 07:15:19 +0000 (18:15 +1100)]
pyglue: added py_samdb_ntds_invocation_id()

10 years agopyglue: don't return 127.0.0.0/8 IPs in interface_ips()
Andrew Tridgell [Thu, 25 Feb 2010 05:29:47 +0000 (16:29 +1100)]
pyglue: don't return 127.0.0.0/8 IPs in interface_ips()

We don't generally want loopback addresses in the python code

10 years agos4-krb5: propogate errors from a lot more kerberos functions
Andrew Tridgell [Thu, 25 Feb 2010 05:16:33 +0000 (16:16 +1100)]
s4-krb5: propogate errors from a lot more kerberos functions

We need to be able to give sensible error messages when a kerberos
calls fails. This propogates the kerberos error up the stack to the
caller.

Pair-Programmed-With: Andrew Bartlett <abartlet@samba.org>

10 years agodns: install samba_dnsupdate
Andrew Tridgell [Thu, 25 Feb 2010 04:13:45 +0000 (15:13 +1100)]
dns: install samba_dnsupdate

10 years agodns: dummy samba_dnsupdate script
Andrew Tridgell [Thu, 25 Feb 2010 04:12:53 +0000 (15:12 +1100)]
dns: dummy samba_dnsupdate script

The merge from metze of my dns tree means we now depend on this. This
is a placeholder until Andrew and I have finished the full script.

10 years agos4-dsdb: fixed the fetch of the server site name
Andrew Tridgell [Fri, 26 Feb 2010 01:36:17 +0000 (12:36 +1100)]
s4-dsdb: fixed the fetch of the server site name

when the ntds objects were moved by a recent change it broke the
calculation of the server site

Pair-Programmed-With: Andrew Bartlett <abartlet@samba.org>

10 years agoRevert "s4:AD content - adequate some revision levels to match Windows Server 2008"
Andrew Tridgell [Fri, 26 Feb 2010 01:31:57 +0000 (12:31 +1100)]
Revert "s4:AD content - adequate some revision levels to match Windows Server 2008"

This reverts commit 973ea198677c581064fad62cdac30baac7103ef8.

This change breaks DRS dcpromo.

10 years agos4:RPC-DSSYNC: pass the ip address to the cldap code instead of a name
Stefan Metzmacher [Thu, 25 Feb 2010 17:10:56 +0000 (18:10 +0100)]
s4:RPC-DSSYNC: pass the ip address to the cldap code instead of a name

metze

10 years agos4:torture: use a connected CLDAP socket.
Stefan Metzmacher [Thu, 25 Feb 2010 11:47:38 +0000 (12:47 +0100)]
s4:torture: use a connected CLDAP socket.

This is needed because we don't (want) to specify an explicit
local address. And the socket family (ipv4 vs. ipv6) needs to
be autodetected based on the remote address before the
socket() syscall.

Otherwise we would try to connect to a ipv4 address through an
ipv6only socket.

metze

10 years agos4:libnet: use a connected CLDAP socket.
Stefan Metzmacher [Thu, 25 Feb 2010 11:47:38 +0000 (12:47 +0100)]
s4:libnet: use a connected CLDAP socket.

This is needed because we don't (want) to specify an explicit
local address. And the socket family (ipv4 vs. ipv6) needs to
be autodetected based on the remote address before the
socket() syscall.

Otherwise we would try to connect to a ipv4 address through an
ipv6only socket.

metze

10 years agoFix up debug info on smb2_rename code.
Jeremy Allison [Thu, 25 Feb 2010 19:57:54 +0000 (11:57 -0800)]
Fix up debug info on smb2_rename code.

Jeremy.

10 years agoImplement rename/move in SMB2 from Windows7.
Jeremy Allison [Thu, 25 Feb 2010 19:15:16 +0000 (11:15 -0800)]
Implement rename/move in SMB2 from Windows7.

Jeremy.

10 years agos3-nltest: fix uninitialized query level.
Günther Deschner [Thu, 25 Feb 2010 18:31:06 +0000 (19:31 +0100)]
s3-nltest: fix uninitialized query level.

Guenther

10 years agos4-smbtorture: define environment in one place in RPC-SPOOLSS test.
Günther Deschner [Thu, 25 Feb 2010 18:06:39 +0000 (19:06 +0100)]
s4-smbtorture: define environment in one place in RPC-SPOOLSS test.

Guenther

10 years agos4:kdc add mit plugin code
Simo Sorce [Sun, 31 Jan 2010 18:29:57 +0000 (13:29 -0500)]
s4:kdc add mit plugin code

10 years agos4:kdc make function static
Simo Sorce [Sat, 13 Feb 2010 23:30:36 +0000 (18:30 -0500)]
s4:kdc make function static

10 years agos3-spoolss: Save entire devicemode.
Günther Deschner [Thu, 25 Feb 2010 12:12:38 +0000 (13:12 +0100)]
s3-spoolss: Save entire devicemode.

found by RPC-SPOOLSS-PRINTER torture test.

Guenther

10 years agos4-smbtorture: try to set every single devicemode component in RPC-SPOOLSS-PRINTER.
Günther Deschner [Thu, 25 Feb 2010 12:11:47 +0000 (13:11 +0100)]
s4-smbtorture: try to set every single devicemode component in RPC-SPOOLSS-PRINTER.

Guenther

10 years agos4-smbtorture: move ChangeID test to the list of tests we run against created printers.
Günther Deschner [Wed, 24 Feb 2010 11:00:36 +0000 (12:00 +0100)]
s4-smbtorture: move ChangeID test to the list of tests we run against created printers.

Guenther

10 years agoMake conn_close_all() safe to call from SMB2 sessions (fix crash bug).
Jeremy Allison [Thu, 25 Feb 2010 02:11:07 +0000 (18:11 -0800)]
Make conn_close_all() safe to call from SMB2 sessions (fix crash bug).

Ensure we don't call close_cnum() with SMB2, also talloc_move the
compat_conn pointer from the NULL context onto the tcon context
in SMB2 as it's conceptually owned by that pointer.

Jeremy.

10 years agos4:ldb Fix segfault in ldbsearch store_referral callback
Andrew Bartlett [Thu, 25 Feb 2010 00:46:41 +0000 (11:46 +1100)]
s4:ldb Fix segfault in ldbsearch store_referral callback

sctx->refs_store was not initialised, and that made talloc_realloc
grumpy once we started actually returning referrals regularly from
Samba4's partitions module (0be57c747825737fa9d64411223e693b055b5f8f
by mdw).

We now just use talloc_zero() and forget about this manual
initialisation work.  Tracking down use of uninitialised variables
with valgrind was the grand idea when this started, but in practice we
just get segfaults in unusual places.

Andrew Bartlett

10 years agoChange the credential handling so that we start with maxmux creds,
Jeremy Allison [Thu, 25 Feb 2010 00:16:30 +0000 (16:16 -0800)]
Change the credential handling so that we start with maxmux creds,
and then return to the client the number of credits per operation
that they asked for. This is a more sensible algorithm than just
blindly returning "20" on every reply, although we will probably
still need more changes to this going forward.
Jeremy.

10 years agoidl:lsa change string type
Simo Sorce [Wed, 24 Feb 2010 22:55:25 +0000 (17:55 -0500)]
idl:lsa change string type

comapring win<-> traces it looks like this is an lsa_StringLarge

10 years agos4:lsa use the correct way to store a domain sid
Simo Sorce [Wed, 24 Feb 2010 22:10:59 +0000 (17:10 -0500)]
s4:lsa use the correct way to store a domain sid

Converting the sid to a string and then storing a string does not save the sid
in the right format. Causing following retrievals to fail to read back a sid
with samdb_result_dom_sid().

10 years agos4:lsa avoid confusing ourselves over sam_ldb
Simo Sorce [Wed, 24 Feb 2010 22:07:26 +0000 (17:07 -0500)]
s4:lsa avoid confusing ourselves over sam_ldb

Do not use policy_state->sam_ldb and trusted_domain_state->policy->sam_ldb
interchangeably all over the place. Just use sam_ldb everywhere and make the
code slightly more readable.

10 years agos4:lsa cleanup trailing spaces and tabs
Simo Sorce [Wed, 24 Feb 2010 21:54:16 +0000 (16:54 -0500)]
s4:lsa cleanup trailing spaces and tabs

10 years agoRevert "s4-smb: Migrate named_pipe_server to tsocket."
Simo Sorce [Wed, 24 Feb 2010 21:35:35 +0000 (16:35 -0500)]
Revert "s4-smb: Migrate named_pipe_server to tsocket."

This reverts commit 69d5cea2e59162f19460e7ce4b6382fc5fdd6ca0.

This commit causes issues with the RPC server, revert it until we find the
exact issue and possibly have a torture test to avoid it happening again.
Found playing with w2k8r2 and forest trusts.

10 years agos4:install Fix bug #7149 reported by JHT.
Andrew Bartlett [Wed, 24 Feb 2010 08:19:41 +0000 (19:19 +1100)]
s4:install Fix bug #7149 reported by JHT.

We need to install named.conf.update for provision to succeed from the
installed setup file.

Andrew Bartlett

10 years agos4:scripting/devel Allow tmpfs script to be re-run
Andrew Bartlett [Mon, 18 Jan 2010 06:24:45 +0000 (19:24 +1300)]
s4:scripting/devel Allow tmpfs script to be re-run

By doing the unmount, we can avoid double-mounting st and bin

10 years agos4:DNS update - change "i" to be unsigned
Matthias Dieter Wallnöfer [Thu, 11 Feb 2010 10:49:26 +0000 (11:49 +0100)]
s4:DNS update - change "i" to be unsigned

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
10 years agos4:ldap_server - make it "signed-safe"
Matthias Dieter Wallnöfer [Sat, 7 Nov 2009 20:21:26 +0000 (21:21 +0100)]
s4:ldap_server - make it "signed-safe"

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
10 years agos4:auth - make some parts "signed-safe"
Matthias Dieter Wallnöfer [Sat, 7 Nov 2009 20:20:12 +0000 (21:20 +0100)]
s4:auth - make some parts "signed-safe"

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
10 years agos4:cldap_server - make it "signed-safe"
Matthias Dieter Wallnöfer [Sat, 7 Nov 2009 20:20:56 +0000 (21:20 +0100)]
s4:cldap_server - make it "signed-safe"

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
10 years agos4:torture/ldap/basic.c - add a basic test for referral return
Matthias Dieter Wallnöfer [Sun, 21 Feb 2010 19:36:34 +0000 (20:36 +0100)]
s4:torture/ldap/basic.c - add a basic test for referral return

I implemented this referral test in C since the LDB python API isn't capable
to extract referrals from search result sets (there the result sets are simple
lists which contain only the matching entries).

First I enhanced the RootDSE test to return all partition base DNs in a new
null-terminated list "partitions". Then I used this in my referrals test which
I've implemented in the LDB api since I needed some certain DN functions.

10 years agos4:partition DSDB module - Generate basic referrals
Matthias Dieter Wallnöfer [Sat, 20 Feb 2010 21:07:12 +0000 (22:07 +0100)]
s4:partition DSDB module - Generate basic referrals

This is a first, very basic implementation of the referrals (more informations
at MS-ADTS 3.1.1.4.6 and 3.1.1.3.4.1.12).

To have the full referral support (and to always point to the right host) the
full implementation using DNS will be needed (at the moment we always point to
the main DC which is referenceable through the DNS domainname).

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
10 years agos4:partition DSDB module - change the search and domain scope control handling
Matthias Dieter Wallnöfer [Mon, 22 Feb 2010 10:19:10 +0000 (11:19 +0100)]
s4:partition DSDB module - change the search and domain scope control handling

The domain scope control is always removed, from the search one only the two
interesting flags (which are handled) and it is marked as non-critical.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
10 years agos4:LDAP server - Enable support for returning referrals through it
Matthias Dieter Wallnöfer [Sun, 21 Feb 2010 10:56:12 +0000 (11:56 +0100)]
s4:LDAP server - Enable support for returning referrals through it

This is needed for my work regarding the referrals when the domain scope control
isn't specified.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
10 years agos4:SAMLDB module - ignore referrals
Matthias Dieter Wallnöfer [Tue, 23 Feb 2010 16:59:55 +0000 (17:59 +0100)]
s4:SAMLDB module - ignore referrals

They don't cause any harm to our functionality - so ignore them were not needed.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
10 years agos4:netlogon remove wrong ZERO_STRUCT of output
Simo Sorce [Wed, 24 Feb 2010 16:02:18 +0000 (11:02 -0500)]
s4:netlogon remove wrong ZERO_STRUCT of output

This was causing marshalling faults when we returned errors.

10 years agos3: Make connections_fetch_record() static
Volker Lendecke [Wed, 24 Feb 2010 14:38:06 +0000 (15:38 +0100)]
s3: Make connections_fetch_record() static

10 years agopython: ntacls, fix a leftover that is not in the try/except branch
Matthieu Patou [Fri, 19 Feb 2010 20:09:57 +0000 (23:09 +0300)]
python: ntacls, fix a leftover that is not in the try/except branch

Signed-off-by: Matthias Dieter Wallnöfer <mwallnoefer@yahoo.de>
10 years agodsdb: Add a more explicit error message for constructed attributes
Matthieu Patou [Tue, 23 Feb 2010 13:03:16 +0000 (16:03 +0300)]
dsdb: Add a more explicit error message for constructed attributes

Signed-off-by: Matthias Dieter Wallnöfer <mwallnoefer@yahoo.de>
10 years agos4/drs_util: 'net drs showrepl' command implementation
Kamen Mazdrashki [Wed, 17 Feb 2010 19:32:50 +0000 (21:32 +0200)]
s4/drs_util: 'net drs showrepl' command implementation

10 years agos4/idl: Regenerate IDL for DRSUAPI interface
Kamen Mazdrashki [Sun, 14 Feb 2010 00:50:13 +0000 (02:50 +0200)]
s4/idl: Regenerate IDL for DRSUAPI interface

10 years agos4/drs: Propagate drsuapi_DsReplicaGetInfoRequest2 changes in source code
Kamen Mazdrashki [Sun, 14 Feb 2010 00:49:40 +0000 (02:49 +0200)]
s4/drs: Propagate drsuapi_DsReplicaGetInfoRequest2 changes in source code

10 years agos4/idl: drsuapi_DsReplicaGetInfoRequest2 - 'string2' to 'value_dn_str'
Kamen Mazdrashki [Sun, 14 Feb 2010 00:49:02 +0000 (02:49 +0200)]
s4/idl: drsuapi_DsReplicaGetInfoRequest2 - 'string2' to 'value_dn_str'

10 years agos4/idl: drsuapi_DsReplicaGetInfoRequest2 - 'string1' to 'attribute_name'
Kamen Mazdrashki [Sun, 14 Feb 2010 00:47:57 +0000 (02:47 +0200)]
s4/idl: drsuapi_DsReplicaGetInfoRequest2 - 'string1' to 'attribute_name'

10 years agos4/drs: Propagate drsuapi_DsReplicaGetInfoRequest... changes into source code
Kamen Mazdrashki [Sun, 14 Feb 2010 00:40:07 +0000 (02:40 +0200)]
s4/drs: Propagate drsuapi_DsReplicaGetInfoRequest... changes into source code

10 years agos4/idl: rename 'guid1' to 'source_dsa_guid' in drsuapi_DsReplicaGetInfoRequest descri...
Kamen Mazdrashki [Sun, 14 Feb 2010 00:38:11 +0000 (02:38 +0200)]
s4/idl: rename 'guid1' to 'source_dsa_guid' in drsuapi_DsReplicaGetInfoRequest description

10 years agos4/drs_util: 'net drs replicate' command implementation
Kamen Mazdrashki [Sat, 13 Feb 2010 01:27:27 +0000 (03:27 +0200)]
s4/drs_util: 'net drs replicate' command implementation

10 years agos4/drs_util: Add public function for binding to a DC
Kamen Mazdrashki [Fri, 12 Feb 2010 04:13:22 +0000 (06:13 +0200)]
s4/drs_util: Add public function for binding to a DC

10 years agos4/drs_util: Refactor code to use net_drs_connection object for DRSUAPI connecitons
Kamen Mazdrashki [Fri, 12 Feb 2010 04:02:05 +0000 (06:02 +0200)]
s4/drs_util: Refactor code to use net_drs_connection object for DRSUAPI connecitons

10 years agos4/drs_util: Move DRSUAPI connection data into separate object
Kamen Mazdrashki [Fri, 12 Feb 2010 04:01:06 +0000 (06:01 +0200)]
s4/drs_util: Move DRSUAPI connection data into separate object

We need this so we can create independent DRS connections to
different DCs.

10 years agos4/net_drs: Utility macros for conditions checking
Kamen Mazdrashki [Thu, 11 Feb 2010 23:47:55 +0000 (01:47 +0200)]
s4/net_drs: Utility macros for conditions checking

10 years agos4/drs: Propagate drsuapi_DsReplicaSync changes in source base
Kamen Mazdrashki [Sat, 13 Feb 2010 02:57:28 +0000 (04:57 +0200)]
s4/drs: Propagate drsuapi_DsReplicaSync changes in source base

10 years agos4/drs: Propagate drsuapi_DsReplicaSyncRequest1 changes in source base
Kamen Mazdrashki [Thu, 11 Feb 2010 22:20:52 +0000 (00:20 +0200)]
s4/drs: Propagate drsuapi_DsReplicaSyncRequest1 changes in source base

10 years agos4/idl: Regenerate IDL for DRSUAPI interface
Kamen Mazdrashki [Thu, 11 Feb 2010 22:17:52 +0000 (00:17 +0200)]
s4/idl: Regenerate IDL for DRSUAPI interface

10 years agos4/idl: drsuapi.idl fix drsuapi_DsReplicaSync definition
Kamen Mazdrashki [Sat, 13 Feb 2010 02:53:20 +0000 (04:53 +0200)]
s4/idl: drsuapi.idl fix drsuapi_DsReplicaSync definition

- Function should accept pointer to drsuapi_DsReplicaSyncRequest.
  While this doesn't generate essentially different code for
  NDR parser, using pointer will make drsuapi_DsReplicaSync
  descritpin with the rest of the functions in DRSUAPI interface.
  Another benefit is that this way we could create Wireshark
  dissector directly from Samba's verions for drsuapi.idl

- 'level' and thus the switch_type() should be uint32

10 years agos4/idl: drsuapi.idl fix drsuapi_DsReplicaSyncRequest1 description
Kamen Mazdrashki [Thu, 11 Feb 2010 22:15:10 +0000 (00:15 +0200)]
s4/idl: drsuapi.idl fix drsuapi_DsReplicaSyncRequest1 description

- pointer to naming_context should be [ref] pointer
  (i.e. not NULL pointer)
- other_info is actually the DNS name for Source DSA and is used
  if DRSUAPI_DRS_SYNC_BYNAME is passed
  ref: [MS-DRSR] 5.39

10 years agos3:selftest: handle spaces in test names
Stefan Metzmacher [Wed, 24 Feb 2010 09:09:32 +0000 (10:09 +0100)]
s3:selftest: handle spaces in test names

metze

10 years agos3:selftest: make wbinfo_s3 work on the "member" server too.
Stefan Metzmacher [Wed, 24 Feb 2010 08:33:19 +0000 (09:33 +0100)]
s3:selftest: make wbinfo_s3 work on the "member" server too.

metze

10 years agos3:test_wbinfo_s3: test --check-secret and --change-secret
Stefan Metzmacher [Tue, 23 Feb 2010 13:15:00 +0000 (14:15 +0100)]
s3:test_wbinfo_s3: test --check-secret and --change-secret

metze

10 years agos3:rpc_transport_np: handle trans rdata like the output of a normal read
Stefan Metzmacher [Sat, 20 Feb 2010 08:08:16 +0000 (09:08 +0100)]
s3:rpc_transport_np: handle trans rdata like the output of a normal read

Inspired by bug #7159.

metze

10 years agos4-smbtorture: verify that the client cpu architecture has no influence on the
Günther Deschner [Tue, 23 Feb 2010 12:05:52 +0000 (13:05 +0100)]
s4-smbtorture: verify that the client cpu architecture has no influence on the
calculated buffer size in RPC-SPOOLSS.

Guenther

10 years agotdb: handle processes dying during transaction commit.
Rusty Russell [Wed, 24 Feb 2010 02:53:58 +0000 (13:23 +1030)]
tdb: handle processes dying during transaction commit.

tdb transactions were designed to be robust against the machine
powering off, but interestingly were never designed to handle the case
where an administrator kill -9's a process during commit.  Because
recovery is only done on tdb_open, processes with the tdb already
mapped will simply use it despite it being corrupt and needing
recovery.

The solution to this is to check for recovery every time we grab a
data lock: we could have gained the lock because a process just died.
This has no measurable cost: here is the time for tdbtorture -s 0 -n 1
-l 10000:

Before:
2.75 2.50 2.81 3.19 2.91 2.53 2.72 2.50 2.78 2.77 = Avg 2.75

After:
2.81 2.57 3.42 2.49 3.02 2.49 2.84 2.48 2.80 2.43 = Avg 2.74

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
10 years agopatch tdb-refactor-tdb_lock-and-tdb_lock_nonblock.patch
Rusty Russell [Wed, 24 Feb 2010 02:48:06 +0000 (13:18 +1030)]
patch tdb-refactor-tdb_lock-and-tdb_lock_nonblock.patch

10 years agotdb: add -k option to tdbtorture
Rusty Russell [Wed, 24 Feb 2010 00:23:05 +0000 (10:53 +1030)]
tdb: add -k option to tdbtorture

To test the case of death of a process during transaction commit, add
a -k (kill random) option to tdbtorture.  The easiest way to do this
is to make every worker a child (unless there's only one child), which
is why this patch is bigger than you might expect.

Using -k without -t (always transactions) you expect corruption, though
it doesn't happen every time.  With -t, we currently get corruption but
the next patch fixes that.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
10 years agotdb: don't truncate tdb on recovery
Rusty Russell [Wed, 24 Feb 2010 00:20:41 +0000 (10:50 +1030)]
tdb: don't truncate tdb on recovery

The current recovery code truncates the tdb file on recovery.  This is
fine if recovery is only done on first open, but is a really bad idea
as we move to allowing recovery on "live" databases.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
10 years agotdb: remove lock ops
Rusty Russell [Wed, 24 Feb 2010 00:19:22 +0000 (10:49 +1030)]
tdb: remove lock ops

Now the transaction code uses the standard allrecord lock, that stops
us from trying to grab any per-record locks anyway.  We don't need to
have special noop lock ops for transactions.

This is a nice simplification: if you see brlock, you know it's really
going to grab a lock.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
10 years agotdb: rename tdb_release_extra_locks() to tdb_release_transaction_locks()
Rusty Russell [Wed, 24 Feb 2010 00:32:55 +0000 (11:02 +1030)]
tdb: rename tdb_release_extra_locks() to tdb_release_transaction_locks()

tdb_release_extra_locks() is too general: it carefully skips over the
transaction lock, even though the only caller then drops it.  Change
this, and rename it to show it's clearly transaction-specific.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
10 years agotdb: cleanup: remove ltype argument from _tdb_transaction_cancel.
Rusty Russell [Wed, 24 Feb 2010 02:12:24 +0000 (12:42 +1030)]
tdb: cleanup: remove ltype argument from _tdb_transaction_cancel.

Now the transaction allrecord lock is the standard one, and thus is cleaned
in tdb_release_extra_locks(), _tdb_transaction_cancel() doesn't need to
know what type it is.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
10 years agotdb: tdb_allrecord_lock/tdb_allrecord_unlock/tdb_allrecord_upgrade
Rusty Russell [Wed, 17 Feb 2010 05:12:15 +0000 (15:42 +1030)]
tdb: tdb_allrecord_lock/tdb_allrecord_unlock/tdb_allrecord_upgrade

Centralize locking of all chains of the tdb; rename _tdb_lockall to
tdb_allrecord_lock and _tdb_unlockall to tdb_allrecord_unlock, and
tdb_brlock_upgrade to tdb_allrecord_upgrade.

Then we use this in the transaction code.  Unfortunately, if the transaction
code records that it has grabbed the allrecord lock read-only, write locks
will fail, so we treat this upgradable lock as a write lock, and mark it
as upgradable using the otherwise-unused offset field.

One subtlety: now the transaction code is using the allrecord_lock, the
tdb_release_extra_locks() function drops it for us, so we no longer need
to do it manually in _tdb_transaction_cancel.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
10 years agotdb: suppress record write locks when allrecord lock is taken.
Rusty Russell [Wed, 24 Feb 2010 00:15:26 +0000 (10:45 +1030)]
tdb: suppress record write locks when allrecord lock is taken.

Records themselves get (read) locked by the traversal code against delete.
Interestingly, this locking isn't done when the allrecord lock has been
taken, though the allrecord lock until recently didn't cover the actual
records (it now goes to end of file).

The write record lock, grabbed by the delete code, is not suppressed
by the allrecord lock.  This is now bad: it causes us to punch a hole
in the allrecord lock when we release the write record lock.  Make this
consistent: *no* record locks of any kind when the allrecord lock is
taken.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
10 years agotdb: cleanup: always grab allrecord lock to infinity.
Rusty Russell [Wed, 24 Feb 2010 00:15:14 +0000 (10:45 +1030)]
tdb: cleanup: always grab allrecord lock to infinity.

We were previously inconsistent with our "global" lock: the
transaction code grabbed it from FREELIST_TOP to end of file, and the
rest of the code grabbed it from FREELIST_TOP to end of the hash
chains.  Change it to always grab to end of file for simplicity and
so we can merge the two.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
10 years agotdb: remove num_locks
Rusty Russell [Wed, 17 Feb 2010 04:31:07 +0000 (15:01 +1030)]
tdb: remove num_locks

This was redundant before this patch series: it mirrored num_lockrecs
exactly.  It still does.

Also, skip useless branch when locks == 1: unconditional assignment is
cheaper anyway.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
10 years agotdb: use tdb_nest_lock() for seqnum lock.
Rusty Russell [Wed, 17 Feb 2010 02:10:57 +0000 (12:40 +1030)]
tdb: use tdb_nest_lock() for seqnum lock.

This is pure overhead, but it centralizes the locking.  Realloc (esp. as
most implementations are lazy) is fast compared to the fnctl anyway.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
10 years agotdb: use tdb_nest_lock() for active lock.
Rusty Russell [Wed, 24 Feb 2010 00:14:40 +0000 (10:44 +1030)]
tdb: use tdb_nest_lock() for active lock.

Use our newly-generic nested lock tracking for the active lock.

Note that the tdb_have_extra_locks() and tdb_release_extra_locks()
functions have to skip over this lock now it is tracked.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
10 years agotdb: use tdb_nest_lock() for open lock.
Rusty Russell [Mon, 22 Feb 2010 03:28:07 +0000 (13:58 +1030)]
tdb: use tdb_nest_lock() for open lock.

This never nests, so it's overkill, but it centralizes the locking into
lock.c and removes the ugly flag in the transaction code to track whether
we have the lock or not.

Note that we have a temporary hack so this places a real lock, despite
the fact that we are in a transaction.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
10 years agotdb: use tdb_nest_lock() for transaction lock.
Rusty Russell [Wed, 17 Feb 2010 02:07:34 +0000 (12:37 +1030)]
tdb: use tdb_nest_lock() for transaction lock.

Rather than a boutique lock and a separate nest count, use our
newly-generic nested lock tracking for the transaction lock.

Note that the tdb_have_extra_locks() and tdb_release_extra_locks()
functions have to skip over this lock now it is tracked.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
10 years agotdb: cleanup: find_nestlock() helper.
Rusty Russell [Wed, 17 Feb 2010 02:05:54 +0000 (12:35 +1030)]
tdb: cleanup: find_nestlock() helper.

Factor out two loops which find locks; we are going to introduce a couple
more so a helper makes sense.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
10 years agotdb: cleanup: tdb_release_extra_locks() helper
Rusty Russell [Wed, 24 Feb 2010 00:11:15 +0000 (10:41 +1030)]
tdb: cleanup: tdb_release_extra_locks() helper

Move locking intelligence back into lock.c, rather than open-coding the
lock release in transaction.c.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
10 years agotdb: cleanup: tdb_have_extra_locks() helper
Rusty Russell [Wed, 17 Feb 2010 02:04:26 +0000 (12:34 +1030)]
tdb: cleanup: tdb_have_extra_locks() helper

In many places we check whether locks are held: add a helper to do this.

The _tdb_lockall() case has already checked for the allrecord lock, so
the extra work done by tdb_have_extra_locks() is merely redundant.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
10 years agotdb: don't suppress the transaction lock because of the allrecord lock.
Rusty Russell [Wed, 17 Feb 2010 02:01:49 +0000 (12:31 +1030)]
tdb: don't suppress the transaction lock because of the allrecord lock.

tdb_transaction_lock() and tdb_transaction_unlock() do nothing if we
hold the allrecord lock.  However, the two locks don't overlap, so
this is wrong.

This simplification makes the transaction lock a straight-forward nested
lock.

There are two callers for these functions:
1) The transaction code, which already makes sure the allrecord_lock
   isn't held.
2) The traverse code, which wants to stop transactions whether it has the
   allrecord lock or not.  There have been deadlocks here before, however
   this should not bring them back (I hope!)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
10 years agotdb: cleanup: tdb_nest_lock/tdb_nest_unlock
Rusty Russell [Wed, 17 Feb 2010 01:56:13 +0000 (12:26 +1030)]
tdb: cleanup: tdb_nest_lock/tdb_nest_unlock

Because fcntl locks don't nest, we track them in the tdb->lockrecs array
and only place/release them when the count goes to 1/0.  We only do this
for record locks, so we simply place the list number (or -1 for the free
list) in the structure.

To generalize this:

1) Put the offset rather than list number in struct tdb_lock_type.
2) Rename _tdb_lock() to tdb_nest_lock, make it non-static and move the
   allrecord check out to the callers (except the mark case which doesn't
   care).
3) Rename _tdb_unlock() to tdb_nest_unlock(), make it non-static and
   move the allrecord out to the callers (except mark again).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
10 years agotdb: cleanup: rename global_lock to allrecord_lock.
Rusty Russell [Wed, 17 Feb 2010 01:49:47 +0000 (12:19 +1030)]
tdb: cleanup: rename global_lock to allrecord_lock.

The word global is overloaded in tdb.  The global_lock inside struct
tdb_context is used to indicate we hold a lock across all the chains.

Rename it to allrecord_lock.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
10 years agotdb: cleanup: rename GLOBAL_LOCK to OPEN_LOCK.
Rusty Russell [Wed, 17 Feb 2010 01:48:33 +0000 (12:18 +1030)]
tdb: cleanup: rename GLOBAL_LOCK to OPEN_LOCK.

The word global is overloaded in tdb.  The GLOBAL_LOCK offset is used at
open time to serialize initialization (and by the transaction code to block
open).

Rename it to OPEN_LOCK.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
10 years agotdb: make _tdb_transaction_cancel static.
Rusty Russell [Wed, 24 Feb 2010 00:09:59 +0000 (10:39 +1030)]
tdb: make _tdb_transaction_cancel static.

Now tdb_open() calls tdb_transaction_cancel() instead of
_tdb_transaction_cancel, we can make it static.

Signed-off-by: Rusty Russell<rusty@rustcorp.com.au>
10 years agotdb: cleanup: split brlock and brunlock methods.
Rusty Russell [Wed, 17 Feb 2010 01:47:19 +0000 (12:17 +1030)]
tdb: cleanup: split brlock and brunlock methods.

This is taken from the CCAN code base: rather than using tdb_brlock for
locking and unlocking, we split it into brlock and brunlock functions.

For extra debugging information, brunlock says what kind of lock it is
unlocking (even though fnctl locks don't need this).  This requires an
extra argument to tdb_transaction_unlock() so we know whether the
lock was upgraded to a write lock or not.

We also use a "flags" argument tdb_brlock:
1) TDB_LOCK_NOWAIT replaces lck_type = F_SETLK (vs F_SETLKW).
2) TDB_LOCK_MARK_ONLY replaces setting TDB_MARK_LOCK bit in ltype.
3) TDB_LOCK_PROBE replaces the "probe" argument.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>