talloc: Test that talloc magic differs between processes.

Signed-off-by: Adrian Cochrane <adrianc@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Fri Oct  9 23:15:06 CEST 2015 on sn-devel-104

talloc: Increment minor version due to added talloc_test_get_magic.

Signed-off-by: Adrian Cochrane <adrianc@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

talloc: Provide tests access to talloc_magic

Signed-off-by: Adrian Cochrane <adrianc@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

talloc: Test magic protection measures.

Signed-off-by: Adrian Cochrane <adrianc@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

lib/talloc: Provide multiple-loading detection for libtalloc via rand()

The use of rand() is strongly discrouanged, but here is it ideal, as we just want to select a different
set of random bytes if we are called again within the same process.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Signed-off-by: Adrian Cochrane <adrianc@catalyst.net.nz>
Reviewed-by: Jeremy Allison <jra@samba.org>

lib/talloc: Disrupt buffer overflow attacks on Samba by using random magic

By setting the talloc magic to a random value, we make it much harder to overwrite a talloc_chunk
in a valid way with a simple buffer overflow.

The flags are placed before more senstive variables so they have to be overwritten first.

Inspired by the exploit in: http://blog.csnc.ch/wp-content/uploads/2012/07/sambaexploit_v1.0.pdf

Andrew Bartlett

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Signed-off-by: Adrian Cochrane <adrianc@catalyst.net.nz>
Reviewed-by: Jeremy Allison <jra@samba.org>

build: Move __attribute__ ((destructor)) and ((constructor)) tests to wafsamba

This allows us to use them in talloc as well.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Signed-off-by: Adrian Cochrane <adrianc@catalyst.net.nz>
Reviewed-by: Jeremy Allison <jra@samba.org>

Fix a few small spelling mistakes in DEBUG messages to reduce confusion
for those trying to debug stuff.

Signed-off-by: Richard Sharpe <rsharpe@samba.org>
Reviewed-by: Ira Cooper <ira@samba.org>
Autobuild-User(master): Richard Sharpe <sharpe@samba.org>
Autobuild-Date(master): Thu Oct  8 08:48:06 CEST 2015 on sn-devel-104

vfs_commit: set the fd on open before calling SMB_VFS_FSTAT

BUG: https://bugzilla.samba.org/show_bug.cgi?id=11547

Signed-off-by: Uri Simchoni <uri@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Thu Oct  8 02:56:41 CEST 2015 on sn-devel-104

lib: Make messaging_send_iov_from return 0/errno

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

lib: Make messaging_ctdbd_init return 0/errno

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

lib: Use poll_intr_one_fd in ctdb_read_packet

This is an actual bug fix if someone sets "ctdb timeout" to something != 0

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

lib: Remove messaging_tevent_context() dependency from ctdbd_conn.c

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

lib: Make ctdbd_probe return 0/errno

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

lib: Make ctdb_unwatch return 0/errno

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

lib: Make ctdb_watch_us return 0/errno

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

lib: Make ctdbd_control_local return 0/errno

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

lib: Make ctdbd_register_ips return 0/errno

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

lib: Make ctdbd_traverse return 0/errno

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

lib: Make ctdbd_parse return 0/errno

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

lib: Make ctdbd_migrate return 0/errno

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

lib: Make ctdbd_db_attach return 0/errno

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

lib: Make ctdbd_messaging_send_iov return 0/errno

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

lib: Make ctdbd_messaging_connection return 0/errno

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

lib: Make ctdbd_init_connection return 0/errno

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

lib: Make ctdbd_register_msg_ctx return 0/errno

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

lib: Make get_cluster_vnn return 0/errno

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

lib: Make register_with_ctdbd return 0/errno

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

lib: Rename ctdbd_control_unix to ctdbd_control

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

lib: Remove ctdbd_control

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

lib: Use ctdbd_control_unix in ctdbd_register_ips

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

lib: Use ctdbd_control_unix in ctdbd_db_attach

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

lib: Use ctdbd_control_unix in ctdbd_db_attach

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

lib: Use ctdbd_control_unix in ctdbd_dbpath

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

lib: Use ctdbd_control_unix in ctdbd_working

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

lib: Rename a variable

We'll have "int ret" in the next commit

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

lib: Use ctdbd_control_unix in get_cluster_vnn

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

lib: Use ctdbd_control_unix in register_with_ctdbd

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

lib: Add ctdbd_control_unix

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

lib: Fix error talloc leaks in ctdb_read_packet()

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

ctdb-recoverd: Always check for recmaster before doing recovery

Recovery daemon checks if it is the recovery master before performing
certain checks.  During those checks it's possible that re-election can
change the recmaster.  In such a case, the recovery daemon should never
do a database recovery.

This is not complete fix since the recovery master can still change
while the recovery is going on.  The correct fix is to abort recovery
if the recovery master changes.

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Wed Oct  7 17:55:05 CEST 2015 on sn-devel-104

ctdb-recoverd: Get rid of connected-ness comparison in election

The reason for favouring more connected node is to create a larger
cluster in case of a split brain.  In split brain condition, the nodes
are not communicating across partitions and each partition will run its
own election.  Among all the partitions, the node which holds the recovery
lock will eventually "win".  All the other nodes which won election but
could not grab recovery lock will end up banning themselves.

This also prevents the recovery master role from bouncing between nodes
during startup when the entire cluster is restarted.

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-recoverd: Do not freeze databases for election

If election occurs during SMB activity, then trying to freeze all the
databases can cause samba/ctdb deadlock which parallel database recovery
is trying to avoid.

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-call: Improve a log message

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-recoverd: Add code for parallel database recovery

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-daemon: Add parallel database recovery capability

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-daemon: Use a define for default capabilities

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-recoverd: Update flags on all nodes before database recovery

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-recoverd: Update capabilities before the database recovery

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-recovery: Factor out existing database recovery code

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-recoverd: Add parallel database recovery helper

Parallel database recovery fixes the samba/ctdb deadlock during recovery.

Many times samba tries to grab multiple record locks in sequence.
Consider a case when samba is already holding a record lock on a database
and tries to get a record lock on second database.  If the second record
is not available on the local node, samba asks ctdb to migrate the record.
If recovery occurs at this time (e.g. node becoming inactive), ctdb
cannot freeze all the databases since samba is already holding a lock
and waiting for the second lock.  CTDB can process the second record
request only after the recovery is complete, thus causing a deadlock.

In parallel database recovery, each database is frozen and recovered
independent from each other.  So as soon as the second database is
recovered, CTDB will resend all the pending migration requests and Samba
can get the second lock.  Once samba releases both the locks, ctdb can
freeze the first database and recover it completing recovery process.

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-client: Add client API for new database controls

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-protocol: Add controls for parallel DB recovery

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-client: Add new client API implementation

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-protocol: Add ctdb protocol serialization routines

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-include: Remove unused structure definitions

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-common: Add logging utilities

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-common: Add communication endpoint abstraction

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-common: Add packet write abstraction

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-common: Add packet read abstraction

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-packaging: Package public library and header for tevent-unix-util

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-build: Placeholder for public headers from lib/util

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

lib/util: Create a new library for tevent_unix.c

This is required for ctdb.  This avoids adding dependency on wstatus
and ntstatus to ctdb build.

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-build: Set PKGCONFIGDIR for public libraries in standalone build

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-packaging: Install header files in ctdb subdirectory

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-daemon: Use reqid abstraction

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-common: Add request id abstraction

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-include: Remove unused definition

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-daemon: whitespace fix

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-daemon: formatting fix

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-daemon: Replace ctdb_message with srvid abstraction

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-common: Add srvid abstraction

Attempt to make common code independent and free from ctdb_context.

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-common: Add db_hash abstraction

A hash table implemented using in-memory tdb backend.

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-daemon: whitespace fix

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-daemon: formatting fix

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-daemon: Remove ctdb from traverse_callback

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-build: Do not split ctdb-system subsystem

The functions defined in common/system_*.c depend on the functions
defined in common/system_common.c.  So keep them together.

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-build: Do not mark ctdb private headers public

These headers are used by the server code and should not be marked public.
Samba builds against the in-tree version of the headers and should not
be built with externally installed CTDB.

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-daemon: Remove freeze requirement for updating vnnmap

In the parallel database recovery model, all the database will not remain
frozen at the same time.  So relax the condition to check if recovery
is active.

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-daemon: Add a check for database generation consistency

Before setting recovery mode to normal, confirm that all the databases are
recovered by matching the database generation with the global generation.

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-daemon: Check packet generation against database generation

CTDB verifies the generation in the packet header matches that of the
current generation.  However, that check now needs to be done where
database context is available.  So add in the check in handlers for
database requests (CTDB_REQ_CALL, CTDB_REQ_DMASTER, CTDB_REPLY_DMASTER
and CTDB_REPLY_CALL).

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-call: Delete old defer queue if recovery occurs

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-daemon: Use database generation in packet headers for database requests

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-daemon: Introduce per database generation

The database generation for each database is updated only during recovery.
After recovery is complete the database generation would be the same as
the global generation.

The database generation is required for parallel database recovery.

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-freeze: Use individual database freeze in blocking freeze

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-freeze: Ensure all databases get frozen during freeze

It's possible that the databases can get attached after initial freeze.
This typically happens during startup as CTDB will only attach persistent
databases and go in to startup freeze.  During recovery, the recovery
master will attach all the missing databases and then send freeze
controls.

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-freeze: Make function ctdb_start_freeze static

Since the banning code does not call this function anymore, it can be
made static.

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-banning: Do not freeze databases on ban in the daemon

Once the node is marked as banned, the recovery daemon on that node will
start freezing the databases anyway.  This check happens every second
in recovery daemon, so the delay in freezing the databases on ban would
be a second.

The main reason for freezing databases is to prevent any database access
from samba clients.  However, banning code also drops the public IPs
and thus disconnecting the clients.  In addition, the generation gets
set to INVALID_GENERATION, thus database record migration requests on
that node will not be processed.

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-freeze: Fix a log message

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-freeze: Use database specific information in wipe database

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-freeze: Use single database transactions for global transactions

This cascades the global transaction controls to individual database
transaction operations.  This ensures that the individual database state
is correctly set when processing global transaction controls.

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-daemon: Add controls for transactions on a single database

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-daemon: Rename ctdb_control_wipe_database to ctdb_control_transdb

The same structure is required in new controls for database transactions.

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-freeze: simplify code with TALLOC_FREE

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-freeze: Improve log message to indicate subsequent freeze operation

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-daemon: Use database specific mark/unmark routines

Instead of marking all the databases with priority, mark only the database
which is currently being processed.

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-daemon: Use database specific freeze check routine

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-locking: Add mark/unmark functions for a single database

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-freeze: Use single database freeze/thaw code for existing controls

This changes the locking behaviour when the databases are frozen.
Instead of a single lock helper locking all databases with a priority,
a lock helper is launched for every database.

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>

ctdb-freeze: Move destructor closer to where it is used

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>