an experimental patch for fixing ldb bloat
authorAndrew Tridgell <tridge@samba.org>
Tue, 16 Dec 2008 07:45:28 +0000 (18:45 +1100)
committerAndrew Tridgell <tridge@samba.org>
Tue, 16 Dec 2008 20:15:29 +0000 (07:15 +1100)
commite294c4799bf6ad8da043aadf8341391644147056
treebb11a627b978aca5cd75db4505ce536b7cb82111
parent4380a374c1dec46ad77939604e548f0c79d9e2ec
an experimental patch for fixing ldb bloat

ldb indexing can cause huge files, and huge memory usage. This
experiment allows us to keep indexes in memory during a transaction,
then to write the indexes to disk when the transaction completes. The
result is that the db is much smaller (we have seen improvements of
about 100x in file size) and memory usage during large transactions is
also greatly reduced

Note that this patch uses the unusual strategy of putting pointers
into a ldb (and thus into a tdb). This works because the pointers are
only there during a transaction, so the pointers are not exposed to
any other users of the database. The pointers allow us to avoid some
really bad allocation problems with tdb record allocation during the
re-indexing.
source4/lib/ldb/ldb_tdb/ldb_index.c
source4/lib/ldb/ldb_tdb/ldb_tdb.c
source4/lib/ldb/ldb_tdb/ldb_tdb.h