sahlberg/remote-cache.git
9 years agoversion 3.6 master
root [Thu, 23 Jul 2009 22:47:45 +0000 (08:47 +1000)]
version 3.6

9 years agoremote-cache-clean: add a final pass to remove empty directories.
Martin Schwenke [Thu, 23 Jul 2009 06:35:02 +0000 (16:35 +1000)]
remote-cache-clean: add a final pass to remove empty directories.

Signed-off-by: Martin Schwenke <martin@meltin.net>
9 years agoAdditional cache cleaning algorithms.
Martin Schwenke [Thu, 23 Jul 2009 02:38:37 +0000 (12:38 +1000)]
Additional cache cleaning algorithms.

This adds 2 new options that enable additional algorithms:

* Delete all files with ctime less than some value.

* Delete all files with atime less than some value.

The offsets are specified in hours via the --trim-old and
--trim-unused options.

Also provided is an option --no-lru to disabled the LRU pass.

The code has been heavily refactored to allow the new features to be
neatly implemented.  New function _trim() now does most of the grunt
work.

Documentation is updated to reflect the new features.

Signed-off-by: Martin Schwenke <martin@meltin.net>
9 years agoRejig code and logging in preparation for extra cleanup methods.
Martin Schwenke [Tue, 21 Jul 2009 07:46:13 +0000 (17:46 +1000)]
Rejig code and logging in preparation for extra cleanup methods.

* Make Cache.scan() no longer private.

* Prefix log messages with function name.  This could be done by using
  %(funcName)s in the logging.Formatter but that option was only added
  in Python 2.5 and this will need to run on Python 2.4.

* Move summary log messages to main and add some extra logging to make
  things clearer.

Signed-off-by: Martin Schwenke <martin@meltin.net>
9 years agoUpdate RPM spec file to cope with moved and new files.
Martin Schwenke [Tue, 21 Jul 2009 07:18:08 +0000 (17:18 +1000)]
Update RPM spec file to cope with moved and new files.

Signed-off-by: Martin Schwenke <martin@meltin.net>
9 years agoremote-cache should be installed in sbin rather than bin.
Martin Schwenke [Tue, 21 Jul 2009 07:11:09 +0000 (17:11 +1000)]
remote-cache should be installed in sbin rather than bin.

remote-cache is a system administration command, so should be
installed in sbin rather than bin.

Signed-off-by: Martin Schwenke <martin@meltin.net>
9 years agoCleanups to doc/remote-cache.8.xml and commit of generated files.
Martin Schwenke [Tue, 21 Jul 2009 06:31:00 +0000 (16:31 +1000)]
Cleanups to doc/remote-cache.8.xml and commit of generated files.

Cleanups to get rid of some "fixme"s and warnings from recent
docbook.xsl.  Also some simple changes to make the output more
reliably pretty.  Regenerated doc/remote-cache.8 and
doc/remote-cache.8.html.

Signed-off-by: Martin Schwenke <martin@meltin.net>
9 years agoMove remote-cache documentation to section 8 of the manual.
Martin Schwenke [Tue, 21 Jul 2009 05:57:13 +0000 (15:57 +1000)]
Move remote-cache documentation to section 8 of the manual.

Signed-off-by: Martin Schwenke <martin@meltin.net>
9 years agoAdd remote-cache-clean derived documentation files.
Martin Schwenke [Tue, 21 Jul 2009 05:52:21 +0000 (15:52 +1000)]
Add remote-cache-clean derived documentation files.

Signed-off-by: Martin Schwenke <martin@meltin.net>
9 years agoFix circular dependency error with autoconf 2.6.3.
Martin Schwenke [Tue, 21 Jul 2009 05:38:20 +0000 (15:38 +1000)]
Fix circular dependency error with autoconf 2.6.3.

As per the following Samba commit:

  commit b39611c36bb904774fd4032bf2f8003fbdeb5d34
  Author: Andreas Schneider <anschneider@suse.de>
  Date:   Wed Oct 29 14:12:04 2008 +0100

Signed-off-by: Andreas Schneider <anschneider@suse.de>
Signed-off-by: Martin Schwenke <martin@meltin.net>
9 years agoNew remote-cache-clean script.
Martin Schwenke [Tue, 21 Jul 2009 05:32:15 +0000 (15:32 +1000)]
New remote-cache-clean script.

This adds a cache cleanup script that can be used to trim the cache.
This script is written in Python.  Also includes manpage and
associated Makefile.in updates.

Signed-off-by: Martin Schwenke <martin@meltin.net>
9 years agonew version 3.5
root [Wed, 15 Jul 2009 03:10:44 +0000 (13:10 +1000)]
new version 3.5

9 years agoremove the switch_to_real_user() function and switch_back_to_root()
root [Wed, 15 Jul 2009 00:25:40 +0000 (10:25 +1000)]
remove the switch_to_real_user() function and switch_back_to_root()

9 years agoremove the last use of switch_to_real_user()
root [Wed, 15 Jul 2009 00:24:57 +0000 (10:24 +1000)]
remove the last use of switch_to_real_user()

9 years agoremote an additional switch_to_real_user()
root [Wed, 15 Jul 2009 00:01:40 +0000 (10:01 +1000)]
remote an additional switch_to_real_user()

9 years agoremove a switch_to_real_user()
root [Tue, 14 Jul 2009 07:37:06 +0000 (17:37 +1000)]
remove a switch_to_real_user()

9 years agoremove a dead function
root [Tue, 14 Jul 2009 06:26:10 +0000 (16:26 +1000)]
remove a dead function

9 years agoremove a call to switch_to_real_user
root [Tue, 14 Jul 2009 05:34:29 +0000 (15:34 +1000)]
remove a call to switch_to_real_user

9 years agoremove the use of one switch_to_real_user()
root [Tue, 14 Jul 2009 04:59:44 +0000 (14:59 +1000)]
remove the use of one switch_to_real_user()

10 years agonew version 3.4
root [Tue, 7 Apr 2009 00:37:59 +0000 (10:37 +1000)]
new version 3.4

10 years agomake sure we replace any existing man page with a new manpage if/when we upgrade...
root [Tue, 7 Apr 2009 00:32:12 +0000 (10:32 +1000)]
make sure we replace any existing man page with a new manpage if/when we upgrade the rpm

10 years agoadd a simple array of hashes so that we dont keep adding the same file to the migrati...
root [Tue, 7 Apr 2009 00:24:11 +0000 (10:24 +1000)]
add a simple array of hashes so that we dont keep adding the same file to the migration queue over and over

10 years agocheck that the umount actually worked and keep running if fusermount returned error...
root [Mon, 6 Apr 2009 23:41:41 +0000 (09:41 +1000)]
check that the umount actually worked and keep running if fusermount returned error by the filesystembeing busy

10 years agouse a mutex to make sur eonly one thread at a time tries to manipulate the tdb databases
root [Mon, 6 Apr 2009 23:31:28 +0000 (09:31 +1000)]
use a mutex to make sur eonly one thread at a time tries to manipulate the tdb databases

10 years agoadd pregenerated versions of the manpage and the html page for systems
Ronnie Sahlberg [Mon, 6 Apr 2009 22:27:50 +0000 (08:27 +1000)]
add pregenerated versions of the manpage and the html page for systems
that cant build with docbook

10 years agonew version 3.3
root [Mon, 30 Mar 2009 00:42:25 +0000 (11:42 +1100)]
new version 3.3

10 years agofix an error in a prototype
root [Mon, 30 Mar 2009 00:36:51 +0000 (11:36 +1100)]
fix an error in a prototype

10 years agoupdate documentation with the new
Ronnie Sahlberg [Mon, 30 Mar 2009 00:31:38 +0000 (11:31 +1100)]
update documentation with the new
 remote-cache --unexport=...    command

10 years agoCreate a new option --unexport=<path>" to be used to perform a
root [Mon, 30 Mar 2009 00:24:52 +0000 (11:24 +1100)]
Create a new option --unexport=<path>" to be used to perform a
controlled shutdown and unexport of a remote-cache filesystem in the new two process model.

10 years agouse a lock on the cache file to ensure that only one remote-cache node at a time...
root [Sun, 29 Mar 2009 23:15:04 +0000 (10:15 +1100)]
use a lock on the cache file to ensure that only one remote-cache node at a time tries to migrate the same file

10 years agoFork() and return from the parent back to the shell to ensure remote-cache is run...
root [Sun, 29 Mar 2009 23:04:21 +0000 (10:04 +1100)]
Fork() and return from the parent back to the shell to ensure remote-cache is run in the background.

Add a handler for SIGUSR2.    Kill -USR2 <pid of migration daemon>
is now the approved was to stop and unmount a remote cache filesystem

10 years agonew version 3.2
root [Fri, 20 Mar 2009 05:11:32 +0000 (16:11 +1100)]
new version 3.2

10 years agoremove more switching between user ids
root [Fri, 20 Mar 2009 05:08:07 +0000 (16:08 +1100)]
remove more switching between user ids

10 years agoremove more swhitches between root and the user.
root [Fri, 20 Mar 2009 03:39:25 +0000 (14:39 +1100)]
remove more swhitches between root and the user.
add subroutines to do common operations such as evaluating the permission
and creating the substring to point to the parent directory

10 years agodont switch to/from the real user inmkdir() evaluate the permission bits explicitel...
root [Fri, 20 Mar 2009 02:08:40 +0000 (13:08 +1100)]
dont switch to/from the real user inmkdir()   evaluate the permission bits explicitely in the code and do this as root instead

10 years agoremove all remaining talloc calls
root [Thu, 19 Mar 2009 05:28:43 +0000 (16:28 +1100)]
remove all remaining talloc calls

10 years agoremove the use of talloc from unlink_parent_dir_cache()
root [Thu, 19 Mar 2009 03:50:37 +0000 (14:50 +1100)]
remove the use of talloc from unlink_parent_dir_cache()

10 years agoremove the use of talloc from open_parent_cache_ro()
root [Thu, 19 Mar 2009 03:45:46 +0000 (14:45 +1100)]
remove the use of talloc from open_parent_cache_ro()

10 years agoremove the use of talloc from open_dir_cache_ro()
root [Thu, 19 Mar 2009 03:39:24 +0000 (14:39 +1100)]
remove the use of talloc from open_dir_cache_ro()

10 years agoremove the use of talloc from unlink_object()
root [Thu, 19 Mar 2009 03:32:45 +0000 (14:32 +1100)]
remove the use of talloc from unlink_object()

10 years agoupdate the description in the specfile to explain remote cache
root [Thu, 19 Mar 2009 03:23:58 +0000 (14:23 +1100)]
update the description in the specfile to explain remote cache

10 years agofix typo in the usage printout
Ronnie Sahlberg [Tue, 17 Mar 2009 03:35:06 +0000 (14:35 +1100)]
fix typo in the usage printout

10 years agonew version 3.1
Ronnie Sahlberg [Mon, 9 Mar 2009 22:08:17 +0000 (09:08 +1100)]
new version 3.1

10 years agoremove use of talloc in performance critical paths
Ronnie Sahlberg [Mon, 9 Mar 2009 22:06:55 +0000 (09:06 +1100)]
remove use of talloc in performance critical paths

10 years agoadd the generated manpage and html page for remote-cache so they are
Ronnie Sahlberg [Mon, 9 Mar 2009 03:00:35 +0000 (14:00 +1100)]
add the generated manpage and html page for remote-cache so they are
available also for those that dont want to build the documentation
themselves

10 years agonew version 3.0 2.5
root [Mon, 9 Mar 2009 02:38:27 +0000 (13:38 +1100)]
new version 3.0

10 years agomake sure we use null terminated path strings
root [Fri, 6 Mar 2009 03:44:44 +0000 (14:44 +1100)]
make sure we use null terminated path strings

10 years agomake sure we null terminate the string we return in _readlink()
root [Fri, 6 Mar 2009 03:19:21 +0000 (14:19 +1100)]
make sure we null terminate the string we return in _readlink()

10 years agoremove some DEBUG statements we no longer need.
root [Fri, 6 Mar 2009 01:58:36 +0000 (12:58 +1100)]
remove some DEBUG statements we no longer need.

10 years agosplit remote-cache into a two process model where we spawn a separate process that...
root [Fri, 6 Mar 2009 01:53:48 +0000 (12:53 +1100)]
split remote-cache into a two process model where we spawn a separate process that is dedicated to only manage migrations from the remote site onto the local cache.

Communications between the FUSE module and the migration daemon is through a tdb database holding the list of files to migrate and signals to tell the migration daemon : there is work to do now.

10 years agoanother file required by the events system
root [Thu, 5 Mar 2009 01:29:08 +0000 (12:29 +1100)]
another file required by the events system

10 years agoadd another file that the events system needs
root [Thu, 5 Mar 2009 01:23:06 +0000 (12:23 +1100)]
add another file that the events system needs

10 years agoadd a file for linked list macros that the event system depends on
root [Thu, 5 Mar 2009 01:09:15 +0000 (12:09 +1100)]
add a file for linked list macros that the event system depends on

10 years agoadd the event system from ctdb
root [Thu, 5 Mar 2009 00:58:46 +0000 (11:58 +1100)]
add the event system from ctdb

10 years agoadd gpl boilerplate to migrate.h
root [Thu, 5 Mar 2009 00:22:33 +0000 (11:22 +1100)]
add gpl boilerplate to migrate.h

remote-cache is read-write not rea-only any more

10 years agomake sure to check that fhs is no null before dereferencing it.
Ronnie Sahlberg [Fri, 27 Feb 2009 01:44:02 +0000 (12:44 +1100)]
make sure to check that fhs is no  null before dereferencing it.
it should never happen  but check nevertheless

10 years agoverion 2.9
Ronnie Sahlberg [Fri, 27 Feb 2009 01:09:25 +0000 (12:09 +1100)]
verion 2.9

10 years agouse an explicit chmod to make sure the bits are set correctly
Ronnie Sahlberg [Fri, 27 Feb 2009 01:07:34 +0000 (12:07 +1100)]
use an explicit chmod to make sure the bits are set correctly

10 years agouse file locking on the temporary cache file to prevent two immeidate
Ronnie Sahlberg [Fri, 27 Feb 2009 01:02:37 +0000 (12:02 +1100)]
use file locking on the temporary cache file to prevent two immeidate
migrations simultaneously for the same file

10 years agofix a broken debug statement
Ronnie Sahlberg [Fri, 27 Feb 2009 00:41:26 +0000 (11:41 +1100)]
fix a broken debug statement

10 years agosmall files smaller than 128k are migrated explicitely during first read
Ronnie Sahlberg [Fri, 27 Feb 2009 00:37:18 +0000 (11:37 +1100)]
small files smaller than 128k are migrated explicitely during first read
in the current thread since it this is cheap (cheaper than spawning a
child process and running rsync at least)

10 years agodont switch uid in _unlink()
Ronnie Sahlberg [Thu, 26 Feb 2009 00:50:30 +0000 (11:50 +1100)]
dont switch uid in _unlink()
instead check permissions explicitely and do the operation as root

10 years agodont switch userid to/from the real user and root in _create() since
Ronnie Sahlberg [Thu, 26 Feb 2009 00:44:58 +0000 (11:44 +1100)]
dont switch userid to/from the real user and root in _create() since
this is such an expensive operation.
Instead check the permission bits explicitely

10 years agodont flush the cached-file or the direcache after a write has completed.
Ronnie Sahlberg [Wed, 25 Feb 2009 03:30:55 +0000 (14:30 +1100)]
dont flush the cached-file or the direcache after a write has completed.
while a write does mean that any cached copy would be "stale" deleting
it at this stage is a very expensive operation.
It is much cheaper to leave the stale file in the cache and rely on next
read flushing it from the cache.

10 years agoDont switch uid/gid to the user issuing the operation, instead do it as
Ronnie Sahlberg [Wed, 25 Feb 2009 02:13:33 +0000 (13:13 +1100)]
Dont switch uid/gid to the user issuing the operation, instead do it as
root but check the mode bits explicitely.

the switch of uid/gid are surprisingly expensive

10 years agoAvoid switching to/from "real user"/root during _read()
Ronnie Sahlberg [Wed, 25 Feb 2009 00:53:27 +0000 (11:53 +1100)]
Avoid switching to/from "real user"/root during _read()
and instead evaluate the permissions explicitely before trying
open()/pread().

These switches were very expensive.

10 years agooptimize away some expensive lstat()
Ronnie Sahlberg [Mon, 23 Feb 2009 23:10:01 +0000 (10:10 +1100)]
optimize away some expensive lstat()

10 years agonew version
root [Fri, 6 Feb 2009 05:22:14 +0000 (16:22 +1100)]
new version

10 years agoadd a "big_writes" argument
root [Fri, 6 Feb 2009 05:19:56 +0000 (16:19 +1100)]
add a "big_writes" argument

10 years agomake it possible to tweak the max number of concurrent migrations
root [Thu, 5 Feb 2009 08:13:48 +0000 (19:13 +1100)]
make it possible to tweak the max number of concurrent migrations

10 years agosee if we can get a lock on the migration limiter file first becore we create and...
root [Thu, 5 Feb 2009 07:58:29 +0000 (18:58 +1100)]
see if we can get a lock on the migration limiter file first becore we create and lock the file in the local cache fs

10 years agoreduce the loglevel for an info message from ERR to INFO
root [Thu, 5 Feb 2009 07:11:06 +0000 (18:11 +1100)]
reduce the loglevel for an info message from ERR to INFO

10 years agoincrease the limit on number of concurrent migrations to 1000
root [Thu, 5 Feb 2009 07:03:46 +0000 (18:03 +1100)]
increase the limit on number of concurrent migrations to 1000

10 years agowhen trying to migrate a file, check and make sure we have created all parent directo...
root [Thu, 5 Feb 2009 07:01:50 +0000 (18:01 +1100)]
when trying to migrate a file, check and make sure we have created all parent directories, and create them if not

otherwise we will not be able to migrate and cache files when the parent directory has never been scanned (and thus created) by READDIR

10 years agobuild the docs
root [Mon, 19 Jan 2009 22:32:57 +0000 (09:32 +1100)]
build the docs

10 years agoremove big-writes doenst work with NFS anyway
root [Mon, 19 Jan 2009 22:28:00 +0000 (09:28 +1100)]
remove big-writes   doenst work with NFS anyway

10 years agoremove the lock operation
root [Mon, 19 Jan 2009 22:26:27 +0000 (09:26 +1100)]
remove the lock operation

10 years agoremove a debug statement
root [Tue, 16 Dec 2008 22:37:26 +0000 (09:37 +1100)]
remove a debug statement

10 years agoThe kernel nfs daemon can not handle filesystems offering DIRECT_IO
root [Tue, 16 Dec 2008 08:38:39 +0000 (19:38 +1100)]
The kernel nfs daemon can not handle filesystems offering DIRECT_IO
and thus by design will feed  a write i/o to the application data
in a singlehtreaded sequence of pages to the filesystem, one by one.
But hang on, it gets better,  these 4kb blobs from the idiocy of having
a demon in kernelspace has page alignment problems!
So each page is actually fed twice to the filesystem, once as 4kb-small-delta anf ythen as a second i/o of size small-delta.

Brilliant! Great work.
Thus any filesystem that uses DIRECT_IO for performance reasons will break, immediately.
Second when disabling DIRECT_IO the nice surprise comes that when a client performs 1 32kb write to kNFSd,   kNFSd and the rest of the kernel translates this to a singlethreaded sequence of 8 almost 4kb writes  and another 8 small writes  as far as the filesystem can see.

This is either a mechanism to prevent and punish out of tree filesystems or some other braindamage by the one deciding that a "hey lets put all userspace daemons inside the kernel because it is cool".

why implement a userspace service inside the kernel?   insanity.

10 years agodont build the docs by default
Ronnie Sahlberg [Tue, 16 Dec 2008 01:57:44 +0000 (12:57 +1100)]
dont build the docs by default

10 years agoadd an option to specify direct_io
Ronnie Sahlberg [Tue, 16 Dec 2008 01:57:03 +0000 (12:57 +1100)]
add an option to specify direct_io

10 years agoreactivate big writes
root [Tue, 16 Dec 2008 01:43:40 +0000 (12:43 +1100)]
reactivate big writes

10 years agofix bug when handling gids.
root [Thu, 11 Dec 2008 07:26:54 +0000 (18:26 +1100)]
fix bug when handling gids.

we have to look up all aux gids for a user every time and set them
since this must go in the oncrpc layer to the remote nfs server

create a memory tdb database to store the uid->gidlist mappings since
this operation might be quite expensive

10 years agonew version 2.5
root [Mon, 24 Nov 2008 23:00:32 +0000 (10:00 +1100)]
new version 2.5

10 years agoplug some memory leaks
root [Mon, 24 Nov 2008 22:53:17 +0000 (09:53 +1100)]
plug some memory leaks

10 years agonew version 2.4
root [Sun, 23 Nov 2008 03:15:53 +0000 (14:15 +1100)]
new version  2.4

10 years agorekmove debug code
root [Sun, 23 Nov 2008 03:14:04 +0000 (14:14 +1100)]
rekmove debug code

10 years agouse stat and check if the file exist before we try to delete it.
root [Sun, 23 Nov 2008 02:55:49 +0000 (13:55 +1100)]
use stat and check if the file exist before we try to delete it.

10 years agonew version 2.3
root [Sat, 15 Nov 2008 03:08:02 +0000 (14:08 +1100)]
new version 2.3

10 years agodont leak filedescriptors
root [Sat, 15 Nov 2008 03:06:29 +0000 (14:06 +1100)]
dont leak filedescriptors

10 years agonew version 2.2
root [Wed, 12 Nov 2008 07:57:14 +0000 (18:57 +1100)]
new version 2.2

10 years agoset the "big_write" option when starting fuse.
root [Wed, 12 Nov 2008 07:53:57 +0000 (18:53 +1100)]
set the "big_write" option when starting fuse.
This requires that remote cache is built with a fuse version later than 2.7.4
such as either the 2.8.0 prerelease or later or current cvs.

The resulting remote-cache application will however still be compatible
with a kernel module from an earlier version of fuse.

10 years agoupdate version
root [Tue, 11 Nov 2008 00:56:35 +0000 (11:56 +1100)]
update version

10 years agonew version 2.1.0
root [Tue, 11 Nov 2008 00:54:41 +0000 (11:54 +1100)]
new version 2.1.0

10 years agoupdates to read write support
root [Tue, 11 Nov 2008 00:53:08 +0000 (11:53 +1100)]
updates to read write support

10 years agoupdate the version insid ethe makerpms script
Ronnie Sahlberg [Wed, 23 Jul 2008 03:22:00 +0000 (13:22 +1000)]
update the version insid ethe makerpms script

10 years agoGNU_SOURCE may already be defined so define it inside an ifndef
Ronnie Sahlberg [Wed, 23 Jul 2008 03:18:18 +0000 (13:18 +1000)]
GNU_SOURCE may already be defined so define it inside an ifndef

10 years agoNew version 2.0.0
Ronnie Sahlberg [Wed, 23 Jul 2008 02:14:02 +0000 (12:14 +1000)]
New version 2.0.0
This adds read/write support

Signed-off-by: Ronnie Sahlberg <ronniesahlberg@gmail.com>
10 years agodocument --read-write
Ronnie Sahlberg [Wed, 23 Jul 2008 02:11:18 +0000 (12:11 +1000)]
document --read-write

10 years agoadd read/write functions for all commands we used to return EROFS
Ronnie Sahlberg [Tue, 22 Jul 2008 05:23:37 +0000 (15:23 +1000)]
add read/write functions for all commands we used to return EROFS
for unconditionally.

we can now write (slowly) on the cache site. These write operations
(WRITE, but also everything that modifies metadata like chmod, mkdir etc)
are passed over to the master site and the local cache is wiped.

Signed-off-by: Ronnie Sahlberg <ronniesahlberg@gmail.com>