root [Thu, 23 Jul 2009 22:47:45 +0000 (08:47 +1000)]
version 3.6
Martin Schwenke [Thu, 23 Jul 2009 06:35:02 +0000 (16:35 +1000)]
remote-cache-clean: add a final pass to remove empty directories.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Martin Schwenke [Thu, 23 Jul 2009 02:38:37 +0000 (12:38 +1000)]
Additional cache cleaning algorithms.
This adds 2 new options that enable additional algorithms:
* Delete all files with ctime less than some value.
* Delete all files with atime less than some value.
The offsets are specified in hours via the --trim-old and
--trim-unused options.
Also provided is an option --no-lru to disabled the LRU pass.
The code has been heavily refactored to allow the new features to be
neatly implemented. New function _trim() now does most of the grunt
work.
Documentation is updated to reflect the new features.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Martin Schwenke [Tue, 21 Jul 2009 07:46:13 +0000 (17:46 +1000)]
Rejig code and logging in preparation for extra cleanup methods.
* Make Cache.scan() no longer private.
* Prefix log messages with function name. This could be done by using
%(funcName)s in the logging.Formatter but that option was only added
in Python 2.5 and this will need to run on Python 2.4.
* Move summary log messages to main and add some extra logging to make
things clearer.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Martin Schwenke [Tue, 21 Jul 2009 07:18:08 +0000 (17:18 +1000)]
Update RPM spec file to cope with moved and new files.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Martin Schwenke [Tue, 21 Jul 2009 07:11:09 +0000 (17:11 +1000)]
remote-cache should be installed in sbin rather than bin.
remote-cache is a system administration command, so should be
installed in sbin rather than bin.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Martin Schwenke [Tue, 21 Jul 2009 06:31:00 +0000 (16:31 +1000)]
Cleanups to doc/remote-cache.8.xml and commit of generated files.
Cleanups to get rid of some "fixme"s and warnings from recent
docbook.xsl. Also some simple changes to make the output more
reliably pretty. Regenerated doc/remote-cache.8 and
doc/remote-cache.8.html.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Martin Schwenke [Tue, 21 Jul 2009 05:57:13 +0000 (15:57 +1000)]
Move remote-cache documentation to section 8 of the manual.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Martin Schwenke [Tue, 21 Jul 2009 05:52:21 +0000 (15:52 +1000)]
Add remote-cache-clean derived documentation files.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Martin Schwenke [Tue, 21 Jul 2009 05:38:20 +0000 (15:38 +1000)]
Fix circular dependency error with autoconf 2.6.3.
As per the following Samba commit:
commit
b39611c36bb904774fd4032bf2f8003fbdeb5d34
Author: Andreas Schneider <anschneider@suse.de>
Date: Wed Oct 29 14:12:04 2008 +0100
Signed-off-by: Andreas Schneider <anschneider@suse.de>
Signed-off-by: Martin Schwenke <martin@meltin.net>
Martin Schwenke [Tue, 21 Jul 2009 05:32:15 +0000 (15:32 +1000)]
New remote-cache-clean script.
This adds a cache cleanup script that can be used to trim the cache.
This script is written in Python. Also includes manpage and
associated Makefile.in updates.
Signed-off-by: Martin Schwenke <martin@meltin.net>
root [Wed, 15 Jul 2009 03:10:44 +0000 (13:10 +1000)]
new version 3.5
root [Wed, 15 Jul 2009 00:25:40 +0000 (10:25 +1000)]
remove the switch_to_real_user() function and switch_back_to_root()
root [Wed, 15 Jul 2009 00:24:57 +0000 (10:24 +1000)]
remove the last use of switch_to_real_user()
root [Wed, 15 Jul 2009 00:01:40 +0000 (10:01 +1000)]
remote an additional switch_to_real_user()
root [Tue, 14 Jul 2009 07:37:06 +0000 (17:37 +1000)]
remove a switch_to_real_user()
root [Tue, 14 Jul 2009 06:26:10 +0000 (16:26 +1000)]
remove a dead function
root [Tue, 14 Jul 2009 05:34:29 +0000 (15:34 +1000)]
remove a call to switch_to_real_user
root [Tue, 14 Jul 2009 04:59:44 +0000 (14:59 +1000)]
remove the use of one switch_to_real_user()
root [Tue, 7 Apr 2009 00:37:59 +0000 (10:37 +1000)]
new version 3.4
root [Tue, 7 Apr 2009 00:32:12 +0000 (10:32 +1000)]
make sure we replace any existing man page with a new manpage if/when we upgrade the rpm
root [Tue, 7 Apr 2009 00:24:11 +0000 (10:24 +1000)]
add a simple array of hashes so that we dont keep adding the same file to the migration queue over and over
root [Mon, 6 Apr 2009 23:41:41 +0000 (09:41 +1000)]
check that the umount actually worked and keep running if fusermount returned error by the filesystembeing busy
root [Mon, 6 Apr 2009 23:31:28 +0000 (09:31 +1000)]
use a mutex to make sur eonly one thread at a time tries to manipulate the tdb databases
Ronnie Sahlberg [Mon, 6 Apr 2009 22:27:50 +0000 (08:27 +1000)]
add pregenerated versions of the manpage and the html page for systems
that cant build with docbook
root [Mon, 30 Mar 2009 00:42:25 +0000 (11:42 +1100)]
new version 3.3
root [Mon, 30 Mar 2009 00:36:51 +0000 (11:36 +1100)]
fix an error in a prototype
Ronnie Sahlberg [Mon, 30 Mar 2009 00:31:38 +0000 (11:31 +1100)]
update documentation with the new
remote-cache --unexport=... command
root [Mon, 30 Mar 2009 00:24:52 +0000 (11:24 +1100)]
Create a new option --unexport=<path>" to be used to perform a
controlled shutdown and unexport of a remote-cache filesystem in the new two process model.
root [Sun, 29 Mar 2009 23:15:04 +0000 (10:15 +1100)]
use a lock on the cache file to ensure that only one remote-cache node at a time tries to migrate the same file
root [Sun, 29 Mar 2009 23:04:21 +0000 (10:04 +1100)]
Fork() and return from the parent back to the shell to ensure remote-cache is run in the background.
Add a handler for SIGUSR2. Kill -USR2 <pid of migration daemon>
is now the approved was to stop and unmount a remote cache filesystem
root [Fri, 20 Mar 2009 05:11:32 +0000 (16:11 +1100)]
new version 3.2
root [Fri, 20 Mar 2009 05:08:07 +0000 (16:08 +1100)]
remove more switching between user ids
root [Fri, 20 Mar 2009 03:39:25 +0000 (14:39 +1100)]
remove more swhitches between root and the user.
add subroutines to do common operations such as evaluating the permission
and creating the substring to point to the parent directory
root [Fri, 20 Mar 2009 02:08:40 +0000 (13:08 +1100)]
dont switch to/from the real user inmkdir() evaluate the permission bits explicitely in the code and do this as root instead
root [Thu, 19 Mar 2009 05:28:43 +0000 (16:28 +1100)]
remove all remaining talloc calls
root [Thu, 19 Mar 2009 03:50:37 +0000 (14:50 +1100)]
remove the use of talloc from unlink_parent_dir_cache()
root [Thu, 19 Mar 2009 03:45:46 +0000 (14:45 +1100)]
remove the use of talloc from open_parent_cache_ro()
root [Thu, 19 Mar 2009 03:39:24 +0000 (14:39 +1100)]
remove the use of talloc from open_dir_cache_ro()
root [Thu, 19 Mar 2009 03:32:45 +0000 (14:32 +1100)]
remove the use of talloc from unlink_object()
root [Thu, 19 Mar 2009 03:23:58 +0000 (14:23 +1100)]
update the description in the specfile to explain remote cache
Ronnie Sahlberg [Tue, 17 Mar 2009 03:35:06 +0000 (14:35 +1100)]
fix typo in the usage printout
Ronnie Sahlberg [Mon, 9 Mar 2009 22:08:17 +0000 (09:08 +1100)]
new version 3.1
Ronnie Sahlberg [Mon, 9 Mar 2009 22:06:55 +0000 (09:06 +1100)]
remove use of talloc in performance critical paths
Ronnie Sahlberg [Mon, 9 Mar 2009 03:00:35 +0000 (14:00 +1100)]
add the generated manpage and html page for remote-cache so they are
available also for those that dont want to build the documentation
themselves
root [Mon, 9 Mar 2009 02:38:27 +0000 (13:38 +1100)]
new version 3.0
root [Fri, 6 Mar 2009 03:44:44 +0000 (14:44 +1100)]
make sure we use null terminated path strings
root [Fri, 6 Mar 2009 03:19:21 +0000 (14:19 +1100)]
make sure we null terminate the string we return in _readlink()
root [Fri, 6 Mar 2009 01:58:36 +0000 (12:58 +1100)]
remove some DEBUG statements we no longer need.
root [Fri, 6 Mar 2009 01:53:48 +0000 (12:53 +1100)]
split remote-cache into a two process model where we spawn a separate process that is dedicated to only manage migrations from the remote site onto the local cache.
Communications between the FUSE module and the migration daemon is through a tdb database holding the list of files to migrate and signals to tell the migration daemon : there is work to do now.
root [Thu, 5 Mar 2009 01:29:08 +0000 (12:29 +1100)]
another file required by the events system
root [Thu, 5 Mar 2009 01:23:06 +0000 (12:23 +1100)]
add another file that the events system needs
root [Thu, 5 Mar 2009 01:09:15 +0000 (12:09 +1100)]
add a file for linked list macros that the event system depends on
root [Thu, 5 Mar 2009 00:58:46 +0000 (11:58 +1100)]
add the event system from ctdb
root [Thu, 5 Mar 2009 00:22:33 +0000 (11:22 +1100)]
add gpl boilerplate to migrate.h
remote-cache is read-write not rea-only any more
Ronnie Sahlberg [Fri, 27 Feb 2009 01:44:02 +0000 (12:44 +1100)]
make sure to check that fhs is no null before dereferencing it.
it should never happen but check nevertheless
Ronnie Sahlberg [Fri, 27 Feb 2009 01:09:25 +0000 (12:09 +1100)]
verion 2.9
Ronnie Sahlberg [Fri, 27 Feb 2009 01:07:34 +0000 (12:07 +1100)]
use an explicit chmod to make sure the bits are set correctly
Ronnie Sahlberg [Fri, 27 Feb 2009 01:02:37 +0000 (12:02 +1100)]
use file locking on the temporary cache file to prevent two immeidate
migrations simultaneously for the same file
Ronnie Sahlberg [Fri, 27 Feb 2009 00:41:26 +0000 (11:41 +1100)]
fix a broken debug statement
Ronnie Sahlberg [Fri, 27 Feb 2009 00:37:18 +0000 (11:37 +1100)]
small files smaller than 128k are migrated explicitely during first read
in the current thread since it this is cheap (cheaper than spawning a
child process and running rsync at least)
Ronnie Sahlberg [Thu, 26 Feb 2009 00:50:30 +0000 (11:50 +1100)]
dont switch uid in _unlink()
instead check permissions explicitely and do the operation as root
Ronnie Sahlberg [Thu, 26 Feb 2009 00:44:58 +0000 (11:44 +1100)]
dont switch userid to/from the real user and root in _create() since
this is such an expensive operation.
Instead check the permission bits explicitely
Ronnie Sahlberg [Wed, 25 Feb 2009 03:30:55 +0000 (14:30 +1100)]
dont flush the cached-file or the direcache after a write has completed.
while a write does mean that any cached copy would be "stale" deleting
it at this stage is a very expensive operation.
It is much cheaper to leave the stale file in the cache and rely on next
read flushing it from the cache.
Ronnie Sahlberg [Wed, 25 Feb 2009 02:13:33 +0000 (13:13 +1100)]
Dont switch uid/gid to the user issuing the operation, instead do it as
root but check the mode bits explicitely.
the switch of uid/gid are surprisingly expensive
Ronnie Sahlberg [Wed, 25 Feb 2009 00:53:27 +0000 (11:53 +1100)]
Avoid switching to/from "real user"/root during _read()
and instead evaluate the permissions explicitely before trying
open()/pread().
These switches were very expensive.
Ronnie Sahlberg [Mon, 23 Feb 2009 23:10:01 +0000 (10:10 +1100)]
optimize away some expensive lstat()
root [Fri, 6 Feb 2009 05:22:14 +0000 (16:22 +1100)]
new version
root [Fri, 6 Feb 2009 05:19:56 +0000 (16:19 +1100)]
add a "big_writes" argument
root [Thu, 5 Feb 2009 08:13:48 +0000 (19:13 +1100)]
make it possible to tweak the max number of concurrent migrations
root [Thu, 5 Feb 2009 07:58:29 +0000 (18:58 +1100)]
see if we can get a lock on the migration limiter file first becore we create and lock the file in the local cache fs
root [Thu, 5 Feb 2009 07:11:06 +0000 (18:11 +1100)]
reduce the loglevel for an info message from ERR to INFO
root [Thu, 5 Feb 2009 07:03:46 +0000 (18:03 +1100)]
increase the limit on number of concurrent migrations to 1000
root [Thu, 5 Feb 2009 07:01:50 +0000 (18:01 +1100)]
when trying to migrate a file, check and make sure we have created all parent directories, and create them if not
otherwise we will not be able to migrate and cache files when the parent directory has never been scanned (and thus created) by READDIR
root [Mon, 19 Jan 2009 22:32:57 +0000 (09:32 +1100)]
build the docs
root [Mon, 19 Jan 2009 22:28:00 +0000 (09:28 +1100)]
remove big-writes doenst work with NFS anyway
root [Mon, 19 Jan 2009 22:26:27 +0000 (09:26 +1100)]
remove the lock operation
root [Tue, 16 Dec 2008 22:37:26 +0000 (09:37 +1100)]
remove a debug statement
root [Tue, 16 Dec 2008 08:38:39 +0000 (19:38 +1100)]
The kernel nfs daemon can not handle filesystems offering DIRECT_IO
and thus by design will feed a write i/o to the application data
in a singlehtreaded sequence of pages to the filesystem, one by one.
But hang on, it gets better, these 4kb blobs from the idiocy of having
a demon in kernelspace has page alignment problems!
So each page is actually fed twice to the filesystem, once as 4kb-small-delta anf ythen as a second i/o of size small-delta.
Brilliant! Great work.
Thus any filesystem that uses DIRECT_IO for performance reasons will break, immediately.
Second when disabling DIRECT_IO the nice surprise comes that when a client performs 1 32kb write to kNFSd, kNFSd and the rest of the kernel translates this to a singlethreaded sequence of 8 almost 4kb writes and another 8 small writes as far as the filesystem can see.
This is either a mechanism to prevent and punish out of tree filesystems or some other braindamage by the one deciding that a "hey lets put all userspace daemons inside the kernel because it is cool".
why implement a userspace service inside the kernel? insanity.
Ronnie Sahlberg [Tue, 16 Dec 2008 01:57:44 +0000 (12:57 +1100)]
dont build the docs by default
Ronnie Sahlberg [Tue, 16 Dec 2008 01:57:03 +0000 (12:57 +1100)]
add an option to specify direct_io
root [Tue, 16 Dec 2008 01:43:40 +0000 (12:43 +1100)]
reactivate big writes
root [Thu, 11 Dec 2008 07:26:54 +0000 (18:26 +1100)]
fix bug when handling gids.
we have to look up all aux gids for a user every time and set them
since this must go in the oncrpc layer to the remote nfs server
create a memory tdb database to store the uid->gidlist mappings since
this operation might be quite expensive
root [Mon, 24 Nov 2008 23:00:32 +0000 (10:00 +1100)]
new version 2.5
root [Mon, 24 Nov 2008 22:53:17 +0000 (09:53 +1100)]
plug some memory leaks
root [Sun, 23 Nov 2008 03:15:53 +0000 (14:15 +1100)]
new version 2.4
root [Sun, 23 Nov 2008 03:14:04 +0000 (14:14 +1100)]
rekmove debug code
root [Sun, 23 Nov 2008 02:55:49 +0000 (13:55 +1100)]
use stat and check if the file exist before we try to delete it.
root [Sat, 15 Nov 2008 03:08:02 +0000 (14:08 +1100)]
new version 2.3
root [Sat, 15 Nov 2008 03:06:29 +0000 (14:06 +1100)]
dont leak filedescriptors
root [Wed, 12 Nov 2008 07:57:14 +0000 (18:57 +1100)]
new version 2.2
root [Wed, 12 Nov 2008 07:53:57 +0000 (18:53 +1100)]
set the "big_write" option when starting fuse.
This requires that remote cache is built with a fuse version later than 2.7.4
such as either the 2.8.0 prerelease or later or current cvs.
The resulting remote-cache application will however still be compatible
with a kernel module from an earlier version of fuse.
root [Tue, 11 Nov 2008 00:56:35 +0000 (11:56 +1100)]
update version
root [Tue, 11 Nov 2008 00:54:41 +0000 (11:54 +1100)]
new version 2.1.0
root [Tue, 11 Nov 2008 00:53:08 +0000 (11:53 +1100)]
updates to read write support
Ronnie Sahlberg [Wed, 23 Jul 2008 03:22:00 +0000 (13:22 +1000)]
update the version insid ethe makerpms script
Ronnie Sahlberg [Wed, 23 Jul 2008 03:18:18 +0000 (13:18 +1000)]
GNU_SOURCE may already be defined so define it inside an ifndef
Ronnie Sahlberg [Wed, 23 Jul 2008 02:14:02 +0000 (12:14 +1000)]
New version 2.0.0
This adds read/write support
Signed-off-by: Ronnie Sahlberg <ronniesahlberg@gmail.com>
Ronnie Sahlberg [Wed, 23 Jul 2008 02:11:18 +0000 (12:11 +1000)]
document --read-write
Ronnie Sahlberg [Tue, 22 Jul 2008 05:23:37 +0000 (15:23 +1000)]
add read/write functions for all commands we used to return EROFS
for unconditionally.
we can now write (slowly) on the cache site. These write operations
(WRITE, but also everything that modifies metadata like chmod, mkdir etc)
are passed over to the master site and the local cache is wiped.
Signed-off-by: Ronnie Sahlberg <ronniesahlberg@gmail.com>