NEWS for rsync 2.6.4 (30 March 2005) Protocol: 29 (changed) Changes since 2.6.3: OUTPUT CHANGES: - When rsync deletes a directory and outputs a verbose message about it, it now appends a trailing slash to the name instead of (only sometimes) outputting a preceding "directory " string. - The --stats output will contain file-list time-statistics if both sides are 2.6.4, or if the local side is 2.6.4 and the files are being pushed (since the stats come from the sending side). (Requires protocol 29 for a pull.) - The "%o" (operation) log-format escape now has a third value (besides "send" and "recv"): "del." (with trailing dot to make it 4 chars). This changes the way deletions are logged in the daemon's log file. - When the --log-format option is combined with --verbose, rsync now avoids outputting the name of the file twice in most circumstances. As long as the --log-format item does not refer to any post-transfer items (such as %b or %c), the --log-format message is output prior to the transfer, so --verbose is now the equivalent of a --log-format of '%n%L' (which outputs the name and any link info). If the log output must occur after the transfer to be complete, the only time the name is also output prior to the transfer is when --progress was specified (so that the name will precede the progress stats, and the full --log-format output will come after). - Filenames have non-printable characters replaced with a '?' to avoid corrupting the screen or generating empty lines in the output. BUG FIXES: - Restore the list-clearing behavior of "!" in a .cvsignore file (2.6.3 was only treating it as a special token in an rsync include/exclude file). - The combination of --verbose and --dry-run now mentions the full list of changes that would be output without --dry-run. - Avoid a mkdir warning when removing a directory in the destination that already exists in the --backup-dir. - An OS that has a binary mode for its files (such as cygwin) needed setmode(fd, O_BINARY) called on the temp-file we opened with mkstemp(). (Fix derived from the cygwin's 2.6.3 rsync package.) - Fixed a potential hang when verbosity is high, the client side is the sender, and the file-list is large. - Fixed a potential protocol-corrupting bug where the generator could merge a message from the receiver into the middle of a multiplexed packet of data if only part of that data had been written out to the socket when the message from the generator arrived. - We now check if the OS doesn't support using mknod() for creating FIFOs and sockets, and compile-in some compatibility code using mkfifo() and socket() when necessary. - Fixed an off-by-one error in the handling of --max-delete=N. Also, if the --max-delete limit is exceeded during a run, we now output a warning about this at the end of the run and exit with a new error code (25). - One place in the code wasn't checking if fork() failed. - The "ignore nonreadable" daemon parameter used to erroneously affect readable symlinks that pointed to a non-existent file. - If the OS does not have lchown() and a chown() of a symlink will affect the referent of a symlink (as it should), we no longer try to set the user and group of a symlink. - The generator now properly runs the hard-link loop and the dir-time rewriting loop after we're sure that the redo phase is complete. - When --backup was specified with --partial-dir=DIR, where DIR is a relative path, the backup code was erroneously trying to backup a file that was put into the partial-dir. - If a file gets resent in a single transfer and the --backup option is enabled along with --inplace, rsync no longer performs a duplicate backup (it used to overwrite the first backup with the failed file). - One call to flush_write_file() was not being checked for an error. - The --no-relative option was not being sent from the client to a server sender. - If an rsync daemon specified "dont compress = ..." for a file and the client tried to specify --compress, the libz code was not handling a compression level of 0 properly. This could cause a transfer failure if the block-size for a file was large enough (e.g. rsync might have exited with an error for large files). - Fixed a bug that would sometimes surface when using --compress and sending a file with a block-size larger than 64K (either manually specified, or computed due to the file being really large). Prior versions of rsync would sometimes fail to decompress the data properly, and thus the transferred file would fail its verification. - If a daemon can't open the specified log file (i.e. syslog is not being used), die without crashing. We also output an error about the failure on stderr (which will only be seen if --no-detach was specified) and exit with a new error code (6). - A local transfer no longer duplicates all its include/exclude options (since the forked process already has a copy of the exclude list, there's no need to send them a set of duplicates). - When --progress is specified, the output of items that the generator is creating (e.g. dirs, symlinks) is now integrated into the progress output without overlapping it. (Requires protocol 29.) - When --timeout is specified, lulls that occur in the transfer while the generator is doing work that does not generate socket traffic (looking for changed files, deleting files, doing directory-time touch-ups, etc.) will cause a new keep-alive packet to be sent that should keep the transfer going as long as the generator continues to make progress. (Requires protocol 29.) - The stat size of a device is not added to the total file size of the items in the transfer (the size might be undefined on some OSes). - Fixed a problem with refused-option messages sometimes not making it back to the client side when a remote --files-from was in effect and the daemon was the receiver. - The --compare-dest option was not updating a file that differred in (the preserved) attributes from the version in the compare-dest DIR. - When rsync is copying files into a write-protected directory, fixed the change-report output for the directory so that we don't report an identical directory as changed. ENHANCEMENTS: - Rsync now supports popt's option aliases, which means that you can use /etc/popt and/or ~/.popt to create your own option aliases. - Added the --delete-during (--del) option which will delete files from the receiving side incrementally as each directory in the transfer is being processed. This makes it more efficient than the default, before-the-transfer behavior, which is now also available as --delete-before (and is still the default --delete-WHEN option that will be chosen if --delete or --delete-excluded is specified without a --delete-WHEN choice). All the --del* options infer --delete, so an rsync daemon that refuses "delete" will still refuse to allow any file-deleting options (including the new --remove-sent-files option). - All the --delete-WHEN options are now more memory efficient: Previously an duplicate set of file-list objects was created on the receiving side for the entire destination hierarchy. The new algorithm only creates one directory of objects at a time (for files inside the transfer). - Added the --copy-dest option, which works like --link-dest except that it locally copies identical files instead of hard-linking them. - Added support for specifying multiple --compare-dest, --copy-dest, or --link-dest options, but only of a single type. (Promoted from the patches dir and enhanced.) (Requires protocol 29.) - Added the --max-size option. (Promoted from the patches dir.) - The daemon-mode options are now separated from the normal rsync options so that they can't be mixed together. This makes it impossible to start a daemon that has improper default option values (which could cause problems when a client connects, such as hanging or crashing). - The --bwlimit option may now be used in combination with --daemon to specify both a default value for the daemon side and a value that cannot be exceeded by a user-specified --bwlimit option. - Added the "port" parameter to the rsyncd.conf file. (Promoted from the patches dir.) Also added "address". The command-line options take precedence over a config-file option, as expected. - In _exit_cleanup(): when we are exiting with a partially-received file, we now flush any data in the write-cache before closing the partial file. - The --inplace support was enhanced to work with --compare-dest, --link-dest, and (the new) --copy-dest options. (Requires protocol 29.) - Added the --dirs (-d) option for an easier way to copy directories without recursion. - Added the --list-only option, which is mainly a way for the client to put the server into listing mode without needing to resort to any internal option kluges (e.g. the age-old use of "-r --exclude="/*/*" for a non-recursive listing). This option is used automatically (behind the scenes) when a modern rsync speaks to a modern daemon, but may also be specified manually if you want to force the use of the --list-only option over a remote-shell connection. - Added the --omit-dir-times (-O) option, which will avoid updating the modified time for directories when --times was specified. This option will avoid an extra pass through the file-list at the end of the transfer (to tweak all the directory times), which may provide an appreciable speedup for a really large transfer. (Promoted from the patches dir.) - Added the --filter (-f) option and its helper option, -F. Filter rules are an extension to the existing include/exclude handling that also supports nested filter files as well as per-directory filter files (like .cvsignore, but with full filter-rule parsing). This new option was chosen in order to ensure that all existing include/exclude processing remained 100% compatible with older versions. Protocol 29 is needed for full filter-rule support, but backward-compatible rules work with earlier protocol versions. (Promoted from the patches dir and enhanced.) - Added the --delay-updates option that puts all updated files into a temporary directory (by default ".~tmp~", but settable via the --partial-dir=DIR option) until the end of the transfer. This makes the updates a little more atomic for a large transfer. - If rsync is put into the background, any output from --progress is reduced. - Documented the "max verbosity" setting for rsyncd.conf. (This setting was added a couple releases ago, but left undocumented.) - The sender and the generator now double-check the file-list index they are given, and refuse to try to do a file transfer on a non-file index (since that would indicate that something had gone very wrong). - Added the --itemize-changes (-i) option, which is a way to output a more detailed list of what files changed and in what way. The effect is the same as specifying a --log-format of "%i %n%L" (see both the rsync and rsyncd.conf manpages). Works with --dry-run too. - Added the --fuzzy (-y) option, which attempts to find a basis file for a file that is being created from scratch. The current algorithm only looks in the destination directory for the created file, but it does attempt to find a match based on size/mod-time (in case the file was renamed with no other changes) as well as based on a fuzzy name-matching algorithm. This option requires protocol 29 because it needs the new file-sorting order. (Promoted from patches dir and enhanced.) (Requires protocol 29.) - Added the --remove-sent-files option, which lets you move files between systems. - The hostname in HOST:PATH or HOST::PATH may now be an IPv6 literal enclosed in '[' and ']' (e.g. "[::1]"). (We already allowed IPv6 literals in the rsync://HOST:PORT/PATH format.) - When rsync recurses to build the file list, it no longer keeps open one or more directory handles from the dir's parent dirs. - When building under windows, the default for --daemon is now to avoid detaching, requiring the new --detach option to force rsync to detach. - The --dry-run option can now be combined with either --write-batch or --read-batch, allowing you to run a do-nothing test command to see what would happen without --dry-run. - The daemon's "read only" config item now sets an internal read_only variable that makes extra sure that no write/delete calls on the read-only side can succeed. - The log-format % escapes can now have a numeric field width in between the % and the escape letter (e.g. "%-40n %08p"). - Improved the option descriptions in the --help text. SUPPORT FILES: - Added atomic-rsync to the support dir: a perl script that will transfer some files using rsync, and then move the updated files into place all at once at the end of the transfer. Only works when pulling, and uses --link-dest and a parallel hierarchy of files to effect its update. - Added mnt-excl to the support dir: a perl script that takes the /proc/mounts file and translates it into a set of excludes that will exclude all mount points (even mapped mounts to the same disk). The excludes are made relative to the specified source dir and properly anchored. - Added savetransfer.c to the support dir: a C program that can make a copy of all the data that flows over the wire. This lets you test for data corruption (by saving the data on both the sending side and the receiving side) and provides one way to debug a protocol error. - Added rrsync to the support dir: this is an updated version of Joe Smith's restricted rsync perl script. This helps to ensure that only certain rsync commands can be run by an ssh invocation. INTERNAL: - Added better checking of the checksum-header values that come over the socket. - Merged a variety of file-deleting functions into a single function so that it is easier to maintain. - Improved the type of some variables (particularly blocksize vars) for consistency and proper size. - Got rid of the uint64 type (which we didn't need). - Use a slightly more compatible set of core #include directives. - Defined int32 in a way that ensures that the build dies if we can't find a variable with at least 32 bits. PROTOCOL DIFFERENCES FOR VERSION 29: - A 16-bit flag-word is transmitted after every file-list index. This indicates what is changing between the sender and the receiver. The generator now transmits an index and a flag-word to indicate when dirs and symlinks have changed (instead of producing a message), which makes the outputting of the information more consistent and less prone to screen corruption (because the local receiver/sender is now outputting all the file-change info messages). - If a file is being hard-linked, the ITEM_XNAME_FOLLOWS bit is enabled in the flag-word and the name of the file that was linked immediately follows in vstring format (see below). - If a file is being transferred with an alternate-basis file, the ITEM_BASIS_TYPE_FOLLOWS bit is enabled in the flag-word and a single byte follows, indicating what type of basis file was chosen. If that indicates that a fuzzy-match was selected, the ITEM_XNAME_FOLLOWS bit is set in the flag-word and the name of the match in vstring format follows the basis byte. A vstring is a variable length string that has its size written prior to the string, and no terminating null. If the string is from 1-127 bytes, the length is a single byte. If it is from 128-32767 bytes, the length is written as ((len >> 8) | 0x80) followed by (len % 0x100). - The sending of exclude names is done using filter-rule syntax. This means that all names have a prefixed rule indicator, even excludes (which used to be sent as a bare pattern, when possible). The -C option will include the per-dir .cvsignore merge file in the list of filter rules so it is positioned correctly (unlike in some older transfer scenarios). - Rsync sorts the filename list in a different way: it sorts the subdir names after the non-subdir names for each dir's contents, and it always puts a dir's contents immediately after the dir's name in the list. (Previously an item named "foo.txt" would sort in between directory "foo/" and "foo/bar".) - When talking to a protocol 29 rsync daemon, a list-only request is able to note this before the options are sent over the wire and the new --list-only option is included in the options. - When the --stats bytes are sent over the wire (or stored in a batch), they now include two elapsed-time values: one for how long it took to build the file-list, and one for how long it took to send it over the wire (each expressed in thousandths of a second). - When --delete-excluded is specified with some filter rules (AKA excludes), a client sender will now initiate a send of the rules to the receiver (older protocols used to omit the sending of excludes in this situation since there were no receiver-specific rules that survived --delete-excluded back then). Note that, as with all the filter-list sending, only items that are significant to the other side will actually be sent over the wire, so the filter-rule list that is sent in this scenario is often empty. - An index equal to the file-list count is sent as a keep-alive packet from the generator to the sender, which then forwards it on to the receiver. This normally invalid index is only a valid keep-alive packet if the 16-bit flag-word that follows it contains a single bit (ITEM_IS_NEW, which is normally an illegal flag to appear alone). - A protocol-29 batch file includes a bit for the setting of the --dirs option and for the setting of the --compress option. Also, the shell script created by --write-batch will use the --filter option instead of --exclude-from to capture any filter rules. BUILD CHANGES: - Handle an operating system that use mkdev() in place of makedev(). - Improved configure to better handle cross-compiling. NEWS for rsync 2.6.3 (30 Sep 2004) Protocol: 28 (unchanged) Changes since 2.6.2: SECURITY FIXES: - A bug in the sanitize_path routine (which affects a non-chrooted rsync daemon) could allow a user to craft a pathname that would get transformed into an absolute path for certain options (but not for file-transfer names). If you're running an rsync daemon with chroot disabled, *please upgrade*, ESPECIALLY if the user privs you run rsync under is anything above "nobody". OUTPUT CHANGES (ATTN: those using a script to parse the verbose output): - Please note that the 2-line footer (output when verbose) now uses the term "sent" instead of "wrote" and "received" instead of "read". If you are not parsing the numeric values out of this footer, a script would be better off using the empty line prior to the footer as the indicator that the verbose output is over. - The output from the --stats option was similarly affected to change "written" to "sent" and "read" to "received". - Rsync ensures that a filename that contains a newline gets mentioned with each newline transformed into a question mark (which prevents a filename from causing an empty line to be output). - The "backed up ..." message that is output when at least 2 --verbose options are specified is now the same both with and without the --backup-dir option. BUG FIXES: - Fixed a crash bug that might appear when --delete was used and multiple source directories were specified. - Fixed a 32-bit truncation of the file length when generating the checksums. - The --backup code no longer attempts to create some directories over and over again (generating warnings along the way). - Fixed a bug in the reading of the secrets file (by the daemon) and the password file (by the client): the files no longer need to be terminated by a newline for their content to be read in. - If a file has a read error on the sending side or the reconstructed data doesn't match the expected checksum (perhaps due to the basis file changing during the transfer), the receiver will no longer retain the resulting file unless the --partial option was specified. (Note: for the read-error detection to work, neither side can be older than 2.6.3 -- older receivers will always retain the file, and older senders don't tell the receiver that the file had a read error.) - If a file gets resent in a single transfer and the --backup option is enabled, rsync no longer performs a duplicate backup (it used to overwrite the original file in the backup area). - Files specified in the daemon's "exclude" or "exclude from" config items are now excluded from being uploaded (assuming that the module allows uploading at all) in addition to the old download exclusion. - Got rid of a potential hang in the receiver when near the end of a phase. - When using --backup without a --backup-dir, rsync no longer preserves the modify time on directories. This avoids confusing NFS. - When --copy-links (-L) is specified, we now output a separate error for a symlink that has no referent instead of claiming that a file "vanished". - The --copy-links (-L) option no longer has the side-effect of telling the receiving side to follow symlinks. See the --keep-dirlinks option (mentioned below) for a way to specify that behavior. - Error messages from the daemon server's option-parsing (such as refused options) are now successfully transferred back to the client (the server used to fail to send the message because the socket wasn't in the right state for the message to get through). - Most transfer errors that occur during a daemon transfer are now returned to the user in addition to being logged (some messages are intended to be daemon-only and are not affected by this). - Fixed a bug in the daemon authentication code when using one of the batch-processing options. - We try to work around some buggy IPv6 implementations that fail to implement IPV6_V6ONLY. This should fix the "address in use" error that some daemons get when running on an OS with a buggy IPv6 implementation. Also, if the new code gets this error, we might suggest that the user specify --ipv4 or --ipv6 (if we think it will help). - When the remote rsync dies, make a better effort to recover any error messages it may have sent before dying (the local rsync used to just die with a socket-write error). - When using --delete and a --backup-dir that contains files that are hard-linked to their destination equivalents, rsync now makes sure that removed files really get removed (avoids a really weird rename() behavior). - Avoid a bogus run-time complaint about a lack of 64-bit integers when the int64 type is defined as an off_t and it actually has 64-bits. - Added a configure check for open64() without mkstemp64() so that we can avoid using mkstemp() when such a combination is encountered. This bypasses a problem writing out large temp files on OSes such as AIX and HP-UX. - Fixed an age-old crash problem with --read-batch on a local copy (rsync was improperly assuming --whole-file for the local copy). - When --dry-run (-n) is used and the destination directory does not exist, rsync now produces a correct report of files that would be sent instead of dying with a chdir() error. - Fixed a bug that could cause a slow-to-connect rsync daemon to die with an error instead of waiting for the connection to finish. - Fixed an ssh interaction that could cause output to be lost when the user chose to combine the output of rsync's stdout and stderr (e.g. using the "2>&1"). ENHANCEMENTS: - Added the --partial-dir=DIR option that lets you specify where to (temporarily) put a partially transferred file (instead of over- writing the destination file). E.g. --partial-dir=.rsync-partial Also added support for the RSYNC_PARTIAL_DIR environment variable that, when found, transforms a regular --partial option (such as the convenient -P option) into one that also specifies a directory. - Added --keep-dirlinks (-K), which allows you to symlink a directory onto another partition on the receiving side and have rsync treat it as matching a normal directory from the sender. - Added the --inplace option that tells rsync to write each destination file without using a temporary file. The matching of existing data in the destination file can be severely limited by this, but there are also cases where this is more efficient (such as appending data). Use only when needed (see the man page for more details). - Added the "write only" option for the daemon's config file. - Added long-option names for -4 and -6 (namely --ipv4 and --ipv6) and documented all these options in the man page. - Improved the handling of the --bwlimit option so that it's less bursty, more accurate, and works properly over a larger range of values. - The rsync daemon-over-ssh code now looks for SSH_CONNECTION and SSH2_CLIENT in addition to SSH_CLIENT to figure out the IP address. - Added the --checksum-seed=N option for advanced users. - Batch writing/reading has a brand-new implementation that is simpler, fixes a few weird problems with the old code (such as no longer sprinkling the batch files into different dirs or even onto different systems), and is much less intrusive into the code (making it easier to maintain for the future). The new code generates just one data file instead of three, which makes it possible to read the batch on stdin via a remote shell. Also, the old requirement of forcing the same fixed checksum-seed for all batch processing has been removed. - If an rsync daemon has a module set with "list = no" (which hides its presence in the list of available modules), a user that fails to authenticate gets the same "unknown module" error that they would get if the module were actually unknown (while still logging the real error to the daemon's log file). This prevents fishing for module names. - The daemon's "refuse options" config item now allows you to match option names using wildcards and/or the single-letter option names. - Each transferred file now gets its permissions and modified-time updated before the temp-file gets moved into place. Previously, the finished file would have a very brief window where its permissions disallowed all group and world access. - Added the ability to parse a literal IPv6 address in an "rsync:" URL (e.g. rsync://[2001:638:500:101::21]:873/module/dir). - The daemon's wildcard expanding code can now handle more than 1000 filenames (it's now limited by memory instead of having a hard-wired limit). INTERNAL: - Some cleanup in the exclude code has saved some per-exclude memory and made the code easier to maintain. - Improved the argv-overflow checking for a remote command that has a lot of args. - Use rsyserr() in the various places that were still calling rprintf() with strerror() as an arg. - If an rsync daemon is listening on multiple sockets (to handle both IPv4 and IPv6 to a single port), we now close all the unneeded file handles after we accept a connection (we used to close just one of them). - Optimized the handling of larger block sizes (rsync used to slow to a crawl if the block size got too large). - Optimized away a loop in hash_search(). - Some improvements to the sanitize_path() and clean_fname() functions makes them more efficient and produce better results (while still being compatible with the file-name cleaning that gets done on both sides when sending the file-list). - Got rid of alloc_sanitize_path() after adding a destination-buffer arg to sanitize_path() made it possible to put all the former's functionality into the latter. - The file-list that is output when at least 4 verbose options are specified reports the uid value on the sender even when rsync is not running as root (since we might be sending to a root receiver). BUILD CHANGES: - Added a "gen" target to rebuild most of the generated files, including configure, config.h.in, the man pages, and proto.h. - If "make proto" doesn't find some changes in the prototypes, the proto.h file is left untouched (its time-stamp used to always be updated). - The variable $STRIP (that is optionally set by the install-strip target's rule) was changed to $INSTALL_STRIP because some systems have $STRIP already set in the environment. - Fixed a build problem when SUPPORT_HARD_LINKS isn't defined. - When cross-compiling, the gettimeofday() function is now assumed to be a modern version that takes two-args (since we can't test it). DEVELOPER RELATED: - The scripts in the testsuite dir were cleaned up a bit and a few new tests added. - Some new diffs were added to the patches dir, and some accepted ones were removed. NEWS for rsync 2.6.2 (30 Apr 2004) Protocol: 28 (unchanged) Changes since 2.6.1: BUG FIXES: - Fixed a major bug in the sorting of the filenames when --relative is used for some sources (just sources such as "/" and "/*" were affected). This fix ensures that we ask for the right file-list item when requesting changes from the sender. - Rsync now checks the return value of the close() function to better report disk-full problems on an NFS file system. - Restored the old daemon-server behavior of logging error messages rather than returning them to the user. (A better long-term fix will be sought in the future.) - An obscure uninitialized-variable bug was fixed in the uid/gid code. (This bug probably had no ill effects.) BUILD CHANGES: - Got rid of the configure check for sys/sysctl.h (it wasn't used and was causing a problem on some systems). Also improved the broken-largefile-locking test to try to avoid failure due to an NFS build-dir. - Fixed a compile problem on systems that don't define AI_NUMERICHOST. - Fixed a compile problem in the popt source for compilers that don't support __attribute__. DEVELOPER RELATED: - Improved the testsuite's "merge" test to work on OSF1. - Two new diffs were added to the patches dir. NEWS for rsync 2.6.1 (26 Apr 2004) Protocol: 28 (changed) Changes since 2.6.0: SECURITY FIXES: - Paths sent to an rsync daemon are more thoroughly sanitized when chroot is not used. If you're running a non-read-only rsync daemon with chroot disabled, *please upgrade*, ESPECIALLY if the user privs you run rsync under is anything above "nobody". ENHANCEMENTS: - Lower memory use, more optimal transfer of data over the socket, and lower CPU usage (see the INTERNAL section for details). - The RSYNC_PROXY environment variable can now contain a "USER:PASS@" prefix before the "HOST:PORT" information. (Bardur Arantsson) - The --progress output now mentions how far along in the transfer we are, including both a count of files transferred and a percentage of the total file-count that we've processed. It also shows better current-rate-of-transfer and remaining-transfer-time values. - Documentation changes now attempt to describe some often mis- understood features more clearly. BUG FIXES: - When -x (--one-file-system) is combined with -L (--copy-links) or --copy-unsafe-links, no symlinked files are skipped, even if the referent file is on a different filesystem. - The --link-dest code now works properly for a non-root user when (1) the UIDs of the source and destination differ and -o was specified, or (2) when the group of the source can't be used on the destination and -g was specified. - Fixed a bug in the handling of -H (hard-links) that might cause the expanded PATH/NAME value of the current item to get overwritten (due to an expanded-name caching bug). - We now reset the "new data has been sent" flag at the start of each file we send. This makes sure that an interrupted transfer with the --partial option set doesn't keep a shorter temp file than the current basis file when no new data has been transfered over the wire for that file. - Fixed a byte-order problem in --batch-mode on big-endian machines. (Jay Fenlason) - When using --cvs-exclude, the exclude items we get from a per-directory's .cvsignore file once again only affect that one directory (not all following directories too). The items are also now properly word-split and parsed without any +/- prefix parsing. - When specifying the USER@HOST: prefix for a file, the USER part can now contain an '@', if needed (i.e. the last '@' is used to find the HOST, not the first). - Fixed some bugs in the handling of group IDs for non-root users: (1) It properly handles a group that the sender didn't have a name for (it would previously skip changing the group on any files in that group). (2) If --numeric-ids is used, rsync no longer attempts to set groups that the user doesn't have the permission to set. - Fixed the "refuse options" setting in the rsyncd.conf file. - Improved the -x (--one-file-system) flag's handling of any mount- point directories we encounter. It is both more optimal (in that it no longer does a useless scan of the contents of the mount- point dirs) and also fixes a bug where a remapped mount of the original filesystem could get discovered in a subdir we should be ignoring. - Rsync no longer discards a double-slash at the start of a filename when trying to open the file. It also no longer constructs names that start with a double slash (unless the user supplied them). - Path-specifying options to a daemon should now work the same with or without chroot turned on. Previously, such a option (such as --link-dest) would get its absolute path munged into a relative one if chroot was not on, making that setting fairly useless. Rsync now transforms the path into one that is based on the module's base dir when chroot is not enabled. - Fixed a compatibility problem interacting with older rsync versions that might send us an empty --suffix value without telling us that --backup-dir was specified. - The "hosts allow" option for a daemon-over-remote-shell process now has improved support for IPv6 addresses and a fix for systems that have a length field in their socket structs. - Fixed the ability to request an empty backup --suffix when sending files to an rsync daemon. INTERNAL: - Most of the I/O is now buffered, which results in a pretty large speedup when running under MS Windows. (Craig Barratt) - Optimizations to the name-handling/comparing code have made some significant reductions in user-CPU time for large file sets. - Some cleanup of the variable types make the code more consistent. - Reduced memory requirements of hard link preservation. (J.W. Schultz) - Implemented a new algorithm for hard-link handling that speeds up the code significantly. (J.W. Schultz and Wayne Davison) - The --hard-link option now uses the first existing file in the group of linked files as the basis for the transfer. This prevents the sub-optimal transfer of a file's data when a new hardlink is added on the sending side and it sorts alphabetically earlier in the list than the files that are already present on the receiving side. - Dropped support for protocol versions less than 20 (2.3.0 released 15 Mar 1999) and activated warnings for protocols less than 25 (2.5.0 released 23 Aug 2001). (Wayne Davison and J.W. Schultz, severally) - More optimal data transmission for --hard-links (protocol 28). - More optimal data transmission for --checksum (protocol 28). - Less memory is used when --checksum is specified. - Less memory is used in the file list (a per-file savings). - The generator is now better about not modifying the file list during the transfer in order to avoid a copy-on-write memory bifurcation (on systems where fork() uses shared memory). Previously, rsync's shared memory would slowly become unshared, resulting in real memory usage nearly doubling on the receiving side by the end of the transfer. Now, as long as permissions are being preserved, the shared memory should remain that way for the entire transfer. - Changed hardlink info and file_struct + strings to use allocation pools. This reduces memory use for large file-sets and permits freeing memory to the OS. (J.W. Schultz) - The 2 pipes used between the receiver and generator processes (which are forked on the same machine) were reduced to 1 pipe and the protocol improved so that (1) it is now impossible to have the "redo" pipe fill up and hang rsync, and (2) trailing messages from the receiver don't get lost on their way through the generator over to the sender (which mainly affected hard-link messages and verbose --stats output). - Improved the internal uid/gid code to be more portable and a little more optimized. - The device numbers sent when using --devices are now sent as separate major/minor values with 32-bit accuracy (protocol 28). Previously, the copied devices were sent as a single 32-bit number. This will make inter-operation of 64-bit binaries more compatible with their 32-bit brethren (with both ends of the connection are using protocol 28). Note that optimizations in the binary protocol for sending the device numbers often results in fewer bytes being used than before, even though more precision is now available. - Some cleanup of the exclude/include structures and its code made things clearer (internally), simpler, and more efficient. - The reading & writing of the file-list in batch-mode is now handled by the same code that sends & receives the list over the wire. This makes it much easier to maintain. (Note that the batch code is still considered to be experimental.) BUILD CHANGES: - The configure script now accepts --with-rsyncd-conf=PATH to override the default value of the /etc/rsyncd.conf file. - Fixed configure bug when running "./configure --disable-ipv6". - Fixed compilation problem on Tru64 Unix (having to do with sockaddr.sa_len and sockaddr.sin_len). DEVELOPER RELATED: - Fixed "make test" bug when build dir is not the source dir. - Added a couple extra diffs in the "patches" dir, removed the ones that got applied, and rebuilt the rest. NEWS for rsync 2.6.0 (1 Jan 2004) Protocol: 27 (changed) Changes since 2.5.7: ENHANCEMENTS: * "ssh" is now the default remote shell for rsync. If you want to change this, configure like this: "./configure --with-rsh=rsh". * Added --files-from, --no-relative, --no-implied-dirs, and --from0. Note that --from0 affects the line-ending character for all the files read by the --*-from options. (Wayne Davison) * Length of csum2 is now per-file starting with protocol version 27. (J.W. Schultz) * Per-file dynamic block size is now sqrt(file length). The per-file checksum size is determined according to an algorithm provided by Donovan Baarda which reduces the probability of rsync algorithm corrupting data and falling back using the whole md4 checksums. (J.W. Schultz, Donovan Baarda) * The --stats option no longer includes the (debug) malloc summary unless the verbose option was specified at least twice. * Added a new error/warning code for when files vanish from the sending side. Made vanished source files not interfere with the file-deletion pass when --delete-after was specified. * Various trailing-info sections are now preceded by a newline. BUG FIXES: * Fixed several exclude/include matching bugs when using wild-cards. This has a several user-visible effects, all of which make the matching more consistent and intuitive. This should hopefully not cause anyone problems since it makes the matching work more like what people are expecting. (Wayne Davison) - A pattern with a "**" no longer causes a "*" to match slashes. For example, with "/*/foo/**", "foo" must be 2 levels deep. [If your string has BOTH "*" and "**" wildcards, changing the "*" wildcards to "**" will provide the old behavior in all versions.] - "**/foo" now matches at the base of the transfer (like /foo does). [Use "/**/foo" to get the old behavior in all versions.] - A non-anchored wildcard term floats to match beyond the base of the transfer. E.g. "CVS/R*" matches at the end of the path, just like the non-wildcard term "CVS/Root" does. [Use "/CVS/R*" to get the old behavior in all versions.] - Including a "**" in the match term causes it to be matched against the entire path, not just the name portion, even if there aren't any interior slashes in the term. E.g. "foo**bar" would exclude "/path/foo-bar" (just like before) as well as "/foo-path/baz-bar" (unlike before). [Use "foo*bar" to get the old behavior in all versions.] * The exclude list specified in the daemon's config file is now properly applied to the pulled items no matter how deep the user's file-args are in the source tree. (Wayne Davison) * For protocol version >= 27, mdfour_tail() is called when the block size (including checksum_seed) is a multiple of 64. Previously it was not called, giving the wrong MD4 checksum. (Craig Barratt) * For protocol version >= 27, a 64 bit bit counter is used in mdfour.c as required by the RFC. Previously only a 32 bit bit counter was used, causing incorrect MD4 file checksums for file sizes >= 512MB - 4. (Craig Barratt) * Fixed a crash bug when interacting with older rsync versions and multiple files of the same name are destined for the same dir. (Wayne Davison) * Keep tmp names from overflowing MAXPATHLEN. * Make --link-dest honor the absence of -p, -o, and -g. * Made rsync treat a trailing slash in the destination in a more consistent manner. * Fixed file I/O error detection. (John Van Essen) * Fixed bogus "malformed address {hostname}" message in rsyncd log when checking IP address against hostnames from "hosts allow" and "hosts deny" parameters in config file. * Print heap statistics when verbose >= 2 instead of when >= 1. * Fixed a compression (-z) bug when syncing a mostly-matching file that contains already-compressed data. (Yasuoka Masahiko and Wayne Davison) * Fixed a bug in the --backup code that could cause deleted files to not get backed up. * When the backup code makes new directories, create them with mode 0700 instead of 0755 (since the directory permissions in the backup tree are not yet copied from the main tree). * Call setgroups() in a more portable manner. * Improved file-related error messages to better indicate exactly what pathname failed. (Wayne Davison) * Fixed some bugs in the handling of --delete and --exclude when using the --relative (-R) option. (Wayne Davison) * Fixed bug that prevented regular files from replacing special files and caused a directory in --link-dest or --compare-dest to block the creation of a file with the same path. A directory still cannot be replaced by a regular file unless --delete specified. (J.W. Schultz) * Detect and report when open or opendir succeed but read and readdir fail caused by network filesystem issues and truncated files. (David Norwood, Michael Brown, J.W. Schultz) * Added a fix that should give ssh time to restore the tty settings if the user presses Ctrl-C at an ssh password prompt. INTERNAL: * Eliminated vestigial support for old versions that we stopped supporting. (J.W. Schultz) * Simplified some of the option-parsing code. (Wayne Davison) * Some cleanup made to the exclude code, as well as some new defines added to enhance readability. (Wayne Davison) * Changed the protocol-version code so that it can interact at a lower protocol level than the maximum supported by both sides. Added an undocumented option, --protocol=N, to force the value we advertise to the other side (primarily for testing purposes). (Wayne Davison) NEWS for rsync 2.5.7 (4 Dec 2003) Protocol: 26 (unchanged) Changes since 2.5.6: SECURITY FIXES: * Fix buffer handling bugs. (Andrew Tridgell, Martin Pool, Paul Russell, Andrea Barisani) NEWS for rsync 2.5.6, aka "the dwd-between-jobs release" (26 Jan 2003) Protocol: 26 (unchanged) Changes since 2.5.5: ENHANCEMENTS: * The --delete-after option now implies --delete. (Wayne Davison) * The --suffix option can now be used with --backup-dir. (Michael Zimmerman) * Combining "::" syntax with the -rsh/-e option now uses the specified remote-shell as a transport to talk to a (newly-spawned) server-daemon. This allows someone to use daemon features, such as modules, over a secure protocol, such as ssh. (JD Paul) * The rsync:// syntax for daemon connections is now accepted in the destination field. * If the file name given to --include-from or --exclude-from is "-", rsync will read from standard input. (J.W. Schultz) * New option --link-dest which is like --compare-dest except that unchanged files are hard-linked in to the destination directory. (J.W. Schultz) * Don't report an error if an excluded file disappears during an rsync run. (Eugene Chupriyanov and Bo Kersey) * Added .svn to --cvs-exclude list to support subversion. (Jon Middleton) * Properly support IPv6 addresses in the rsyncd.conf "hosts allow" and "hosts deny" fields. (Hideaki Yoshifuji) * Changed exclude file handling to permit DOS or MAC style line terminations. (J.W. Schultz) * Ignore errors from chmod when -p/-a/--preserve-perms is not set. (Dave Dykstra) BUG FIXES: * Fix "forward name lookup failed" errors on AIX 4.3.3. (John L. Allen, Martin Pool) * Generate each file's rolling-checksum data as we send it, not in a separate (memory-eating) pass before hand. This prevents timeout errors on really large files. (Stefan Nehlsen) * Fix compilation on Tru64. (Albert Chin, Zoong Pham) * Better handling of some client-server errors. (Martin Pool) * Fixed a crash that would occur when sending a list of files that contains a duplicate name (if it sorts to the end of the file list) and using --delete. (Wayne Davison) * Fixed the file-name duplicate-removal code when dealing with multiple dups in a row. (Wayne Davison) * Fixed a bug that caused rsync to lose the exit status of its child processes and sometimes return an exit code of 0 instead of showing an error. (David R. Staples, Dave Dykstra) * Fixed bug in --copy-unsafe-links that caused it to be completely broken. (Dave Dykstra) * Prevent infinite recursion in cleanup code under certain circumstances. (Sviatoslav Sviridov and Marc Espie) * Fixed a bug that prevented rsync from creating intervening directories when --relative-paths/-R is set. (Craig Barratt) * Prevent "Connection reset by peer" messages from Cygwin. (Randy O'Meara) INTERNAL: * Many code cleanups and improved internal documentation. (Martin Pool, Nelson Beebe) * Portability fixes. (Dave Dykstra and Wayne Davison) * More test cases. (Martin Pool) * Some test-case fixes. (Brian Poole, Wayne Davison) * Updated included popt to the latest vendor drop, version 1.6.4. (Jos Backus) * Updated config.guess and config.sub to latest versions; this means rsync should build on more platforms. (Paul Green) NEWS for rsync 2.5.5, aka Snowy River (2 Apr 2002) Protocol: 26 (unchanged) Changes since 2.5.4: ENHANCEMENTS: * With --progress, when a transfer is complete show the time taken; otherwise show expected time to complete. (Cameron Simpson) * Make "make install-strip" works properly, and "make install" accepts a DESTDIR variable for help in building binary packages. (Peter Breitenlohner, Greg Louis) * If configured with --enable-maintainer-mode, then on receipt of a fatal signal rsync will try to open an xterm running gdb, similarly to Samba's "panic action" or GNOME's bug-buddy. (Martin Pool) BUG FIXES: * Fix situation where failure to fork (e.g. because out of process slots) would cause rsync to kill all processes owned by the current user. Yes, really! (Paul Haas, Martin Pool) * Fix test suite on Solaris. (Jos Backus, Martin Pool) * Fix minor memory leak in socket code. (Dave Dykstra, Martin Pool.) * Fix --whole-file problem that caused it to be the default even for remote connections. (Martin Pool, Frank Schulz) * Work around bug in Mac OS X mkdir(2), which cannot handle trailing slashes. (Martin Pool) * Improved network error handling. (Greg A. Woods) NEWS for rsync 2.5.4, aka "Imitation lizard skin" (13 Mar 2002) Protocol: 26 (unchanged) Changes since 2.5.3: BUG FIXES: * Additional fix for zlib double-free bug. (Martin Pool, Andrew Tridgell) (CVE CAN-2002-0059) ENHANCEMENTS: * Merge in changes from zlib 1.1.3 to zlib 1.1.4. (Jos Backus) (Note that rsync still uses a custom version of zlib; you can not just link against a system library. See zlib/README.rsync) * Additional test cases for --compress. (Martin Pool) NEWS for rsync 2.5.3, aka "Happy 26" (11 Mar 2002) Protocol: 26 (unchanged) Changes since 2.5.2: SECURITY FIXES: * Make sure that supplementary groups are removed from a server process after changing uid and gid. (Ethan Benson) (Debian bug #132272, CVE CAN-2002-0080) BUG FIXES: * Fix zlib double-free bug. (Owen Taylor, Mark J Cox) (CVE CAN-2002-0059) * Fixed problem that in many cases caused the error message unexpected read size of 0 in map_ptr and resulted in the wrong data being copied. * Fixed compilation errors on some systems caused by the use of "unsigned int64" in rsync.h. * Fixed problem on systems such as Sunos4 that do not support realloc on a NULL pointer; error was "out of memory in flist_expand". * Fix for rsync server processes hanging around after the client unexpectedly disconnects. (Colin Walters) (Debian bug #128632) * Cope with BSD systems on which mkdir() will not accept a trailing slash. ENHANCEMENTS: * Merge in changes from zlib 1.1.2 to zlib 1.1.3. (Note that rsync still uses a custom version of zlib; you can not just link against a system library. See zlib/README.rsync) * Command to initiate connections is only shown with -vv, rather than -v as in 2.5.2. Output from plain -v is more similar to what was historically used so as not to break scripts that try to parse the output. * Added --no-whole-file and --no-blocking-io options (Dave Dykstra) * Made the --write-batch and --read-batch options actually work and added documentation in the man page (Jos Backus) * If the daemon is unable to fork a child to accept a connection, print an error message. (Colin Walters) NEWS for rsync 2.5.2 (26 Jan 2002) Protocol: 26 (changed) Changes since 2.5.1: SECURITY FIXES: * Signedness security patch from Sebastian Krahmer -- in some cases we were not sufficiently careful about reading integers from the network. BUG FIXES: * Fix possible string mangling in log files. * Fix for setting local address of outgoing sockets. * Better handling of hardlinks and devices on platforms with 64-bit dev_t or ino_t. * Name resolution on machines supporting IPv6 is improved. * Fix for device nodes. (dann frazier) (Debian #129135) ENHANCEMENTS: * With -v, rsync now shows the command used to initiate an ssh/rsh connection. * --statistics now shows memory heap usage on platforms that support mallinfo(). * "The Ted T'so school of program optimization": make progress visible and people will think it's faster. (With --progress, rsync will show you how many files it has seen as it builds the file_list, giving some indication that it has not hung.) * Improvements to batch mode support. This is still experimental but testing would be welcome. (Jos Backus) * New --ignore-existing option, patch previously distributed with Vipul's Razor. (Debian #124286) NEWS for rsync 2.5.1 (3 Jan 2002) Protocol: 25 (unchanged) Changes since 2.5.0: BUG FIXES: * Fix for segfault in --daemon mode configuration parser. (Paul Mackerras) * Correct string<->address parsing for both IPv4 and 6. (YOSHIFUJI Hideaki, SUMIKAWA Munechika and Jun-ichiro "itojun" Hagino) * Various fixes for IPv6 support. (Dave Dykstra) * rsync.1 typo fix. (Matt Kraai) * Test suite typo fixes. (Tom Schmidt) * rsync.1 grammar and clarity improvements. (Edward Welbourne) * Correction to ./configure tests for inet_ntop. (Jeff Garzik) ENHANCEMENTS: * --progress and -P now show estimated data transfer rate (in a multiple of bytes/s) and estimated time to completion. (Rik Faith) * --no-detach option, required to run as a W32 service and also useful when running on Unix under daemontools, AIX's SRC, or a debugger. (Max Bowsher, Jos Backus) * Clearer error messages for some conditions. NEWS for rsync 2.5.0 (30 Nov 2001) Protocol: 25 (changed) Changes since 2.4.6: ANNOUNCEMENTS * Martin Pool is now a co-maintainer. NEW FEATURES * Support for LSB-compliant packaging * Shell wildcards are allowed in "auth users" lines. * Merged UNC rsync+ patch to support creation of standalone patch sets. By Bert J. Dempsey and Debra Weiss, updated by Jos Backus. * IPv6 support based on a patch from KAME.net, on systems including modern versions of Linux, Solaris, and HP-UX. Also includes IPv6 compatibility functions for old OSs by the Internet Software Consortium, Paul Vixie, the OpenSSH portability project, and OpenBSD. ENHANCEMENTS * Include/exclude cluestick: with -vv, print out whether files are included or excluded and why. * Many error messages have more friendly explanations and more details. * Manual page improvements plus scanty protocol documentation. * When running as --daemon in the background and using a "log file" rsyncd.conf directive, close the log file every time it is open when going to sleep on the socket. This allows the log file to get cleaned out by another process. * Change to using libpopt rather than getopt for processing options. This makes the code cleaner and the behaviour more consistent across platforms. popt is included and built if not installed on the platform. * More details in --version, including note about whether 64-bit files, symlinks and hardlinks are supported. * MD4 code may use less CPU cycles. * Use mkstemp on systems where it is secure. If we use mktemp, explain that we do it in a secure way. * --whole-file is the default when source and target are on the local machine. BUG FIXES: * Fix for various bugs causing rsync to hang. * Attempt to fix Large File Summit support on AIX. * Attempt to fix error handling lockup bug. * Give a non-0 exit code if *any* of the files we have been asked to transfer fail to transfer. * For log messages containing ridiculously long strings that might overflow a buffer rsync no longer aborts, but rather prints an ellipsis at the end of the string. (Patch from Ed Santiago.) PLATFORMS: * Improved support for UNICOS (tested on Cray T3E and Cray SV1) * autoconf2.52 (or later) is now required to rebuild the autoconf scripts. It is not required to simply build rsync. * Platforms thought to work in this release: Cray SV1 UNICOS 10.0.0.8 cc Debian Linux 2.2 UltraSparc gcc Debian Linux testing/unstable ARM gcc FreeBSD 3.3-RELEASE i386 cc FreeBSD 4.1.1-RELEASE i386 cc FreeBSD 4.3-STABLE i386 cc HP PA-RISC HP-UX 10.20 gcc HP PA-RISC HP-UX 11.11 cc IRIX 6.5 MIPS cc IRIX 6.5 MIPS gcc Mac OS X PPC (--disable-ipv6) cc NetBSD 1.5 i386 gcc NetBSD Current i386 cc OpenBSD 2.5 Sparc gcc OpenBSD 2.9 i386 cc OpenBSD Current i386 cc RedHat 6.2 i386 gcc RedHat 6.2 i386 insure++ RedHat 7.0 i386 gcc RedHat 7.1 i386 (Kernel 2.4.10) gcc Slackware 8.0 i686 (Kernel 2.4.10) Solaris 8 UltraSparc cc Solaris 8 UltraSparc gcc Solaris 8 i386 gcc SuSE 7.1 i386 gcc2.95.2 SuSE 7.1 ppc gcc2.95.2 i386-pc-sco3.2v5.0.5 cc i386-pc-sco3.2v5.0.5 gcc powerpc-ibm-aix4.3.3.0 cc i686-unknown-sysv5UnixWare7.1.0 gcc i686-unknown-sysv5UnixWare7.1.0 cc TESTING: * The existing test.sh script by Phil Hands has been merged into a test framework that works from both "make check" and the Samba build farm. Partial Protocol History RELEASE DATE VER. DATE OF COMMIT* PROTOCOL ?? May 2005 2.6.5 29 30 Mar 2005 2.6.4 17 Jan 2005 29 30 Sep 2004 2.6.3 28 30 Apr 2004 2.6.2 28 26 Apr 2004 2.6.1 08 Jan 2004 28 01 Jan 2004 2.6.0 10 Apr 2003 27 (MAX=40) 04 Dec 2003 2.5.7 26 26 Jan 2003 2.5.6 26 02 Apr 2002 2.5.5 26 13 Mar 2002 2.5.4 26 11 Mar 2002 2.5.3 26 26 Jan 2002 2.5.2 11 Jan 2002 26 03 Jan 2002 2.5.1 25 30 Nov 2001 2.5.0 23 Aug 2001 25 06 Sep 2000 2.4.6 24 19 Aug 2000 2.4.5 24 29 Jul 2000 2.4.4 24 09 Apr 2000 2.4.3 24 30 Mar 2000 2.4.2 24 30 Jan 2000 2.4.1 29 Jan 2000 24 29 Jan 2000 2.4.0 28 Jan 2000 23 25 Jan 2000 2.3.3 23 Jan 2000 22 08 Nov 1999 2.3.2 26 Jun 1999 21 06 Apr 1999 2.3.1 20 15 Mar 1999 2.3.0 15 Mar 1999 20 25 Nov 1998 2.2.1 19 03 Nov 1998 2.2.0 19 09 Sep 1998 2.1.1 19 20 Jul 1998 2.1.0 19 17 Jul 1998 2.0.19 19 18 Jun 1998 2.0.17 19 01 Jun 1998 2.0.16 19 27 May 1998 2.0.13 27 May 1998 19 26 May 1998 2.0.12 18 22 May 1998 2.0.11 18 18 May 1998 2.0.9 18 May 1998 18 17 May 1998 2.0.8 17 15 May 1998 2.0.1 17 14 May 1998 2.0.0 17 17 Apr 1998 1.7.4 17 13 Apr 1998 1.7.3 17 05 Apr 1998 1.7.2 17 26 Mar 1998 1.7.1 17 26 Mar 1998 1.7.0 26 Mar 1998 17 (MAX=30) 13 Jan 1998 1.6.9 13 Jan 1998 15 (MAX=20) * DATE OF COMMIT is the date the protocol change was committed to CVS.