rsync.git
4 years agoChange man page src format from yodl to markdown.
Wayne Davison [Tue, 9 Jun 2020 02:21:42 +0000 (19:21 -0700)]
Change man page src format from yodl to markdown.

This removes the yodl dependency, which is sometimes hard to track down.
Instead, this uses a python3 script that leverages the cmarkgfm library
to turn the source file into html.  Then, the script parses the html in
order to turn the tag stream into a nroff stream using a simple state
machine. While it's doing that it also implements one added format rule
that turns an ordinal list that starts at 0 into a description list
(since markdown doesn't have an easy description list idiom).

4 years agoTweak the new heading
Wayne Davison [Mon, 8 Jun 2020 02:43:08 +0000 (19:43 -0700)]
Tweak the new heading

4 years agoAdd OpenSSL license exception also to COPYING
Sebastian Andrzej Siewior [Sun, 7 Jun 2020 19:54:49 +0000 (21:54 +0200)]
Add OpenSSL license exception also to COPYING

Add the OpenSSL license exception also to the COPYING file which
contains the license related information.

Signed-off-by: Sebastian Andrzej Siewior <sebastian@breakpoint.cc>
4 years agoUse ZSTD_CLEVEL_DEFAULT define.
Wayne Davison [Mon, 8 Jun 2020 01:58:30 +0000 (18:58 -0700)]
Use ZSTD_CLEVEL_DEFAULT define.

4 years agoPut the rsync-ssl-rsh helper script into a lib dir.
Wayne Davison [Mon, 8 Jun 2020 00:20:58 +0000 (17:20 -0700)]
Put the rsync-ssl-rsh helper script into a lib dir.

4 years agoA couple minor fixes.
Wayne Davison [Fri, 5 Jun 2020 05:54:38 +0000 (22:54 -0700)]
A couple minor fixes.

4 years agoOnly sender can output non-final stats on error
Wayne Davison [Fri, 5 Jun 2020 04:21:05 +0000 (21:21 -0700)]
Only sender can output non-final stats on error

The receiving side's stats are split between 2 processes until the very end.

4 years agoMake use of O_NOFOLLOW if it is defined.
Wayne Davison [Fri, 5 Jun 2020 02:47:59 +0000 (19:47 -0700)]
Make use of O_NOFOLLOW if it is defined.

4 years agoMention more NEWS items.
Wayne Davison [Fri, 5 Jun 2020 02:05:56 +0000 (19:05 -0700)]
Mention more NEWS items.

4 years agoUse a lock to not fail on a left-over pid file.
Wayne Davison [Fri, 5 Jun 2020 00:55:20 +0000 (17:55 -0700)]
Use a lock to not fail on a left-over pid file.

4 years agoUse more switch statements.
Wayne Davison [Thu, 4 Jun 2020 23:17:12 +0000 (16:17 -0700)]
Use more switch statements.

4 years agoMake -V the short opt for --version.
Wayne Davison [Thu, 4 Jun 2020 22:50:57 +0000 (15:50 -0700)]
Make -V the short opt for --version.

4 years agoAvoid a deadlock due to huge amounts of verbose messages.
Wayne Davison [Thu, 4 Jun 2020 21:11:07 +0000 (14:11 -0700)]
Avoid a deadlock due to huge amounts of verbose messages.

Allow the receiver to increase their iobuf.msg xbuf if it fills up. This
ensures that the receiver will never block trying to output a message,
and thus it will always drain the data from the sender and keep the
whole thing from clogging up.

4 years agoTweak --copy-as docs a bit more.
Wayne Davison [Thu, 4 Jun 2020 20:07:50 +0000 (13:07 -0700)]
Tweak --copy-as docs a bit more.

4 years agoAdd hostname "lh" as a --no-cd localhost.
Wayne Davison [Thu, 4 Jun 2020 19:37:07 +0000 (12:37 -0700)]
Add hostname "lh" as a --no-cd localhost.

4 years agoFix change_dir() leaving appended slash in curr_dir on failure.
Wayne Davison [Thu, 4 Jun 2020 19:35:19 +0000 (12:35 -0700)]
Fix change_dir() leaving appended slash in curr_dir on failure.

4 years agoHandle --skip-compress right for new compressors
Wayne Davison [Wed, 3 Jun 2020 00:58:24 +0000 (17:58 -0700)]
Handle --skip-compress right for new compressors

Some compressors can't completely turn off, so minimize the level
when a file is being "skipped".

4 years agoMention that `rsync --version` outputs checksum & compress lists.
Wayne Davison [Wed, 3 Jun 2020 00:21:51 +0000 (17:21 -0700)]
Mention that `rsync --version` outputs checksum & compress lists.

4 years agoUpdate ccpp.yml
Wayne Davison [Wed, 3 Jun 2020 00:20:22 +0000 (17:20 -0700)]
Update ccpp.yml

Switch to ubuntu-20.04 and add a couple dev libs.

4 years agoSome improvements to the release mechanism
Wayne Davison [Tue, 2 Jun 2020 19:43:38 +0000 (12:43 -0700)]
Some improvements to the release mechanism

- Some manpage changes to make them more consistent and to add a section
  that the release script expects in rsync-ssl.
- Fixed some issues in release-rsync pertaining to various file changes,
  such as the .md file changes.
- Change the gpg handling to stop prompting for a passphrase since gpg
  now makes use of gpg-agent (and the old gpg script is apparently not
  passing through fd 2 that git needs to get status).

4 years agoMove the CSUM defines.
Wayne Davison [Tue, 2 Jun 2020 01:49:15 +0000 (18:49 -0700)]
Move the CSUM defines.

4 years agoA few style tweaks.
Wayne Davison [Tue, 2 Jun 2020 01:36:22 +0000 (18:36 -0700)]
A few style tweaks.

4 years agoMove OpenSSL-related MD4/5 defines and imports to lib/mdigest.h
Jorrit Jongma [Mon, 25 May 2020 16:16:19 +0000 (18:16 +0200)]
Move OpenSSL-related MD4/5 defines and imports to lib/mdigest.h

Works just as well, prevents having to repeat them across files

4 years agoSome improvements for --msgs2stderr and --daemon.
Wayne Davison [Sun, 31 May 2020 22:25:11 +0000 (15:25 -0700)]
Some improvements for --msgs2stderr and --daemon.

- Set am_daemon to -1 (from 1) when the daemon is run via rsh.
- Only disable --msgs2stderr for a normal (socket) daemon.
- Forward a -q to the server if --msgs2stderr was also specified.
- Added --no-msgs2stderr option to allow it to be overridden.

4 years agoMention the latest NEWS items.
Wayne Davison [Sat, 30 May 2020 12:54:09 +0000 (05:54 -0700)]
Mention the latest NEWS items.

4 years agoAvoid noop_io_until_death() if --msgs2stderr was specified.
Wayne Davison [Sat, 30 May 2020 12:53:59 +0000 (05:53 -0700)]
Avoid noop_io_until_death() if --msgs2stderr was specified.

4 years agoSearch for double-fuzzy files only when needed
benrubson [Tue, 26 May 2020 14:06:07 +0000 (16:06 +0200)]
Search for double-fuzzy files only when needed

4 years agoSilence a strncpy() warning.
Wayne Davison [Sat, 30 May 2020 00:35:56 +0000 (17:35 -0700)]
Silence a strncpy() warning.

4 years agoSilence some g++ warnings.
Wayne Davison [Fri, 29 May 2020 21:38:19 +0000 (14:38 -0700)]
Silence some g++ warnings.

4 years agoGive configure's snprintf() test a guaranteed short string at the start.
Wayne Davison [Fri, 29 May 2020 21:37:49 +0000 (14:37 -0700)]
Give configure's snprintf() test a guaranteed short string at the start.

4 years agoSilence gcc7.1 warnings about snprintf().
Wayne Davison [Fri, 29 May 2020 21:18:08 +0000 (14:18 -0700)]
Silence gcc7.1 warnings about snprintf().

4 years agoFix regex issue due to python 3.8 bug.
Wayne Davison [Fri, 29 May 2020 03:48:24 +0000 (20:48 -0700)]
Fix regex issue due to python 3.8 bug.

4 years agoA few minor xxhash changes.
Wayne Davison [Fri, 29 May 2020 03:21:54 +0000 (20:21 -0700)]
A few minor xxhash changes.

4 years agoMerge pull request #5 from benrubson/daemonstats
Wayne Davison [Thu, 28 May 2020 20:42:47 +0000 (13:42 -0700)]
Merge pull request #5 from benrubson/daemonstats

Have daemon log data sent/received even when exiting with an error.

4 years agoAdd stub for canonical_checksum().
Wayne Davison [Thu, 28 May 2020 19:46:46 +0000 (12:46 -0700)]
Add stub for canonical_checksum().

4 years agoThe xxh* checksums don't need to be reversed on output.
Wayne Davison [Thu, 28 May 2020 19:05:54 +0000 (12:05 -0700)]
The xxh* checksums don't need to be reversed on output.

4 years agoSwitch to using LZ4_compress_default().
Wayne Davison [Thu, 28 May 2020 18:40:52 +0000 (11:40 -0700)]
Switch to using LZ4_compress_default().

4 years agoUse MSG_FLUSH in a couple more spots.
Wayne Davison [Thu, 28 May 2020 07:41:39 +0000 (00:41 -0700)]
Use MSG_FLUSH in a couple more spots.

4 years agoMerge pull request #2 from benrubson/flush
Wayne Davison [Thu, 28 May 2020 06:35:25 +0000 (23:35 -0700)]
Merge pull request #2 from benrubson/flush

Help final error messages get to the sender. Is particularly helpful when talking with an older rsync client. Adds a new flush type of MSG_FLUSH.

4 years agoA couple minor tweaks.
Wayne Davison [Wed, 27 May 2020 05:28:22 +0000 (22:28 -0700)]
A couple minor tweaks.

4 years agoLog data sent/received even if error
benrubson [Tue, 26 May 2020 17:53:25 +0000 (19:53 +0200)]
Log data sent/received even if error

4 years agoSimplify the capabilities array.
Wayne Davison [Tue, 26 May 2020 14:38:45 +0000 (07:38 -0700)]
Simplify the capabilities array.

4 years agoCorrectly send last error to sender
benrubson [Tue, 26 May 2020 14:24:30 +0000 (16:24 +0200)]
Correctly send last error to sender

4 years agoConvert NEWS & OLDNEWS into .md files.
Wayne Davison [Tue, 26 May 2020 09:14:05 +0000 (02:14 -0700)]
Convert NEWS & OLDNEWS into .md files.

4 years agoFix the parsing of the --version capabilities.
Wayne Davison [Tue, 26 May 2020 06:42:33 +0000 (23:42 -0700)]
Fix the parsing of the --version capabilities.

4 years agoInstall yodl for the CI builds.
Wayne Davison [Tue, 26 May 2020 06:41:39 +0000 (23:41 -0700)]
Install yodl for the CI builds.

4 years agoGet indent right.
Wayne Davison [Tue, 26 May 2020 06:33:11 +0000 (23:33 -0700)]
Get indent right.

4 years agoImprove output of capabilities in --version list.
Wayne Davison [Tue, 26 May 2020 05:43:33 +0000 (22:43 -0700)]
Improve output of capabilities in --version list.

It now wraps better as the "no " prefixes change, and it makes it easier
to maintain patches that add items into the capabilities list.

4 years agoRemove xxhash from capabilities list.
Wayne Davison [Tue, 26 May 2020 04:27:21 +0000 (21:27 -0700)]
Remove xxhash from capabilities list.

It's now listed in the "Checksum list:" output.

4 years agoFix a couple deb names.
Wayne Davison [Tue, 26 May 2020 04:21:58 +0000 (21:21 -0700)]
Fix a couple deb names.

4 years agoImprove CI setup.
Wayne Davison [Tue, 26 May 2020 04:17:51 +0000 (21:17 -0700)]
Improve CI setup.

4 years agoImprove how negotiated info affects batch files.
Wayne Davison [Tue, 26 May 2020 01:03:02 +0000 (18:03 -0700)]
Improve how negotiated info affects batch files.

4 years agoTweak the --zz option to --zc (aka --compress-choice).
Wayne Davison [Mon, 25 May 2020 23:57:47 +0000 (16:57 -0700)]
Tweak the --zz option to --zc (aka --compress-choice).

4 years agoRename a couple files to .md
Wayne Davison [Mon, 25 May 2020 22:00:30 +0000 (15:00 -0700)]
Rename a couple files to .md

4 years agoAdd some extra indent.
Wayne Davison [Mon, 25 May 2020 21:59:51 +0000 (14:59 -0700)]
Add some extra indent.

4 years agoCreate ccpp.yml
Wayne Davison [Mon, 25 May 2020 21:43:25 +0000 (14:43 -0700)]
Create ccpp.yml

4 years agoTwo sparse fixes from Yuxuan Shui.
Wayne Davison [Mon, 25 May 2020 21:01:52 +0000 (14:01 -0700)]
Two sparse fixes from Yuxuan Shui.

- Make "len" parameter of do_punch_hole an OFF_T.
- Clear sparse_past_write in sparse_end(), otherwise when write_sparse()
  is called for the next file, do_punch_hole() will be called with a pos
  that's not actually the current position in file, causing it to fail.

4 years agoInclude lz4 compression support.
Wayne Davison [Mon, 25 May 2020 20:43:25 +0000 (13:43 -0700)]
Include lz4 compression support.

Based on a patch that was emailed to me without a valid return address.

4 years agoInclude zstd compression support.
Wayne Davison [Mon, 25 May 2020 20:31:30 +0000 (13:31 -0700)]
Include zstd compression support.

Based on a patch by Sebastian A. Siewior. Fixes bug #14338.

4 years agoFix handling of a compressor that has no off_level.
Wayne Davison [Mon, 25 May 2020 20:02:56 +0000 (13:02 -0700)]
Fix handling of a compressor that has no off_level.

4 years agoOutput the default checksum & compress lists in the --version output.
Wayne Davison [Mon, 25 May 2020 19:45:28 +0000 (12:45 -0700)]
Output the default checksum & compress lists in the --version output.

4 years agoA couple minor configure.ac tweaks.
Wayne Davison [Mon, 25 May 2020 18:50:44 +0000 (11:50 -0700)]
A couple minor configure.ac tweaks.

4 years agoMake compression-level handling generic.
Wayne Davison [Mon, 25 May 2020 17:47:57 +0000 (10:47 -0700)]
Make compression-level handling generic.

4 years agoSome minor tweaks & tidying up.
Wayne Davison [Mon, 25 May 2020 05:31:28 +0000 (22:31 -0700)]
Some minor tweaks & tidying up.

4 years agoChange odd-ball map_ptr() to use remainder like the others.
Wayne Davison [Mon, 25 May 2020 03:38:48 +0000 (20:38 -0700)]
Change odd-ball map_ptr() to use remainder like the others.

4 years agoImprove performance of file_checksum()
Jorrit Jongma [Sun, 24 May 2020 22:30:55 +0000 (00:30 +0200)]
Improve performance of file_checksum()

Previously files were hashed in blocks of CSUM_CHUNK (64) bytes. This
causes significant overhead. The CSUM_CHUNK define cannot be changed as
md5.c depends on it, but there is no obvious reason to use it in
file_checksum(). By using CHUNK_SIZE (32 kB) instead, in some test
cases throughput more than doubles.

4 years agoMore NEWS updates.
Wayne Davison [Mon, 25 May 2020 03:19:15 +0000 (20:19 -0700)]
More NEWS updates.

4 years agoPrefer zlibx compression consistently instead of having 2 possible default preference...
Wayne Davison [Mon, 25 May 2020 02:51:11 +0000 (19:51 -0700)]
Prefer zlibx compression consistently instead of having 2 possible default preference orders.

4 years agoDon't auto-foward debug options to the server side anymore.
Wayne Davison [Mon, 25 May 2020 02:35:22 +0000 (19:35 -0700)]
Don't auto-foward debug options to the server side anymore.

4 years agoFix -z choice with older rsyncs.
Wayne Davison [Mon, 25 May 2020 02:16:05 +0000 (19:16 -0700)]
Fix -z choice with older rsyncs.

4 years agoA few more compression tweaks.
Wayne Davison [Mon, 25 May 2020 01:08:09 +0000 (18:08 -0700)]
A few more compression tweaks.

4 years agoMore compress changes
Wayne Davison [Sun, 24 May 2020 22:45:59 +0000 (15:45 -0700)]
More compress changes

- Add the zlibx (external-code compatible) compression name.
- Re-enable zlib support with the external library so it can be
  tried as a fallback if zlibx isn't available.
- Add --compress-choice=STR (aka -zz=STR) option.
- Make --cc=STR an alias for --checksum-choice=STR.
- Hook up the new compression negotiation logic.

4 years agoFix negotiation of none & improve NSTR debug msgs.
Wayne Davison [Sun, 24 May 2020 20:34:47 +0000 (13:34 -0700)]
Fix negotiation of none & improve NSTR debug msgs.

4 years agoAdd support for negotiated checksum names.
Wayne Davison [Sun, 24 May 2020 19:56:00 +0000 (12:56 -0700)]
Add support for negotiated checksum names.

4 years agoMake xxh64 the "main_name" for the current xxhash.
Wayne Davison [Sun, 24 May 2020 09:07:43 +0000 (02:07 -0700)]
Make xxh64 the "main_name" for the current xxhash.

4 years agoFix some warnings.
Wayne Davison [Sun, 24 May 2020 09:04:14 +0000 (02:04 -0700)]
Fix some warnings.

4 years agoSome more checksum improvements
Wayne Davison [Sun, 24 May 2020 05:05:16 +0000 (22:05 -0700)]
Some more checksum improvements

- Add/improve --debug=CSUM2 messages.
- Add an "xxh64" alias for "xxhash" name because we should be
  getting a few more xxhash variations in the future.
- Tweak the matching code to handle entries that have multiple
  names.
- Tweak some of the vars/defines.

4 years agoImprove some early debug-message newlines.
Wayne Davison [Sun, 24 May 2020 04:46:15 +0000 (21:46 -0700)]
Improve some early debug-message newlines.

Avoid a newline issue during the output of --DEBUG=CSUM info from
both the server and the client -- we need to output the full message
with its newline as much as possible.

4 years agoAvoid re-evaluating the args of SIVAL* w/CAREFUL_ALIGNMENT.
Wayne Davison [Sun, 24 May 2020 04:43:53 +0000 (21:43 -0700)]
Avoid re-evaluating the args of SIVAL* w/CAREFUL_ALIGNMENT.

4 years agoUnify the checksum context memory, since we only use one at a time.
Wayne Davison [Sun, 24 May 2020 01:25:11 +0000 (18:25 -0700)]
Unify the checksum context memory, since we only use one at a time.

4 years agoFixed the use of openssl MD4 for transfer checksums.
Wayne Davison [Sat, 23 May 2020 23:22:36 +0000 (16:22 -0700)]
Fixed the use of openssl MD4 for transfer checksums.

4 years agoOptional openssl support for MD4 pre-transfer checksums (but, sadly, not transfer...
Wayne Davison [Sat, 23 May 2020 19:26:06 +0000 (12:26 -0700)]
Optional openssl support for MD4 pre-transfer checksums (but, sadly, not transfer checksums).

4 years agoAdd optional use of the openssl crypto lib for MD5.
Wayne Davison [Sat, 23 May 2020 16:23:01 +0000 (09:23 -0700)]
Add optional use of the openssl crypto lib for MD5.

4 years agoMention a few more news items.
Wayne Davison [Sat, 23 May 2020 06:26:25 +0000 (23:26 -0700)]
Mention a few more news items.

4 years agoAdd "input" handling for cmd_txt_*() pkglib.py.
Wayne Davison [Sat, 23 May 2020 05:40:54 +0000 (22:40 -0700)]
Add "input" handling for cmd_txt_*() pkglib.py.

4 years agoOptimized assembler version of md5_process() for x86-64
Jorrit Jongma [Fri, 22 May 2020 17:38:37 +0000 (19:38 +0200)]
Optimized assembler version of md5_process() for x86-64

Originally created by Marc Bevand and placed in the public domain.
Enable/disabled via the same --enable-simd configure switch as
the rolling checksum optimizations.

4 years agoA little more safety in negotiate_checksum().
Wayne Davison [Sat, 23 May 2020 02:26:30 +0000 (19:26 -0700)]
A little more safety in negotiate_checksum().

4 years agoPromoting xxhash support.
Wayne Davison [Sat, 23 May 2020 00:40:08 +0000 (17:40 -0700)]
Promoting xxhash support.

4 years agoSome checksum improvements
Wayne Davison [Fri, 22 May 2020 22:06:20 +0000 (15:06 -0700)]
Some checksum improvements

- Improve csum negotation logic.
- Define the csum names in a single structure.
- Add --debug=CSUM.

4 years agoAVX2 optimized version of get_checksum1() for x86-64
Jorrit Jongma [Fri, 22 May 2020 11:03:55 +0000 (13:03 +0200)]
AVX2 optimized version of get_checksum1() for x86-64

Additionally restructures build switches and defines from SSE2 to SIMD,
to allow potential reuse should patches become available with SIMD
instructions for other processor architectures.

(Some minor tweaks of Jorrit's patch to avoid requiring GNU make and to
avoid C++ comments in .c files.)

4 years agoChecksum negotiation & more bits for compat_flags
Wayne Davison [Fri, 22 May 2020 15:23:26 +0000 (08:23 -0700)]
Checksum negotiation & more bits for compat_flags

- Add checksum negotiation to the protocol so that we can easily add new
  checksum algorithms and each will be used when both sides support it.
- Increase the size of the compat_flags value in the protocol from a
  byte to an int.

4 years agoFix a bug in the writing of the batch.sh file
Wayne Davison [Fri, 22 May 2020 15:10:41 +0000 (08:10 -0700)]
Fix a bug in the writing of the batch.sh file

Fix the code that writes the options and the default destination path
into the batch.sh file to be able to handle options being specified
after source/dest args.

4 years agoSSE2/SSSE3 optimized version of get_checksum1() for x86-64
Jorrit Jongma [Tue, 19 May 2020 12:52:40 +0000 (14:52 +0200)]
SSE2/SSSE3 optimized version of get_checksum1() for x86-64

Requires compilation using GCC C++ front end, build scripts have been
modified accordingly. C++ is only used when the optimization is enabled
(g++ as compiler, x86-64 build target, --enable-sse2 is passed to
configure).

(Wayne made a few tweaks, including making it disabled by default.)

4 years agoTweak the accept/refuse strings a bit.
Wayne Davison [Mon, 18 May 2020 07:03:09 +0000 (00:03 -0700)]
Tweak the accept/refuse strings a bit.

4 years agoAdd back a lost "*" and document the refusing of log-file* opts.
Wayne Davison [Mon, 18 May 2020 06:26:13 +0000 (23:26 -0700)]
Add back a lost "*" and document the refusing of log-file* opts.

4 years agoMore manpage improvements for "refuse options".
Wayne Davison [Mon, 18 May 2020 05:16:58 +0000 (22:16 -0700)]
More manpage improvements for "refuse options".

4 years agoAdd the ability to negate matches for the daemon's "refuse options".
Wayne Davison [Mon, 18 May 2020 04:29:11 +0000 (21:29 -0700)]
Add the ability to negate matches for the daemon's "refuse options".

4 years agoNeed to flush early errors before we exit.
Wayne Davison [Mon, 18 May 2020 03:21:45 +0000 (20:21 -0700)]
Need to flush early errors before we exit.

4 years agoSome pkglib improvements.
Wayne Davison [Sun, 17 May 2020 21:39:18 +0000 (14:39 -0700)]
Some pkglib improvements.

4 years agoA couple more manpage fixes.
Wayne Davison [Wed, 13 May 2020 07:20:03 +0000 (00:20 -0700)]
A couple more manpage fixes.