Wayne Davison [Tue, 9 Jun 2020 02:21:42 +0000 (19:21 -0700)]
Change man page src format from yodl to markdown.
This removes the yodl dependency, which is sometimes hard to track down.
Instead, this uses a python3 script that leverages the cmarkgfm library
to turn the source file into html. Then, the script parses the html in
order to turn the tag stream into a nroff stream using a simple state
machine. While it's doing that it also implements one added format rule
that turns an ordinal list that starts at 0 into a description list
(since markdown doesn't have an easy description list idiom).
Wayne Davison [Mon, 8 Jun 2020 02:43:08 +0000 (19:43 -0700)]
Tweak the new heading
Sebastian Andrzej Siewior [Sun, 7 Jun 2020 19:54:49 +0000 (21:54 +0200)]
Add OpenSSL license exception also to COPYING
Add the OpenSSL license exception also to the COPYING file which
contains the license related information.
Signed-off-by: Sebastian Andrzej Siewior <sebastian@breakpoint.cc>
Wayne Davison [Mon, 8 Jun 2020 01:58:30 +0000 (18:58 -0700)]
Use ZSTD_CLEVEL_DEFAULT define.
Wayne Davison [Mon, 8 Jun 2020 00:20:58 +0000 (17:20 -0700)]
Put the rsync-ssl-rsh helper script into a lib dir.
Wayne Davison [Fri, 5 Jun 2020 05:54:38 +0000 (22:54 -0700)]
A couple minor fixes.
Wayne Davison [Fri, 5 Jun 2020 04:21:05 +0000 (21:21 -0700)]
Only sender can output non-final stats on error
The receiving side's stats are split between 2 processes until the very end.
Wayne Davison [Fri, 5 Jun 2020 02:47:59 +0000 (19:47 -0700)]
Make use of O_NOFOLLOW if it is defined.
Wayne Davison [Fri, 5 Jun 2020 02:05:56 +0000 (19:05 -0700)]
Mention more NEWS items.
Wayne Davison [Fri, 5 Jun 2020 00:55:20 +0000 (17:55 -0700)]
Use a lock to not fail on a left-over pid file.
Wayne Davison [Thu, 4 Jun 2020 23:17:12 +0000 (16:17 -0700)]
Use more switch statements.
Wayne Davison [Thu, 4 Jun 2020 22:50:57 +0000 (15:50 -0700)]
Make -V the short opt for --version.
Wayne Davison [Thu, 4 Jun 2020 21:11:07 +0000 (14:11 -0700)]
Avoid a deadlock due to huge amounts of verbose messages.
Allow the receiver to increase their iobuf.msg xbuf if it fills up. This
ensures that the receiver will never block trying to output a message,
and thus it will always drain the data from the sender and keep the
whole thing from clogging up.
Wayne Davison [Thu, 4 Jun 2020 20:07:50 +0000 (13:07 -0700)]
Tweak --copy-as docs a bit more.
Wayne Davison [Thu, 4 Jun 2020 19:37:07 +0000 (12:37 -0700)]
Add hostname "lh" as a --no-cd localhost.
Wayne Davison [Thu, 4 Jun 2020 19:35:19 +0000 (12:35 -0700)]
Fix change_dir() leaving appended slash in curr_dir on failure.
Wayne Davison [Wed, 3 Jun 2020 00:58:24 +0000 (17:58 -0700)]
Handle --skip-compress right for new compressors
Some compressors can't completely turn off, so minimize the level
when a file is being "skipped".
Wayne Davison [Wed, 3 Jun 2020 00:21:51 +0000 (17:21 -0700)]
Mention that `rsync --version` outputs checksum & compress lists.
Wayne Davison [Wed, 3 Jun 2020 00:20:22 +0000 (17:20 -0700)]
Update ccpp.yml
Switch to ubuntu-20.04 and add a couple dev libs.
Wayne Davison [Tue, 2 Jun 2020 19:43:38 +0000 (12:43 -0700)]
Some improvements to the release mechanism
- Some manpage changes to make them more consistent and to add a section
that the release script expects in rsync-ssl.
- Fixed some issues in release-rsync pertaining to various file changes,
such as the .md file changes.
- Change the gpg handling to stop prompting for a passphrase since gpg
now makes use of gpg-agent (and the old gpg script is apparently not
passing through fd 2 that git needs to get status).
Wayne Davison [Tue, 2 Jun 2020 01:49:15 +0000 (18:49 -0700)]
Move the CSUM defines.
Wayne Davison [Tue, 2 Jun 2020 01:36:22 +0000 (18:36 -0700)]
A few style tweaks.
Jorrit Jongma [Mon, 25 May 2020 16:16:19 +0000 (18:16 +0200)]
Move OpenSSL-related MD4/5 defines and imports to lib/mdigest.h
Works just as well, prevents having to repeat them across files
Wayne Davison [Sun, 31 May 2020 22:25:11 +0000 (15:25 -0700)]
Some improvements for --msgs2stderr and --daemon.
- Set am_daemon to -1 (from 1) when the daemon is run via rsh.
- Only disable --msgs2stderr for a normal (socket) daemon.
- Forward a -q to the server if --msgs2stderr was also specified.
- Added --no-msgs2stderr option to allow it to be overridden.
Wayne Davison [Sat, 30 May 2020 12:54:09 +0000 (05:54 -0700)]
Mention the latest NEWS items.
Wayne Davison [Sat, 30 May 2020 12:53:59 +0000 (05:53 -0700)]
Avoid noop_io_until_death() if --msgs2stderr was specified.
benrubson [Tue, 26 May 2020 14:06:07 +0000 (16:06 +0200)]
Search for double-fuzzy files only when needed
Wayne Davison [Sat, 30 May 2020 00:35:56 +0000 (17:35 -0700)]
Silence a strncpy() warning.
Wayne Davison [Fri, 29 May 2020 21:38:19 +0000 (14:38 -0700)]
Silence some g++ warnings.
Wayne Davison [Fri, 29 May 2020 21:37:49 +0000 (14:37 -0700)]
Give configure's snprintf() test a guaranteed short string at the start.
Wayne Davison [Fri, 29 May 2020 21:18:08 +0000 (14:18 -0700)]
Silence gcc7.1 warnings about snprintf().
Wayne Davison [Fri, 29 May 2020 03:48:24 +0000 (20:48 -0700)]
Fix regex issue due to python 3.8 bug.
Wayne Davison [Fri, 29 May 2020 03:21:54 +0000 (20:21 -0700)]
A few minor xxhash changes.
Wayne Davison [Thu, 28 May 2020 20:42:47 +0000 (13:42 -0700)]
Merge pull request #5 from benrubson/daemonstats
Have daemon log data sent/received even when exiting with an error.
Wayne Davison [Thu, 28 May 2020 19:46:46 +0000 (12:46 -0700)]
Add stub for canonical_checksum().
Wayne Davison [Thu, 28 May 2020 19:05:54 +0000 (12:05 -0700)]
The xxh* checksums don't need to be reversed on output.
Wayne Davison [Thu, 28 May 2020 18:40:52 +0000 (11:40 -0700)]
Switch to using LZ4_compress_default().
Wayne Davison [Thu, 28 May 2020 07:41:39 +0000 (00:41 -0700)]
Use MSG_FLUSH in a couple more spots.
Wayne Davison [Thu, 28 May 2020 06:35:25 +0000 (23:35 -0700)]
Merge pull request #2 from benrubson/flush
Help final error messages get to the sender. Is particularly helpful when talking with an older rsync client. Adds a new flush type of MSG_FLUSH.
Wayne Davison [Wed, 27 May 2020 05:28:22 +0000 (22:28 -0700)]
A couple minor tweaks.
benrubson [Tue, 26 May 2020 17:53:25 +0000 (19:53 +0200)]
Log data sent/received even if error
Wayne Davison [Tue, 26 May 2020 14:38:45 +0000 (07:38 -0700)]
Simplify the capabilities array.
benrubson [Tue, 26 May 2020 14:24:30 +0000 (16:24 +0200)]
Correctly send last error to sender
Wayne Davison [Tue, 26 May 2020 09:14:05 +0000 (02:14 -0700)]
Convert NEWS & OLDNEWS into .md files.
Wayne Davison [Tue, 26 May 2020 06:42:33 +0000 (23:42 -0700)]
Fix the parsing of the --version capabilities.
Wayne Davison [Tue, 26 May 2020 06:41:39 +0000 (23:41 -0700)]
Install yodl for the CI builds.
Wayne Davison [Tue, 26 May 2020 06:33:11 +0000 (23:33 -0700)]
Get indent right.
Wayne Davison [Tue, 26 May 2020 05:43:33 +0000 (22:43 -0700)]
Improve output of capabilities in --version list.
It now wraps better as the "no " prefixes change, and it makes it easier
to maintain patches that add items into the capabilities list.
Wayne Davison [Tue, 26 May 2020 04:27:21 +0000 (21:27 -0700)]
Remove xxhash from capabilities list.
It's now listed in the "Checksum list:" output.
Wayne Davison [Tue, 26 May 2020 04:21:58 +0000 (21:21 -0700)]
Fix a couple deb names.
Wayne Davison [Tue, 26 May 2020 04:17:51 +0000 (21:17 -0700)]
Improve CI setup.
Wayne Davison [Tue, 26 May 2020 01:03:02 +0000 (18:03 -0700)]
Improve how negotiated info affects batch files.
Wayne Davison [Mon, 25 May 2020 23:57:47 +0000 (16:57 -0700)]
Tweak the --zz option to --zc (aka --compress-choice).
Wayne Davison [Mon, 25 May 2020 22:00:30 +0000 (15:00 -0700)]
Rename a couple files to .md
Wayne Davison [Mon, 25 May 2020 21:59:51 +0000 (14:59 -0700)]
Add some extra indent.
Wayne Davison [Mon, 25 May 2020 21:43:25 +0000 (14:43 -0700)]
Create ccpp.yml
Wayne Davison [Mon, 25 May 2020 21:01:52 +0000 (14:01 -0700)]
Two sparse fixes from Yuxuan Shui.
- Make "len" parameter of do_punch_hole an OFF_T.
- Clear sparse_past_write in sparse_end(), otherwise when write_sparse()
is called for the next file, do_punch_hole() will be called with a pos
that's not actually the current position in file, causing it to fail.
Wayne Davison [Mon, 25 May 2020 20:43:25 +0000 (13:43 -0700)]
Include lz4 compression support.
Based on a patch that was emailed to me without a valid return address.
Wayne Davison [Mon, 25 May 2020 20:31:30 +0000 (13:31 -0700)]
Include zstd compression support.
Based on a patch by Sebastian A. Siewior. Fixes bug #14338.
Wayne Davison [Mon, 25 May 2020 20:02:56 +0000 (13:02 -0700)]
Fix handling of a compressor that has no off_level.
Wayne Davison [Mon, 25 May 2020 19:45:28 +0000 (12:45 -0700)]
Output the default checksum & compress lists in the --version output.
Wayne Davison [Mon, 25 May 2020 18:50:44 +0000 (11:50 -0700)]
A couple minor configure.ac tweaks.
Wayne Davison [Mon, 25 May 2020 17:47:57 +0000 (10:47 -0700)]
Make compression-level handling generic.
Wayne Davison [Mon, 25 May 2020 05:31:28 +0000 (22:31 -0700)]
Some minor tweaks & tidying up.
Wayne Davison [Mon, 25 May 2020 03:38:48 +0000 (20:38 -0700)]
Change odd-ball map_ptr() to use remainder like the others.
Jorrit Jongma [Sun, 24 May 2020 22:30:55 +0000 (00:30 +0200)]
Improve performance of file_checksum()
Previously files were hashed in blocks of CSUM_CHUNK (64) bytes. This
causes significant overhead. The CSUM_CHUNK define cannot be changed as
md5.c depends on it, but there is no obvious reason to use it in
file_checksum(). By using CHUNK_SIZE (32 kB) instead, in some test
cases throughput more than doubles.
Wayne Davison [Mon, 25 May 2020 03:19:15 +0000 (20:19 -0700)]
More NEWS updates.
Wayne Davison [Mon, 25 May 2020 02:51:11 +0000 (19:51 -0700)]
Prefer zlibx compression consistently instead of having 2 possible default preference orders.
Wayne Davison [Mon, 25 May 2020 02:35:22 +0000 (19:35 -0700)]
Don't auto-foward debug options to the server side anymore.
Wayne Davison [Mon, 25 May 2020 02:16:05 +0000 (19:16 -0700)]
Fix -z choice with older rsyncs.
Wayne Davison [Mon, 25 May 2020 01:08:09 +0000 (18:08 -0700)]
A few more compression tweaks.
Wayne Davison [Sun, 24 May 2020 22:45:59 +0000 (15:45 -0700)]
More compress changes
- Add the zlibx (external-code compatible) compression name.
- Re-enable zlib support with the external library so it can be
tried as a fallback if zlibx isn't available.
- Add --compress-choice=STR (aka -zz=STR) option.
- Make --cc=STR an alias for --checksum-choice=STR.
- Hook up the new compression negotiation logic.
Wayne Davison [Sun, 24 May 2020 20:34:47 +0000 (13:34 -0700)]
Fix negotiation of none & improve NSTR debug msgs.
Wayne Davison [Sun, 24 May 2020 19:56:00 +0000 (12:56 -0700)]
Add support for negotiated checksum names.
Wayne Davison [Sun, 24 May 2020 09:07:43 +0000 (02:07 -0700)]
Make xxh64 the "main_name" for the current xxhash.
Wayne Davison [Sun, 24 May 2020 09:04:14 +0000 (02:04 -0700)]
Fix some warnings.
Wayne Davison [Sun, 24 May 2020 05:05:16 +0000 (22:05 -0700)]
Some more checksum improvements
- Add/improve --debug=CSUM2 messages.
- Add an "xxh64" alias for "xxhash" name because we should be
getting a few more xxhash variations in the future.
- Tweak the matching code to handle entries that have multiple
names.
- Tweak some of the vars/defines.
Wayne Davison [Sun, 24 May 2020 04:46:15 +0000 (21:46 -0700)]
Improve some early debug-message newlines.
Avoid a newline issue during the output of --DEBUG=CSUM info from
both the server and the client -- we need to output the full message
with its newline as much as possible.
Wayne Davison [Sun, 24 May 2020 04:43:53 +0000 (21:43 -0700)]
Avoid re-evaluating the args of SIVAL* w/CAREFUL_ALIGNMENT.
Wayne Davison [Sun, 24 May 2020 01:25:11 +0000 (18:25 -0700)]
Unify the checksum context memory, since we only use one at a time.
Wayne Davison [Sat, 23 May 2020 23:22:36 +0000 (16:22 -0700)]
Fixed the use of openssl MD4 for transfer checksums.
Wayne Davison [Sat, 23 May 2020 19:26:06 +0000 (12:26 -0700)]
Optional openssl support for MD4 pre-transfer checksums (but, sadly, not transfer checksums).
Wayne Davison [Sat, 23 May 2020 16:23:01 +0000 (09:23 -0700)]
Add optional use of the openssl crypto lib for MD5.
Wayne Davison [Sat, 23 May 2020 06:26:25 +0000 (23:26 -0700)]
Mention a few more news items.
Wayne Davison [Sat, 23 May 2020 05:40:54 +0000 (22:40 -0700)]
Add "input" handling for cmd_txt_*() pkglib.py.
Jorrit Jongma [Fri, 22 May 2020 17:38:37 +0000 (19:38 +0200)]
Optimized assembler version of md5_process() for x86-64
Originally created by Marc Bevand and placed in the public domain.
Enable/disabled via the same --enable-simd configure switch as
the rolling checksum optimizations.
Wayne Davison [Sat, 23 May 2020 02:26:30 +0000 (19:26 -0700)]
A little more safety in negotiate_checksum().
Wayne Davison [Sat, 23 May 2020 00:40:08 +0000 (17:40 -0700)]
Promoting xxhash support.
Wayne Davison [Fri, 22 May 2020 22:06:20 +0000 (15:06 -0700)]
Some checksum improvements
- Improve csum negotation logic.
- Define the csum names in a single structure.
- Add --debug=CSUM.
Jorrit Jongma [Fri, 22 May 2020 11:03:55 +0000 (13:03 +0200)]
AVX2 optimized version of get_checksum1() for x86-64
Additionally restructures build switches and defines from SSE2 to SIMD,
to allow potential reuse should patches become available with SIMD
instructions for other processor architectures.
(Some minor tweaks of Jorrit's patch to avoid requiring GNU make and to
avoid C++ comments in .c files.)
Wayne Davison [Fri, 22 May 2020 15:23:26 +0000 (08:23 -0700)]
Checksum negotiation & more bits for compat_flags
- Add checksum negotiation to the protocol so that we can easily add new
checksum algorithms and each will be used when both sides support it.
- Increase the size of the compat_flags value in the protocol from a
byte to an int.
Wayne Davison [Fri, 22 May 2020 15:10:41 +0000 (08:10 -0700)]
Fix a bug in the writing of the batch.sh file
Fix the code that writes the options and the default destination path
into the batch.sh file to be able to handle options being specified
after source/dest args.
Jorrit Jongma [Tue, 19 May 2020 12:52:40 +0000 (14:52 +0200)]
SSE2/SSSE3 optimized version of get_checksum1() for x86-64
Requires compilation using GCC C++ front end, build scripts have been
modified accordingly. C++ is only used when the optimization is enabled
(g++ as compiler, x86-64 build target, --enable-sse2 is passed to
configure).
(Wayne made a few tweaks, including making it disabled by default.)
Wayne Davison [Mon, 18 May 2020 07:03:09 +0000 (00:03 -0700)]
Tweak the accept/refuse strings a bit.
Wayne Davison [Mon, 18 May 2020 06:26:13 +0000 (23:26 -0700)]
Add back a lost "*" and document the refusing of log-file* opts.
Wayne Davison [Mon, 18 May 2020 05:16:58 +0000 (22:16 -0700)]
More manpage improvements for "refuse options".
Wayne Davison [Mon, 18 May 2020 04:29:11 +0000 (21:29 -0700)]
Add the ability to negate matches for the daemon's "refuse options".
Wayne Davison [Mon, 18 May 2020 03:21:45 +0000 (20:21 -0700)]
Need to flush early errors before we exit.
Wayne Davison [Sun, 17 May 2020 21:39:18 +0000 (14:39 -0700)]
Some pkglib improvements.
Wayne Davison [Wed, 13 May 2020 07:20:03 +0000 (00:20 -0700)]
A couple more manpage fixes.