Updating rsync from 2.x to 3.x (because GNU v3 blows chunks)

[donmontalvo]

Apologies, title should read "Updating rsync from 2.x to 3.x (because GPLv3 blows chunks)"

Apple bundles rsync 2.6.9 with macOS, and never went past that version because of changes from GPLv2 to GPLv3 .

Without diving into the weeds on why, we needed to update rsync to 3.x, because it can preserve macOS metadata (resource forks [don't laugh PostScript fonts are still a thing], etc.).

While you can use -E with rsync 2.6.9 to preserve resource forks, a side effect is you lose the ability to run delta syncs. So every time you run rsync, it copies the whole enchilada.

Using rsync 3.x, with -X preserves macOS metadata including resource forks, but you don't lose full/delta syncing.

The concern we always have when updating anything bundled with macOS is we don't want to cause any issues by tinkering, luckily Apple's /usr/bin/rsync is not touched when you install the new version which installs in /usr/local/bin/rsync.

But...that's a problem too, since it makes the new version the one that's used when you invoke rsync in Terminal...which can piss off developers, and cause problems with any scripts or apps that expect rsync to just work.

We decided to install into /Library/COMPANY/Applications/rsync folder, where the binary path would be /Library/COMPANY/Applications/rsync/bin/rsync.

On a computer that has Xcode installed, we updated to 3.0.6 (since that's the version Mike Bombich trusts enough to bundle with CCC), and applied the patches related to preserving macOS metadata...

# mkdir -p /Library/COMPANY/Applications/rsync
# cd /Library/COMPANY/Applications/rsync
# curl -O http://rsync.samba.org/ftp/rsync/src/rsync-3.0.6.tar.gz
# tar -xzvf rsync-3.0.6.tar.gz
# rm rsync-3.0.6.tar.gz
# curl -O http://rsync.samba.org/ftp/rsync/src/rsync-patches-3.0.6.tar.gz
# tar -xzvf rsync-patches-3.0.6.tar.gz
# rm rsync-patches-3.0.6.tar.gz
# cd rsync-3.0.6
# patch -p1 <patches/fileflags.diff
# patch -p1 <patches/crtimes.diff
# ./prepare-source
# ./configure --prefix=/Library/COMPANY/Applications/rsync
# make
# make install
# cd /Library/COMPANY/Applications/rsync
# rm -Rf rsync-3.0.6

To confirm the new version run:

# /Library/COMPANY/Applications/rsync/bin/rsync --version
rsync  version 3.0.6  protocol version 30
Copyright (C) 1996-2009 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
    64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
    socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
    append, ACLs, xattrs, no iconv, symtimes, file-flags

rsync comes with ABSOLUTELY NO WARRANTY.  This is free software, and you
are welcome to redistribute it under certain conditions.  See the GNU
General Public Licence for details.

Apple confirmed the newly installed /Library/COMPANY/Applications/rsync/ folder is self contained, so it can be deployed, making it possible to use the new version of rsync without stomping on anything (and pissing off your developers), as long as you specify the new path (can't just use rsync in your commands, else you'll be invoking the old version).

Once installed, this shows how macOS will continue to use bundled version:

$ rsync --version | head -1
rsync  version 2.6.9  protocol version 29
$ /Library/COMPANY/Applications/rsync/bin/rsync --version | head -1
rsync  version 3.0.6  protocol version 30

Might also be worth passing word along to your developers so they know the new version is installed, pretty sure they'll use it if they into the macOS metadata problems we ran into.

Now just update your scripts to replace every instance of rsync with /Library/COMPANY/Applications/rsync/bin/rsync and you should be all set.

EDIT: Removed this line: "# patch -p1 <patches/hfs-compression.diff" since it does not apply to this version.

[donmontalvo]

Next phase...getting the commands right on the switch from rsync 2.6.9 to 3.0.6...

macOS - updating scripts - moving from rsync 2.6.9 to 3.0.6

[djdavetrouble]

This is great, I have been doing something similar with a homebrew recipe. One question, why that path and not /usr/local/bin or /usr/local/$company/bin or something like that? Also, do you have backup / restore script examples for JAMF ? I have been looking at this one by @loceee https://github.com/loceee/OSXCasperScripts/tree/master/BackupRestoreUsers

[djdavetrouble]

@donmontalvo I don't see hfs-compression in the patches download:

acls.diff           link-by-hash.diff
adaptec_acl_mods.diff       log-checksum.diff
atimes.diff         munge-links.diff
backup-dir-dels.diff        nameconverter.diff
catch_crash_signals.diff    netgroup-auth.diff
checksum-reading.diff       omit-dir-changes.diff
checksum-updating.diff      openssl-support.diff
checksum-xattr.diff     osx-xattr-nodev.diff
copy-devices.diff       preallocate.diff
crtimes.diff            remote-option.diff
cvs-entries.diff        slow-down.diff
daemon-forward-lookup.diff  slp.diff
date-only.diff          soften-links.diff
db.diff             source-backup.diff
detect-renamed-lax.diff     source-filter_dest-filter.diff
detect-renamed.diff     sparse-block.diff
downdate.diff           stdout.diff
dparam.diff         time-limit.diff
drop-cache.diff         transliterate.diff
fileflags.diff          tru64.diff
fsync.diff          usermap.diff
ignore-case.diff        xattrs.diff

[djdavetrouble]

Been looking at this a little big more. I am not sure why Bombich uses 3.0.6 but he does not use hfs-compression patches, he uses these patches and some of his own modifications: "crtimes.diff, fileflags.diff, log-checksum.diff, and backup-dir-dels.diff] and my modifications"

Your instructions won't work because there is no hfs-compression.diff in the patches file that you reference. It does exist in 3.1.3, however.

If you want to save yourself some time, just download CCC and copy Mike's precompiled rsync from Contents > MacOS > rsync. Not sure why Mike doesn't use hfs-compression anymore since he reportedly wrote the original diff.

[djdavetrouble]

Just an aside as a response to the perplexing dos equis gnu meme you posted, Linus' objection to GPLv3 was that it left no room for DRM in the linux kernal, not that it sucks. As a matter of fact, Linux owes much of its success to the groundwork that gnu laid. Linux was RELEASED under the GPL even. Also, Linux refers to a Kernal, usually built with tons of gnu components into a full operating system distribution.

Love it or hate it, you have to respect Stallman for drawing the line and never moving or crossing it.

[djdavetrouble]

Follow up - hfs-compression is marked as broken as of 3.1.3. I ended up repackaging the version from homebrew for my project and repackaged it into /usr/local/bin

[donmontalvo]

@djdavetrouble Thanks for the heads up. That option was used during testing of several versions, but isn't needed for 3.0.6. It is safe to remove. I updated he original post:

# patch -p1 <patches/hfs-compression.diff

Mike's patch patch is available if hfs-compression.diff Is needed.

https://youtu.be/7SofmXIYvGM?t=47m28s

[edgardeveij]

You can also 'force' the use of your newly compiled rsync in terminal sessions by adding a file in /etc/paths.d

In this case, call this file rsync and put the path inside.

After you've done this, you can call your version directly from the command line. No need for entering paths.

[donmontalvo]

@edgardevelj thanks, yes, many ways to skin the cat. Its not something we're deploying for users, else we'd go that route. We deployed this new(er) version of rsync to our team's support folder, so we can call it in our script. If users poke around and find it, they're welcome to use it but we can't/won't support their use of it. Developers or any other folks with admin rights know how to get it into the paths they want to install it in.