Rsync for the layman... there's so much crap info out there

General Linux Question
1

I love using rsync…

I use it all the time… it’s a lifesaver!

I will give kudos to things like tar, which can be considerably faster…

e.g. compare these :

$SOURCE
$DEST

Where $SOURCE could be local, or remove, but proably contains gigs (or tibs) or data… $DEST could be remote, or local, and will contain those gigs (or tibs) of data, eventually…

This I have proven is faster than rsync :

cd $SOURCE ; tar cvpf - * | ( cd $DEST ; tar xvpf - )

If there was a network in there, then “tar czvfp”, followed by “tar xzvfp” might make it even faster - but unlikely e.g. if it was just NFS or SMB, cpu overhead might rob any network overhead cost saving…

But - what happens if you need to update / refresh from some kinda “Delta” - well rsync is your tool… piece of cake :

cd $DEST ; rsync -av $SOURCE/ .

But what happens if I wanna delete stuff in $DEST, that might have got copied last time, but has since been deleted off $SOURCE in the intervening period?

Honestly - I don’t know. I’ve just been doing some reading on sourceforge / stack overflow, and it’s DOWNRIGHT confusing, and there’s always uninformed morons piping up and offering opinions, which you read further down are just PLAIN WRONG! Shouldn’t their posts get deleted and them get banned from the whole of the intertoobs?

Some of the recommended options, recommended by some MORONS, will actually delete a file form the f–king SOURCE if it doesn’t exist in the DEST!

Can of worms anyway…

I want to use rsync to keep an exernal SSD refreshed with stuff, for my employer’s MacBook, which has a SHITTY policy of banning writing to external media! That cheeses me off! And it’s got such a tiny shonky little SSD (256) there’s not a chance in hell it could host MacOs AND my music collection… But if I’m going to have script a refresh, I’d rather also delete stuff on the destination, if I’ve already removed it form the source. Is that so hard?

I started writing a script to run in Mac 1, to refresh the external USB SSD contents, to view on Mac 2…

Hidden incase there’s prudes here who can’t handle some colourful four letter words now and then :

Summary 
#!/usr/bin/env bash
# fucking corporate piece of shit rules and JAMF SHITE on MacOS
# enterprising sods come up with workarounds...
# in this case we're rsync'ing mp3 / music collection off one Mac, to look at
# read-only on a SOE MOE bullshit macbook
SRC_CUNT=~/ResilioSync/Music
DESTCUNT=/Volumes/SHJTSHEAU/Music
# rsync -avhn --delete $SRC_CUNT/. $DESTCUNT/.

Sorry - I can’t help but swearing…

Now - I’m still unsure, because there’s so many F–KING opinions expressed as “knowledge” on tech help forums - that look dodgy and risky, which one is right???

What will “–delete” do if used BEFORE $SOURCE???

2

Don’t quote me on this, as I haven’t tried it explicitly, but as far as I know --delete should only remove from the destination.

 --del                   an alias for --delete-during
 --delete                delete extraneous files from dest dirs
 --delete-before         receiver deletes before transfer (default)
 --delete-during         receiver deletes during xfer, not before
 --delete-delay          find deletions during, delete after
 --delete-after          receiver deletes after transfer, not before
 --delete-excluded       also delete excluded files from dest dirs

If SRC and DEST are synced, then deletions may also happen on the SRC side of the situation.

3

If you want exactly that, use – delete as you suggest
but
be aware it is no longer what people want in a backup
If you accidentally delete something in $SOURCE it will disappear from $DEST as well.
Might be OK for a one-off copyj

I dont think rsync was designed to be used in both directions. Too complicated . Stick to simple stuff you can manage

Neville

4

I just read the man page, normally I don’t bother with them, too obtuse, WAY TOO MUCH information…

But I found this tiny example (in the man page):

       I mirror a directory between my "old" and "new" ftp sites with the command:

           rsync -az -e ssh --delete ~ftp/pub/samba nimbus:"~ftp/pub/tridge"

I remember Solaris man pages ALWAYS had good examples of nearly every option at the bottom of each man page - but not so much with Linux man pages.

That looks to do everything I need… Keeping files isn’t a major problem, but I do want it to look as much like the source as possible - so I will delete.

What’s interesting about this man page, and when you look right at the bottom at the AUTHOR section, Andrew Tridgell (he’s Autralian, and not only wrote rsync, he also wrote Samba!) is one…

He’s either infamous, or famous…

He tried (semi-successfully) to reverse engineer the source control system (Bitkeeper) that Linus was using to manage the Linux kernel. The owner of that software, who let Linus use it for free, PULLED it and refused to let anyone use it in “retaliation” for Tridge reverse engineering it. Linus fired off a “pithy” email to Andrew Tridgell - and in a huff, he quickly started developing GIT.

If “Tridge” hadn’t transgressed, we might never had GIT! Thanks Tridge!

And I wonder if anyone’s still using Bitkeeper? If the copyright owner hadn’t kicked up a stink, and refused free use of his product, people might still be using it, Linus might still be using it!

5

If you really want two filesystems kept exactly the same, with files being dynamically edited in both, rsync is not the tool for bidirectional synchronisation. Try Unison… I have not used it, but it is supposed to deal with that situation properly without having to bend your brain.

Nice story about git
Neville

6

I used to use it (Unison) - all the time, between Debian Jessie systems…

But it’s problematic if you then want to start using some other “non Jessie” platform - Unison breaks if the remote side is too different a version…

And I’m not using rsync as a remote thing anyway…

I think rsync is the right tool for my use case :

╭─x@titan ~/bin  ‹main*› 
╰─➤  bat synxunt.bash
───────┬────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
       │ File: synxunt.bash
───────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
   1   │ #!/usr/bin/env bash
   2   │ # f--king corporate piece of crap rules and JAMF crap on MacOS
   3   │ # enterprising sods come up with workarounds...
   4   │ # in this case we're rsync'ing mp3 / music collection off one Mac, to look at
   5   │ # read-only on a SOE MOE garbage macbook
   6   │ OS=$(uname -s)
   7   │ if [ ! $OS == "Darwin" ] ; then
   8   │     echo "only interested in running this on a Mac..."
   9   │     exit 1
  10   │ fi
  11   │ SRC_THING=/Users/x/ResilioSync/Music
  12   │ DESTTHING=/Volumes/SHJTSHEAU/Music
  13   │ if [ ! -d $DESTTHING ] ; then
  14   │     echo "not there to copy too moron!..."
  15   │     exit 1
  16   │ elif [ ! -d $SRC_THING ] ; then
  17   │     echo "source not there to copy from you moron!"
  18   │     exit 1
  19   │ fi
  20   │ rsync -avhn --delete $SRC_THING/. $DESTTHING/. 
───────┴────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Which does the job… just gotta remember to do it - especially if I’m planning on “Road Warrioring” it with the work Macbook somewhere…

7

Thanks,
We need to know that. I will not waste my time.

8

Yeah - as an example, as well as Debian Jessie (on ARM) systems, I had Ubuntu 16…

I had to disable Ubuntu from installing it from its repos, and locate the same (or very similar revision) version as a DEB file for x86, and install that on Ubuntu, then block Ubuntu from updating it…

I did have a few shell scripts that used Unison, but I seem to have deleted them (found them).

That version incompatibility was what convinced me to stop using it - plus, I also found my own self hosting cloud solution (Resilio Sync) to run on all platforms (Dropbox doesn’t really run “properly” on non x86 kit).

That version incompatibility may not be a thing any more, I haven’t used Unison since around 2017???

Note also - unison is not in Fedora’s default repo’s, so another reason to cross it off my list…

I tried updating my rsync script above so I could run it on on Linux, it sorta runs, but for some reason there’s bizarro permissions ownership on a bunch of files on there 'cause it’s HFS+ format - so it has trouble writing files… Give up… too hard basket… works from one mac to another… I’ll rest…

1 Like
9

Back to rsync :

I really kinda have a love hate relationship with rsync…

Its a lifesaver and timesaver - BUT - and this is a HUGE BUT in 48 point Helvetica Bold…

Why does it do crappy things like try to copy ALL the same data, over again, even though it already exists?

Most times - nearly ALL times, e.g. I’m rsync’ing between something with some kinda POSIX compliant filesystem, e.g. ext4 on a remote system to a folder on an ext3 fileystem… and it works… only copies the freshest shit…

rsync -av $SOURCE $DESTINATION
and most times I use the ‘/.’ notation :
rsync -av $SOURCE/. $DESTINATION/.

And e.g.

rsync -av --delete --exclude .sync $SOURCE/. $DESTINATION/.

And it works - everytime… And the ADB “adb-sync” version also works just as well…

But then I get complacent and expect the same behaviour when rsync’ing to Fat32 or some other mongrel-of-a-stopgap filesystem (I can’t believe it’s 2022 and we’re still using it!) and I have to then remember all the other switches…

So I have to remember to EXPLICITLY state “–ignore-existing” :

rsync -av --ignore-existing --delete --exclude ".sync" ~/ResilioSync/Music/. /media/x/413B-8828/Music/.

What cheeses me off, is that “adb-sync”, copies just fine to Fat32 system mounted on my Samsung phone - and I don’t need to worry about or consider “existing” files, it knows not to copy, again, something that’s already there… Also - adb-sync is also smart enough to ignore “.sync” anyway (that’s a folder that Resilio Sync creates to store file changes, e.g. make backups where files have been modified or deleted).

There’s also this “switch” for rsync, but I really have NO IDEA what it does, but saw it on several of the many forums / stackexchange posts I trawled through :

“--no-r”

Which sometimes seems to do what I want, or need, othertimes it does sweet FA…

10

Cant find that anywhere on my system.

I always use the same options
rsync -aAXvH
with or without excludes
because all I do is copy OS or data between disks.
I avoid mistakes by keeping a list of examples… cant trust memory with important things like moving data.
I have yet to meet anyone who needs the whole 200 options. Its worse than gcc.

1 Like
11

These days - if I found some tricky finicky piece of cli crap that had me scratching my head - I save it into a non-executable bash script in my scripts folder… When I need it again I just “grep rsync ~/bin/*.bash”…

12

Well thats neat. I just have a Directory with a heap of clearly named files.

13

I keep my scripts folder sync’d across all my computers, even my iPad(s) and Android phone… I can’t run them on either (too much hard work to make them executable in TermUx on Android, but I’m sure it’s do-able - and forget about it on iPadOS)… Naturally enough, I’ve mentioned the product often enough, but I sync my scripts folder across about 12 devices using Resilio Sync…

Yeah - I now have three iPads… The first one (circa 2012 Gen 3) hasn’t been charged or powered up for months or years… But I use my 2017/18 12.9" iPad Pro as a 2nd monitor on my MacBook Pro M1 (using a “feature” called “Sidecar”) and I’m using a 5th gen iPad mini as an e-book reader - but I can use it for work stuff, if and when it’s the only device in my backpack

Having everything “volatile” sync’d, or stored on RAID on my NAS, means I’ve nearly always got some form of backup (yeah RAID’s not a backup solution - but file versioning in Resilio Sync IS)… A complete rebuild of a desktop, the biggest “job” data-wise, is re-installing my games library from Steam…

Just hit a kinda “wall” with Resilio Sync - I have a larger sync folder with images and documents and binaries, just trying to sync it to the single core RPi Zero W pretty much keeps sorta/kinda crashing the Pi Zero (it doesn’t die, it just stops responding for minutes at a time) … it’s about ~60 GB, but the Pi Zero’s only got a single core… This will be addressed when I move its boot SD card to a Pi Zero 2 W I’ve got coming in the mail (4 cores! on a credit card sized computer! 512 MB is plenty for a headless application, for my use case anyway).

14

I must try one of these Pi things. .
I dont sync anything. It exists in one place, and is backed up externally . Thats all, no fancy stuff with files.

That Termux thing is second rate. Try GnURootDebian.