Rsync for the layman... there's so much crap info out there

I love using rsync…

I use it all the time… it’s a lifesaver!

I will give kudos to things like tar, which can be considerably faster…

e.g. compare these :

$SOURCE
$DEST

Where $SOURCE could be local, or remove, but proably contains gigs (or tibs) or data… $DEST could be remote, or local, and will contain those gigs (or tibs) of data, eventually…

This I have proven is faster than rsync :

cd $SOURCE ; tar cvpf - * | ( cd $DEST ; tar xvpf - )

If there was a network in there, then “tar czvfp”, followed by “tar xzvfp” might make it even faster - but unlikely e.g. if it was just NFS or SMB, cpu overhead might rob any network overhead cost saving…

But - what happens if you need to update / refresh from some kinda “Delta” - well rsync is your tool… piece of cake :

cd $DEST ; rsync -av $SOURCE/ .

But what happens if I wanna delete stuff in $DEST, that might have got copied last time, but has since been deleted off $SOURCE in the intervening period?

Honestly - I don’t know. I’ve just been doing some reading on sourceforge / stack overflow, and it’s DOWNRIGHT confusing, and there’s always uninformed morons piping up and offering opinions, which you read further down are just PLAIN WRONG! Shouldn’t their posts get deleted and them get banned from the whole of the intertoobs?

Some of the recommended options, recommended by some MORONS, will actually delete a file form the f–king SOURCE if it doesn’t exist in the DEST!

Can of worms anyway…

I want to use rsync to keep an exernal SSD refreshed with stuff, for my employer’s MacBook, which has a SHITTY policy of banning writing to external media! That cheeses me off! And it’s got such a tiny shonky little SSD (256) there’s not a chance in hell it could host MacOs AND my music collection… But if I’m going to have script a refresh, I’d rather also delete stuff on the destination, if I’ve already removed it form the source. Is that so hard?

I started writing a script to run in Mac 1, to refresh the external USB SSD contents, to view on Mac 2…

Hidden incase there’s prudes here who can’t handle some colourful four letter words now and then :

Summary
#!/usr/bin/env bash
# fucking corporate piece of shit rules and JAMF SHITE on MacOS
# enterprising sods come up with workarounds...
# in this case we're rsync'ing mp3 / music collection off one Mac, to look at
# read-only on a SOE MOE bullshit macbook
SRC_CUNT=~/ResilioSync/Music
DESTCUNT=/Volumes/SHJTSHEAU/Music
# rsync -avhn --delete $SRC_CUNT/. $DESTCUNT/.

Sorry - I can’t help but swearing…

Now - I’m still unsure, because there’s so many F–KING opinions expressed as “knowledge” on tech help forums - that look dodgy and risky, which one is right???

What will “–delete” do if used BEFORE $SOURCE???

Don’t quote me on this, as I haven’t tried it explicitly, but as far as I know --delete should only remove from the destination.

 --del                   an alias for --delete-during
 --delete                delete extraneous files from dest dirs
 --delete-before         receiver deletes before transfer (default)
 --delete-during         receiver deletes during xfer, not before
 --delete-delay          find deletions during, delete after
 --delete-after          receiver deletes after transfer, not before
 --delete-excluded       also delete excluded files from dest dirs

If SRC and DEST are synced, then deletions may also happen on the SRC side of the situation.

If you want exactly that, use – delete as you suggest
but
be aware it is no longer what people want in a backup
If you accidentally delete something in $SOURCE it will disappear from $DEST as well.
Might be OK for a one-off copyj

I dont think rsync was designed to be used in both directions. Too complicated . Stick to simple stuff you can manage

Neville

I just read the man page, normally I don’t bother with them, too obtuse, WAY TOO MUCH information…

But I found this tiny example (in the man page):

       I mirror a directory between my "old" and "new" ftp sites with the command:

           rsync -az -e ssh --delete ~ftp/pub/samba nimbus:"~ftp/pub/tridge"

I remember Solaris man pages ALWAYS had good examples of nearly every option at the bottom of each man page - but not so much with Linux man pages.

That looks to do everything I need… Keeping files isn’t a major problem, but I do want it to look as much like the source as possible - so I will delete.

What’s interesting about this man page, and when you look right at the bottom at the AUTHOR section, Andrew Tridgell (he’s Autralian, and not only wrote rsync, he also wrote Samba!) is one…

He’s either infamous, or famous…

He tried (semi-successfully) to reverse engineer the source control system (Bitkeeper) that Linus was using to manage the Linux kernel. The owner of that software, who let Linus use it for free, PULLED it and refused to let anyone use it in “retaliation” for Tridge reverse engineering it. Linus fired off a “pithy” email to Andrew Tridgell - and in a huff, he quickly started developing GIT.

If “Tridge” hadn’t transgressed, we might never had GIT! Thanks Tridge!

And I wonder if anyone’s still using Bitkeeper? If the copyright owner hadn’t kicked up a stink, and refused free use of his product, people might still be using it, Linus might still be using it!

If you really want two filesystems kept exactly the same, with files being dynamically edited in both, rsync is not the tool for bidirectional synchronisation. Try Unison… I have not used it, but it is supposed to deal with that situation properly without having to bend your brain.

Nice story about git
Neville

I used to use it (Unison) - all the time, between Debian Jessie systems…

But it’s problematic if you then want to start using some other “non Jessie” platform - Unison breaks if the remote side is too different a version…

And I’m not using rsync as a remote thing anyway…

I think rsync is the right tool for my use case :

╭─x@titan ~/bin  ‹main*› 
╰─➤  bat synxunt.bash
───────┬────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
       │ File: synxunt.bash
───────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
   1   │ #!/usr/bin/env bash
   2   │ # f--king corporate piece of crap rules and JAMF crap on MacOS
   3   │ # enterprising sods come up with workarounds...
   4   │ # in this case we're rsync'ing mp3 / music collection off one Mac, to look at
   5   │ # read-only on a SOE MOE garbage macbook
   6   │ OS=$(uname -s)
   7   │ if [ ! $OS == "Darwin" ] ; then
   8   │     echo "only interested in running this on a Mac..."
   9   │     exit 1
  10   │ fi
  11   │ SRC_THING=/Users/x/ResilioSync/Music
  12   │ DESTTHING=/Volumes/SHJTSHEAU/Music
  13   │ if [ ! -d $DESTTHING ] ; then
  14   │     echo "not there to copy too moron!..."
  15   │     exit 1
  16   │ elif [ ! -d $SRC_THING ] ; then
  17   │     echo "source not there to copy from you moron!"
  18   │     exit 1
  19   │ fi
  20   │ rsync -avhn --delete $SRC_THING/. $DESTTHING/. 
───────┴────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Which does the job… just gotta remember to do it - especially if I’m planning on “Road Warrioring” it with the work Macbook somewhere…

Thanks,
We need to know that. I will not waste my time.

Yeah - as an example, as well as Debian Jessie (on ARM) systems, I had Ubuntu 16…

I had to disable Ubuntu from installing it from its repos, and locate the same (or very similar revision) version as a DEB file for x86, and install that on Ubuntu, then block Ubuntu from updating it…

I did have a few shell scripts that used Unison, but I seem to have deleted them (found them).

That version incompatibility was what convinced me to stop using it - plus, I also found my own self hosting cloud solution (Resilio Sync) to run on all platforms (Dropbox doesn’t really run “properly” on non x86 kit).

That version incompatibility may not be a thing any more, I haven’t used Unison since around 2017???

Note also - unison is not in Fedora’s default repo’s, so another reason to cross it off my list…

I tried updating my rsync script above so I could run it on on Linux, it sorta runs, but for some reason there’s bizarro permissions ownership on a bunch of files on there 'cause it’s HFS+ format - so it has trouble writing files… Give up… too hard basket… works from one mac to another… I’ll rest…

1 Like