The unforgiven and the complacent

Damn - when you screw up with ResilioSync - it doesn’t forgive…

I stupidly sync’d (using rsync) the wrong folder to speed up RSL (ReSilioSync) and screwed up ALL my computers! It had been taking me ALL afternoon to sync a “smallish” folder of shit onto my new Pi Zero 2 W - and I thought - I’ll preseed it direct to the SDCARD - STUPID MISTAKE and FAMOUS LAST WORDS! Instead of thinking “why is it taking so long in the first place”, I decided to speed it up!

Unforgiving piece of crap… Which happily lets me input garbage so I can get garbage out…

Oh well - it mostly doesn’t delete stuff… It’s forced me to completely rethink how I do stuff anyway… And thankfully - I’m working out my notice period (with no actual work to do) and I can devote myself 9-5 sorting out the mess I created…

  1. don’t have folders with the same names! That’s the tantamount GOLDEN RULE
  2. don’t be complacent
  3. DON’T BE COMPLACENT

How did I notice what had happened? My main desktop computer lost its wallpaper (which was file in a symlinked folder on one of my RSL sync’d “shares”).

Not to worry - it’s all there in ~/ResilioSync/$SHARENAME/.sync/Archive…

BUT F–K it’s a headf–k trying to remember where everything goes! Damn - and it’s only the 2nd time this EXACT same thing has happened to me on this desktop build in the last 2 months!

Most recent instance was when I powered up a Pi Zero that I hadn’t booted in months and it (RSL) decided to OVERWRITE a whole bunch of “newer” files (thanks Pi Zero for NOT having a realtime clock! And RSL - just how the f–k does a file with a 1970 timestamp over-write one with a 2022 timestamp?)…

Blame :

  • lack of a clock/realtime clock on Raspberry Pi pieces of crap
  • FAT32’s PURE garbage liabilities

I can almost guarantee I will still be sorting this shit out 24 hours from now… NOT HAPPY

1 Like

Thats very disturbing, I am sorry.

Why dont you draw back a bit and replan what you sync and why.
I never sync anything , I just leave it where I first made it. So I dont really understand why sync is so important to you. It seems to make life complicated. Would not a central file server be simpler? You have some sort of FreeNAS setup… is that just for backup?.. why not use it instead of sync?

Perhaps I dont understand the issues

There are pros and cons for each method. The mainly important aspect is that the backup method works well enough.

1 Like

the biggest drawback is my personal complacency…

No doubt I may not cross this bridge again this year, touch wood, this decade (men in my ancestry usually drop off the perch just after retirement age - so I have about five years - anything longer than that I’ll count as a bonus, not a blessing) - but I reckon I’ll probably become complacent again sooner, rather than later, and attempt to steamroll a shortcut for something that’s irking me…

According to physics, information is never lost… so apparently, if I accidentally delete something, I can restore it from some cosmic backup silo :smiley: :heart:


The data’s all there - but - FFS - RSL changes “some” of the timestamps (I’ve NO idea what their algorithm is) so if I think “hmm - what was that script I wrote the other day” and ls -alRt my ~/bin/ - they’re NOW in the order that F–KING RSL sorts them in when it syncs!

Grrr :rage:


I think RSL’s “sort order” is F–ING ALPHABETICAL! Good work genius! Seriously WTF?

Started taking a look at SyncThing the other day, but it all got “too hard” - but I might re-evaluate that rash decision… Trouble is I think SyncThing needs some kinda server backend (like my NAS) to run a database, a proper SQL thingie or whatever… seems like overkill for my use case scenario…

1 Like

According to physics, we all are the Bing Bang. :smiley:

1 Like

I just had a light bulb moment…

The way I use RSL - is the “free” way… And it’s their way, or the freeway… and they really don’t give a fig for the customer…

But I bought a pro license in 2018…

I have been guilty of kicking myself in the teeth :

All I have to do is have EVERYTHING sync’d in ONE single f–king folder, and 'cause I paid for a pro license, is selective sync! i.e. just sync computer to computer, not folder to folder FFS!

GODDAMN! I’ve been wrongthinking this f–king thing for FOUR LONG YEARS!

With “small” computers, like Pi Zero, I just sync ~/ResilioSync/bin, on bigger computers I sync ~ResilioSync/bin and Music and Document and Pictures and whatever the F–K!

But I reckon I’ll keep my encrypted secrets share separate…

One of the problems with “selective sync” however (default behaviour on tablets and smartphones) is you have to remember to choose selective sync before it fills up your tiny 8 / 16 / 32 GB filesystem… (and turn off “keep deleted files forever” option)…

DAMN - too many options!


Not game to change my method with my Music share…


Actually not game to use it “out of the box” featurewise and will continue to use my old method… for the time being… I’ll just make sure I don’t have a “Pictures” or “Photos” folder in MORE THAN ONE SHARE… dickhead (me)…


Also - NEVER ever cheat and try to pre-seed - because RSL will probably think the WRONG copy is the master and OVERWRITE f–king everything…

I only ever did this 'cause I was unemployed in 2017 and needed a much cheaper solution than Dropbox, and was happy that RSL worked on ARM seemlessly (Dropbox only supports Linux on x86)…


Going to sleep on this - but I might actualy re-visit the idea of going back to the drawing board, and use Resilio Sync the way they intended (i.e. you buy a pro license and sync nearly everything, computer to computer)…

i.e. :
here’s the “master” - sync off this…

Don’t need all those FLAC and MP3 files? Then don’t sync that f–king folder…


This is where Resilio Sync fails to prove its mettle as some kinda “business solution” - it makes up new time stamps as it goes… bugger whatever timestamp the file ever had in the first place…

You cant live with that. Permissions and attributes have to go with the file.

According to physics, entopy is always increasing, so information IS being lost.
But that doesnt apply to biological systems, they increase information… at the expense of the rest of the universe.
Thats the problem with climate change solutions… it is not possible for bilogical systems to exist without deentropizing the environment.

1 Like

Took me about 36 hours - but got there in the end… ended up having to make a manual checklist and ensure I’d done everything, on everything…

I didn’t merge all my sync targets - I still have 4 of them… I’m still using Resilio Sync like it’s the free version…

Thankfully I didn’t succeed in trashing my Music folder or my encrypted secrets folder…

And now - my main sync share, that I share to EVERYTHING that has a Resilio Sync client, only has 3 folders, my shell scripts, a few shared config files, my ansible playbooks…

So you did centralize to some extent.?

With a decentralised model, it is not uncommon to have asymmetrically sized nodes. The vast majority is usually tiny, some are bigger, few are really big with lots of data and a couple of select few have everything, offer everything and run the biggest business.

1 Like

100%

No I didn’t - if anything, further decentralized…

The “main” sync, that goes to everything that can run a client, is mostly shell scripts, it’s mostly text, some of those computers running sync clients have as little as 16 GB storage (and I have managed to fill one of them up once, when I forgot to disable the default setting of “keep deleted files”)… There’s 14 devices… The default behaviour on portable devices (Android + iOS) is to “selective sync” (pro feature available for free on portable devices)… i.e. it only writes the file, if you want it (and bad luck if you’re not on the same wifi / LAN - e.g. on the bus or a train), the rest of the time, it’s just a pointer or something (I haven’t browsed the folder tree to see what those look like).

And my main issue, that I’ve caused several times over the last 5 years I’ve been using RSL as my self hosted cloud solution harks back to what @Akito often refers to, the sheer DODGYness and kludge-factor with a considerable proportion of applications written for Linux desktops : and here - with RSL? THERE IS NO DESKTOP APPLICATION!

You have to point a browser at a web server hosted on LOCALHOST (and get this - the DEFAULT in RSL is to only let browsers running on LOCALHOST connect - I have to “hack” my config file EVERY time I install the piece of crap - so that it allows “0.0.0.0/0” instead of restricting it to “127.0.0.1/30”).

As if that wasn’t diabolical enough, to setup a sync folder, I go to the master (running on my NAS), and either use a QR code (e.g. phone or table camera), or grab a hex string key, then paste that into the RSL HTTP/HTTPS instance running on the new machine, by adding a folder… Now this is what I’ve managed to do a couple of times - I’ve COPIED the WRONG hex string to the remote machine, which then asks me what local folder I want at the target, this is where it gets accident prone, I’ve pasted the key for my smaller sync into the folder for my bigger data sync - and guess what? If I’m not paying attention, ResilioSync happily merrily OVERWRITES EVERYTHING - and metastasizes that clusterf–k to everything else, just like I told it to! So - VERY unforgiving - and VERY far from being prime time ready…

The MacOS and Windows clients, are actual applications, and a tad more forgiving than the Linux WEB UI solution, and in both cases, much better integrated (there’s NO desktop integration on Linux!) with the desktop OS, e.g. double click on a sync folder in the Windows or MacOS client, and it will launch file explorer or Finder…

Here’s the breakdown :

  1. scripts (14 peers - it’s all peer to peer, so there’s no middleman, some are readonly)
  2. encrypted stuff (only 3 peers host unencrypted, 2 peers readonly, another 2 peers host encrypted copies)
  3. big stuff (7 targets / peers : e.g. documents, pictures/photos, cross platform binaries for a bunch of stuff - all are read/write)
  4. Music (4 peers all are read/write) - I actually cleared it out recently (it was 150+GB) and have it down to 60 GB - most of the files on there are FLAC - this is too big to peer to my phone - I use an adb-sync shell script to keep my phone’s music concurrent with my p2p sync solution.

Note : my “main” music collection, hosted on my NAS is actually now about 1 TB - from ~600 GB 5 years ago - I’ve since been replacing much of it with FLAC versions… And in many cases, I have BOTH mp3 and FLAC versions of the same content. I intend to fix this - I have a shell script I can downsample FLAC to mp3 (uses ffmpeg) - so I can get some savings by removing mp3 copies.

In “theory” my NAS is capable of de-dup, ZFS de-dup, but that’s a ONE WAY street and no going back, and I have no idea if my dual AMD Turion will have enough grunt to do that work (it does have 16 GB of ECC RAM) - and - I know for a fact I have some instances of duplicate data, there’s just so much of it - it would be a fulltime job for a fortnight or longer, to do housekeeping on that…

1 Like