Compare rsync backup times with USB drive and SATA drive

I use csh.
[*,.*] is supposed to glob all files and directories and all dot files and dot directories
I had better check bash… dont want to mislead anyone

1 Like

Won’t “.*” include “.” and “..” in csh ?

1 Like

I checked
leave out the square brackets and comma

cp * .* ....

1 Like

Yes, this is the usual bash syntax (given you have globskipdots option set, to avoid current and parent directories (. and ..) included in ".*`").

2 Likes

I had better patch that reply… it may confuse people. thank you

1 Like

What we used to do to avoid that was
.[A-Z]* .[a-z]* .[0-9]*

I found something better

.[!.]*

it works because it matches at least two characters, the second of which must not be a dot 

That does not require any environment variable like globskipdots be set.
I modified the original reply again.

1 Like

You want to exclude exactly (and only) “.” and “..”.
I mean: “...” or “..a” are perfectly valid. The best bet is globskipdots, IMHO. As anyway you are using a bash syntax now, this would be fine (this option is the default for some distribs, at least on all I have: Debian, Ubuntu, and MX)…

Edit: if I understand what you try to do, what about “./”, instead of trying to list all files and directories in current directory ? Or did I miss something ?

I will have to check. ./ may prepend something at the destination.

Yes .[!.] will miss files that start with two dots

1 Like

It will not, you simply copy current directory :

$ pwd
/tmp/src

$ ls -laR
.:
total 24
drwxr-xr-x  3 br   br    4096 Sep 28 09:44 ./
drwxrwxrwt 13 root root 16384 Sep 28 09:46 ../
-rw-r--r--  1 br   br       0 Sep 27 22:54 ..a
drwxr-xr-x  2 br   br    4096 Sep 28 09:39 dir/
-rw-r--r--  1 br   br       0 Sep 28 09:39 file

./dir:
total 8
drwxr-xr-x 2 br br 4096 Sep 28 09:39 ./
drwxr-xr-x 3 br br 4096 Sep 28 09:44 ../
-rw-r--r-- 1 br br    0 Sep 28 09:39 dirfile

$ rm -rf ../dst-{cp,rsync,tar}; mkdir ../dst-{cp,rsync,tar}

$ cp -R . ../dst-cp
$ rsync -a . ../dst-rsync/
$ tar cf - . | (cd ../dst-tar/; tar xf -)

$ ls -laR ../dst-{cp,rsync,tar}
../dst-cp:
total 24
drwxr-xr-x  3 br   br    4096 Sep 28 09:46 ./
drwxrwxrwt 13 root root 16384 Sep 28 09:46 ../
-rw-r--r--  1 br   br       0 Sep 28 09:46 ..a
drwxr-xr-x  2 br   br    4096 Sep 28 09:46 dir/
-rw-r--r--  1 br   br       0 Sep 28 09:46 file

../dst-cp/dir:
total 8
drwxr-xr-x 2 br br 4096 Sep 28 09:46 ./
drwxr-xr-x 3 br br 4096 Sep 28 09:46 ../
-rw-r--r-- 1 br br    0 Sep 28 09:46 dirfile

../dst-rsync:
total 24
drwxr-xr-x  3 br   br    4096 Sep 28 09:44 ./
drwxrwxrwt 13 root root 16384 Sep 28 09:46 ../
-rw-r--r--  1 br   br       0 Sep 27 22:54 ..a
drwxr-xr-x  2 br   br    4096 Sep 28 09:39 dir/
-rw-r--r--  1 br   br       0 Sep 28 09:39 file

../dst-rsync/dir:
total 8
drwxr-xr-x 2 br br 4096 Sep 28 09:39 ./
drwxr-xr-x 3 br br 4096 Sep 28 09:44 ../
-rw-r--r-- 1 br br    0 Sep 28 09:39 dirfile

../dst-tar:
total 24
drwxr-xr-x  3 br   br    4096 Sep 28 09:44 ./
drwxrwxrwt 13 root root 16384 Sep 28 09:46 ../
-rw-r--r--  1 br   br       0 Sep 27 22:54 ..a
drwxr-xr-x  2 br   br    4096 Sep 28 09:39 dir/
-rw-r--r--  1 br   br       0 Sep 28 09:39 file

../dst-tar/dir:
total 8
drwxr-xr-x 2 br br 4096 Sep 28 09:39 ./
drwxr-xr-x 3 br br 4096 Sep 28 09:44 ../
-rw-r--r-- 1 br br    0 Sep 28 09:39 dirfile

Note: I replaced “./” with “.”, which is the normal way to do, but there will be exactly no difference (even for rsync exception discussed previously, as the missing trailing ‘/’ in source will not create “.” at dst, for obvious reasons). I used “./” previously for pure cosmetic reasons.

My own backup script always (1) cd to source dir, and (2) perform the rsync on “.”.

2 Likes

OK, that is the best solution, because it does not depend on any bashisms
I will add a further note to my original reply, with due acknowledgement of course.
I dont want people seeing that and getting half the message
Thank you.

PS Your script is too complicated for me. I want a one line command that I understand.

1 Like

It looks like complicated, only because it is a incremental backup script, not a copy one :slight_smile:
I did not want from beginning to use a backup software, as, would it disappear/be unavailable, I would not be able to find easily my data (where is my 5 days or 2 weeks or 6 months or 1 year data without the backup software ?).
This is a personal preference only: I like to get this data with a simple “cd” :slight_smile: Even if I disappear, all is easily available, by date (without any software).

I agree. cp is unlikely to disappear.
I originally used tar without unpacking the destination
I tried borg for a while… it frightened me.
I reverted to rsync, but cp is probably the simplest… provided it deals with ACL’s…
we still have to resolve that.
Here is a test… if you wanted to copy a whole OS to another partition… would you use cp?.. would the new copy work?.. I know rsync -aAXvH works.

No: no cp, no rsync… I would do my own way: Hard copy the disk, then install grub.

But if you want to only care the OS (no grub) a simple copy will do it. No magic. And no boot (as you experienced).

Oh yes, you have to do a few things after the copy to make it boot

You have some interesting scripts, but too complicated for me. I like to do things by hand… so I understand.

1 Like

Frankly speaking, these scripts started with one line command (similar to the “rsync” discussed here), that I had to type again and again, until I had that command in a script (sync.sh).

That script simply evolved (very slowly) later:
I just improved it to allow some parameters/config to specify files to exclude and destination, and to keep previous backup versions (day/week/…) without losing disk space. Nothing more. The rest is mostly cosmetic.

If you type a rsync backup command from time to time, I am 100% sure it will become a one line script soon… If yes, I guess you will add one first option in a not so far future… Voilà :slight_smile:

You are right, things evolve.
At the moment I have a file with a collection of rsync commands that I copy/past… at least it avoids typos.
I will be adding some tar and cp recipes.

This has been interesting and educational. Thanks

I have to get back to timing clonezilla… yes I use that, I dont depend entirely on rsync copies.

Regards
Neville

1 Like

Perfect. Ensure that you can get any backup without clonezilla, just an advise:

r$ ssh mydomain.xxx
Last login: Sat Sep 28 13:27:04 2024 from 133.159.120.39
$ ls -lrt /mnt/nas2/xxx/
total 0
drwxr-xr-x 1 root root 114 Jan  1  2022 yearly-03/
drwxr-xr-x 1 root root 114 Jan  1  2023 yearly-02/
drwxr-xr-x 1 root root  54 Mar  1  2023 monthly-12/
drwxr-xr-x 1 root root  54 Apr  1  2023 monthly-11/
drwxr-xr-x 1 root root  54 May  1  2023 monthly-10/
drwxr-xr-x 1 root root  54 Jun  1  2023 monthly-09/
drwxr-xr-x 1 root root  54 Oct  1  2023 monthly-08/
drwxr-xr-x 1 root root 114 Jan  1  2024 yearly-01/
drwxr-xr-x 1 root root  54 Feb  1  2024 monthly-07/
drwxr-xr-x 1 root root  54 Mar  1  2024 monthly-06/
drwxr-xr-x 1 root root  54 Apr  1 03:36 monthly-05/
drwxr-xr-x 1 root root  54 May  1 03:38 monthly-04/
drwxr-xr-x 1 root root  54 Jun  1 03:36 monthly-03/
drwxr-xr-x 1 root root  54 Jul  1 03:36 monthly-02/
drwxr-xr-x 1 root root  54 Aug 18 03:37 weekly-06/
drwxr-xr-x 1 root root  54 Aug 25 03:36 weekly-05/
drwxr-xr-x 1 root root  54 Sep  1 03:39 weekly-04/
drwxr-xr-x 1 root root  54 Sep  1 03:39 monthly-01/
drwxr-xr-x 1 root root  54 Sep  8 03:40 weekly-03/
drwxr-xr-x 1 root root  54 Sep 15 03:37 weekly-02/
drwxr-xr-x 1 root root  54 Sep 19 03:36 daily-10/
drwxr-xr-x 1 root root  54 Sep 20 03:36 daily-09/
drwxr-xr-x 1 root root  54 Sep 21 03:35 daily-08/
drwxr-xr-x 1 root root  54 Sep 22 03:35 weekly-01/
drwxr-xr-x 1 root root  54 Sep 22 03:35 daily-07/
drwxr-xr-x 1 root root  54 Sep 23 03:35 daily-06/
drwxr-xr-x 1 root root  54 Sep 24 03:35 daily-05/
drwxr-xr-x 1 root root  54 Sep 25 03:36 daily-04/
drwxr-xr-x 1 root root  54 Sep 26 03:35 daily-03/
drwxr-xr-x 1 root root  54 Sep 27 03:35 daily-02/
drwxr-xr-x 1 root root  54 Sep 28 03:36 daily-01/

Will clonezilla allow you to access your history without clonezilla itself ?
If yes, go for it, I will happily follow, please let me know. The backups above will survive me, and my family will have easy access (no software needed).

I know I can go into a Clonezilla image directory and look at the files. Each partition can have multiple image files if it is large. I will have to investigate … by default clonezilla uses the partclone utility and zip compression.

Clonezilla is not meant for archiving. It does not do incrementals. It is meant for full recovery when your disk fails.

1 Like

And if you don’t have clonezilla, can you recover ? My point was to access backups without any software (ls and cp will be enough). Again, I will love to use clonezilla, if I can get data back without clonezilla.

I am investigating. It looks complicated at the moment.

It seems what you do is restore the image to a temporary file using partclone then loop mount it and you can see the files.

Note what I said… clonezilla is not intended for archives.

1 Like