If you run that every night to the same backup directory, it will create an archive … all the files you ever had
If you want a snapshot to recover from, either
I make an archive … it is simple and fast after the first run.
You used /home/pete
it should be
/home/pete/
The source path should always end in /
Nice catch! I also noticed that it still copied my Downloads.
time rsync -axHAXS --exclude={/home/pete/Downloads*,swapfile} /home/pete /mnt/backup
should be
time rsync -axHAXS --exclude={/home/pete/Downloads*,swapfile} /home/pete /mnt/backup/
but still the exclude didn’t work for my Downloads. Should it also have /* like this --exclude={/home/pete/Downloads/*,swapfile}
Yes… you want to keep the Downloads directory, but not its contents.
What is swapfile? Does it need a path?
That is good… you are checking.One of the good things about rsync is you can browse the backup easily. …and easily get back one file if you make a simple mistake editing.
Correction
Trailing slash has a different meaning in an exclude list
“Example Exclude a Directory: Exclude option: --exclude=dir/ Excludes the directory dir and its contents. Exclude Files in a Directory: Exclude option: --exclude=dir Excludes both dir and any files named dir.”
see reply 123
OK!! Now it’s working! I used these commands:
gentoo ~ # time rsync -axHAXS --dry-run --exclude={Downloads/*,Thunderbird/*,powerlevel10k/*,efi/*,temp/*} /home/pete /mnt/backup/
real 0m0.423s
user 0m0.165s
sys 0m0.248s
gentoo ~ # time rsync -axHAXS --exclude={Downloads/*,Thunderbird/*,powerlevel10k/*,efi/*,temp/*} /home/pete /mnt/backup/
real 1m7.345s
user 0m25.211s
sys 0m49.943s
and my Downloads folder is empty in the backup (like the others I excluded):
gentoo ~ # df -h /mnt/backup/pete/
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/pete 25G 15G 9.1G 62% /mnt
gentoo ~ # sync
gentoo ~ # ls /mnt/backup/pete/Downloads/
gentoo ~ #
it didn’t need to be on the exclude list because it is not in my /home folder. You can create a swapfile if you don’t have a partition for swap. Here’s how it is created:
gentoo ~ # fallocate -l 8GiB swapfile
gentoo ~ # chmod 600 swapfile
gentoo ~ # mkswap swapfile
Setting up swapspace version 1, size = 8 GiB (8589930496 bytes)
no label, UUID=a9356e9a-f88c-46d9-adc5-0dff04a89090
gentoo ~ # swapon swapfile
gentoo ~ # swapon --show
NAME TYPE SIZE USED PRIO
/root/swapfile file 8G 0B -2
so it’s on /root and I was backing my home folder.
and edit /etc/fstab:
#SWAPFILE:
/swapfile none swap sw 0 0
So you did not use /home/pete/ and that meant the pete directory was included in the backup …
If you had used /home/pete/ , it would have been
/mnt/backup/Downloads
Be careful when you recover that you dont end up with 2 nested pete directories … ie you would need to recover to /home
Also
You do not need a trailing slash on the destination … but it is harmless.
I am still not sure I understand exclude lists, but Download/* seems to be right.
This is a painful learning curve.
I like to find one simple usage pattern that works, and then build on that with small variations.
Well done.
Note that tar is also an option to backup a home directory. Tar can compress the backup so it uses less disk space, but it is not as easy to browse the backup.
You are right, I need to change the target mount point. This is still a learning experience
You have just provided a good answer to this topic
It is amazing how things cross fertilize.
And remember, this all started because of discussion under this topic
Backups seem to be a hot discussion point
It is called ‘directory creep’ . It is an issue with tar backups too.
Copy A to B. Then copy B to A. You end up with A/A
@Laszlo and I put forward some ideas for using rsync and tar for backups ( see Reply No 1)
In relation to rsync, I have been able to get some of the suggestions in Reply No 1 to work, but others have proved difficult. Tar has yet to be tested on live systems.
Here is a brief summary of progress and additional ideas, some from other users.
mount /dev/sdxx /mnt/backup
rsync -axHAXS --exclude={/dev/*,/proc/*,/sys/*,/run/*,/mnt/*,/media/*,/tmp/*} /mnt/source/ /mnt/backup
Note the trailing / on the source directory, that ensures all files( including dot files) in source are copied.
mount /dev/sdxx -o ro /mnt/backup
mount /dev/sdyy /mnt/dest
rsync -axHAXS --delete --exclude={/dev/*,/proc/*,/sys/*,/run/*,/mnt/*,/media/*,/tmp/*} /mnt/backup/ /mnt/dest
Note the use of --delete. You do not have to delete the system being recovered to… rsync will wind it back to the backup state in terms of files and file dates and file ownership
mount /dev/sdyy /mnt/backup
rsync -axHAXS --exclude={/dev/*,/proc/*,/sys/*,/run/*,/mnt/*,/media/*,/tmp/*} / /mnt/backup
Note it is very important to exclude /mnt/*. The / for source is OK, because it is a trailing slash.
It is important to close any running processes, and to exclude any active files like databases.
Prevent any updates from running.
The -x flag will exclude /home if it is a separate filesystem. (Thanks Rosika)
mount /dev/sdxx -o ro /mnt/backup
rsync -axHAXS --delete --exclude={/dev/*,/proc/*,/sys/*,/run/*,/mnt/*,/media/*,/tmp/*,/etc/mtab} /mnt/backup/ /
Note /etc/mtab is excluded. That came from a systemback example ( Thanks Laszlo).
It is important to exclude /mnt/*. Always mount the backup read-only. There is a trailing / on the source path.
Example
rsync --dry-run -axHAXS --delete --exclude={/dev/*,/proc/*,/sys/*,/run/*,/mnt/*,/media/*,/tmp/*,/etc/mtab} /mnt/backup/ /
do a sync after rsync finishes. To be sure it completed successfully.
the arguments in a --exclude=.. option can be placed on a file.
rsync -av --exclude-from=$EXL / $ARSYDIR/
Where $EXL is a filename containing a list of exclude criteria. ( Thank you Daniel)
Or like this
rsync -av --exclude-from=- /source/ /dest/ <<_EOF
/run
/proc
/sys
_EOF
(Thanks Laszlo).
the --filter=protect_{/dev/*,...} option should protect directories from being deleted or owerwritten during a restore with --delete… in the same way as --exclude.
I tried it witjh a live restore and it crashed the live system.
in Peppermint/Devian OS an rsync of a live system does not properly terminate. It seems to not flush the ram buffers. A sync after the rsync rerminates hangs forever.
Not mounting the backup filesystem read only is a bad idea. One mistake with rsync and you can destroy your backup. I know, I did it twice. Slow learner.
leaving out -x to allow it to follow links to other filesystem. One might do this if /boot were a separate partition, or if /home were a separate partiton and one wanted to include it.
*systemback uses --include='* as well as –exclude=`. This to me seems unnecessaritly complicated but there may be reasons.
systemback excludes /etc/mtab for live recoveries. I have not tried without it, but I suspect it is necessaryfor a live system.
Thanks to all who have contributed, and to @easyt50 and @ihasama for suggesting the topic, and to @ihasama for having the courage to try rsync.
Suggested additions/corrections to summary welcome.
I tried to repeat the Peppermint live backup sync problem
root@trinity:/home/nevj# mkdir /mnt/backup
root@trinity:/home/nevj# mount /dev/sdc3 /mnt/backup
root@trinity:/home/nevj# ls /mnt/backup
root@trinity:/home/nevj# time rsync -axHAXS --exclude={/dev/*,/proc/*,/sys/*,/run/*,/mnt/*,/media/*,/tmp/*} / /mnt/backup
real 1m54.852s
user 0m18.467s
sys 1m6.003s
root@trinity:/home/nevj# time sync
real 0m11.337s
user 0m0.001s
sys 0m0.003s
root@trinity:/home/nevj# ls /mnt/backup
bin common etc initrd.img lib64 media opt root sbin sys usr vmlinuz
boot dev home lib lost+found mnt proc run srv tmp var
root@trinity:/home/nevj#
This time the sync took an entirely reasonable 11 secs.
Looks like I had an unlucky gliche first try…it really confused the whole situation.
Then I restored from it
root@trinity:/home/nevj# umount /mnt/backup
root@trinity:/home/nevj# mount /dev/sdc3 -o ro /mnt/backup
root@trinity:/home/nevj# time rsync -axHAXS --delete --exclude={/dev/*,/proc/*,/sys/*,/run/*,/mnt/*,/media/*,/tmp/*,/etc/mtab} /mnt/backup/ /
real 0m26.827s
user 0m5.445s
sys 0m9.468s
root@trinity:/home/nevj# time sync
real 0m0.047s
user 0m0.000s
sys 0m0.001s
root@trinity:/home/nevj#
The root file system went from 6635856 bytes to 6645692 bytes.
It still seems to run.
Maybe the extra was mozilla doing this reply ?
root@trinity:/home/nevj# du -s .
137056 .
root@trinity:/home/nevj# du -s /mnt/backup/home/nevj
124096 /mnt/backup/home/nevj
So my home directory has grown since the recovery.
It shows what a dynamic entity a live system is. Trying to snapshot it live requires some caution.
For example
root@trinity:/home/nevj# du -s /etc
70624 /etc
root@trinity:/home/nevj# du -s /mnt/backup/etc
59412 /mnt/backup/etc
That is an enormous difference?
Another look
root@trinity:/home/nevj# du -s /var
436148 /var
root@trinity:/home/nevj# du -s /mnt/backup/var
435796 /mnt/backup/var
/var is growing too. That might be expected
Try again
root@trinity:/home/nevj# du -s /usr/bin
168460 /usr/bin
root@trinity:/home/nevj# du -s /mnt/backup/usr/bin
167220 /mnt/backup/usr/bin
Why would /usr/bin grow? I have not added anything.
Maybe the du figures are not comparable for different media?
We have lots to learn about live systems.
root@trinity:/home/nevj# du -s /usr/bin
168460 /usr/bin
root@trinity:/home/nevj# du -s /usr/bin
168460 /usr/bin
root@trinity:/home/nevj# du -s /usr/bin
168460 /usr/bin
That is it… it is not growing… it is different media.
Wow it is hard to see what is going on with a live recovery.
My stupid fault
“du doesn’t add up the files byte sizes, it measures occupied disk space.”
Well, using du to compare filesystems was a bad idea.
Here is a correct way… using rsync
root@trinity:/home/nevj# mount /dev/sdc3 -o ro /mnt/backup
root@trinity:/home/nevj# time rsync -avn --delete --exclude={/dev/*,/proc/*,/sys/*,/run/*,/mnt/*,/media/*,/tmp/*} / /mnt/backup/
....
home/nevj/.mozilla/firefox/50z3o7ov.default-esr/storage/permanent/chrome/idb/3870112724rsegmnoittet-es.files/596
home/nevj/.mozilla/firefox/50z3o7ov.default-esr/storage/permanent/chrome/idb/3870112724rsegmnoittet-es.files/597
....
proc/
root/.bash_history
run/
sys/
tmp/
usr/lib/rc/cache/deptree
usr/lib/rc/cache/softlevel
var/cache/cups/
var/cache/cups/job.cache
var/cache/cups/job.cache.O
var/lib/NetworkManager/
var/lib/NetworkManager/NetworkManager.state
var/lib/NetworkManager/internal-84100763-19a3-307d-a459-fbda281fb6d7-eth1.lease
var/lib/NetworkManager/seen-bssids
var/lib/NetworkManager/timestamps
var/lib/alsa/
var/lib/alsa/asound.state
var/lib/dbus/
var/lib/dbus/machine-id
var/lib/lightdm/.Xauthority
var/lib/lightdm/.cache/lightdm-gtk-greeter/
var/lib/lightdm/.cache/lightdm-gtk-greeter/state
var/lib/lightdm/.dbus/session-bus/
var/lib/lightdm/.dbus/session-bus/d5d95b62296aa2dcf499cbf4697b359e-0
var/lib/lightdm/data/
var/lib/lightdm/data/lightdm/
var/lib/plymouth/boot-duration
var/lib/smartmontools/
var/lib/smartmontools/attrlog.ST1000LM048_2E7172-ZKP4R53V.ata.csv
var/lib/smartmontools/attrlog.ST2000DM001_9YN164-W2F08WRX.ata.csv
....
var/log/lightdm/seat0-greeter.log.old
var/log/lightdm/x-0.log
var/log/lightdm/x-0.log.old
sent 13,058,309 bytes received 20,128 bytes 2,906,319.33 bytes/sec
total size is 5,936,881,871 speedup is 453.94 (DRY RUN)
real 0m4.637s
user 0m2.848s
sys 0m2.344s
root@trinity:/home/nevj#
The rsync options used are
So what it does is list files that would be changed or transferred , if this were not a dry run
It found some dot files, some files in /usr/lib, and lots of files in /var.
That is all OK. The live system has gone on running after I did a backup and restore yesterday.
You can add the option -c which makes rsync do a content based comparison based on checksums. It takes longer
time rsync -avnc --delete --exclude={/dev/*,/proc/*,/sys/*,/run/*,/mnt/*,/media/*,/tmp/*} / /mnt/backup/
.....
var/log/lightdm/x-0.log.old
sent 20,258,625 bytes received 20,433 bytes 134,744.57 bytes/sec
total size is 5,937,298,182 speedup is 292.78 (DRY RUN)
real 2m30.136s
user 0m8.823s
sys 0m17.659s
root@trinity:/home/nevj#
It listed exactly the same files and took 30x longer.
Now I have a tool to prove that a backup run has done what it should … and the tool works on a live system, or on a non-live source file.
I didn’t read all of this very long thread, but I used to lurk on tihe rsync listserv and there were a lot of discussions there about the order of excludes and how to specify them. A lot of how that works is less than obvious. God bless the dryrun parameter!
There is a summary in reply 189.
Beware , some things in the original post could not be made to work thus far.
When we get to the end of thread, I think we revise the original post … maybe add amendments in bold or italics ? What do you think?
I think order matters if you use a general exclude rule like --exclude=/* , and if you use both --include and --exclude options.
I am trying to avoid such complexities.
This is a good reference
I had another try at rsync’ing Peppermit live , to a USB flash drive
root@trinity:/home/nevj# mount /dev/sdj2 /mnt/backup
root@trinity:/home/nevj# cd /mnt/backup
root@trinity:/mnt/backup# ls
bin common etc initrd.img lib64 media opt root sbin sys usr vmlinuz
boot dev home lib lost+found mnt proc run srv tmp var
root@trinity:/mnt/backup# rm -r *
root@trinity:/mnt/backup# ls
root@trinity:/mnt/backup# cd
root@trinity:~#
root@trinity:~# time rsync --dry-run -axHAXS --exclude={/dev/*,/proc/*,/sys/*,/run/*,/mnt/*,/media/*,/tmp/*} / /mnt/backup
real 0m11.918s
user 0m3.097s
sys 0m4.391s
root@trinity:~# time rsync -axHAXS --exclude={/dev/*,/proc/*,/sys/*,/run/*,/mnt/*,/media/*,/tmp/*} / /mnt/backup
real 3m46.767s
user 0m18.140s
sys 1m2.510s
root@trinity:~# time sync
That final sync has been going about 10 minutes
So the buffer flushing problem occurs when the backup drive is a USB flash drive, at least in Peppermint.
There is a sync process running, but it is asleep?
nevj@trinity:~$ ps alx | grep sync
0 0 4130 3993 20 0 5476 908 - D+ pts/0 0:00 sync
and, I can not do a shutdown… that hangs too … I had to use the switch.
I dont understand.
Further to the sync problem with live rsync to a usb flash drive
I tried with antiX instead of Peppermint. It was 19GB so it took a while
root@trinity:/mnt/backup# du -s /mnt/backup
8379276 /mnt/backup
root@trinity:/mnt/backup# rm -r *
root@trinity:/mnt/backup# time rsync --dry-run -axHAXS --exclude={/dev/*,/proc/*,/sys/*,/run/*,/mnt/*,/media/*,/tmp/*} / /mnt/backup
real 0m9.086s
user 0m1.513s
sys 0m2.125s
root@trinity:/mnt/backup# time rsync -axHAXS --exclude={/dev/*,/proc/*,/sys/*,/run/*,/mnt/*,/media/*,/tmp/*} / /mnt/backup
real 54m17.710s
user 0m41.776s
sys 1m47.975s
root@trinity:/mnt/backup# time sync
real 0m0.068s
user 0m0.002s
sys 0m0.000s
root@trinity:/mnt/backup#
The whole thing went smoothly, and there was no sync problem
Therefore:
So, I shall close this annoying side issue… and get back to testing rsync
Rsync has been used by so many people for so long that it’s not usually the problem. Like anything else, it does have bugs, but they get fixed pretty fast once identified.
True. I just could not let that sync thing lie without finding the culprit.
It seems Peppermint was a really bad choice for my testing.
I wanted a small distro… I might switch to antiX
Rewriting thread posts tends to break the responses to the original post. If necessary, I would add a line at the top of a post saying something like this post has been revised at this link.
Rsync is very sensitive to filter ordering. You can get it to do almost anything if you can figure out how. I was never an advanced rsync user. I did write my own backup system in bash using rsync forever ago and if I ever figure out ssh again on my new LAN, I’ll use it with my NAS which is currently a door stop.
I think I will take that advice. Thanks.
Disable all the keys rubbish, and use it like rsh used to be.
Nobody is going to break into your LAN, so treat the whole LAN like it was one computer. You only need one firewall to the outside world… your internal world should be open… not conventional advice but think about where you need security and where you dont.