Using the CLI to backup or copy a Linux system

[nevj (Neville Jackson)]

There is a nice set of rsync examples here

Artikel_rsync.pdf

Last try was:

@nevj

rsync -avxHAXS --delete --filter=protect_{/dev/*,/proc/*,/sys/*,/run/*,/mnt/*,/media/*,/tmp/*} /mnt /

There are problems with that

• I think /dev/* etc should be /dev… ie protect a directory
• the source should be /mnt/* … according to examples I find

So I ran

rsync  -avxHAXS --delete  --filter=protect_{/dev,/proc,/sys,/run,/mnt,/media,/tmp} /mnt/*  /

It ran and displayed a long incremental list of files … so /mnt/* was effective
but
on completion the OS collapsed… could not find ls or cat … the background and menu disappeared, and it would not boot… so not sure the --filter option was correct.

So… golden opportunity, … I am going to try with rsync from MX to do a recovery from my backup with both the source and destination directories mounted.
Wish me luck!

Well, my luck is out. … it seems the above rsync attempt not only messed up the destination OS, but also destroyed the source backup filesystem… when I mount it from MX, there is nothing there… it seems --delete deleted the source???

I think the problem was… the --filter=protect option was wrong, so it did not protect /mnt and therefore the backup directory itself came under the --delete operation.

Oh well… start again with a fresh install of Peppermint

[ihasama]

I’m also trying this rsync to USB now. Actually it’s been running 8hours now.. I’m on Mint, trying to rsync my Gentoo. I tried to exclude my /home/pete/Downloads but it still copied it. At the moment it’s syncing

/gentoo/var/tmp/portage/dev-lang/
ghc-9.0.2-r4  mujs-1.3.3  rust-1.71.1  rust-1.74.1-r101

which takes a lot of time. I don’t know how to exclude these because I used the (added /home/pete/Downloads/*) command as per on the first message.

@nevj

rsync -avxHAXS <source directory> <destination directory>

@nevj

--exclude={/dev/*,/proc/*,/sys/*,/run/*}

I can’t give the exact command at the moment for my rsync because it’s still running but will edit this after it stops.

Maybe tar is the right way to do a full system backup? Or just copy /home folder for backing up the data and just re-install OS if needed or use clonezilla?

EDIT: the rsync finished. The command was:

sent 147.839.839.015 bytes received 31.709.647 bytes 4.374.444,91 bytes/sec
total size is 150.170.251.842 speedup is 1,02
root@pete:~# rsync --exclude={/dev/,/proc/,/sys/,/run/} -avxHAXS /gentoo /mnt

It seems that I didn’t add /home/pete/Downloads/* as I thought

[nevj (Neville Jackson)]

@ihasama

It seems that I didn’t add /home/pete/Downloads/* as I thought

Well, wecome to the mistake makers club… that is how you learn , but it can be painful.

@ihasama

–exclude={/dev/,/proc/,/sys/,/run/}

It rendered in discourse as /dev/ but when I quoted it above I got /dev/* in the edit window. ?? The codebox is supposed to protect that??
That started me thinking… /dev/* does not include dot files.
That does not matter in /dev but for /home/pete/Downloads/* I think you might use /home/pete/Downloads/ to include dot files.

That issue arises in specifying the <source> too.
/ is OK, but /abc is not right… it needs to be /abc/
But your /gentoo worked? My feeling is it should be /gentoo/ ?
That only applies to <source>… <destination> can be a directory

The ‘trailing / rule’ applies to pathnames. I am not sure if it applies to items in an --exclude option? The man page is not helpful on this?
I asked duck.ai

"The trailing slash rule in rsync applies differently to source paths compared to exclude patterns.

---

Trailing Slash in Source Paths When specifying source paths, the presence or absence of a trailing slash determines how rsync interprets the path.

---

With Trailing Slash: Includes the contents of the directory. Without Trailing Slash: Includes the directory itself and its contents."

---

"Exclude Patterns For exclude options, the trailing slash has a different effect:

---

With Trailing Slash: Indicates that you want to exclude directories that match the pattern. This means only directories with that exact name (and their contents) will be excluded. Without Trailing Slash: Excludes all files and directories that match the pattern, regardless of whether they are files or folders."

---

"Example Exclude a Directory: Exclude option: --exclude=dir/ Excludes the directory dir and its contents. Exclude Files in a Directory: Exclude option: --exclude=dir Excludes both dir and any files named dir.

---

In summary, yes, the trailing slash rule does apply to exclude options in rsync, and it affects the scope of what gets excluded."

It does not say what dir/* does? That seems to be what everyone uses.
It does say that your /gentoo included the gentoo directory on the backup partition. That is OK , but be careful when you do a recovery.

@ihasama

takes a lot of time

I am using a hotplug sata disk. rsync copies a 2Gb linux in about a minute.
Usb3 to a flash drive should be about the same time.
You must have heaps in Downloads

Keep at it… There is lots to learn, and even more mistakes are waiting.

What I learnt… when doing a recovery, mount the backup read only … you dont want a mistake to clobber the backup.

Update:

Yes, the trailing slash rule for pathnames

does apply to pathnames within rsync filter rules, including the filter='P ...' (protect) option. The meaning is consistent with the general rsync filter rules:

"* A pattern that ends with a / only matches a directory, not a file. This is useful for protecting a directory and all its contents.

• A pattern without a trailing / can match either a file or a directory with that name. "

Update 2:
This is the best doc on exclude rules and filter rules

rsync filter rules - 0ink.net

[ihasama]

Thanks Neville! I’ll check the linked filter rules. I can confirm that the rsync did copy my system files to the USB. It’s 3rd gen USB and took more than 8 hours. I need to clean my laptop before running the sync again. And also need to pay attention to exclude some of the folders the right way

[nevj (Neville Jackson)]

@ihasama

It’s 3rd gen USB and took more than 8 hours

Something is wrong. How many Gb is your Gentoo?
I can do 2Gb in 1 minute to my sata disk rated at 6Gb per sec … I think usb3 is rated about the same.
Maybe there is ram buffering? Do you have plenty of ram?

@ihasama

exclude

That is the big issue . The documentaion is hard to follow and there are few examples.

Sorry if I gave the impression that rsync for backup is all solved… it is not…there are too many different cases for anyone to be able to write down one simple command or script that would work for everyone.

[ihasama]

@nevj

Something is wrong. How many Gb is your Gentoo?

sudo du -hs /
du: cannot access ‘/proc/20089/task/20089/fd/4’: No such file or directory
du: cannot access ‘/proc/20089/task/20089/fdinfo/4’: No such file or directory
du: cannot access ‘/proc/20089/fd/3’: No such file or directory
du: cannot access ‘/proc/20089/fdinfo/3’: No such file or directory
du: cannot access ‘/run/user/1000/doc’: Permission denied
du: cannot access ‘/run/user/1000/gvfs’: Permission denied
74G /

root is 74G
I found one place where I can free space, the /var/tmp/portage is 21G. These files are some leftovers which I can delete. I have only 8GB RAM.

@nevj

Sorry if I gave the impression that rsync for backup is all solved… it is not…there are too many different cases for anyone to be able to write down one simple command or script that would work for everyone.

This is a difficult issue and I really appreciate how you two handled this! I really learned a lot (as many of us here have). I think I’ll continue backing up my data and system separately. Using rsync for my home folder is fine.

EDIT: after removing the /var/tmp/portage my system is now 53G, so freed 21G of temp files is great! I wouldn’t have noticed these files if the backup wouldn’t take so long.

[nevj (Neville Jackson)]

@ihasama

root is 74G

At 2Gb per minute, it should finish in half an hour.

I looked up usb3.0 … it is 5Gb per second… similar to my 6Gbps for sata. That is bits per second… divide by 8 for bytes. … still way faster than I am getting (2GB per min). … Line speed is not the problem.

There is something wrong … is it maybe following links to data outside of Gentoo?

You can mount the backup and look at its filesystem … does it contain more than 74Gb? Are there extra folders?

[ihasama]

@nevj

You can mount the backup and look at its filesystem … does it contain more than 74Gb? Are there extra folders?

Interesting… sudo du -hs /mnt/gentoo (the USB stick) gives 81G. But 5G difference don’t take 8 hours.. Maybe my USB stick is just fake and it’s not that fast?

I have removed the /var/tmp/portage from my system. Should I try to rsync the USB stick with remove?

Talking about mistakes I once rsynced my system which had a 8Gb Swap file

EDIT: Here’s the root and backup, looks the same.

┌─[pete@gentoo]─[~]
└──╼ ls -a /
.         backup              boot      dots  gentoo  lib64       mnt   root  sys   usr
..        backup19102025.zip  deja-dup  efi   home    lost+found  opt   run   temp  var
.xinitrc  bin                 dev       etc   lib     media       proc  sbin  tmp
┌─[pete@gentoo]─[~]
└──╼ ls -a /mnt/gentoo/
.         backup              boot      dots  gentoo  lib64       mnt   root  sys   usr
..        backup19102025.zip  deja-dup  efi   home    lost+found  opt   run   temp  var
.xinitrc  bin                 dev       etc   lib     media       proc  sbin  tmp

[abu (Alfred Bühler)]

Usually I run rsync with the option -n first, for verifying what will be included.

[easyt50 (Howard)]

@ihasama

root is 74G

My God, that’s huge. I have a partition size for root of 35 GB with 22 GB reported as free for my Mint OS.

[Rosika (Rosika Schreck)]

Hi @nevj and all,

sorry, this may be a bit off-topic, because my question aims at backing up not the root partition but some content of either the home or any other data partition.

I was wondering about the parameters of rsync to use in this case.

Would this still be considered to be the preferred method:

rsync -avxHAXS  <source directory> <destination directory>

I was wondering about the -x parameter.

-x, --one-file-system
This tells rsync to avoid crossing a filesystem boundary when recursing.

I´m not sure what that exactly means:
Filesystem boundary is not the same as crossing partitions, or am I getting it all wrong?

A practical example of my query would be saving the hidden .thunderbird folder (containing all e-mails and other stuff), which is located on my 3rd data partition.

Until now I´ve done it this way:

rsync -a --info=progress2 /media/rosika/f14a27c2-0b49-4607-94ea-2e56bbf76fe1/thunderbird_neu/.thunderbird /media/rosika/verbatim2/für_thunderbird/

Sorry again for asking this.

Many greetings from Rosika

[ihasama]

@easyt50

My God, that’s huge.

Not heard that before But yes, you’re right! I learned that on Gentoo the package manager leaves all unfinished builds to a directory and since it’s an old laptop there was appr 20gigs of .tar files. One thing with Gentoo is that you need to make cleaning and I hadn’t checked if there’s something else to clean than just use a program called eclean. My laptop would need a proper cleaning, for an example there’s Sway, Hyprland and Cinnamon and I only use Sway. Then there’s some virtual machine things I could uninstall and I’ve also learned some Python with it so there’s environment for coding also.

[nevj (Neville Jackson)]

@ihasama

. Maybe my USB stick is just fake and it’s not that fast?

I assume you used a usb3 flash drive?

@ihasama

Here’s the root and backup, looks the same.

Yeah , no extra folders and 81G versus 74G is probably just filesystem specs varying on the flash drive

I am wondering about your hardware. … if that laptop is old it may have a slow data bus, or not much memory for buffering. ?
You did not use compression… that would slow it.
I will try an rsync to flash drive on my 2 machines… just to get some idea of timing with a flash drive.

[nevj (Neville Jackson)]

Hi Rosika

@Rosika

backing up not the root partition but some content of either the home or any other data partition.

I was wondering about the parameters of rsync to use in this case.

Would this still be considered to be the preferred method:

Yes, but for data some of those parameters (eg S) would be unlikely to be needed.

With dot files like your Thunderbird example, be careful how you specify the <source>

@Rosika

Filesystem boundary is not the same as crossing partitions, or am I getting it all wrong

No. Filesystem boundry would be the same as crossing partitions. … unless you use something like LVM where partitions mean nothing.

@Rosika

sorry, this may be a bit off-topic

I dont think I intended the topic to apply only to system files… no need to apologise.

Regards
Neville

[nevj (Neville Jackson)]

@easyt50

My God, that’s huge.

There are backup files in there.
@ihasama should probably have excluded those… they may be very large… rsync goes best copying lots of small files.

You may have pointed out the solution. Lets see what @ihasama says

[nevj (Neville Jackson)]

@abu

run rsync with the option -n first

Yes I have been using --dry-run. I am not very practised at interpreting the output.