Rsync to Data Drive in Home Server

Shell Scripting & Linux Command Line
1

As I have been reading a great article on here about rsync, I attempted my first sync of files from my LM to home server.

I got errors (see below) and from what I can tell, it has something to do with using the 2nd internal data drive as the target drive/directory. Might also have some syntax errors? The job was to sync my /Documents folder (from my current LM machine) to the Fedora server, onto the mounted SSD drive “Server Backups”, to a folder already created there called MSI Linux Mint.

Here is what I used:

myviolinsings@mint-desktop:~$ rsync -aAXvz /home/myviolinsings/Documents sflanagan@192.168.1.157 /run/media/sflanagan/Server Backups/MSI Linux Mint

And here are the errors returned:

rsync: [sender] link_stat "/home/myviolinsings/sflanagan@192.168.1.157" failed: No such file or directory (2)
rsync: [sender] change_dir "/run/media/sflanagan" failed: No such file or directory (2)
rsync: [sender] change_dir "/home/myviolinsings/Backups" failed: No such file or directory (2)
rsync: [sender] link_stat "/home/myviolinsings/Linux" failed: No such file or directory (2)
created directory Mint
sent 10,225,316,902 bytes  received 67,665 bytes  97,850,570.02 bytes/sec
total size is 14,935,289,964  speedup is 1.46
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1338) [sender=3.2.7]

I looked at where the second drive was mounted on Fedora in order to choose that drive for the rsync command, and then used the name of the folder I wanted /Documents to go into.

So clearly there is something wrong. And even though I saw hundreds of files named in terminal, I could not find any of them on the Fedora machine, but I did find an empty “Mint” folder on the root directory of the OS disk. Assuming that is where rsync “created directory Mint?”

I just need some help understanding how to use that non-OS drive with rsync with the current mount point.

Thanks,
Sheila Flanagan

1 Like
2

Should there just be a colon or maybe just no space after the IP address and before the /run?

The rsync command is basically from and to. The to in your case would include your username@serverip:/location.

Maybe try a colon or nothing at all in there.

1 Like
3

@pdecker I had previously used a colon and that gave an error immediately without any further output. I just now tried removing the space and got this:

sending incremental file list
rsync: [sender] change_dir "/home/myviolinsings/sflanagan@192.168.1.157/run/media/sflanagan" failed: No such file or directory (2)
rsync: [sender] change_dir "/home/myviolinsings/Backups" failed: No such file or directory (2)
rsync: [sender] link_stat "/home/myviolinsings/Linux" failed: No such file or directory (2)
Documents/Sheila's Business Vault/.obsidian/workspace.json
Documents/Sheila's Business Vault/Customers/Angel Steadmon.md
Documents/Sheila's Personal Vault/.obsidian/workspace.json
Documents/Sheila's Personal Vault/.obsidian/plugins/recent-files-obsidian/data.json
Documents/Sheila's Personal Vault/Linux Tips & CLI/Using Rsync Commands on Home Network.md
Documents/Sheila's Personal Vault/To Do Lists/Thurs 02.22.24.md

sent 167,267 bytes  received 830 bytes  112,064.67 bytes/sec
total size is 14,935,290,843  speedup is 88,849.24
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1338) [sender=3.2.7]

The mention of /home/myviolinsings/[directory name] makes me think it does not like spaces in folder names (i.e., Server Backups, MSI Linux Mint) because the change dir reported no such file or directory in a single word part of the folder I am using as target. Could that be an issue in Fedora, or rsync itself?

Thanks,
Sheila

4

Yes, I think it needs a colon separating the host from the filename.

Backups/MSI Linux Mint

What are they there for?

It is thinking the destination is another filename to copy, because there are things after it. The destination has to be the last parameter.

5

The Backups Drive on Fedora Server have a file folder for each machine. This was a test to see if I had the syntax correct. I wanted to sync all of my Linux Mint machine Documents folder to the Server on the 2nd drive mounted on /run /media/sflanagan/Server Backups/`to the folder on that drive named MSI Linux Mint. So I included the name of the folder (as there are several) that I want that sync to go into. Does that make sense?

Thanks,
Sheila

6

I think you can only sync into one folder at a time.
Must admit I have never tried syncing to multiple folders… how would rsync know which folder to use for a given file?

How about you simplify it to 3 separate rsync statements, each to only one folder. You could put them in a script.

7

I am only syncing one folder to one remote folder. Let me explain.

The server machine has 2 drives: one with the OS and one for data. This second disk has folders for each machine on the network. I always use the name of the computer and the OS when I create these folders; thus, MSI Linux Mint is the folder on the server data drive that I wish to sync my /Documents folder from LM machine.

My understanding is I put the server name “sflanagan@192.168.1.157”
and then to ensure it went to the right drive, I list the directory where this drive is mounted
" /run/media/sflanagan/Server Backups/"

But that drive has 10 folders, so to ensure the sync is copied to the right folder, I added
“MSI Linux Mint”

Hopefully that’s clear as mud. :grin:

Sheila

8

My understanding is different to yours.

If I want to sync to 3 folders in the destination, I have to have 3 folders in the source.

I dont think rsync is capable of changing the folder structure of what it is syncing. Its purpose is to make a perfect copy, without modification of the folder structure.
You just need to name the MInt folder in the destination when syncing from Mint

9

Let’s forget about the data drive on the server. I want to sync my Documents folder (single folder) from one computer to another computer that also has a Documents folder (single folder already on it.

The difference here is because we are not syncing from /home/myviolinsings/documents TO /home/sflanagan/documents (on the OS drive where home is located on the server), I have to include the location of the data drive (using its mount point and label) and the /Documents folder on local machine needs to sync to a different single folder on that data drive, which is named MSI Linux Mint.

I have a different folder for each machine so that I can keep track of the different syncs between 4 machines and the server where the syncs reside.

Would it be easier to sync to the OS drive and then somehow move the syncs (or copy) to the data drive? And if I did that, wouldn’t my /Documents from LM just fall into the /Documents on Fedora? As would all the other machine /Documents folders? My goal is not to have all document folders from 4 different machines be all in one folder.

Thanks,
Sheila

1 Like
10

I’m not sure if you’re thinking of rsync the way I think of it. I just think of it as a cp command that copies between one computer and another. It could also be used locally.

Maybe just try to work out what the command would be for cp fromX toY and then subtitute in the rsync and the remote user@server/path.

To avoid possible issues with embedded spaces could you avoid using spaces in the directory names and use camel case maybe?

Spaces:
sflanagan@192.168.1.157/run/media/sflanagan/Server Backups/MSI Linux Mint

No spaces:
sflanagan@192.168.1.157/run/media/sflanagan/ServerBackups/MSILinuxMint
sflanagan@192.168.1.157/run/media/sflanagan/ServerBackups/Server1
sflanagan@192.168.1.157/run/media/sflanagan/ServerBackups/Server2

It could be I am not quite following your intent too.

11

That is how I thought it worked. The issue, to me, seems to be
a) copying to a remote computer, and
b) copying to a different drive on remote computer that does not have the same directory structure

So it seemed logical that telling rsync to copy FROM (whatever that may be) TO (wherever that may be) would be the correct way to write the command.

I did wonder about the spaces only because the error output said no such directory and name a single word of a 2-word folder–like no such directory Backups when the actual name was Server Backups. It did the same with MSI Linux Mint in that it said no such directory /Mint. Those two instances say it is using the last word of each folder name.

So I renamed /dev/sda1 to ServerBackups (no spaces) and the folder on ServerBackups to MSILinuxMint. Using a colon to separate from and to resulted in error below:

myviolinsings@mint-desktop:~$ rsync -aAXvz /home/myviolinsings/Documents:sflanagan@192.168.1.157/run/media/sflanagan/ServerBackups/MSILinux Mint
sending incremental file list
rsync: [sender] change_dir "/home/myviolinsings/Documents:sflanagan@192.168.1.157/run/media/sflanagan/ServerBackups" failed: No such file or directory (2)

sent 19 bytes  received 12 bytes  62.00 bytes/sec
total size is 0  speedup is 0.00
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1338) [sender=3.2.7]

And using a space instead of colon:

myviolinsings@mint-desktop:~$ rsync -aAXvz /home/myviolinsings/Documents sflanagan@192.168.1.157/run/media/sflanagan/ServerBackups/MSILinux Mint
sending incremental file list
rsync: [sender] change_dir "/home/myviolinsings/sflanagan@192.168.1.157/run/media/sflanagan/ServerBackups" failed: No such file or directory (2)
Documents/Sheila's Business Vault/.obsidian/workspace.json
Documents/Sheila's Business Vault/Customers/
Documents/Sheila's Business Vault/Customers/Chetan Patel.md
Documents/Sheila's Business Vault/Estimates to be done/
Documents/Sheila's Business Vault/Estimates to be done/Chetan Patel 1.md
Documents/Sheila's Personal Vault/.obsidian/workspace.json
Documents/Sheila's Personal Vault/.obsidian/plugins/recent-files-obsidian/data.json

sent 164,649 bytes  received 817 bytes  330,932.00 bytes/sec
total size is 14,935,291,877  speedup is 90,261.00
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1338) [sender=3.2.7]

Thing I do not undertstand is sending incremental file list actually includes a folder & file that just changed an hour ago. So is it finding the changed files but just cannot copy them to the specified location?

Notice the error:

rsync: [sender] change_dir "/home/myviolinsings/sflanagan@192.168.1.157/run/media/sflanagan/ServerBackups" failed: No such file or directory (2)

And now I don’t know if that means changing the directory in the target (since I renamed those folders) or if it is trying to say it cannnot do it since the directory target is not the same as source.

Thanks,
Sheila

12

I just ran it without the data drive & folder designation in the target (just used sflanagan@192.168.1.157) and it did the sync. Here is the output minus the 1000 files:

myviolinsings@mint-desktop:~$ rsync -aAXvz /home/myviolinsings/Documents sflanagan@192.168.1.157
sending incremental file list
created directory sflanagan@192.168.1.157
 <LIST OF FILES>
sent 10,225,317,956 bytes  received 67,718 bytes  94,243,185.94 bytes/sec
total size is 14,935,291,877  speedup is 1.46

And no errors occurred. But it created the directory on my Linux Mint machine, not on the server at all. So that means we did not sync to the other machine. What does that mean I did wrong with my syntax?

Sheila

13

‘MSI Linux Mint’
you need quotes or Linux sees it as 3 names

Not sure that is the only issue
The rsync command has to be of the form

rsync options source destination

Hou cant hzve multiple destinations.
so

has to be all one word … no blanks… so use quotes .

Relax. I have several gliches like that every day

14

I think the real issue here is using rsync and ssh. I found an article that was more explicit on using ssh with rsync and here is what it said:

How to Create Backups Using Rsync over SSH?

Making incremental backups is one of Rsync over SSH use cases. Please pay attention to the following command to understand how it works:


rsync -av -e ssh /sourceDir/ user@remote:/path/to/destinationDir

Note that -e ssh option demands Rsync connection to be over SSH protocol.

If you want the SSH connection to be accepted over a port other than defaul (port 22), you need to mention the port number after -e ssh. Look at the example command given:

rsync -av -e ssh 1998 sourceDir/ user@remote:/path/to/destinationDir

My options were -aAXvz so I would then add -e ssh 9090 (port used in Cockpit ssh) and then the source dir
Notice the slash after sourceDir? I did not include that after home/myviolinsings/Documents in my previous tries. I know I read that it matters in certain cases.

Then sflanagan@192.168.1.157 should cover user@remote:/
so colon is used but here is where I am unsure if I need to include Fedora Linux/run/media/sflanagan/ServerBackups/MSILinuxMint.

I don’t think so, but remember that /run is on the OS drive called Fedora Linux and that is where the mount is for the 2nd drive. Do I need that?

So I think we are closer to a resolution.

Sheila

1 Like
15

Okay, that makes sense why I kept seeing single folders not created like /Linux and /Mint.
See my next post and we might just get it right.

Thanks,
Sheila

16

@Sheila_Flanagan

Afterthought

You may need to quote the whole destination rather than just the folder name part.

17

I’d start by trying 100% to avoid SPACE in any filenames or paths… And you don’t seem to be escaping them or using quotes eitehr.

rsync -aAXvz /home/myviolinsings/Documents sflanagan@192.168.1.157:/run/media/sflanagan/Server_Backups/MSI_Linux_Mint

And somebody else already pointed out you need a “:” between the DEST HOST and folder.

You could also shorten it like this - create an entry (on the SOURCE system) in ~/.ssh/config e.g.


Host RSYNCTARGET rsynctarget
   Hostname 192.168.1.157
   User sflanagan

And your CLI is now :

rsync -aAXvz /home/myviolinsings/Documents rsynctarget:/run/media/sflanagan/Server_Backups/MSI_Linux_Mint

(because the .ssh/config file has username and a “nickname” for the destination host…

I don’t recall what “AX” does - I mostly only ever use “rsync -av SOURCE DEST” And “z” does on the fly compression - right? Do you need that? I don’t use it myself.

1 Like
18

I wonder if the rsync target folder is relative to the home or default directory for the user specified.

sflanagan@192.168.1.157:/run/media/sflanagan/serverbackups/

Would that end up in /home/sflanagan/run/media/sflanagan/serverbackups/ on the target?

If it isn’t relative does the user have permissions to /run/media/… on the target?

19

So now I tried:

myviolinsings@mint-desktop:~$ rsync -aAXvz -e ssh 9090 /home/myviolinsings/Documents/ 'sflanagan@192.168.1.157:/run/media/sflanagan/Server Backups/MSI Linux Mint'
sflanagan@192.168.1.157's password: 
sending incremental file list
rsync: [sender] link_stat "/home/myviolinsings/9090" failed: No such file or directory (2)
created directory /run/media/sflanagan/Server Backups/MSI Linux Mint
./

And that got us to the server, at least

That is what I was wondering…since one article talked about making a source on the local machine for the target machine. Thanks a lot.

But you are not using the -e ssh [Port#]?

Sheila

20

@pdecker not sure, but that last command I ran with quotes as @nevj suggested created a folder on the ROOT directory of the server and put my files there.

oy vey!!
Sheila