Borg backup questions

Applications
1

I was doing a lot of work in my Solus home directory recently and decided it needed more frequent backups. Instead of going to my usual tar backup procedure, I decided to try borg as it seemed to make more sense to use incrementals when things are changing rapidly.

So I installed borg in Solus, and setup a dedicated borg repository area on a 2Tb USB backup drive. The area automounted as /run/media/nevj/Borg
I decided to make a repository just for Solus backups inside that so

# cd /run/media/nevj/Borg
# mkdir Solus

Then I initialised a repository

# borg init --encryption=none /run/media/nevj/Borg/Solus

and created a backup of /home/nevj

#cd ~nevj
#borg create /run/media/nevj/Borg/Solus::home-{now} .

Note the dot
The backup goes into an archive named by the stuff after the ::
Borg can list all the archives in a repo

#borg list /run/media/nevj/Borg/Solus
home-2022-07-02T17:02:17             Sat, 2022-07-02 17:02:17 
[41e6e90a72984a217ea783d54fc7aa7f4c1b3c977e99c310d332d83655660643]

So I have one archive called home-2022-07-02T17:02:17
I needed to test file recovery so

#cd
#mkdir tmp
#cd tmp

So I will recover to a subdir called tmp

#borg extract /run/media/nevj/Borg/Solus::home-2022-07-02T17:02:17
#ls
 borgtmp          Downloads   Shared               update20220611.out
 common           Music       solusupgrades.txt    Vboxaids
 dawn             packages    Templates            Videos
 Desktop          Pictures    ud140222a.out       'VirtualBox VMs'
 Documents        Public      ud140222.out
 downloadhelper   Recipes     update20220411.out
#

and all the files come back.
You can retrieve files selectively by name. I tried that too.

So I am using borg OK in a very simple case.
But there are questions?

  • What if I now use borg from another Linux version - lets say Debian. It will probably be a different version of borg. My gut feeling is that I should make a separate Debian repository and keep borg archived by Solus separate from borg archived by Debian. That is why I made the Solus subdirectory. But does it matter? Are all borg versions compatable?
  • It is more convenient to run borg from within the distro that I happen to be working in. But it would be possible to do all borg work in one distro and just mount the partitions I want to backup, like I do with tar. Which is a better strategy?
  • The attraction of borg for me is that it is a quick and easy way to backup workfiles… so I am more likely to do it often. Dont plan to use it for system files. Does anyone know of disadvantages or is there an even better solution to my issues?
  • I used the simple command line interface. There is a desktop client for borg called vorta. Has anyone used vorta?
  • I did all that as root. Is that necessary?

References:

1 Like
2

Hi Neville, :wave:

it´s been a while since I last used borg; therefore my knowledge regarding your questions is a bit limited. I may not be totally up-to-date in that respect. Sorry. :blush:

Nevertheless I just looked up (in my fish-history) what borg-related commands I was using in the past and came up with this:

/media/rosika/f14a27c2-0b49-4607-94ea-2e56bbf76fe1/DATEN-PARTITION/Dokumente/Ergänzungen_zu_Programmen/zu_borg/alte_Version/borg-linux64 mount /media/rosika/A492-CD29/borgbackups::7pythonkursbackup /media/rosika/f14a27c2-0b49-4607-94ea-2e56bbf76fe1/DATEN-PARTITION/Dokumente/Ergänzungen_zu_Programmen/zu_borg/borg_mountpoint
Enter passphrase for key /media/rosika/A492-CD29/borgbackups:

So it was just the passphrase that was required, not sudo. :smiley:

I was doing it this way:
I didn´t install borg from the repositories but opted for the standalone binary.

So I simply take it over for a fresh install of my OS (Lubuntu).

usage says:

Borg does not necessarily have to be run with administrator or root privileges.

However, if the user does not have write permission for the storage location or no read permission for all files to be backed up, the following commands may have to be preceded by sudo […]

(translated from German)

Perhaps someone else may be of better help to you. :neutral_face:

Many greetings from Rosika :slightly_smiling_face:

1 Like
3

Well, thanks for the translation…
I think in my case, where I just use borg on my home directory, I could get away with operating as nevj.
And I can get away without the passphrase, if I do not encrypt the repo.

It seems like a good option, where I want a lot of rapid backups because I am changing things rapidly.

Regards
Neville

1 Like
4

Hi Neville, :wave:

You´re very welcome.
Of course you don´t have to use encryption. I just posted the command the way I used it at the time. :blush:
In fact I issued the command right now and was delighted to see everything still works fine after all this time. :wink:

Many greetings.
Rosika :slightly_smiling_face:

5

Hi Rosika,
I am confused. What do you mean by standalone binary?
Did you use a flatpak or something ?

I chose to have no encryption to make it as simple as possible

Regards
Neville

1 Like
6

Hi Neville, :wave:

Uh, let me just think about it…

Well, I´m pretty sure I was following the instructions given by
BorgBackup › Wiki › ubuntuusers.de at the time:

As an alternative to official package sources, standalone binaries can also be used.

To do this, download the desired binary with the name borg-linux32 or borg-linux64 (depending on the system architecture) from the release page and install it as follows:

32-Bit:

sudo cp borg-linux32 /usr/local/bin/borg
sudo chown root:root /usr/local/bin/borg
sudo chmod 755 /usr/local/bin/borg

64-Bit:

sudo cp borg-linux64 /usr/local/bin/borg
sudo chown root:root /usr/local/bin/borg
sudo chmod 755 /usr/local/bin/borg

Advantage: you can also get newer versions than those available in the official package sources.
Disadvantage: you have to take care of updates yourself

(translated from German)

(also: Installation — Borg - Deduplicating Archiver 1.2.7 documentation )

So no alternative packaging is needed; no snap, no flatpak… just a “standalone binary” which just works.

Hope it helps…

Many greetings from Rosika :slightly_smiling_face:

1 Like
7

Hi Rosika,
OK I get it, thank you.
You can only do that if all the dependencies are statically compiled into the binary. That is probably quite easy for borg, it is a simple program.

Sorry to be such a nuisance. There are a lot of things about modern computing that I am not up to date with.
Regards
Neville

1 Like
8

Hi Neville, :wave:

Thanks for the clarification. See, now it´s me who has learnt something new again. :smiley:

No, no. Not at all. Don´t worry. :slightly_smiling_face:

You know so much more that I do about those things (Linux and computer-related topics).
In fact I´m glad I could be of some slight help at all… :blush:

Have a really nice weekend.
Many greetings from Rosika :slightly_smiling_face:

1 Like
9

First of all, great choice! You picked my favourite. It’s a fascinatingly awesome piece of software.
Second of all, it’s obvious you had a good read of the manual. You got all the basics right.
The official manual is really good, though. With loads of examples and sufficient explanations. I wish, that’d be the default case among open source projects…

They are not all compatible. However, don’t worry, you just started recently which means you will never encounter the incompatibility from versions released years ago. So, from your perspective, everything will be compatible and safe, except you are using an alpha, beta or explicitly marked “use at your own risk” version. Otherwise, if you use a recent stable version, they are all compatible.

To maximise compatibility and safety, I would recommend to natively install the package.

pip3 install -U pip setuptools wheel
pip3 install pkgconfig
pip3 install borgbackup[llfuse]
Source

This way, you make absolutely sure that you have precisely the same borg version everywhere. Then, over time, they may become superseded by newer versions, but as previously mentioned, that does not matter. If you update them at least once a year or every other year, you are fine.

I had an article explaining all that, but it is currently unavailable due to server structure migrations.

Not sure, if I understand correctly. Are you simply talking about using the borg CLI or about the location your archives reside? In my case, I have a central borg server, where I put all my backups and from there I ensure data redundancy by spreading the backup across other servers.

I think borg covers everything regarding backups. It’s an astonishing tool. If you find it quick and easy, you should start using it everywhere, exclusively, in my opinion.

The only thing it cannot do is full image backups. However, these bloat monsters are rarely ever useful. They are easy, but eat tons of space for no reason but to have an exact copy, which is almost never needed.

I did not use it, because I almost exclusively work with servers, not having any GUI available, but I heard that piece of software is worth looking into for a more convenient experience.

Yes and no. Technically, it’s not necessary from the view of borg itself. However, if you want to backup an entire Linux installation, you won’t get around working as the root user, otherwise you wouldn’t be able to create a full backup.

However, it’s recommended to only use the root user if necessary. If you are, for example, only backing up your home directory, you should definitely use the user owning that directory.

Another issue may arise if you mix root and normal users on a single repository. If I remember correctly, it might be the case some things in the repository might be blocked off, if you used the root user once to create an archive. However, not sure about that, anymore. It also depends on which user is actually accessing the repository.
If you are backing up as root but the borg server is using a normal user, I think that’s the best and most general as well as convenient solution to the issue at hand.

https://borgbackup.readthedocs.io/en/stable/index.html

2 Likes
10

Thank you @Akito , that is a comprehensive answer.

1 Like
11

I read the doc
https://borgbackup.readthedocs.io/en/stable/index.html

One thing not clear about how Borg works.
What happens if a file is modified by a user, while it is being copied by Borg?
Traditional archivers like tar make a mess of your archive in this sitation. Does Borg lock a file while it is copying it? … The way vi does when you edit a file?
If not, Borg should only be used in a single user machine, and even then taking care not to do other things while it is running.

12

Does it? I used tar a lot on log files that would change each second. Nothing broke, at all.

Not sure. I think, it would make a copy of it in memory or something and then use this “old” version for backup. At least, that’s what my guess is.

I use single user machines, where I use Borg, but I never was watching out for not doing anything else while it runs. Still, all backups work perfectly fine. No problem.

13

I will setup a test. I believe you, it is probably OK.
I have never had trouble with tar or rsync either.

If there is no issue, Borg should be OK for system snapshots too.

1 Like
14

There is some info on the Borg site

Important note about files changing during the backup process

Borg does not do anything about the internal consistency of the
 data it backs up. It just reads and backs up each file in whatever
 state that file is when Borg gets to it. On an active system, this 
can lead to two kinds of inconsistency:


    By the time Borg backs up a file, it might have changed since 
the backup process was initiated

    A file could change while Borg is backing it up, making the file 
internally inconsistent


If you have a set of files and want to ensure that they are backed
 up in a specific or consistent state, you must take steps to 
prevent changes to those files during the backup process. There 
are a few common techniques to achieve this.

    Avoid running any programs that might change the files.
    Snapshot files, filesystems, container storage volumes, or 
logical volumes. LVM or ZFS might be useful here.
    Dump databases or stop the database servers.
    Shut down virtual machines before backing up their images.
    Shut down containers before backing up their storage volumes.

For some systems Borg might work well enough without these 
precautions. If you are simply backing up the files on a system 
that isn’t very active (e.g. in a typical home directory), Borg 
usually works well enough without further care for consistency. 
Log files and caches might not be in a perfect state, but this is 
rarely a problem.


For databases, virtual machines, and containers, there are 
specific techniques for backing them up that do not simply use 
Borg to backup the underlying filesystem. For databases, check 
your database documentation for techniques that will save the 
database state between transactions. For virtual machines, 
consider running the backup on the VM itself or mounting the 
filesystem while the VM is shut down. For Docker containers,
 perhaps docker’s “save” command can help.

Site is here
https://borgbackup.readthedocs.io/en/stable/quickstart.html

It would seem serially written files are safe, but random access
files like databases , or containers, might be an issue.
Also this does not only apply to Borg. All backup programs, even tar, have the same issue, if used while Liux is running.

15

I think, it sounds more impactful than it actually is in real life. Most things that change continuously have integrity mechanisms, in place, anyway. Let’s say, you are backing up a database and some file changes in the process. If you are using any major database, it should recover easily, most likely just discarding the newest change.

Additionally, I already made tons of backups in the past few years on a variety of systems and had to restore quite a few full operating system backups, including home and whatever else was on the storage medium. Still, no problem. I never experienced an issue, when restoring. Actually the opposite. I was initially surprised how easy and painless it went.

That said, in the worst case scenario, you can just dump databases beforehand, if it’s really that important. Or, you make very frequent backups, so you can maybe restore an older backup, too, without losing too many new additions or changes.

Containers are never an issue. They are designed to be ephemeral and reproducible. So, you can break the hell out of them and it does not matter. The only things you can break is when a container uses a database or configuration setup, which is stored in a local volume, not in the container. However, in that case the above measures can be taken, like e.g. dumping a database beforehand.

16

I like that. Gives me a bit of confidence.