How BorgBackup works and User Guide for BorgBackup

guide
#1

Moved this topic to there.

Decided to copy over this reply to its own thread so more people can profit from it. Currently a work in progress, as I will add some examples on how to use the program and what commands are good for certain oprations.


So apparently many people are interested in a good backup solution in general, so I will present to you the one I am mainly using and I personally think it is better than all the backup solutions I have seen being talked about in the whole forum, together.

BorgBackup Github

BorgBackup Documentation

What is BorgBackup?

BorgBackup (short: Borg) is a deduplicating backup program. Optionally, it supports compression and authenticated encryption.

The main goal of Borg is to provide an efficient and secure way to backup data. The data deduplication technique used makes Borg suitable for daily backups since only changes are stored. The authenticated encryption technique makes it suitable for backups to not fully trusted targets.

Main Features

Space efficient storage

Deduplication based on content-defined chunking is used to reduce the number of bytes stored: each file is split into a number of variable length chunks and only chunks that have never been seen before are added to the repository.

Speed
  • performance critical code (chunking, compression, encryption) is implemented in C/Cython
  • local caching of files/chunks index data
  • quick detection of unmodified files
Data encryption

All data can be protected using 256-bit AES encryption, data integrity and authenticity is verified using HMAC-SHA256. Data is encrypted clientside.

Compression

All data can be optionally compressed:

  • lz4 (super fast, low compression)
  • zstd (wide range from high speed and low compression to high compression and lower speed)
  • zlib (medium speed and compression)
  • lzma (low speed, high compression)
Off-site backups

Borg can store data on any remote host accessible over SSH. If Borg is installed on the remote host, big performance gains can be achieved compared to using a network filesystem (sshfs, nfs, …).
Backups mountable as filesystems

Free and Open Source Software
  • security and functionality can be audited independently
  • licensed under the BSD (3-clause) license, see License for the complete license

The catch with BorgBackup is that it takes some time to understand the concept and use it appropriately. It definitely takes some time getting used to, especially for people not familiar with advanced backup solutions. But once you get the gist of it, it will definitely be a pleasure and I am sure a majority would stick to this solution for most backup targets.

Here’s how I personally would explain how it works and how I use it

The program has 2 main functions. The first is creating a repository, the second is creating an archive within a repository.

Repository

The repository is basically the world that stores all the content as archives for you. The special thing about this world is that every single thing exists only once. Everyone can have a simulation of all things, e.g. everyone can have an orange and use it in this world, but actually there is literally only a single orange in the whole world.

Archive

An archive is a certain state of the world, defined by the time the state (snapshot, basically) was captured. Let’s say, yesterday your brother had an orange, but your sister and you had none. Today, your brother gives you an orange, so you have one now, but your sister and brother have none. If you create a snapshot, i.e. archive, every day in the evening, then there will be 2 archives. 1 from yesterday where your brother had an orange but your sister and you didn’t, the second one is from today where you have an orange but your brother and sister have none. That means that you have 2 entire archives that are basically standalone (you can delete one of them and the other one will remain as it should) while the space used for both archives equals to the space only 1 archive uses because all that changed in the world is that the orange changed its owner, so no additional data was added, which means that the size of the new archive seems to increase by 0 because the other archive already contains the needed data for the new archive.
Now if your sister gets an orange tomorrow, so that you and your sister have one each now, then the archive from tomorrow will only increase the size of the respository by a couple of bytes (if the size of owning an orange would be a couple of bytes, that is; NOTE: the orange itself does not get duplicated, the only thing that gets saved additionally, is that your sister has the orange now, but the orange exists only once in the whole world, as explained above).

Now comes the even more interesting part. Let’s say, every day there are major changes in the whole world but the only thing you care about is the orange situation at home, for now. Your very first archive already contains the whole world ( i.e. e.g. root directory / ). Now further backups only make a snapshot of the orange situation ( i.e. e.g. /home/*/oranges-directory ). This directory is part of the root directory so all the data is already in the initial backup and doesn’t need to be additionally stored. The only thing that is stored in the newest archive, are the changes in the oranges-directory, effectively ignoring all other changes in other places.

Real world example

I had a repository containing an initial archive of my root directory /. Yesterday, I created an additional archive of my Downloads folder, because I downloaded some .deb files; i.e. /home/user/Downloads. Today, I updated my Debian archive mirror, so I only backed up the /var/debian folder. Tomorrow, I will update the whole root directory / once again.

How much space will all this use? I have 2 separate backups from 2 separate days from the whole root directory / and yet all the space that will be used is pretty much the space that the whole root directory / + the couple of .deb files I downloaded, need. Nothing else. My Debian archive mirror only updated the packages, didn’t add any new ones. My system overall didn’t change much, except I have a couple more .deb files in my Downloads folder. So you can pretty much have 100s of different archives, each saving the state of when the snapshot was taken and at what location, but the size won’t increase, at all, except you actually add entirely new data. Therefore it already takes almost no space to backup everything you need to backup, and yet you can optionally compress everything, too, so the space needed is EVEN SMALLER.

Real world example from my Raspberry Pi system:

The root directory / of my Raspberry Pi 3B takes about 12-14GB of space on my SD card. The actual initial Borg archive of the whole SD card takes up about 4GB in space, after low compression (so you can compress the data even higher if you have a more compute ready machine).
Now, do you have several Raspberry Pis but don’t want to use ~4GB for each Raspberry Pi? No problem, just make archives of all the different Pis in the same repository and if the data on all the Raspberry Pis is more or less the same datawise, then the repository will be maybe ~4.5-5GB in size, despite backing up 4 Raspberry Pis (real world example from my own setup).


I hope I could explain the system well enough to you, since I had to try out BorgBackup several times to finally get the gist of how to use it at best.

P.S.: You can also safely encrypt all your backup data. I personally don’t need that option, but it definitely pumps up the value of this backup solution by a whole lot, as well.


Borg 1.1.9

Installation

Debian

Install Dependencies
sudo apt install python3 python3-dev python3-pip python-virtualenv libssl-dev openssl libacl1-dev libacl1 build-essential libfuse-dev fuse pkg-config
Install BorgBackup through Python pip

Note that this part can take some time, especially on very old machines or SBCs, for example.

sudo pip3 install borgbackup[fuse]

Ubuntu

Install Dependencies
sudo usermod -aG fuse #appendyourusernamehere
sudo apt install python3 python3-dev python3-pip python-virtualenv libssl-dev openssl libacl1-dev libacl1 build-essential libfuse-dev fuse pkg-config
Install BorgBackup through Python pip

Note that this part can take some time, especially on very old machines or SBCs, for example.

sudo pip3 install borgbackup[fuse]

User Guide

Create backup of your development files to another hard drive

  • ~/src is the folder with your development files
  • /dev/sda i.e. / is where your system is installed and /dev/sdb i.e. /mnt/usb-hdd0 is an attached external hard drive
mkdir -p /mnt/usb-hdd0/backups/borg/repos/$HOSTNAME/    # creates directory for backed up files
cd /mnt/usb-hdd0/backups/borg/repos/                    # changes to this directory
borg --verbose init --encryption=none $HOSTNAME          # initializes a borg repository
cat README                                              # verify the creation was executed successfully
borg --verbose --progress create --stats --comment "Initial backup of my dev files." /mnt/usb-hdd0/backups/borg/repos/$HOSTNAME/::firstbackup /home/$USER/src
# creates the initial archive for your dev files in the above created repository

Now backing up your dev files the next day, again…

borg --verbose --progress create --stats --comment "Backup of my dev files after several commits to my main project." /mnt/usb-hdd0/backups/borg/repos/$HOSTNAME/::secondbackup /home/$USER/src
# creates the second archive for your dev files in the above created repository

Wait, when did I back up my dev files the last time and how much space do they take, exactly?

borg --verbose info --last 1 /mnt/usb-hdd0/backups/borg/repos/$HOSTNAME/
# shows all information about the newest backup in this repository
# e.g. time/date of creation, comment, size, etc.

Wait, what files exactly did I back up in my last backup?

borg --verbose list --short --last 1 /mnt/usb-hdd0/backups/borg/repos/$HOSTNAME/
# prints all file and directory names contained
# within the last archive of this repository

Create a complete backup of your entire OS

In case you didn’t create a repository as in the above example, yet:

# creates directory for backed up files
mkdir -p /mnt/usb-hdd0/backups/borg/repos/$HOSTNAME/
# changes to this directory
cd /mnt/usb-hdd0/backups/borg/repos/$HOSTNAME/    
# initializes a borg repository      
borg --verbose init --encryption=none 
# verify the creation was executed successfully                   
cat README                                             

Now the actual backup of the whole system:

sudo borg --progress --verbose create --comment "Use as root. My first system backup." -e /dev -e /run -e /tmp -e /sys -e /proc -e /mnt -s /mnt/usb-hdd0/backups/borg/repos/$HOSTNAME/::firstsystembackup /
# This creates a backup of the whole system, excluding locations
# that are re-generated at every boot up, as well as, the `/mnt` location
# as we don't want to back up our backups, especially when you
# have a backup drive containing several hundred GB of data.

Note that it is important to not mix up owners when backing up and later restoring backups. If you are unsure that you completely own a location, you should execute the backup as root. When doing a whole system backup, of course you have no choice but to execute the backup as root.

To be continued…
1 Like

Another Simple Question
Why do my backups not completely restore all the files? Solved
Deja-dup not working on schedule. in Lite 4.2
#2

thank you for moving this to a more visible location. i still want to give it a go one of these days and think this will make it easier to find than trying to remember to dig it out of the clonezilla thread :slight_smile:

1 Like

#3

I am also adding examples for use cases I find common and/or useful. So it should be even easier to use, since the documentation is extensive but it’s not easy to puzzle everything together that quickly, so I summarize all the stuff on point.

1 Like