Do-It-Yourself Backup System Using Rsync and Btrfs
Written by Kevin Korb
as a presentation for GOLUG
Presented 2011-04-06
This document is available at http://www.sanitarium.net/golug/rsync+btrfs_backups_2011.html
Note: This presentation is designed to be a follow-up of my old presentation on using rsync for hard link based backups. In it I discussed many different things about backups in general and how to use various rsync features. I do not intend to repeat much of that information but instead will concentrate on the differences and benefits of Btrfs when using rsync for backups. There will be a few exceptions that I think need to be in this document and I will put a * in the bullet list for items that are simply copied from the old presentation. The old presentation can be found here: http://www.sanitarium.net/golug/rsync_backups_2010.html
- *What is rsync?
Rsync is a program for synchronizing two directory trees across different file systems even if they are on different computers. It can run its host to host communications over ssh to keep things secure and to provide key based authentication. If a file is already present in the target and is the same as on the source the file will not be transmitted. If the file on the target is different than the one on the source then only the parts of it that are different are transferred. These features greatly increase the performance of rsync over a network.
- What is Btrfs?
Btrfs is a "next generation" filesystem created for the Linux operating system. It has many new features that previous Linux based filesystems (ext2, ext3, ext4, xfs, reiserfs) did not have. The new features that are useful for rsync backups are subvolumes and subvolume snapshots. With Btrfs we can make many subvolumes within a filesystem that will mostly operate like subdirectories. We can then do block level snapshots of those subvolumes allowing us to maintain older data at the block level without having to do the snapshot at the LVM2 layer which would snapshot the entire filesystem. This allows for much faster purging of old rsync backups when they are no longer needed.
- What is a subvolume?
In Btrfs a subvolume is a virtual filesystem contained within the main filesystem. They have some of the properties of subdirectories and some of the properties of mount points. All of the disk space used by all of the subvolumes is allocated from the main filesystem so you don't have to worry about disk space on each subvolume. If you mv a file from one subvolume to another it will act as if they are separate filesystems and it will copy and delete rather than simple moving. Hard links do not work across subvolumes. However, subvolumes do not show up in df or mount since they are all part of the same btrfs filesystem.
- What is a subvolume snapshot?
This is where it becomes interesting. A subvolume can also be a block level CoW based snapshot of another subvolume. This means you can have the same data in multiple subvolume snapshots without taking up any extra disk space. If a file is modified on any of them then the new version will be unique to that snapshot and it will consume disk space as needed through the CoW mechanism.
- Faster? How much faster?
The performance of the backup itself will not change much. Rsync will not be running with --link-dest so it will not have to check other directories for files or create hard links but that isn't that much of a performance difference. The real difference comes when you delete a backup. With hard link based backups the deletion procedure is a simple 'rm -rf' of the backup you want to delete. This can take minutes, hours, or even days depending on how many links, files, and directories need to be deleted. With Btrfs subvolume snapshots you would instead simply delete the snapshot which takes up to a few seconds regardless of the size or how many files are involved.
- What about ZFS?
While I do not plan to cover ZFS it is worth mentioning that ZFS also has these features and can do backups the same way. The commands of course will be different and different operating systems have different degrees of support for ZFS (I only recommend using it on Solaris or OpenSolaris).
- *Why not just use RAID / Is this like using RAID-1? / Is this like DRBD?
I don't think I can ever say this enough times.... RAID is NOT a backup system! RAID (other than level 0) does a wonderful job of protecting your data from disk failures. However, it provides absolutely NO protection against file corruption, files destroyed by a virus or a hacker, or the "oops, I deleted the wrong file" problem which most of us have encountered. There is a time and a place for RAID and RAID is not always needed however data should ALWAYS be backed up regardless of what media it is stored on or how redundant that media may be. Networked mirroring solutions like DRBD have the same drawbacks as RAID as they are a simple real-time mirror. My general rule of thumb is that if you can't restore your data to the way it was last Monday or last night using a storage device other than the one the data was on last Monday then you don't have a backup system.
- How is this different from using LVM2 snapshots? I already do that to backup my databases!
These are two completely different things. When you do an LVM2 snapshot to backup a database you are doing that on the client so that rsync has a frozen copy of the files to copy. You should STILL DO THIS. Using Btrfs subvolumes and snapshots is done on the backup server to organize and maintain backups like what --link-dest used to do. Btrfs will store block level differences between backups done on different dates instead of storing complete new files and hard links to files that didn't change.
- Btrfs is still listed as experimental/unstable. Is this safe?
Btrfs is in fact still both experimental and unstable. IMHO it is good enough for backups. Meaning that it is unlikely for it to fail and very unlikely for it to fail when you happen to need it. Also, it will not be unstable for long and now is the time to prepare for it. I suggest using a kernel that is version 2.6.36 or later and running btrfs-progs from the git repository at least for now.
- How do you do off-site backups with rsync+btrfs?
Unfortunately I don't see an easy way to do this yet. In order to do it correctly the other end would have to have the same Btrfs subvolumes and snapshots of them. Btrfs does not yet provide a mechanism to copy this and tools like rsync can't yet tell the difference between a subvolume and a subdirectory. The off-site backup can't be hard link based because there is nothing in the filesystem to tie files from different subvolume snapshots together (they will have a link count of 1, different size+mtime, and yet the same inode number). Currently the only way I see to do this is to DRBD the entire thing or to simply have 2 backup serves operating independently. Neither solution is optimal. Hopefully as the Btrfs programs evolve a better method will emerge.
- Organizing backups
Since this is a do-it-yourself system this is totally up to you to design. I will explain my system as an example...
- I have my rsync+Btrfs backup filesystem mounted under /backup/rsync.
- Within that filesystem I make a subvolume for each host that gets backed up.
- Then for each backup of each file system I change '/' to '_' in the mount point name and time stamp the file system so my backup of /home/asylum done at 17:47 on 2010-07-25 would be stored in /backup/rsync/asylum/_home_asylum.2010-07-25.17-47-42 which will be a subvolume snapshot.
- When the backup is done I would create a symlink from that directory to /backup/rsync/asylum/_home_asylum.current to make it easier to find especially from scripts.
- The rsync target is always the same directory which is a subvolume (not a snapshot) and I just use the _path notation with no extension. All of the date+time stamped backups are snapshots of this subvolume.
- So, /backup/rsync/asylum/_home_asylum.2010-07-25.17-47-42 would be a snapshot of /backup/rsync/asylum/_home_asylum created as soon as that rsync finished on that date+time.
- The actual backup procedure
This pseudo code would be run for each filesystem being backed up (assuming my organization and assuming we are backing up asylum:/home/asylum)...
- if there is no directory named /backup/rsync/asylum then btrfs subvolume create /backup/rsync/asylum
- if there is no directory named /backup/rsync/asylum/_home_asylum then btrfs subvolume create /backup/rsync/asylum/_home_asylum
- rsync --archive --one-file-system --hard-links --inplace --numeric-ids --delete --delete-excluded --exclude-from=excludefile asylum:/home/asylum/ /backup/rsync/asylum/_home_asylum/
- if rsync failed interpret the error and blow up if needed
- btrfs subvolume snapshot /backup/rsync/asylum/_home_asylum /backup/rsync/asylum/_home_asylum.`date +'%Y-%m-%d.%H-%M-%S'`
- Rotating backups
Once you have enough backups in place you will want to start deleting the old ones while maintaining however many old copies you need. To do this you simply delete the subvolume snapshots of the oldest backups until there is no longer too many of them. Unlike with hard link based backups there is no reason to not do this during the backup process as it does not take much time. The command is simply 'btrfs subvolume delete' instead of 'rm -rf'.
- Can I use --sparse instead of --inplace?
Yes, you can. However, if you do rsync will completely rewrite any file that has changed between backups causing the btrfs subvolume snapshot to grow by entire files instead of just the blocks that have changed. If you have significant sparse file usage this might still be smaller that writing out all the null space of a sparse file but for most people --inplace will save more space than --sparse. Hopefully some day rsync will allow these two features to work together.
- What can this system not do yet do to the incompleteness of Btrfs?
- Currently Btrfs does not provide any mechanism to determine how much disk space is used by each snapshot.
- Currently Btrfs does not have a fully functioning fsck. Unfortunately this means that there is a possibility of filesystem corruption that can only be fixed by a reformat. This is being worked on and should be corrected soon.
- Currently there is no easy method of determining what files are different between two snapshots. Since the same inode numbers are in use the tool I wrote for --link-dest based backups will not work for this. I will probably eventually write something but for now I am waiting to see if Btrfs itself adds something that can do it.
- Currently the only way to determine if a directory within a btrfs filesystem is just a directory or if it is really a subvolume is to check the inode number. All subvolumes will be inode number 256.
- What can you do with --link-dest based backups that you can't do with this by basic design?
- Manually move something from one backup to another. I often do this in my personal backups if data is moved from one system to another or from one filesystem to another. It allows me to keep large things from being re-backed up when they are moved. Unfortunately subvolumes act like different filesystems when it comes to moving data so this just isn't useful anymore.