Introduction: Backup Your Server
More by the author:
About: Software developer, Placethings co-founder, and technologist. Currently attending graduate school in the Emerging Media and Communications program at the University of Texas at Dallas.
Learn how to backup your *nix box to an external harddrive (or tapedrive without much effort). I cover installation of the backup medium, using `dump`, restoring, and also backing up files from a remote server to an external harddrive.
To backup a Windows PC, read the lifehacker.com article here.
Step 1: Get stuff ready
Step 2: Take a `dump`
Step 3: `restore
Step 4: Remote backups
Step 5: Automation
The FreeBSD Logo is a trademark of The FreeBSD Foundation and is used
by John Syrinek with the permission of The FreeBSD Foundation.
To backup a Windows PC, read the lifehacker.com article here.
Step 1: Get stuff ready
Step 2: Take a `dump`
Step 3: `restore
Step 4: Remote backups
Step 5: Automation
The FreeBSD Logo is a trademark of The FreeBSD Foundation and is used
by John Syrinek with the permission of The FreeBSD Foundation.
Step 1: Get Stuff Ready
Before you make a backup, you need to have something to backup to. Tradionally, this has been tape drives; however, a (quality) external harddrive will work just fine. You don't HAVE to use an external harddrive, but with an external drive you have the convenience of being able to bring the harddrive home (or to some other off-site location).
I used a couple of Western Digital MyBook's. Circuit City had an 80% off sale on (select) external drives, and I got two 250GB drives for dirt cheap. In my experience, Western Digital has very high quality drives (meaning they last forever). This makes them great for a backup. If you're going to go through the hassle of backing something up, you don't want your backup medium crapping out on you.
OK, just like my other article, I'll be using FreeBSD®; however, most of the things I'll be covering can be done in any flavor of Linux, Unix, or BSD.
(Skip the next paragraph if you already have USB 2.0 support or aren't using an external USB drive)
The MyBooks are USB 2.0 drives. FreeBSD 5.4-STABLE doesn't have the EHCI driver (basically the thing that gives you USB 2.0) enabled by default. This is an easy fix, though some might think recompiling the kernel to be scary (it's not). If you're one of the overly cautious types, I recommend doing a backup BEFORE recompiling your kernel. USB 2.0 might not be enabled, but USB 1.1 still works. It's just much slower. To enable EHCI, read this page of the Handbook. You will probably also have to refer to this section of the Handbook which explains how to actually recompile the kernel.
If you're using an external drive, or even an internal drive, you will need to mount the drive before you can use it. This is done with the `mount` command, and is pretty straightforward. Here's mount's man page. If mount is complaining about not being able to determine the filesystem type, you'll probably need to format the drive. To do this, you will need to know the correct device to format. For me, it was /dev/da0, but for you it may be different. Consult your distro's documentation. After determining your which device your external HD is attached to, you will need to do the actual formatting of the drive (well, partition). If you need help partitioning your drive, just ask me. FreeBSD uses mkfs to create filesystems on partitions. Any type of filesystem will work, but I opted to use UFS because that's what FreeBSD uses by default. FAT32 is probably the most compatible with other operating systems, and Ext3 is what most Linux flavors are using nowdays (or atleast they did last time I used Linux).
So, I used this command to mount my drive: mount -t ufs /dev/da0 /backup
OK, you should have your backup medium ready to go. If not, just ask :)
Proceed to Step 2.
The mark FreeBSD is a registered trademark of The FreeBSD Foundation
and is used by John Syrinek with the permission of The FreeBSD Foundation.
I used a couple of Western Digital MyBook's. Circuit City had an 80% off sale on (select) external drives, and I got two 250GB drives for dirt cheap. In my experience, Western Digital has very high quality drives (meaning they last forever). This makes them great for a backup. If you're going to go through the hassle of backing something up, you don't want your backup medium crapping out on you.
OK, just like my other article, I'll be using FreeBSD®; however, most of the things I'll be covering can be done in any flavor of Linux, Unix, or BSD.
(Skip the next paragraph if you already have USB 2.0 support or aren't using an external USB drive)
The MyBooks are USB 2.0 drives. FreeBSD 5.4-STABLE doesn't have the EHCI driver (basically the thing that gives you USB 2.0) enabled by default. This is an easy fix, though some might think recompiling the kernel to be scary (it's not). If you're one of the overly cautious types, I recommend doing a backup BEFORE recompiling your kernel. USB 2.0 might not be enabled, but USB 1.1 still works. It's just much slower. To enable EHCI, read this page of the Handbook. You will probably also have to refer to this section of the Handbook which explains how to actually recompile the kernel.
If you're using an external drive, or even an internal drive, you will need to mount the drive before you can use it. This is done with the `mount` command, and is pretty straightforward. Here's mount's man page. If mount is complaining about not being able to determine the filesystem type, you'll probably need to format the drive. To do this, you will need to know the correct device to format. For me, it was /dev/da0, but for you it may be different. Consult your distro's documentation. After determining your which device your external HD is attached to, you will need to do the actual formatting of the drive (well, partition). If you need help partitioning your drive, just ask me. FreeBSD uses mkfs to create filesystems on partitions. Any type of filesystem will work, but I opted to use UFS because that's what FreeBSD uses by default. FAT32 is probably the most compatible with other operating systems, and Ext3 is what most Linux flavors are using nowdays (or atleast they did last time I used Linux).
So, I used this command to mount my drive: mount -t ufs /dev/da0 /backup
OK, you should have your backup medium ready to go. If not, just ask :)
Proceed to Step 2.
The mark FreeBSD is a registered trademark of The FreeBSD Foundation
and is used by John Syrinek with the permission of The FreeBSD Foundation.
Step 2: Take a `dump`
<lame joke here>
Let's backup our stuffs. There are a few ways to do this. Dump and Tar are probably the two most common, and both have their own strengths and weaknesses. Dump is hands-down the most reliable way to backup your system; however, it can only backup entire partitions. Tar is quick and easy to use on individual folders, but takes some time to backup larger volumes. Tar also compresses the files, adding a layer of complexity to potentially corrupt your backups. Read this page for more information
I decided to use dump because of the reliablity. Storage space wasn't an issue, and since I have cron performing backups for me automatically while I sleep, I don't have to worry about timeframes. One of dump's quirks is that it backups up entire partitions. This means that you have to dump each partition individually (eg. the /usr, /var, and /tmp partitions, as well as the / partition). Dump lets you specify the "level" of backup as well. I'll be performing weekly and nightly backups. For my weekly backups, I use level 0, and for my nightly backups, I use level 2. If you're using a tape drive, or if you want to conserve storage space, consider using a Tower of Hanoi backup scheme (Google it.
(Remember, /backup is where I have my external HD mounted)
The commands I used for weekly dumps are:
dump -0Lna -C 100 -f /backup/weekly/usr/`date "+%Y-%B-%d"` /usr
Of course, you will need to create any appropriate backup destination directories before running the dump command, but you should be able to figure that out.
And now you should have a snapshot of your system, or atleast know how to create one.
The next step is how to use restore and how to make "fixit" floppies. DON'T SKIP THIS STEP or you're wasting your time.
Let's backup our stuffs. There are a few ways to do this. Dump and Tar are probably the two most common, and both have their own strengths and weaknesses. Dump is hands-down the most reliable way to backup your system; however, it can only backup entire partitions. Tar is quick and easy to use on individual folders, but takes some time to backup larger volumes. Tar also compresses the files, adding a layer of complexity to potentially corrupt your backups. Read this page for more information
I decided to use dump because of the reliablity. Storage space wasn't an issue, and since I have cron performing backups for me automatically while I sleep, I don't have to worry about timeframes. One of dump's quirks is that it backups up entire partitions. This means that you have to dump each partition individually (eg. the /usr, /var, and /tmp partitions, as well as the / partition). Dump lets you specify the "level" of backup as well. I'll be performing weekly and nightly backups. For my weekly backups, I use level 0, and for my nightly backups, I use level 2. If you're using a tape drive, or if you want to conserve storage space, consider using a Tower of Hanoi backup scheme (Google it.
(Remember, /backup is where I have my external HD mounted)
The commands I used for weekly dumps are:
- dump -0Lna -C 100 -f /backup/weekly/root /
- dump -0Lna -C 100 -f /backup/weekly/usr /usr
- dump -0Lna -C 100 -f /backup/weekly/var /var
- dump -0Lna -C 100 -f /backup/weekly/tmp /tmp
- dump -2Lna -C 100 -f /backup/nightly/root /
- dump -2Lna -C 100 -f /backup/nightly/usr /usr
- dump -2Lna -C 100 -f /backup/nightly/var /var
- dump -2Lna -C 100 -f /backup/nightly/tmp /tmp
dump -0Lna -C 100 -f /backup/weekly/usr/`date "+%Y-%B-%d"` /usr
Of course, you will need to create any appropriate backup destination directories before running the dump command, but you should be able to figure that out.
And now you should have a snapshot of your system, or atleast know how to create one.
The next step is how to use restore and how to make "fixit" floppies. DON'T SKIP THIS STEP or you're wasting your time.
Step 3: `restore`
It's been a while since I've had to restore a backup, so bear with me.
To restore a backup, you have to have some kind of minimal OS to transfer the backup from your backup medium to the live machine, a LIVE machine (eg, no faulty hardware), and your backups themselves.
For the minimal OS, I use the same CD that I used to install FreeBSD. Sysinstall has a "Fixit" mode to restore backups. If you have non-standard hardware, you may need to create your own custom bootable disk. This will not be covered in this article, but it basically consists of creating a barebones kernel and putting it on a bootable disk. Note: A VERY minimal FreeBSD 5.4 kernel is around 2.3MB, meaning it won't fit on a single floppy.
So basically, if the poop hits the fan (ger ger ger), you boot from your CD, enter "Fixit" mode, mount your harddrive, and then run the restore command.
I believe you have to mount and unmount the partitions that you're restoring one at a time. Also, your partition tables must be clean, meaning you may have to use `bsdlabel` to fix your partitions.
Restore command:
(after mounting a clean partition and changing to the directory of the destination partition)
restore vrf /dev/da0
Please note that it's possible to restore parts (individual files or directories) of backups created using dump if you need to.
To restore a backup, you have to have some kind of minimal OS to transfer the backup from your backup medium to the live machine, a LIVE machine (eg, no faulty hardware), and your backups themselves.
For the minimal OS, I use the same CD that I used to install FreeBSD. Sysinstall has a "Fixit" mode to restore backups. If you have non-standard hardware, you may need to create your own custom bootable disk. This will not be covered in this article, but it basically consists of creating a barebones kernel and putting it on a bootable disk. Note: A VERY minimal FreeBSD 5.4 kernel is around 2.3MB, meaning it won't fit on a single floppy.
So basically, if the poop hits the fan (ger ger ger), you boot from your CD, enter "Fixit" mode, mount your harddrive, and then run the restore command.
I believe you have to mount and unmount the partitions that you're restoring one at a time. Also, your partition tables must be clean, meaning you may have to use `bsdlabel` to fix your partitions.
Restore command:
(after mounting a clean partition and changing to the directory of the destination partition)
restore vrf /dev/da0
Please note that it's possible to restore parts (individual files or directories) of backups created using dump if you need to.
Step 4: Remote Backups
Remote backups can be done using rdump, scp, or custom software.
Most hosting companies provide (for a fee) nightly backups. I HIGHLY recommend this if you value your data. I've had two dedicated servers crap out on me from two different hosts. Although these backups are typically stored on a separate drive, they are usually in the same building, so if something happens to the building (which is unlikely, but definately possible), then you're SOL. This is why I recommend making your own off-site (relative to the system being backed up) backup as well as local, nightly backups. Please note that I had to add the .txt extension to upload the script (you can remove this).
Because the company hosting my remote machine performs nightly `tar` backups, I decided to just copy these files to a local machine on a nightly basis. I've written a PHP-driven (because that's what I know) shell script that bascially syncs a remote system's backup files with a local copy. It downloads new files, (optionally) re-downloads backup files that have discrepencies in filesizes, and removes local copies that don't exist on the remote system. This saves bandwidth, time, and storage space. It's basically just a `diff` wrapper for `scp`. The source can be found at the bottom of this page. If you use it, be sure to chmod it to have execute permission (chmod u=+rx fetchbackups). I recommend having the operator user run this script (chown operator fetchbackups).
Alternative remote backup methods include using `scp` on an entire directory, using `rdump`, or, as clievers pointed out, using `rsnapshot` or `backuppc`. If you don't have root access to your remote server, some of these methods might not be possible, and `scp` often requires a lot of time and bandwidth.
Most hosting companies provide (for a fee) nightly backups. I HIGHLY recommend this if you value your data. I've had two dedicated servers crap out on me from two different hosts. Although these backups are typically stored on a separate drive, they are usually in the same building, so if something happens to the building (which is unlikely, but definately possible), then you're SOL. This is why I recommend making your own off-site (relative to the system being backed up) backup as well as local, nightly backups. Please note that I had to add the .txt extension to upload the script (you can remove this).
Because the company hosting my remote machine performs nightly `tar` backups, I decided to just copy these files to a local machine on a nightly basis. I've written a PHP-driven (because that's what I know) shell script that bascially syncs a remote system's backup files with a local copy. It downloads new files, (optionally) re-downloads backup files that have discrepencies in filesizes, and removes local copies that don't exist on the remote system. This saves bandwidth, time, and storage space. It's basically just a `diff` wrapper for `scp`. The source can be found at the bottom of this page. If you use it, be sure to chmod it to have execute permission (chmod u=+rx fetchbackups). I recommend having the operator user run this script (chown operator fetchbackups).
Alternative remote backup methods include using `scp` on an entire directory, using `rdump`, or, as clievers pointed out, using `rsnapshot` or `backuppc`. If you don't have root access to your remote server, some of these methods might not be possible, and `scp` often requires a lot of time and bandwidth.
Attachments
Step 5: Automation
So you've figured out to make a snapshot of your system, and now you want to do it every night. Cron works great for this, and it's pretty simple to setup.
Cron jobs are simply commands that are run on a regular basis. They can run monthly, nightly, or even at half-past 7am on monday mornings.
Cron jobs are specified in the crontab file. In FreeBSD this file is located at /etc/crontab
View the man pages for /etc/crontab to figure out how it works
I've attached my crontab to this page (remove the .txt extension).
All you do is add your cron jobs and save the file. The file is re-evaluated every minute, so you're done.
Cron jobs are simply commands that are run on a regular basis. They can run monthly, nightly, or even at half-past 7am on monday mornings.
Cron jobs are specified in the crontab file. In FreeBSD this file is located at /etc/crontab
View the man pages for /etc/crontab to figure out how it works
I've attached my crontab to this page (remove the .txt extension).
All you do is add your cron jobs and save the file. The file is re-evaluated every minute, so you're done.
Attachments
Step 6: Conclusion and Final Thoughts
Hopefully you've managed to backup your machine(s). What follows are just a few notes on the topic.
If you're serious about backups, then TEST TEST TEST. Make sure that your backup and restore procedures are flawless. Backups that you can't restore are worthless.
One problem I ran into was setting my cache size too big. This can (basically) DoS your system and cause it to freeze. You're cache should always be a fraction of your RAM to be effective (mine's one-fifth), and should NEVER exceed your swap space size. 32MB is what dump's man page recommends. While this information may be outdated, having a large cachesize won't make much of a difference if you have all night to backup your system.
If you've automated your backups, make sure they're working. It would be a real nightmare for your system to crash and THEN realize that your backup crons quit working 6 months ago due to insufficient disk space. Cron jobs just automate the "complacency" process. If you do backups manually, don't become complacent and forget. Make it a routine. Don't rely on cron jobs either, because they can fail.
Backups are just copies of your files. This means that backups should be secured just as well, if not better than, your live systems. Keep your external harddrive in a secure location (like away from both water AND burglars). Run backup cronjobs as the 'operator' user. This is a limited account that exists for things such as this. Also make sure that normal users cannot run backups. If you feel you could potentially be the target of a sophisticated attack (or even if you don't), always encrypt data transfered during remote backups. Due to the amount of information, as well as regularity of backups (if you're using cronjobs), hackers can take their time in stealing your information. Encryption's easy, so use it. Make sure that normal users can't run backups to their own devices. Also, `scp` requires authentication. I HIGHLY recommend preshared public/private keys. You don't want your password to be transmitted everytime a backup is run.
If you're serious about backups, then TEST TEST TEST. Make sure that your backup and restore procedures are flawless. Backups that you can't restore are worthless.
One problem I ran into was setting my cache size too big. This can (basically) DoS your system and cause it to freeze. You're cache should always be a fraction of your RAM to be effective (mine's one-fifth), and should NEVER exceed your swap space size. 32MB is what dump's man page recommends. While this information may be outdated, having a large cachesize won't make much of a difference if you have all night to backup your system.
If you've automated your backups, make sure they're working. It would be a real nightmare for your system to crash and THEN realize that your backup crons quit working 6 months ago due to insufficient disk space. Cron jobs just automate the "complacency" process. If you do backups manually, don't become complacent and forget. Make it a routine. Don't rely on cron jobs either, because they can fail.
Backups are just copies of your files. This means that backups should be secured just as well, if not better than, your live systems. Keep your external harddrive in a secure location (like away from both water AND burglars). Run backup cronjobs as the 'operator' user. This is a limited account that exists for things such as this. Also make sure that normal users cannot run backups. If you feel you could potentially be the target of a sophisticated attack (or even if you don't), always encrypt data transfered during remote backups. Due to the amount of information, as well as regularity of backups (if you're using cronjobs), hackers can take their time in stealing your information. Encryption's easy, so use it. Make sure that normal users can't run backups to their own devices. Also, `scp` requires authentication. I HIGHLY recommend preshared public/private keys. You don't want your password to be transmitted everytime a backup is run.