You are probably well aware of the need for offsite backups; as a technology professional this is one of the first arrangements I look into for any permanent storage of business information. When I started two years ago for an internal "startup" for a large company, one of the first things we did was set up an SVN repository and then work out an arrangement with an offsite data storage provider. However, the cobbler's children have no shoes: I've never set up a proper backup scheme for my own data at home, and it's about time to take care of business.
Fortunately, now that we've moved to using Macs at home and with the advent of cheap UN*X hosting providers, it's about time I stopped putting this off. The scheme here is pretty simple: get a hosted Linux server from someone like 1and1.com, dreamhost.com, or rackspace.com where the storage is backed up and they take care of security updates for the OS. Then set up a pretty simple combination of the UN*X utilities rsync, ssh, cron, and bash scripts to get secure nightly backups going. Just to make it more fun, I'm going to challenge myself to have this all working in under an hour! I'll keep notes as I'm doing it as to how long it takes, not counting the writeup before or afterward.
I decided to register a domain name with a hosting provider, since it was included. My basic requirements were:
- SSH access
- rsync installed
- enough storage for my data
Be sure to acquire the following information from your hosting provider:
- username/password with SSH access (preferably root, if you want to use the server for other purposes, but this is not necessary)
- IP address
- SSH host key of the server
I ended up registering a new domain with at dreamhost.com at $9.95/month. As it happened, DreamHost was running a promotion with unlimited disk space and bandwidth for the lifetime of my account. Score! I did have to email tech support to get the ssh host key. If you find yourself in a similar position, you can ask for the output of:
Ok, great! Now you have a destination for your offsite storage. Next step is to make sure we can securely log in over the network (we'll use ssh for this). On the Mac you want to back up, open up a Terminal window, and ssh into your server using your username and the server's hostname, as in the following. N.B. Do not finish connecting if the ssh server host key you got from your hosting provider does not match the key you see when you try this!
Ok, so far so good. Now we need to make sure we can do it without needing a password; this is where user ssh keys come into play. First, let's create an ssh key to use for backups. We'll want to do this as the root user on our Mac, so that when we run the backup script out of cron, we won't run into permissions problems. You can use the "sudo" command to become root on your Mac:
Now we need to create an SSH public/private key pair; this is a similar concept to PGP email encryption/signing; you can read a really interesting description of the chronology behind public key cryptography in the book Crypto by Steven Levy. We'll keep the private key locally on our Mac, and take a copy of the public key and copy it securely up to our backup server; then ssh will use the private key when we connect, allowing the backup server to verify using the public key that we are who we say we are, without having to send a password. Nice.
Specifically, we will want to do the following (still as root):
Use an empty passphrase (i.e. just hit return when prompted for the passphrase), as this will allow the ssh program to load the key without interaction from you. Also note, however, that anyone who gets root access to your Mac will be able to ssh into your backup server at will. Given that our backup server contains a copy of what this would-be hacker would be able to see on the actual Mac anyway, I don't really see this being a big risk....
Now, we need to copy the public key over to the backup server:
You'll have to verify the server SSH key one more time, because now you are connecting from root rather than from your normal user account. Now we'll tell the backup host to accept a login from this key pair:
Now, we should be able to log in without a password from our Mac:
Sweet. Now we create a directory where our mirrored filesystems will live:
The utility we'll use to do the mirroring is the rsync utility, which can be invoked to run securely over ssh. This actually makes a nice backup utility for regular use, as the rsync protocol is actually pretty smart about being able to find just the small subsets of data that changed since the last sync; after the first big sync, for most personal file use, there won't be much work to do every night.
For now, let's set up a test directory on our local Mac.
Now, to make the magic happen, we do this:
Now we can keep a window open on our backups host, and we should see everything show up there:
Just for fun, run the same rsync command above and see that nothing happens if there have been no changes (or rather, just that a very small amount of data gets exchanged to verify no changes).
Let's just make sure changes show up:
Ok, looking good. Next step is to identify all the directories you want to back up; let's keep a list of them in a config file on our mac:
Note that it is important *not* to have trailing slashes on these directory names, as this changes rsync's behavior slightly in a way that you will probably find annoying (it won't copy the directory name over, just the contents).
Ok, now the next step is to set up a script that can sync each of the directories:
Now we run it once by hand to make sure it works:
Finally, we install this in root's crontab as follows:
Nice and simple. Now the backups are off and running every night without your intervention.
If you ever need to restore from the backup, you can always reverse the rsync process like this:
for each of the directories you have backed up over there.
Enjoy, and sleep well tonight....P.S. Total elapsed time for the exercise was 2 hours from the time I placed the hosting order to the time the crontab was installed, but I took a one hour break in the middle for dessert and bedtime with the kids. So I'll claim this really did only take one hour of "CPU time" for me.