Best backup option

[ancientnoise]

Hi

Previously I had squeezed all of the important files from my server into a tarball and scp'd it to a separate host. It's a) a big tarball that takes ages to create and b) a big tarball that takes ages to transfer.

Not the best backup solution is it? It works, but I have to let it run overnight (sometimes longer) for it to be effective.

Caveats:

The backup host is a VM, that may or may not be powered up, so serverA and serverB should do this when they are both online;
I'd like to copy only those dirs/files that are different/updated from the existing backup;
I'd prefer it to be faster that scp!

What do you use? I'm reading about rsync, but thought I'd ask what others do before I set anything up. Any other options? NFS?

Thanks in advance, any assistance much appreciated.

[tunk]

I'm using rsync to make a 1:1-copy and it works fine.
The directories are NFS exported (read-only) to another server which runs a rsync script every night.

[ancientnoise]

I used ssh-keygen, ssh-copy-id <target_server> and added <target_server> to /etc/hosts before running.

Attempted the rsync initially as a dry-run:

rsync --dry-run -avzhe ssh /dir-to-be-copied/* root@<target_server>:/backupdir/

Worked fine. Then tried it as the real deal:

rsync -avzhe ssh /dir-to-be-copied/* root@<target_server>:/backupdir/

Also worked fine.

Then used crontab -e to set up the following, for a nightly transfer:

1 2 * * * rsync -auvzhe ssh /dir-to-be-copied/* root@<target_server>:/backupdir/

This seems to work fine too. I updated one txt file on the dir-to-be-copied and was glad to see that with the "u" option in the command only that file was updated on the <target_server>.

So happy with this for the time being. The first copy took ages, but that won't be the case now, unless of course I add a ton of data to be copied.

But:

a) want to avoid using root wherever possible;
b) tunk - do you mean your rsync is doing a local copy on the other server from the NFS share?
c) Still considering other methods if anyone has any ideas.

[tunk]

b) yes
If you want an 1:1-copy then you may want to add "--delete" to the rsync command.
Edit: If you use "--delete", you should be absolutely sure that both source and destination are online. If the source is e.g. unmounted then everything on your backup will be deleted.

[mkrsn]

Give dirvish a try. Its a kind of a wrapper around rsync. Its checking if a file already exist in a old backup and hardlinks it. Otherwise the new file is saved to the backup. So every day is a differential backup filled with hardlinks from old backups to be a full Backup.

I use it for all my servers - its imho a very nice backup-tool.

[MartinR]

Worrying about how you do the backup is only part of the planning. Backups are for disaster recovery, so let's start with a couple of disasters: (1) a thief breaks in overnight and steals all the servers and disks he can lay his hands on, or (2) there is a fire/earthquake/plane crash and the whole building (including contents) are destroyed. You find new accommodation and buy new servers, now how do you proceed?

Clearly you need to ensure your backups are fire/thief/bomb proof. Two ways: offsite secures or a serious professional firesafe. Is your bandwidth high enough for a network backup? If you can't backup over a network then you need to look at some form of removable media: USB disk or tape? Have you checked that the backups are usable without the nice shiny backup program that's on the destroyed servers? If the data is going off site does it need to be encrypted, and if so, where is the key kept (hint: not in the destroyed computer room I hope)?

You also need to consider time scales. If it's pictures of Auntie Maud at last year's Christmas party then you can probably do without them for days if not weeks. If it's the payroll that needs to go out tomorrow without which you'll have a strike on your hands, then your boss will have some input - maybe you'll see a big finger: "you're fired".

FWIW, rsync and its friends are great if you have a network connection. Recovery is at network speeds and should give you an immediate copy of the disks' state at the time of the last backup. If you can't use a network, then have a look at Amanda (it's in the base repository). At home I use Amanda to backup my machines to one of three USB disks, the other 2 of which are kept at work, and are periodically rotated. You can use rsync in a similar manner, when you rotate the disks there will be a bigger catchup phase but otherwise it will look after you. You might have problems with multiple filesystems and multiple machines though.

[schadfield]

My server backups are a combination of mysql-zrm for the database + duplicity for file backups. The data is stored on an on-site disk and copied to an off-site server using rsync. All scripted and and driven by cron jobs. It has proved its worth by saving my bacon a few times already.

[bertalanimre]

I'm still using the tar.gz, then scp method and I'll tell you why.

It is secure (cause it goes over SSH) and it is less exausting for the server to do this. I'm compressing about 300.000 files each day + making 1-2GB of MySQL dumps as well. Then, when they are created, I simply copy them to my local backup server. I've tried rsync and some other methods as well, but they are always had the following issues. It was slower to copy the files one-by-one (no matter what application I've used, it is faster to download one huge file than thousands of small files) and therefore it used up more resources on the server, thus the websites working on the server were slower.

This is why I still recommend sticking to your guns and keep using this method of saving stuff. There is an example script I'm using.

Code: Select all

#!/bin/bash

source /backup/scripts/pass

ssh [USER]@[MYSERVER] "mkdir /home/[USER]/sqldump; cd /home/[USER]/sqldump/; echo \"/////////////////////////\" >> /home/[USER]/backupfile; echo \"$(date +%Y-%m-%d)\" >> /home/[USER]/backupfile; echo \"Start time: $(date +\"%T\")\" >> /home/[USER]/backupfile; mysqldump -u root -p$[MYSERVER1] --all-databases > /home/[USER]/sqldump/sqldump-[MYSERVER]-$(date +%Y-%m-%d_%H:%M).sql; tar -pzcvf /home/[USER]/serverdump/[MYSERVER]-$(date +%Y-%m-%d_%H:%M).tar.gz /var/www/[WEBSITE1] /var/www/[WEBSITE2] /var/www/[WEBSITE3] --newer-mtime '7 days ago'"

scp -r [USER]@[MYSERVER]:/home/[USER]/sqldump /backup/backup/websites/[MYSERVER]
scp -r [USER]@[MYSERVER]:/home/[USER]/serverdump /backup/backup/websites/[MYSERVER]

ssh [USER]@[MYSERVER] 'echo "Stop time: $(date +"%T")" >> /home/[USER]/backupfile; rm -r /home/[USER]/sqldump/*; rm -rf /home/[USER]/serverdump/*'
exit

in my /backup/scripts/pass file, I have: (without the [ ]-s )

Code: Select all

[MYSERVER1]="MYPASSWD"
[MYSERVER2]="MYPASSWD"
[MYSERVER3]="MYPASSWD"
...

This is one of my scripts, obviously with the sensitive information [TAG]-ed out. Keep in mind please, I'm using a seperate file to store the passwords in plain text format, but it is at a seperate space, unaccessable by any attacker + if they somehow capture the network packaged and recreate the script, they only can see that I'm using a password mentioned in a file. And that file is never sent via the network. So if your backup server is in a safe environment, there is almost no way, anyone can hack your system. (OFC use SSH Keypairs to connect and not passwords).

Just put the script file in crontab to run automatically and periodically and there you go.

[ancientnoise]

or (2) there is a fire/earthquake/plane crash and the whole building (including contents) are destroyed. You find new accommodation and buy new servers, now how do you proceed?

Or the somewhat less catasrophic, but more frequent hard drive failure! Still, I get your point.

FWIW, rsync and its friends are great if you have a network connection. Recovery is at network speeds and should give you an immediate copy of the disks' state at the time of the last backup. If you can't use a network, then have a look at Amanda (it's in the base repository). At home I use Amanda to backup my machines to one of three USB disks, the other 2 of which are kept at work, and are periodically rotated. You can use rsync in a similar manner, when you rotate the disks there will be a bigger catchup phase but otherwise it will look after you. You might have problems with multiple filesystems and multiple machines though.

Yes, I have considered an offline backup. Not sure about USB sticks though, we're talking about 1.4TBs of data here. I used to have one of those old IcyBox media streaming devices, with a 1TB drive in it, which I starting using as a ext-HDD when technology moved on making it otherwise fairly redundant. It's USB 1.1 though, and the backup I originally put on there took about 4 days! I also try to avoid using USB drives (HDD or stick) as I've often found them to be a bit flaky. If I had a £ for every time someone's asked me if I can get their data back from a busted ext-HDD, I'd be able to splash out on a enterprise-class SAN (OK, maybe exagerrating a wee bit there )

[ancientnoise]

Give dirvish a try. Its a kind of a wrapper around rsync. Its checking if a file already exist in a old backup and hardlinks it. Otherwise the new file is saved to the backup. So every day is a differential backup filled with hardlinks from old backups to be a full Backup.

I use it for all my servers - its imho a very nice backup-tool.

Will check this out, thanks.

http://www.dirvish.org/