Is LVM a complete waste of time?

If it doesn't fit in another category, ask it here.
fthomas
Posts: 47
Joined: 2006/10/19 17:51:00

Is LVM a complete waste of time?

Post by fthomas » 2007/01/16 18:53:25

I've been using LVM with servers for a little while and everything I read mentions that I can do a better and safer backup using conventional ext3 partitions and just remount them as read only. I do realize that you can perform a live dump using lvm's, but one thing no one ever mentions is that you have to leave a bit of space on the lvm partition free to perform it operation. So I have 3 servers out there that I'm going to have to shedule a downtime to transfer the info to another disk and increase the size of the lvm..

So, to not spit against the wind, what is your opinions?

Frank.

pjwelsh
Posts: 2589
Joined: 2007/01/07 02:18:02
Location: Central IL USA

Re: Is LVM a complete waste of time?

Post by pjwelsh » 2007/01/16 19:05:38

LVM is a abstraction layer for hardware. ext3 is a filesytem partion. LVM offers MANY benifits *including* non-destructive expansion (PLEASE backup JUST in case) over standard disk partioning schemes.
Utilities of interest include:
lvresize
resize2fs or e2fsadm

Please carefully go through:
http://www.faqs.org/docs/Linux-HOWTO/LVM-HOWTO.html
find the section:
Extending a logical volume

google "lvm resize howto" for other info just in case.

fthomas
Posts: 47
Joined: 2006/10/19 17:51:00

Re: Is LVM a complete waste of time?

Post by fthomas » 2007/01/16 20:48:33

Agreed, but I've read mixed reports regarding backing up lvm's (especially with running db's on them) and recovery. Any hints, procedures, gottas would be a blessing.

thanks

Frank.

pjwelsh
Posts: 2589
Joined: 2007/01/07 02:18:02
Location: Central IL USA

Re: Is LVM a complete waste of time?

Post by pjwelsh » 2007/01/16 21:13:41

Now the question gets more fun... I don't know the DB you are running, I would STRONGLY advise using the "normal" tools to backup for the DB. LVM offers snapshot capabilities, but at what state is the database in (eg flushed writes of OS or DB, data page creation etc). You may end up with a corrupted "backup" (is that realy a backup after all).

Just think of what you are going to do with the snapshot. Will it realy be needed for offsite DR? How big extra space for the snapshot? Of course if you have a 3TB database you are likely in a different class setup (EMC + array snapshot + DR clones etc).

The normal tools also offer the nice historical/transportable options. Nothing better than database/table dbexport archive to get the developer "lost" data from 3 weeks ago. You will still need to find a way to rebuild your data from the snapshot if you need it.

If you give some info on the class hardware, DB, DB size etc. I could offer some suggestions (taken with a grain of salt, of course).

For something like a small medium sized Mysql (that uses OS cache only... basically), I would trush a "sync ; rsync -aP /some/mysqldir bakcupbox:/backup/" (ssh key assumed in place) more than a snapshot.

Naturally, TEST/TRY/BREAK on you own. Just because *I* think something will work doesn't mean it makes sense for *you*.

User avatar
WhatsHisName
Posts: 1542
Joined: 2005/12/19 20:21:43
Location: /earth/usa/nj

Re: Is LVM a complete waste of time?

Post by WhatsHisName » 2007/01/16 21:24:08

fthomas: Like pjwelsh pointed out, LVM2 logical volume snapshots make doing backups simple and fairly painless, in CentOS4 at least.

Don’t know which discussions you are referring to, but getting stable snapshots in FC4&5 was almost impossible. At least it’s greatly improved in FC6 and hopefully in RHEL5, too.

For my own systems, I would not even entertain the idea of going back to simple partitions and not using LVM.

For those committed to fs imaging as a backup method (like I once was), learning the ins-and-outs of [img]http://rsync.samba.org/[/img][url=http://rsync.samba.org/]rsync[/url] will help you acclimatize to LVM. Although not directly related to LVM, [url=http://www.rsnapshot.org/]rsnapshot[/url] is an interesting use of rsync to do backups.

pjwelsh
Posts: 2589
Joined: 2007/01/07 02:18:02
Location: Central IL USA

Re: Is LVM a complete waste of time?

Post by pjwelsh » 2007/01/17 01:41:38

To second @WhatsHisName, I choose to use LVM (+ext3) as a normal course of action. There are far to many pluses with LVM to not have it used! I just don't like the snapshot for DB's and some other uses.

fthomas
Posts: 47
Joined: 2006/10/19 17:51:00

Re: Is LVM a complete waste of time?

Post by fthomas » 2007/01/19 22:05:13

OK, I'm seeing the light. As you can guess gwelsh, that I am alluding to creating a perfect backup protocol for my linux servers. I do make use of rsync to perform diff's in user folders offsite, but I wish to create a bull proof backup/recovery schema for my linux servers. Any more info on your insights (and others) would be greatly appreciated!

Thanks
Frank.

pjwelsh
Posts: 2589
Joined: 2007/01/07 02:18:02
Location: Central IL USA

Re: Is LVM a complete waste of time?

Post by pjwelsh » 2007/01/19 22:37:29

If you like the command line and want a "bull proof" backup, I suggest rdiff-backup (http://www.nongnu.org/rdiff-backup/) VERY VERY MUCH! It's a full backup + differential all rolled up using the rsync libs. There is even a gui for it also...

I like to stage DR differently than Backup as a general note. In other words, I don't like to mix the two (paranoid!). So, some mediocre local box with some disks for the full + daily incrementals (I keep 90 days) and offsite (DR) only the daily diffs. Local recovery is much faster. Plus you only care about the most recent for DR.

If you have SMP box. and you have BIG FILES to compress, you MUST have "pbzip2" (http://www.compression.ca/pbzip2/)! Unlike bzip2, you can use those extra procs(configurable) and factor the time of compression by the number of processors (dual core slighly less). We have 20GB files that pbzip2 compress in under 1 hour using 2 opteron 248's (single core)

fthomas
Posts: 47
Joined: 2006/10/19 17:51:00

Re: Is LVM a complete waste of time?

Post by fthomas » 2007/01/26 19:21:34

pjwelsh,

thanks a bunch for your reply! Very, very interesting. I am going to take a bigger peek into rdiff-backup. Do you use it for both your onsite and offsite backups/dr's?

Does the compression offer you faster backups, or is it faster to just backup uncompressed data?

Frank.

pjwelsh
Posts: 2589
Joined: 2007/01/07 02:18:02
Location: Central IL USA

Re: Is LVM a complete waste of time?

Post by pjwelsh » 2007/01/26 20:50:14

[quote]
fthomas wrote:
thanks a bunch for your reply! Very, very interesting. I am going to take a bigger peek into rdiff-backup. Do you use it for both your onsite and offsite backups/dr's?[/quote]

For us, the DR site is about valid and current data offsite (oh crap a tornado). Onsite is about the normal restoration issues (last hour, last week, last month etc). We use rdiff* for onsite and rsync for current to DR (plus programatic DB restore for our Informix DB's to validate archives).

[quote]Does the compression offer you faster backups, or is it faster to just backup uncompressed data?[/quote]

rdiff* handles normal user files and we keep 60 days "live" on the NAS box (plus we put to LTO2 tapes because we are paranoid). The DB current archives are pbzip'd (keeping original file) into a new name for historical purposes. We keep 60 days on the NAS box of all of the level 0 and Level 1's for the DB's (pbzip'd). We do NOT use rdiff* to handle the DB archives. We find that we are better off compressing -> rsync'ing -> uncompressing -> restoring based on the DR I'net connection than using the rsync "-z" option. Nothing like squezing a 20GB DB archive into 3ish GB's in under an hour. This issue is also about the *amount* I get to keep online. So, more of the BIG files compressed are better than fewer (days) uncompressed.

Honestly, I do end up keeping 30 days of DB archives on the DR site "just to make sure".

Post Reply