I'm very new to centos and pretty new to linux.I run the network for a small business and I'm looking to add some much needed redundancy and there is just about no budget. But we do have several moderate to high end server for the task.

I've been playing with the idea of clustering a xen server and was looking for some options. I've seen where people have effectivly used opensuse to cluster virtual servers but I've also been told that opensuse is about the most foreign flavor of linux so I'm looking to centos as my option as it's also the best idea. I've built out a couple of test centos boxes and I've got clustering and xen virtualization installed. I'm now looking to cluster them for the xen services and virtual machines. Is there any how to or suggestion on how to accomplish this or have i headed in the wrong direction.

I'm looking to do this as an all encompassing solution because I have 3 servers I can make this happen with and then I'm looking to host the servers that were on these boxes on these three boxes so that if one fails we don't lose a critical servers.

Thanks in advance from an mcse slowly turning to the penguin.

Welcome to the CentOS fora. Glad to see you turning away from the dark side.

I don't have personal experience with Xen clusters, but what you propose should be eminently doable. The basic docs are at http://www.centos.org/docs/5/

A google on turns up some 970 hits. A more general query on nets some 23,600 references. Some of the early ones look quite promising. Please post specific questions as you encounter them, and good luck with your CentOS endeavors.

What I've found was more ubuntu related or was how to use drbd and heartbeat. I was more looking for how to add xen as a service under resources. The docs have a section on it, but it is blank??? When you check the service under resources there is option for Xen or virtualization. So should I hunt the drbd method and not use the built in Xen options? Sorry, maybe I should have been more specific.

There's is tons on the drbd and heartbeat method, I was hoping for something more along the lines of the built in centos clustering methodology. But I'll dig some more and come back in tears begging for help later.

Also on a more general question; does using clustering in the method I mention sound reasonable. By that I mean is hosting both the file server resource and the xen instances on the same boxes a good idea? I realize that is a broad question and open to debate and very dependent on the hardware but I'm just looking for a general sense as to if I am heading in the right direction or need to re-evaluate the idea.

If you find any good solution to this I'd be more than happy to know! :)

We are using xen+drbd+heartbeat to provide this. Our configuration does not do a live migration because I was unable to get GFS working correctly over drbd. The servers we run as vms are not so critical that 10 minutes service interruption while migrating is a problem.

BTW- I am interested if anyone has good step by step documentation for using GFS and drbd. The documentation I found was not working for me.

So, the process we used:

Follow drbd directions at http://wiki.centos.org/HowTos/Ha-Drbd

Xen Cluster

Uses drbd, heartbeat, ext3 partitions

Disk Layout

/dev/VGdrbd/LVxenimages /dev/drbd1 Primary storage location for vm image files mounted at /xenimages
/dev/VGdrbd/LVxenmigration /dev/drbd2 Storage location of save files for migration mounted at /xenmigration
/dev/VGdrbd non-drbd Empty space for snapshot backups of LVxenimages

Linux-HA resources

group_xencluster group contains all services which grouped for colocation/order
xenimages_device drbddisk Manages primary/secondary role of vm image files resources
xenimages_mount Filesystem Mounts/unmounts volume on the primary server
xenmigration_device drbddisk Manages primary/secondary role of vm migration files resource
xenmigration_mount Filesystem Mounts/unmounts volume on the primary server
xen_migration xendom-migrate Controls movement of vms. Uses xm save/restore. Must increase the timeout of this resource in heartbeat to give time for the move

File system modifications made on BOTH nodes. make the node that you are working with the primary node and mount it, then

create symbolic links from the default location of the xen files to the drbd volume
copy /etc/xen to /xenimages/etc/xen
create a symbolic link from /etc/xen -> /xenimages/etc/xen
You can do the same for the default location of the images as well. We simply change the location when creating a vm

xen_migration is a custom script that is located on /xenimages/etc/ha.d/resource.d that is symbolically linked at /etc/ha.d/resource.d/xendom-migrate.

Xendom-migrate script

#!/bin/bash
#
# This script is inteded to be used as resource script by heartbeat
#
# Script will list running VMs and save them to be restored on the other
# cluster node
#
###

XM="/usr/sbin/xm"
PATH_TO_SAVE_FILES="/xenmigration"

if [ "$#" -eq 1 ]; then
CMD="$1"
fi

## EXIT CODES
# since this is a "legacy heartbeat R1 resource agent" script,
# exit codes actually do not matter that much as long as we conform to
# http://wiki.linux-ha.org/HeartbeatResourceAgent
# but it does not hurt to conform to lsb init-script exit codes,
# where we can.
# http://refspecs.linux-foundation.org/LSB_3.1.0/
# LSB-Core-generic/LSB-Core-generic/iniscrptact.html
####

case "$CMD" in
start)
# Ensure that xend is running on the host
STATUS=$(/sbin/service xend status | grep "xend is" | /bin/gawk '{print $3}')
if [ $STATUS != "running" ]; then
/sbin/service xend start
fi
# List files in xenmigration. A file represents that a VM was running on the
# other node and should therefore be restored to a running state
for FILE in $(find $PATH_TO_SAVE_FILES -type f); do
$XM restore $FILE
done
;;
stop)
# Remove old migration files
for FILE in $(find $PATH_TO_SAVE_FILES -type f); do
/bin/rm -f $FILE
done
# Save each running VM
while read line # read each line of output from the command redirected in at the done statement
do
echo $line
VHOST=$(echo $line | /bin/gawk '{print $1}')
STATE=$(echo $line | /bin/gawk '{print substr($5,0,1)}')
if [ $VHOST != "Name" ] && [ $VHOST != "Domain-0" ]; then # do not attempt anything with column header or Domain-0
$XM save $VHOST $PATH_TO_SAVE_FILES/$VHOST
fi
done < <($XM list)
;;
status)
$XM list
exit 0
;;
*)
echo "Usage: xendomain-migrate {start|stop|status}"
exit 1
;;
esac

exit 0

Virtual Hosts

The virtual hosts running are limited to a 100Mbps network connection unless paravirtualization drivers are installed on the virtual host. This is similar to installing vmware-tools. gplpv drivers may be downloaded from http://www.meadowcourt.org/downloads/

Determine the correct pv drivers

The new drivers look something like this: gplpv_fre_whl_AMD64_0.10.0.69.msi
So you start with "gplpv_".
You then need to select whether you want the ones with the bug check code in them or not (unless you're debugging problems, you don't need the bug check code). The bug check has "chk" in it, the ones without have "fre"
"gplpv_fre"
After that you decide if you're using XP or not: "wxp" for XP, "wnet" for 2003, and "whl" for 2008, Vista, etc.
"gplpv_fre_whl"
And then you need to know whether your 32 bit "x86" or 64 bit "AMD64".
"gplpv_fre_whl_AMD64"
And then the version "0.10.0.69".
If you're somewhat normal you've ended up with something like this: gplpv_fre_whl_AMD64_0.10.0.69.msi (64 bit 2008 without bug check code)

The snapshots portion of the VG is for backup of the vm image files. We take a lvm snapshot, mount the snapshot, tar the snapshot to another server. This allows us to restore any vm to a specific point in time should a problem occur. WARNING: disk I/O is severely affected by the presence of a lvm snapshot. If you perform the same task, you should delete the snapshot as soon as you have finished creating your backup.