Please note: since I first posted this entry, documentation on this subject has evolved considerably. Please check out the corresponding section in the DRBD User’s Guide for more details.
Undoubtedly, many of you out there are using Xen as the virtualization platform of your choice. Many of you have asked us about doing Xen live migration for DRBD-backed domU’s. Ask no longer, we’ve heard your call.
DRBD 8.0.6, released today, contains a block-drbd device helper script for use with Xen. This greatly simplifies the task of integrating Xen with DRBD.
Xen integration with DRBD: the old way
Previously, if you wanted to use DRBD as a Xen virtual block device (VBD), you would have added entry like this to your domU config file:
disk = [ 'phy:/dev/drbd0,xvda,w',
'phy:/dev/drbd1,xvdb,w' ]
… and you would have had to make sure by yourself that those devices were in the Primary role on the node where you were starting your domain. Heartbeat could assist you with managing your DRBD, but Xen couldn’t do it on its own. It was all just a wee bit clumsy.
Xen integration with DRBD: the new way
With the new helper script, you do just this:
disk = [ 'drbd:myresource,xvda,w',
'drbd:myotherresource,xvdb,w']
Now, when you fire up your domain, the helper script will put your DRBD resources named myresource and myotherresource in the Primary role, and hand the corresponding device nodes over to the Xen daemon for use as VBDs for the newly started domain. Likewise, when you issue a domain shutdown or destroy command, Xen will restore your DRBDs to the Secondary role.
The extra plus: Xen live migration with DRBD
And, what’s even better, with this DRBD directly hooks into Xen’s live migration capabilities. Just include these entries in your DRBD resource configuration:
net {
allow-two-primaries;
after-sb-0pri discard-zero-changes;
after-sb-1pri discard-secondary;
}
Add a resource configured like that to your Xen disk configuration, and voilà, you’ll be able to do xm migrate and even xm migrate --live with that domU.
The added bonus: simplified Xen clustering
Use a Xen domain as a Heartbeat cluster service? No problem, just add it to your Heartbeat V2 CRM configuration. The Xen OCF Resource Agent is your friend. Configure DRBD for Heartbeat? Not necessary, Xen will take care of everything all by itself!
September 5, 2007 at 11:06 |
With this setup, there is one DRBD resource for each domU.
I was wondering :
– is there a limitation of the number of DRBD resources that can be handled ?
– can each resource’s meta-data be safely stored within it or is it better to have an external meta-disk partition ?
September 6, 2007 at 15:41 |
Maxim,
Re question #1: the maximum number of device nodes DRBD can handle is 255 — 32 by default, and up to 255 if you set the minor_count module parameter for the drbd.ko kernel module. I am not sure if there is a lower limit on VBDs per domU in Xen, though.
Re question #2: both internal and external meta data is fine with DRBD. If you can place the meta data on a different spindle than the rest of your data, you might gain a slight performance benefit.
December 20, 2007 at 17:01 |
Does this setup require GFS or other clustering filesystem and all the fencing stuff for live migration? or does it somehow handle it so that the source vm blocks writes/goes secondary when the destination vm becomes primary? at least allow-two-primaries is enabled so i guess gfs is needed?
January 1, 2008 at 23:33 |
“Q”,
this does not require GFS. Nor OCFS2. Indeed, the point is to avoid the need for a clustering file system.
The helper script makes your VBDs Primary on domU creation, and Secondary on domU destruction. The allow-to-primaries option is there to allow live migration, since Xen assumes to have write access to the backing both the source and the destination host at the time of migration. If you’re not using live migration, you can do fine without allow-two-primaries.
January 2, 2008 at 3:43 |
Can I use DRBD 8 on top of LVM, so that I can take advantage of LVM snapshot for backup purpose? Or is there an alternative way to accomplish this?
Thanks in advance
January 9, 2008 at 18:45 |
Marco,
yes you can use LVM snapshots with DRBD, however it’s advisable to make sure that the file system is “frozen” at the time the snapshot is created. That’s not too easily feasible when using DRBD as Xen VBDs. YMMV.
July 16, 2008 at 18:28 |
[...] DRBD 8.0.6 brings full live migration for Xen on DRBD http://fghaas.wordpress.com/2007/09/03/drbd-806-brings-full-live-migration-for-xen-on-drbd/ [...]
November 26, 2008 at 19:11 |
[...] DRBD 8.0.6 ist es möglich auf einer DRBD Ressource das Live Migration Feature von Xen zu verwenden. Dazu [...]
December 24, 2008 at 10:52 |
Hi Florian,
I was wondering if there’s this kind of facilities for KVM ?
I stumbled upon http://www.mail-archive.com/kvm@vger.kernel.org/msg02105.html that’s why I’m asking.
At least KVM live migration works on DRBD according to http://www.mail-archive.com/kvm@vger.kernel.org/msg02112.html
May 4, 2009 at 9:28 |
Some questions.
In drbd active/active mode configuration, is it possible to use drbd to grant either HA either manual load balancing and computing load sharing assigning different active VMs to different physical hosts in the cluster?
What could be a good solution for backup? It is, I was thinking to lvm over drbd snapshotting, but you stated that for this to work is needed that the overlying file system is freezed and that such condition is hard to reach. But is it a filesystem needed? It is not possible to use logical volumes only containing VM images? And if it is possible, are there yet problem in lvm snapshot backups?