XEN Live Migration with iSCSI & OCFS2

Its been slow in coming but I've previously promised to blog about setting up iSCSI / OCFS2 as a means to share XEN domU images so as to facilitate Live Migration. Here we go...

Background:
Live migration in XEN is the moving of a running virtual machine (domU) from one physical machine to another. XEN will replicate the running domU's RAM across the wire between the source and target physical machine (both running XEN and having the right configurations, of course). It is up to us to ensure that the target machine has read/write access to the domU's virtual disks.

Hence, the most common way to do this is via NFS. For most, especially when setting up a demo system or in a class setting, this is sufficient. For larger scale deployments, customers usually have a SAN environment and that's that.

What about folks who are between the demo/class and larger-scale deployment scenarios? This is where this iSCSI / OCFS2 solution fits nicely. This is also commonly known as the poor man's SAN. There are quite a number of articles online that discuss this in detail (like pointing out the various SPOFs) so I'll not bother to blog about it here. Here's a good link for further reading though -> Build your own iSCSI SAN appliance, save money.

Solution:

SUSE Linux Enterprise Server 10 has both iSCSI and OCFS2 support. The following is based on SLES 10 SP2. iSCSI is a means to present a remote disk via TCP/IP to a local server just like its a local SCSI device. Hence, the little i stands for internet. OCFS2 is a proven cluster-aware filesystem and its the recommended filesystem when deploying Oracle RAC on Linux.

Here are the steps in general, you can find the details via online SLES documentation (or a one time 9Mb PDF download).

Link to documentation -> http://www.novell.com/documentation/sles10/index.html
Link to the specific section (12 & 14) -> http://www.novell.com/documentation/sles10/sles_admin/data/part_administration.html

Step 1: Setting aside a disk (or partition) to be shared via iSCSI

With reference to section 12.1, you will need to identity a disk or a disk partition to be used. Next, YaST -> Misc -> iSCSI Target to configure (if iSCSI is not installed, it will do so at this time).

Step 2: Configuring XEN servers as iSCSI clients to the shared disk

With reference to section 12.2, we configure the XEN servers to connect using YaST -> Network Services -> iSCSI Initiator so that the shared disk (or partition) will appear as local.

Step 3: Installing and Configuring OCFS2 on all servers

If you are not familiar with OCFS2, its recommended that you read section 14.0 through to 14.4.

From section 14.5, we install OCFS2 packages on all servers (storage and XEN servers). In section 14.6, we configure OCFS2 and format the disk (or partition) on the storage server.

In section 14.7, we configure OCFS2 to mount the shared storage on a common directory on the XEN servers (ie /mnt/xen/).

Step 5: Edit the domU configuration file

Copy the XEN domU configuration and virtual disk (eg disk0) into the mounted /mnt/xen/vm and /mnt/xen/images subdirectories respectively.

You will need to edit the domU VM configuration file to ensure that the disk parameter is not pointing to the new virtual disk (eg disk0) location in /mnt/xen/images/.

Don't forget to propagate these changes via xm delete and xm new commands.

Step 6: Live Migrate!

Live migrate the running domU from one physical XEN server to another via:

xm migrate [domU ID/Name] [target server hostname/IP] --live


Good luck.