Xen Performance Guide

Disk utilisation is the biggest bottleneck in virtualisation.

The best way to install the guest OS (DomU) is also not an obvious one. Most people have a RAID subsystem as storage for all the DomU virtual disks - but having this misaligned or not optimised can lead to significant performance degradation. This mostly comes about when having an LVM scheme where you use an LV to store the DomUs filesystem - but you use the ISO installer - so you end up offsetting everything by the partition table + boot data.

This means in most cases, when you write a block to the DomU's filesystem, it translates to two block writes in the physical storage array - you're essentially writing twice as much data as you need.

So, my way to solve this was to use the LV directly - with no partition table. We also start the installation via the Dom0 shell - that way the tools have full visibility as to the layout of the underlying block storage. This guide focuses on EL6 - specifically Scientific Linux 6. You can probably substitute easily enough for CentOS or RHEL.

1) Create a new LV to start with:

lvcreate -L 50G -n template.vm vg_domains

2) Create a filesystem on this LV:

mkfs.ext4 -j /dev/vg_domains/template.vm

3) Mount the newly created filesystem to /mnt/template:

mkdir -p /mnt/template
mount /dev/vg_domains/template.vm /mnt/template

4) Now we want to create the bare directory template to allow installation.

mkdir -p /mnt/template/tmp /mnt/template/var/lib/rpm /mnt/template/var/log
mkdir -p /mnt/template/dev /mnt/template/proc /mnt/template/sys

5) Create the new RPM database:

rpm --root=/mnt/template --initdb

6) Install the release file for the distro. In my case, I use the following:

wget http://ftp.scientificlinux.org/linux/scientific/6x/x86_64/os/Packages/sl-release-6.4-1.x86_64.rpm
rpm -ivh --nodeps --root /mnt/template sl-release-6.4-1.x86_64.rpm

7) Install the base set of packages:

yum --installroot=/mnt/template -y groupinstall base

8) Now we can mount a few things and then chroot into the new filesystem to do what we need.

mount -o bind /dev /mnt/template/dev
mount -o bind /proc /mnt/template/proc
chroot /mnt/template

9) Install a kernel (this isn't included normally in the base group)

yum -y install kernel

10) Create a file: /boot/grub/grub.conf then populate it with the correct kernel versions (look in /boot and substitute where needed):

boot=/dev/sda
default=0
timeout=1
splashimage=(hd0)/grub/splash.xpm.gz
hiddenmenu

title Scientific Linux (2.6.32-358.23.2.el6.x86_64)
    root (hd0)
    kernel /boot/vmlinuz-2.6.32-358.23.2.el6.x86_64 ro root=/dev/xvda rd_NO_LUKS rd_NO_DM LANG=en_AU.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us crashkernel=auto console=hvc0
    initrd /boot/initramfs-2.6.32-358.23.2.el6.x86_64.img

11) The last file we need to create is /etc/fstab. It will be something like this:

/dev/xvda               /                       ext4    defaults        1 1
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0

12) Our last real step to prepare the template now is to set a root password:

passwd

13) To configure the DomU now, we’ll need to unmount everything:

exit
umount /mnt/template/dev
umount /mnt/template/proc
umount /mnt/template

14) Create the Xen DomU config file as /etc/xen/template.vm:

memory = 512
name = "template.vm"
vcpus = 1
disk = [ 'phy:/dev/vg_domains/template.vm,xvda,w' ]
vif = [ "bridge=br0,script=vif-bridge" ]
kernel="/usr/lib/xen/boot/pv-grub-x86_64.gz"
extra="(hd0)/boot/grub/grub.conf"

15) Start the DomU and cross your fingers!

xl create /etc/xen/template.vm -c