My Projects: Virtualization - part 1: XEN to ESX

Here is the next post in "My Projects" series, and this time it is about the Virtualization.

Well, if think about that, I used virtualization for a very long time. First time I used jailed environments in FreeBSD about 10 years ago. After that it was VMware workstation server used mainly for home or some tests. But my first serious Project connected with virtualization took place just three years ago.

What we had:

When I moved to a main R&D center in Poland they had two or three XEN hosts for virtualization, and two new ESX servers with Standard license connected to a vCenter Server. Appreciating all benefits of using ESX it was decided to move all old services from the XEN-based virtualization to the ESX-based. In order to complete this task two more Standard licenses was bought and I was assigned to proceed with migration of 10-20 (don't really remember) of VMs.

The funny thing is that VMware Converter 4.0 has been released just in a few weeks after my project has been finished, but I don't feel sorry about that, because I've got the invaluable experience. Moreover, some of the migrated VMs couldn't be migrated with Converter because they were very old, like Fedora Core 1 for example.

Process:

Maybe, the way I did this migration looks too tricky, but it worked and I've migrated few tens of VMs from XEN. Most of guests in XEN had a single disk with /boot and / (root) primary partitions without LVM. Generally, in case with CentOS 5.x the migration looks like that:
Important note: This is just an example without the full description of commands used. In your case procedure can be completely different and single mistake can cause a real harm to your system!

• Create a VM on ESX
• Boot from newly created VM with CentOS CD/ISO in rescue mode with network enabled
• Create all necessary partitions and mount them (e.g. /mnt/newsys)
• On source VM allow ssh for root, or configure rsyncd
• Copy all data from source (e.g. 192.168.1.10) to destination by executing rsync on a new VM:
  # rsync -ah --progress --delete --exclude="dev/" --exclude="proc/" --exclude="sys/" --exclude="tmp/" //192.168.1.10:/ /mnt/newsys/ 
• Mount /dev /sys and /proc to /mnt/newsys
• # chroot /mnt/newsys su -
• Now it's time to change some values in fstab (to use /dev/sda instead of /dev/xvda) and grub.conf
• After this is changed you must (in most cases) rebuild your initrd (mkinitrd tool) and reinstall grub (grub-install)
• Temporary change the IP and boot the system on new VM
• Stop all services on source guest and use rsync to copy the changes
• Turn off the old VM and change back the IP on the new VM.

In most cases it worked, but there was number of cases when it won't work by different reasons (e.g. Fedora Core 1). So, I also had another way to make the same thing. It was more complicated, however it always works:

• Create a VM on ESX with Virtual Disk of the same size or a bit more like it was on XEN
• Boot from CentOS CD in rescue mode with network enabled
• Start netcat to listen for a data an pipe it directly to the disk:
  # nc -l 2121 | dd bs=1M of=/dev/sda
• On XEN host stop the source VM (wasn't necessary in all cases, but is more safe)
• Send the contents of virtual disk to the remote VM booted in rescue mode like mentioned above:
  # dd if=/path/to/disk.img bs=1M|nc 123.123.123.123 2121
• After all data is copied you can open /dev/sda with fdisk and extend second partition (by removing and adding it again with a bigger size)
  # fdisk /dev/sda
  : d (delete partition)
  : 2 (choose 2nd partition)
  : n (new partition)
  p (let it be primary partition)
  : w (write changes and exit)
• After second partition raised you must use file-system tool to extend it (like resie2fs)
  # resize2fs /dev/sda2
• Next you can mount this partition somewhere (don't forget to mount /boot after /), mount system FSes like /dev /sys and /proc, and chroot there.
• Now it's time to change values in fstab (to use /dev/sda instead of /dev/xvda) and grub.conf
• After this is changed you must (in most cases) rebuild your initrd (mkinitrd tool) and reinstall grub (grub-install)
• Temporary change the IP and boot with the new system
• Stop all services on source guest and use rsync to copy the changes
• Turn off the old VM and change back the IP on the new VM.

Of course, now with VMware Converter this procedure is not more needed for a range of Linux servers, but I had pretty interesting experience.

Result:

When project was finished all XEN hosts were reinstalled with ESX Server and configured in a cluster with HA enabled.
In progress of migration the switching between old and new VM for most important services was about 1-2 minutes (sync differences and restart the networking services).
Some services was successfully migrated from old Linux versions to newest CentOS.
Was created a detailed procedure of migration, that allowed engineers in remote offices to complete migration as well.