My Projects: Virtualization - part 2: P2V

Here is the next post in "My Projects" series, and the second one about Virtualization.

After migration from XEN was finished, we started to migrate all existing physical machines to virtual. Generally, this was one of the most light project for the last couple of years. First of all - VMware Converter 4.0 has been released, so all CentOS-es were migrated smoothly, without an issue. Second - while migrating VMs from XEN, I've collected so much experience, that some tasks which seemed very complicated before, now was a kind of obvious for me (e.g. like playing with LVM volumes).

What we had:

A bunch of  physical servers (IBM) with a range of OSes installed (mainly RH-based).

What was done:

• Most of servers were migrated using VMware Converter.
• Some of those old ones were migrated manually as in previous project.
• Some servers were reinstalled as VMs and just services were migrated.

Result:

The result is pretty obvious for this kind of migration. However I'll provide a couple of benefits:

• Increased reliability. Very small  system outage after H/W failure (That even happened twice. HA automatically migrated all VMs to the another ESXi host.)
• Uninterrupted maintenance. I just migrate all VMs to another host in cluster during the upgrade.
• Energy savings. I cannot provide exact quotas, but I was really surprised when review last report. We save really a lot.
• Convenience. Add/remove some disk space/RAM/vCPUs is just several mouse clicks now.

My Projects: Virtualization - part 1: XEN to ESX

Here is the next post in "My Projects" series, and this time it is about the Virtualization.

Well, if think about that, I used virtualization for a very long time. First time I used jailed environments in FreeBSD about 10 years ago. After that it was VMware workstation server used mainly for home or some tests. But my first serious Project connected with virtualization took place just three years ago.

What we had:

When I moved to a main R&D center in Poland they had two or three XEN hosts for virtualization, and two new ESX servers with Standard license connected to a vCenter Server. Appreciating all benefits of using ESX it was decided to move all old services from the XEN-based virtualization to the ESX-based. In order to complete this task two more Standard licenses was bought and I was assigned to proceed with migration of 10-20 (don't really remember) of VMs.

The funny thing is that VMware Converter 4.0 has been released just in a few weeks after my project has been finished, but I don't feel sorry about that, because I've got the invaluable experience. Moreover, some of the migrated VMs couldn't be migrated with Converter because they were very old, like Fedora Core 1 for example.

Process:

Maybe, the way I did this migration looks too tricky, but it worked and I've migrated few tens of VMs from XEN. Most of guests in XEN had a single disk with /boot and / (root) primary partitions without LVM. Generally, in case with CentOS 5.x the migration looks like that:
Important note: This is just an example without the full description of commands used. In your case procedure can be completely different and single mistake can cause a real harm to your system!

• Create a VM on ESX
• Boot from newly created VM with CentOS CD/ISO in rescue mode with network enabled
• Create all necessary partitions and mount them (e.g. /mnt/newsys)
• On source VM allow ssh for root, or configure rsyncd
• Copy all data from source (e.g. 192.168.1.10) to destination by executing rsync on a new VM:
  # rsync -ah --progress --delete --exclude="dev/" --exclude="proc/" --exclude="sys/" --exclude="tmp/" //192.168.1.10:/ /mnt/newsys/ 
• Mount /dev /sys and /proc to /mnt/newsys
• # chroot /mnt/newsys su -
• Now it's time to change some values in fstab (to use /dev/sda instead of /dev/xvda) and grub.conf
• After this is changed you must (in most cases) rebuild your initrd (mkinitrd tool) and reinstall grub (grub-install)
• Temporary change the IP and boot the system on new VM
• Stop all services on source guest and use rsync to copy the changes
• Turn off the old VM and change back the IP on the new VM.

In most cases it worked, but there was number of cases when it won't work by different reasons (e.g. Fedora Core 1). So, I also had another way to make the same thing. It was more complicated, however it always works:

• Create a VM on ESX with Virtual Disk of the same size or a bit more like it was on XEN
• Boot from CentOS CD in rescue mode with network enabled
• Start netcat to listen for a data an pipe it directly to the disk:
  # nc -l 2121 | dd bs=1M of=/dev/sda
• On XEN host stop the source VM (wasn't necessary in all cases, but is more safe)
• Send the contents of virtual disk to the remote VM booted in rescue mode like mentioned above:
  # dd if=/path/to/disk.img bs=1M|nc 123.123.123.123 2121
• After all data is copied you can open /dev/sda with fdisk and extend second partition (by removing and adding it again with a bigger size)
  # fdisk /dev/sda
  : d (delete partition)
  : 2 (choose 2nd partition)
  : n (new partition)
  p (let it be primary partition)
  : w (write changes and exit)
• After second partition raised you must use file-system tool to extend it (like resie2fs)
  # resize2fs /dev/sda2
• Next you can mount this partition somewhere (don't forget to mount /boot after /), mount system FSes like /dev /sys and /proc, and chroot there.
• Now it's time to change values in fstab (to use /dev/sda instead of /dev/xvda) and grub.conf
• After this is changed you must (in most cases) rebuild your initrd (mkinitrd tool) and reinstall grub (grub-install)
• Temporary change the IP and boot with the new system
• Stop all services on source guest and use rsync to copy the changes
• Turn off the old VM and change back the IP on the new VM.

Of course, now with VMware Converter this procedure is not more needed for a range of Linux servers, but I had pretty interesting experience.

Result:

When project was finished all XEN hosts were reinstalled with ESX Server and configured in a cluster with HA enabled.
In progress of migration the switching between old and new VM for most important services was about 1-2 minutes (sync differences and restart the networking services).
Some services was successfully migrated from old Linux versions to newest CentOS.
Was created a detailed procedure of migration, that allowed engineers in remote offices to complete migration as well.

My Projects: Backula backup system

This is third post in the "My Projects" series. This time, like my previous post, it is also dedicated to the World Backup Day. Two posts in a day - why not?

Although the project was about the same, there were few significant differences. 1st – 5 years difference (2009), and 2nd – mainly Linux environment.


It was a middle-size company. In a central office they had all things IT departments must have, including backup system (TSM). But I was hired in a remote division in another country, where was no System Administrator before me. There were many things to do, but today we are talking about backups.

What we had:

Three Linux servers, one external storage, limited budget.

Well, in situation with a single storage I have no possibility to protect data from a failure of that particular storage. But I split the discs in storage in two different arrays, and dedicate one of them for backups only.

This time I wanted something Enterprise-level, reliable and scalable. At that time Bacula fitted all my needs. The only problem was to get used to it. When I have opened User's manual and found out that there are 764 pages.... I was encouraged! Why? My previous job was quite boring so I was "hungry" for challenges like that. In a few days I was ready to propose a solution and install it in production.

Solution:

Quite simple, as reliable things should be

• Bacula as a backup system
• Standard schedule:
• Daily incremental backups
• Weekly differential backups
• Monthly full backups
• Workstation were also added if user desire

 Result:

Backup system is not intended to have configuration changes often. Configured once, it requires just to watch on daily notifications and make some periodical data restore.
After a new server added it was enough to create a single file with a list of directories to exclude from backup (/tmp, /media, /sys, etc. exluded by default).

Have a nice World Backup Day!

My Projects: History: backup scripts

This is second post in the "My Projects" series, and this time it dedicated to the World Backup Day.

Again, I will describe a project which I have quite some time ago, so some details could be missed.

It was 2004 and one of the first things I have decided to improve when come to new company was backups. Well, "improve" is not a correct word here, since there was nothing to improve – I had to create it from zero. It was quite surprising that 6-years old company doesn't have any backup system.

What we had:

No backup system. No money for that.

My experience at that time didn't allow me to propose any free backup solution, in the same time I was fascinated by writing different CMD scripts – I already did most of administrative tasks in windows using command line. So, instead of trying something that is ready, I decide to write my own bunch of scripts for backups.

Solution:

After a few days of work I've got something like following:

• Nightly script scanned all folders listed in a text file.
• All files with "Archive" attribute set were archived to a compressed file named like I_YYYYMMDD-HHSS.rar. After that the "Archive" attribute was cleared. This allowed me to have Incremental backups.
• Once a month a Full backup was made.
• Full backup older that 3 month was removed with all Incremental backups related.
• Intelligent restore script allowed me to restore any file or folder from any day from last two month.
• I was able to run the backup script at any moment, not just at night, and that allowed me to make several different backup copies during the day if needed.

To be honest – there is nothing special in this project. I can create something like that in few hours now. But it was quite interesting experience for a young admin that allows me to understand better how CMD scripts works.

Result:

Scripts-based backup system completely fulfilled our needs at that time. The restoring procedure was used many time during several years without a single failure.

My Projects: My very first commercial project - VoIP

This is the first post in the "My Projects" series, and I decided to start with my very first project completed in a commercial organization.

It was the beginning of 2004 and my first (big enough) idea at the new place was to improve the telephony. Since pretty much time has gone, I cannot remember all details, but the main is the idea.

What we had:

Equipment:

• 1 main office with Panasonic KX-TD500 with about 8 external lines and about 150 extensions.
• 2 big branches with some other PBXes
• about 5 small branches with or without dedicated phone lines.

Situation:
We didn't have any kind of IVR implemented, so any call to the office was served by two operators and forwarded manually. The main problem was that internal company calls was made also via PSTN, and our customers claimed quite often that they can't break through the busy lines.

The second problem was that even two operators at the same time cannot serve all flow of the calls, so some percent of inter-company calls was dropped, and of course there was quite important calls also.

The same problem was experienced by our two branches. Number of the phone pick-ups was about 85% from the calls initiated! 1 of 6 of our users or customers got "busy" during the day!

Along with that, all branches were connected to the main office other by SHDSL lines (double in some cases) and almost wasn't utilized (RDP traffic only).

Budget:
As usual: "Please do cheap, good and reliable".

Solution:

First we started with pretty cheap VoIP gateways from Dynamix. We had very good impression of their gateways with FXS port, and even installed them in 3 or 4 branches. But when we get to the next step of our project (integration with the existing telephone network) we have found out that FXO gateways connected to PBXes generates unacceptable echo. Support from Dynamix couldn't help us with that, so we had to resign from their hardware and started to search for another brand.

Next one was Planet. This time we've got VoIP gateways with FXO and FXS to test simultaneously. And this time tests was successful. We began to install our solution step by step, and in two month we have the following configuration:

Equipment:

• Main office - Planet VoIP PBX (don't remember the model, but something like IPX-2000) with 2 modules 4 ports each connected to Panasonic TDS-500 to extension ports.
• Big branches - Planet VoIP h323 gateways with 4 FXO ports connected to PBXes to extension ports
• Small branches - Planet VoIP h323 gateways with 2 FXS with phones connected directly to them.

Gentoo  linux with GnuGK was used as a H323 gatekeeper (SIP wasn't really popular then).

The IVR has been configured on the Planet PBX and was also used for the external incoming call. There was a standard invitation like "Hello, this is company ABC, please enter extension or wait for an operator". But this made our users enormously happy.

Results:
After the Project was completed 100% of all calls initiated reached the target extensions or the operator.

Bills from the public telephony provider lowed by 2 (!) times. All new equipment payed off in several month.

Side effect:
One of operators was raised to the Office Manager, since there was no more need to hold two operators at the same time.

My Projects: prologue

I'm going to begin a series of posts aimed to create some kind of portfolio of projects in which I was involved.

I'm not sure there is a dedicated target of this action, but this will be useful first of all for me.

However, if somebody find some of the posts interesting, I just would be happy about that.

Currently I have 16 drafts, that means 16 project witch I found interesting enough to share. But for sure there will be more, because those 16 was remembered just in 10 minutes.

You can find them all under "My Projects" label.