Showing posts with label CentOS. Show all posts
Showing posts with label CentOS. Show all posts

Friday, 18 May 2012

How to setup diskless Ubuntu 12.04 with read-write root partition


Well, this how-to for the diskless Ubuntu setup is not usual. The main point is that all our infrastructure is based on CentOS, but sometimes (like in this case) we must support different Linux distributions for our clients. That is why all server-side configuration will be related to CentOS, and client-side to Ubuntu.

In difference to the official DisklessHowto, this one allows you to prepare a shared installation which can be used by any number of clients simultaneously. All changes to the mounted root will resist in ramfs, so no changes will be written to the NFS, and everything will be gone after restart. This feature in some (many) cases is treated as an advantage.

However, it is recommended to read  DisklessHowto from the Ubuntu site, since all basics related to network boot for Ubuntu described there much better. Also, I took some parts from their documentations to my howto. I believe they wouldn't mind.

Requirements

  • DHCP server
  • TFTP server
  • NFS server
 All my servers uses CentOS, so I will describe server-side the configuration only for this system.
Current server setup is based on CentOS 6.2 x86_64 with IP 192.168.1.10.

Step by step

I. DHCP

I will not describe basics of DHCP configuration, you can easily google it. In order to make PXE work, you need to put just these two option to your DHCP configuration:

filename pxelinux.0;
next-server 192.168.1.10;

Where next-server provides with an address of your TFTP server. It can be put in almost any section of dhcpd.conf: global, class, subnet, pool, group or host.

II. TFTP

1. Install tftp daemon and syslinux package. Syslinux is available from rpmforge.
# yum install tftp-server syslinux
# chkconfig tftp on
# service tftp start 

2. Copy pxelinux files to tftp directory:
# cp /usr/share/syslinux/pxelinux.0 /tftpboot/
# cp /usr/share/syslinux/vesamenu.c32 /tftpboot/

3. Configure pxelinux
# mkdir /tftpboot/pxelinux.cfg
# cat << EOF > /tftpboot/pxelinux.cfg/default
DEFAULT vesamenu.c32
TIMEOUT 600
ONTIMEOUT BootLocal
PROMPT 0
MENU TITLE My PXE Server (by TORNADO)
ALLOWOPTIONS 1
menu width 80
menu rows 15
MENU TABMSGROW 24
MENU MARGIN 10
NOESCAPE 1
LABEL BootLocal
    localboot 0
    TEXT HELP
    Boot to local hard disk
    ENDTEXT
LABEL UBUNTU_1204_DISKLESS
    MENU LABEL Ubuntu 12.04 (64-bit) DISKLESS
    KERNEL Ubuntu/12.04/x86_64/vmlinuz-3.2.0-20-generic
    APPEND root=/dev/nfs nfsroot=192.168.1.10:/srv/nfsroot/Ubuntu/12.04/x86_64,ro initrd=Ubuntu/12.04/x86_64/initrd.img-3.2.0-20-generic ip=dhcp aufs=tmpfs
    TEXT HELP
    Boot the Ubuntu 12.04 64-bit Diskless
    ENDTEXT
EOF 

III. NFS

1. Install nfs
# yum install nfs-utils

# chkconfig nfs on

# service nfs start

2. Add /srv/nfsroot to exports
# cat << EOF > /etc/exports
/srv/nfsroot *(ro,async,no_root_squash,no_subtree_check,no_all_squash)
EOF

3. Apply exports
# exportfs -r

IV. Prepare installation

1. Install Ubuntu
Generally you have two ways.
  1. Use debootstrap.
    I used this option to prepare some tiny installations (like network boot for number of POS terminals).
  2. Install the Ubuntu on the real or virtual system and copy it to NFS server.
    In this article I will follow this way, because I want to prepare a usual Ubuntu installation.
After system was installed and configured as you want, you need to prepare it for network boot.

2. Modify /etc/network/interfaces to set eth0 configuration type to manual:

iface eth0 inet manual

3. Configure /etc/fstab to be looking like this:
# /etc/fstab: static file system information.
#
#                
proc            /proc           proc    defaults        0       0
/dev/nfs       /               nfs    defaults          1       1

4.  Change the following options in /etc/initramfs-tools/initramfs.conf:
MODULES=netboot
BOOT=nfs
DEVICE=eth0
NOTE: If the client source installation you copied the files from should remain bootable and usable from local hard disk, restore the former BOOT=local and MODULES=most options you changed in /etc/initramfs-tools/initramfs.conf. Otherwise, the first time you update the kernel image on the originating installation, the initram will be built for network boot, giving you "can't open /tmp/net-eth0.conf" and "kernel panic". Skip this step if you no longer need the source client installation.

5. Add to /etc/initramfs-tools/modules line:
aufs

6. Copy aufs module to /etc/initramfs-tools/scripts/modules
$ sudo cp /lib/modules/$(uname -r)/kernel/ubuntu/aufs/aufs.ko /etc/initramfs-tools/scripts/modules

7. Copy the following script to /etc/initramfs-tools/scripts/init-bottom as 00_aufs_init (0755):
#!/bin/sh -e

case $1 in
  prereqs)
    exit 0
    ;;
esac

for x in $(cat /proc/cmdline); do
  case $x in
    root=*)
      ROOTNAME=${x#root=}
      ;;
    aufs=*)
      UNION=${x#aufs=}
        case $UNION in
          LABEL=*)
            UNION="/dev/disk/by-label/${UNION#LABEL=}"
            ;;
          UUID=*)
            UNION="/dev/disk/by-uuid/${UNION#UUID=}"
            ;;
        esac    
      ;;
  esac
done

echo "Union=$UNION"

if [ -z "$UNION" ]; then
    exit 0
fi

modprobe -b aufs && echo "OK: modprobe -b aufs" || echo "ERR: modprobe -b aufs"

# make the mount points on the init root file system
mkdir /aufs /ro /rw && echo "OK: mkdir /aufs /ro /rw" || echo "ERR: mkdir /aufs /ro /rw"

# mount read-write file system
if [ "$UNION" = "tmpfs" ]; then
  mount -t tmpfs rw /rw -o noatime,mode=0755 && echo "OK:  mount -t tmpfs rw /rw -o noatime,mode=0755 " || echo "ERR:  mount -t tmpfs rw /rw -o noatime,mode=0755"
else
  mount $UNION /rw -o noatime
fi

# move real root out of the way
mount --move ${rootmnt} /ro && echo "OK: mount --move ${rootmnt} /ro" || echo "ERR: mount --move ${rootmnt} /ro"

mount -t aufs aufs /aufs -o noatime,dirs=/rw:/ro=ro && echo "OK: mount -t aufs aufs /aufs -o noatime,dirs=/rw:/ro=ro" || echo "ERR: mount -t aufs aufs /aufs -o noatime,dirs=/rw:/ro=ro"

# test for mount points on union file system
[ -d /aufs/ro ] || mkdir /aufs/ro
[ -d /aufs/rw ] || mkdir /aufs/rw

mount --move /ro /aufs/ro && echo "OK: mount --move /ro /aufs/ro" || echo "ERR: mount --move /ro /aufs/ro"
mount --move /rw /aufs/rw && echo "OK: mount --move /rw /aufs/rw" || echo "ERR: mount --move /rw /aufs/rw"

# strip fstab off of root partition
grep -v $ROOTNAME /aufs/ro/etc/fstab > /aufs/etc/fstab

mount --move /aufs /root && echo "OK: mount --move /aufs /root" || echo "ERR: mount --move /aufs /root"

exit 0 

To be honest this script isn't mine. I found it some time ago and I don't remember where. If you are the author or you know who he is, please tell me in comments so I put your name here.

8. Build a new initrd image:
$ sudo update-initramfs -k $(uname -r) -c -b /root/

9. Now you can copy this all on server by executing the following command on client:

$ sudo rsync -a --exclude=tmp/* --exclude=proc/* --exclude=sys/* --exclude=dev/* / username@192.168.1.10:/srv/nfsroot/Ubuntu/12.04/x86_64/
Be careful with all slashes! rsync treats source or destination in different way if slash is omitted.
"username" must have write permissions to /srv/nfsroot/Ubuntu/12.04/x86_64.

10. Copy kernel and new initrd image to a proper folder on tftp. Run it ON THE SERVER
# cp /srv/nfsroot/Ubuntu/12.04/x86_64/boot/vmlinuz-$(uname -r) username@192.168.1.10:/tftpboot/Ubuntu/12.04/x86_64/
# cp /srv/nfsroot/Ubuntu/12.04/x86_64/root/initrd.img-$(uname -r) username@192.168.1.10:/tftpboot/Ubuntu/12.04/x86_64/


Notes

Remember, that changes to the root filesystem is limited by you RAM. This means that you will not be able to copy a 4GB video to /tmp if you have only 2GB of RAM.

I have prepared this post without access to my test environment, so some small mistakes are possible. If you find any, please comment.

Sunday, 1 April 2012

My Projects: Virtualization - part 1: XEN to ESX

Here is the next post in "My Projects" series, and this time it is about the Virtualization.

Well, if think about that, I used virtualization for a very long time. First time I used jailed environments in FreeBSD about 10 years ago. After that it was VMware workstation server used mainly for home or some tests. But my first serious Project connected with virtualization took place just three years ago.

What we had:

When I moved to a main R&D center in Poland they had two or three XEN hosts for virtualization, and two new ESX servers with Standard license connected to a vCenter Server. Appreciating all benefits of using ESX it was decided to move all old services from the XEN-based virtualization to the ESX-based. In order to complete this task two more Standard licenses was bought and I was assigned to proceed with migration of 10-20 (don't really remember) of VMs.

The funny thing is that VMware Converter 4.0 has been released just in a few weeks after my project has been finished, but I don't feel sorry about that, because I've got the invaluable experience. Moreover, some of the migrated VMs couldn't be migrated with Converter because they were very old, like Fedora Core 1 for example.

Process:

Maybe, the way I did this migration looks too tricky, but it worked and I've migrated few tens of VMs from XEN. Most of guests in XEN had a single disk with /boot and / (root) primary partitions without LVM. Generally, in case with CentOS 5.x the migration looks like that:
Important note: This is just an example without the full description of commands used. In your case procedure can be completely different and single mistake can cause a real harm to your system!
  • Create a VM on ESX
  • Boot from newly created VM with CentOS CD/ISO in rescue mode with network enabled
  • Create all necessary partitions and mount them (e.g. /mnt/newsys)
  • On source VM allow ssh for root, or configure rsyncd
  • Copy all data from source (e.g. 192.168.1.10) to destination by executing rsync on a new VM:
    # rsync -ah --progress --delete --exclude="dev/" --exclude="proc/" --exclude="sys/" --exclude="tmp/" //192.168.1.10:/ /mnt/newsys/ 
  • Mount /dev /sys and /proc to /mnt/newsys
  • # chroot /mnt/newsys su -
  • Now it's time to change some values in fstab (to use /dev/sda instead of /dev/xvda) and grub.conf
  • After this is changed you must (in most cases) rebuild your initrd (mkinitrd tool) and reinstall grub (grub-install)
  • Temporary change the IP and boot the system on new VM
  • Stop all services on source guest and use rsync to copy the changes
  • Turn off the old VM and change back the IP on the new VM.
In most cases it worked, but there was number of cases when it won't work by different reasons (e.g. Fedora Core 1). So, I also had another way to make the same thing. It was more complicated, however it always works:
  • Create a VM on ESX with Virtual Disk of the same size or a bit more like it was on XEN
  • Boot from CentOS CD in rescue mode with network enabled
  • Start netcat to listen for a data an pipe it directly to the disk:
    # nc -l 2121 | dd bs=1M of=/dev/sda
  • On XEN host stop the source VM (wasn't necessary in all cases, but is more safe)
  • Send the contents of virtual disk to the remote VM booted in rescue mode like mentioned above:
    # dd if=/path/to/disk.img bs=1M|nc 123.123.123.123 2121
  • After all data is copied you can open /dev/sda with fdisk and extend second partition (by removing and adding it again with a bigger size)
    # fdisk /dev/sda
    : d (delete partition)
    : 2 (choose 2nd partition)
    : n (new partition)
    p (let it be primary partition)
    : w (write changes and exit)
  • After second partition raised you must use file-system tool to extend it (like resie2fs)
    # resize2fs /dev/sda2
  • Next you can mount this partition somewhere (don't forget to mount /boot after /), mount system FSes like /dev /sys and /proc, and chroot there.
  • Now it's time to change values in fstab (to use /dev/sda instead of /dev/xvda) and grub.conf
  • After this is changed you must (in most cases) rebuild your initrd (mkinitrd tool) and reinstall grub (grub-install)
  • Temporary change the IP and boot with the new system
  • Stop all services on source guest and use rsync to copy the changes
  • Turn off the old VM and change back the IP on the new VM.
Of course, now with VMware Converter this procedure is not more needed for a range of Linux servers, but I had pretty interesting experience.

Result:

When project was finished all XEN hosts were reinstalled with ESX Server and configured in a cluster with HA enabled.
In progress of migration the switching between old and new VM for most important services was about 1-2 minutes (sync differences and restart the networking services).
Some services was successfully migrated from old Linux versions to newest CentOS.
Was created a detailed procedure of migration, that allowed engineers in remote offices to complete migration as well.

Monday, 20 February 2012

How to migrate data to the smaller disks using LVM.

Well, there could be any other subject for this post taken, because I'm going to write about the Physical Extents in LVM, and this could be used for different purposes.

My particular issue was related to the fact that if you want to use snapshots or cloning of a turned-on VM, you must be aware of some overhead for VMDK files. You can find more details in VMware KB 1012384: "Creating a snapshot for a virtual machine fails with the error: File is larger than maximum file size supported".

The problem was that when I try to clone a VM that have a virtual drive with size of 256GB (the maximum size for the specified datastore), I faced the following errors:

Create virtual machine snapshot VIRTUALMACHINE File <unspecified filename> is larger than the maximum size supported by datastore '<unspecified datastore>'
File is larger than the maximum size supported by datastore
 That was a server in production, so I couldn't just turn it off for 30 minutes. At this moment I realized, that I have number of servers with the problem like that and I can face the same problems sooner or later.

Generally you cannot just decrease disk size from the configuration of VM. (Even if there was this possibility, this is fairy not to use. Almost any file system will go crazy with the sudden change like that. In most cases this will lead to the loss of data).

Fortunately, I use LVM for all my servers, mainly for the purpose of extending volumes if needed. This time I had a task to decrease the size of Physical Volumes and make it without a second of downtime.

So, steps was as following:
  • Check your backups
  • Add the virtual hard disks holding in mind the .vmdk sizes as it's specified in VMware KB 1012384.
    Depending on your space assigning policy you can follow the following rules:
    a) add 2 disks of 250GB (in my case) with thin provisioning;
    b) add 1 of 250GB and 1 disk of 10GB. (I use round numbers to simplify the setup)
  • Re-scan scsi bus inside the VM:
# echo "- - -" > /sys/class/scsi_host/host0/scan
  • Create lvm partitions for all the added devices:
# fdisk /dev/sdX
Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-32635, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-32635, default 32635):
Using default value 32635

Command (m for help): t
Selected partition 1
Hex code (type L to list codes): 8e
Changed system type of partition 1 to 8e (Linux LVM)

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

  • Extend the Volume Group by adding those new disk

# lvm vgextend VolGroupXX /dev/sdX1 /dev/sdY1
  • Check the number of Physical Extents to migrate

# lvm pvdisplay
  --- Physical volume ---
  PV Name               /dev/sdf1
  VG Name               VolGroupXX
  PV Size               250.00 GiB / not usable 4.69 MiB
  Allocatable           yes
  PE Size               4.00 MiB
  Total PE              63998
  Free PE               63998
  Allocated PE          0
  PV UUID               Zp7uJR-YsIQ-AjRP-hdGL-OXSl-XbJG-N1GFn2
  --- Physical volume ---
  PV Name               /dev/sdg1
  VG Name               VolGroupXX
  PV Size               250.00 GiB / not usable 4.69 MiB
  Allocatable           yes
  PE Size               4.00 MiB
  Total PE              63998
  Free PE               63998
  Allocated PE          0
  PV UUID               eKdiqW-eMjI-ck4a-grM3-ogX3-6BOP-Q1ldlC
  --- Physical volume ---
  PV Name               /dev/sdb1
  VG Name               VolGroupXX
  PV Size               255.99 GiB / not usable 2.72 MiB
  Allocatable           yes (but full)
  PE Size               4.00 MiB
  Total PE              65534
  Free PE               0
  Allocated PE          65534
  PV UUID               PajbnY-65II-Aeyt-WamI-Xzr8-M3b2-kWfy6U

In this example /dev/sdf and /dev/sdg are the new added hard drivse with the size of 250GB, and /dev/sdb is an old one with the size of 256GB. The important part for as is "Total PE". If we call the pvmove command without specifying the amount of Physical Extents to move, we will receive an error.
  • Migrate the Extents to the new hard drives
# lvm pvmove /dev/sdb1:1-63998 /dev/sdf1
# lvm pvmove /dev/sdb1 /dev/sdg1
As you noticed I didn't specify the number of extents to move to the second drive. In fact this command will move only the used ones.
Now, you might extend the Logical Volume to use all the space added
  • Extend the Logical Volume
# lvm lvextend +100%FREE VolGroupXX/LogVolXX

  • extend the partition with the proper file system tools
# <resize2fs|resize_reiserfs|...> /dev/mapper/VolGroupXX-LogVolXX
  • And, finally, remove the old drives
# lvm vgreduse VolGroupXX /dev/sdb1
# lvm pvremove /dev/sdb1

Now you can remove the old drive from the VM configuration... zero downtime.

Well, many of the experienced Linux admins might not find anything new in this article. However, I didn't know how it works till I faced a need of a change like that. For instance, I have extended so many volumes so much times, that this operation (extending) takes about a minute for me (if not count the time spent by the FS extending tool).

Friday, 2 December 2011

CentOS 6 rpm sign issue (V4 signature is used by default)

Well, the problem I'm going to write about is known and the appropriate bug-reports exists. But I didn't found the proper threads on at the first pages of search results while googling by the error messages. So, I hope, this post will help somebody to find out what is happening when "rpm --checksig" returns "Header V3 DSA signature: OK, key ID xxxxxxxx" when "signature: OK" is expected.

I faced the problem when was deploying a Spacewalk server. I choose CentOS 6.0 to be the OS on that server and I was simply following install instructions (many thanks to the Spacewalk community, it was really easy to install the server) until I tried to create a custom Software Channel for some individual packages.

In fact, instructions about how to sign RPMs is the same on most of the howto pages... let's remind them (simple version):

gpg --gen-key

gpg --export -a XXXXXXX > RPM-GPG-KEY-Mycompany

put to ~/.rpmmacros:
%_signature gpg
%_gpg_name XXXXXXXX

and then just use:

rpm --resign some-package.rpm

On the client system it's enough to get the public key and import it by rpm command:

rpm --import /path/to/RPM-GPG-KEY-Mycompany

This was a very simplified recipe without expected outputs, but if everything is going  well there shouldn't be any unexpected questions.

After all is done, rpm --checksign some-package.rpm should return something like that:
some-package.rpm: rsa sha1 (md5) pgp md5 OK

And it was like that on the CentOS 6 server... But when I tried to install the package via yum on CentOS 5.7, I've received the following error:

error: rpmts_HdrFromFdno: Header V4 RSA/SHA1 signature: BAD, key ID XXXXXXXX

And rpm -v --checksig some-package.rpm was returning:

    Header V4 RSA/SHA1 signature: BAD, key ID xxxxxxxx
    Header SHA1 digest: OK (835b77fb70d2a6075c428b9eb57bbfcdc2a0d1ce)
    V4 RSA/SHA1 signature: BAD, key ID xxxxxxxx
    MD5 digest: OK (ede2464b724b0bafef0db4a53c02c1d0)
 
More weird thing, is that when I sign it with the same key on CentOS 5.7 the rpm was OK.
It was my first time signing the RPMs, so I have spent some time while found out the difference from the package with proper signature:

$ rpm -v --checksig rpmforge-release-0.5.1-1.el5.rf.i386.rpm
    Header V3 DSA signature: OK, key ID 6b8d79e6
    Header SHA1 digest: OK (56871fe945ed2b2c868430b0002bb47dc129e981)
    MD5 digest: OK (69c4cbf8229ba4b319d58f99ddebddf3)
    V3 DSA signature: OK, key ID 6b8d79e6

So, with that insight I found an old bug with a description how to force GPG signature version to v3. To do that your ~/.rpmmacros should look like this:

%_signature gpg
%_gpg_name  XXXXXXXX
%__gpg_sign_cmd %{__gpg} \
    gpg --force-v3-sigs --digest-algo=sha1 --batch --no-verbose --no-armor \
    --passphrase-fd 3 --no-secmem-warning -u "%{_gpg_name}" \
    -sbo %{__signature_filename} %{__plaintext_filename}

Seems like the "rpm --resign" command  uses GPG signature V4 by default, despite to the following text in rpm manual page:

For compatibility with older versions of GPG, PGP, and rpm, only V3 OpenPGP signature packets should be configured.  Either DSA or RSA  verification algorithms can be used, but DSA is preferred.

Hope this helps...

Saturday, 13 August 2011

How to preserve eth0 interface name for cloned or migrated CentOS or Ubuntu?

I'm sure that many people faced this nasty thing: after you clone or reconfigure a Virtual Machine, OS reconfigures the interface because it has another MAC. I will not say that this is bad behavior, but what can you do, if you need to preserve the eth0 name and configuration?

In CentOS it's very simple - all you need is just to remove a string like "HWADDR=00:11:22:aa:bb:cc" in file /etc/sysconfig/network-scripts/ifcfg-eth0. That's it! Now even after you clone this machine or in any other way change the ethernet card - config will not be changed. Of course, in some cases there is a risk to face an IP conflict, but you're good admin, right :)?

Well, I don't use Ubuntu for server often, so I didn't look for a persistent solution. But what I do time to time, is just removing all strings from file /etc/udev/rules.d/70-persistent-net.rules right before cloning. It makes Ubuntu to forget, that some ethX name is already assigned for some MAC address.

Tuesday, 10 May 2011

Use of yum priorities

Some time ago I faced a need to have some useful tools installed with the stable core packages on my CentOS servers. How I did it? I've added an rpmforge repository to the yum configuration and disabled it with enabled=0 in the repository configuration, because if you leave it enabled, some packages that you need to be more stable could be upgraded with the newest versions from rpmforge. Then, when I need to install something out of the main, I did it like that:

# yum --enablerepo=rpmforge install some-tool

Of course it was inconvenient, I had problems with updates and so on.

Then, a single look on the yum documentation showed me the yum-priorities package.
Now I have all the repositories enabled and prioritized. For example, the [base] repository has priority=1 and [rpmforge] has priority=20.
This way, yum update will not replace httpd with the latest version from rpmforge, and  yum install htop  will succeed. Of course, you can make it in another way. For example, you can add a PostgreSQL repository and make it higher priority than base/updates - this way you'll always keep your postgresql packages up-to-date using just  yum update  without a need to upgrade it manually.