vSphere: A general system error occurred: Authorize Exception

This article may help you, if  solution from VMware Knowledge Base titled vCenter Server login fails with error: A general system error occurred: Authorize Exception not helps.

Symptoms (as from KB)

• vCenter Server services are running, but a user that was previously able to log into vCenter Server no longer can
• A local admin account is able to log in, but domain users cannot
• You see this error:
  
  A general system error occurred: Authorize Exception

Additionally

• Re-joining to domain don't help
• Your primary (and secondary)  Domain Controllers which was used before were changed
• C:\Program Files\VMware\Infrastructure\SSOServer\webapps\ims\WEB-INF\classes\krb5.conf contains wrong kdc entries.
  NB: Don't try to edit this file. It's automatically generated.

Cause

•  Single-Sign-On service uses old DC name(s) when binds to Active Directory

Resolution

1.  Install vSphere WebClient (don't forget that you should use admin@System-Domain username in order to connect it with SSO)
2. Login to Web Client (https://vcenter.company.com:9443/vsphere-client/) using SSO admin account - admin@System-Domain
3. On Administration page select Configuration menu under Sign-on and Discovery section
4. Select the desired identity source (type - Active Directory), click Edit and write down (printscreen) all of the connection options
   Want to point out, that in my case, changing server URLs has no effect - no changes was saved after OK was pressed, so...
5. Remove old identity source and add a new one, with the same parameters, but with new server URLs
6. Done

Not important

To be honest - it was the most interesting issue for last couple of month. Mostly because any other issue I faced was already solved by someone else, so any problem was solved by following the obvious scenario: Problem -> logs - > google -> solution.

Now I have to switch on my imagination, because all solutions for "Authorize Exception" problem suggested to re-join to AD and/or fix AD/DNS problems. So we spent several hours fixing non-existing problems.

Well, we knew that Domain Controllers were changed, but we forgot completely about SSO, and nobody knew/remember that SSO uses it's own configuration (based on MIT kreberos) in order to bind to AD.

But even when the problem was located, I've spent next couple hours examining SSO logs and trying to find where AD discovery configuration can be changed. It's a pity, that it's not possible to configure by some CLI (at least I didn't find anything).

Hope this article helps. If so, I would appreciate if you consider to leave a comment.

VMWare: Remove datastore problem - "The resource xxxxxxxxxxxxxxxxxxxxx" is in use

Today I've faced a situation, when I removed all VMs from a datastore, but I still couldn't remove it because of the following error:

Status: The resource 'xxxxxxxxxxxxxxxxxx' is in use.

What I did first, is unmounting the datastore on all hosts in the cluster. This allowed me to find two hosts which has locked this datastore. While all other hosts allowed me to unmount the datastore with no problem, those two returned me an error:

Status: The resource 'xxxxxxxx' is in use.
Erorr Stack: Cannot unmount volume 'xxxxxxxx' because file system is busy. Correct the problem and retry the operation.

At this moment I've remembered, that some time ago in order to install some upgrade, I needed to configure a ScrathConfig option to use some external location on those two hosts.

Tip:
Go to "Inventory > Hosts and Clusters", select host that uses a datastore, goto "Configuration" tab, and click on "Advanced Settings" option. Find "ScratchConfig" section and change to something else (e.g. /tmp). Restart the host. Now you will be able to remove the datastore.

PS: Of course, this tip is a kind of useless if it was you who configured this option. But if you've got some legacy which you haven't configured before.

PPS: I didn't have this situation, but I'd advice to check also Syslog.global.logDir option.

VMWare Cluster - Remove datastore failed - The vSphere HA agent on host '10.0.0.1' failed to quiesce file activity on datastore '/vmfs/volumes/XXXXXXXXXXXXX'. To proceed with the operation to unmount or remove a datastore, ensure that the datastore is accessible, the host is reachable and its vSphere HA agent is running.

For the last few days I do some reorganization of our Virtual Infrastructure. One of the steps of this reorganization is upgrade from VMFS 3 to VMFS 5 for all our storages connected to the main HA cluster.

Although it is possible in ESXi to upgrade to VMFS5 in-place, I decided to completely remove and re-create storages. The main reason was that previously all arrays were sliced in several 2TiB(-1MiB) logical drives, and I wanted to create single logical unit for each storage subsystem.

But almost each time I tried to remove an old datastore from ESXi, I've received an error:


Status: The vSphere HA agent on host '10.0.0.1' failed to quiesce file activity on datastore '/vmfs/volumes/XXXXXXXXXXXXX'. To proceed with the operation to unmount or remove a datastore, ensure that the datastore is accessible, the host is reachable and its vSphere HA agent is running.


Tip:
Well, solution is quite simple. Just go to the host which generated this error (in my example it's 10.0.0.1) in "Inventory > Hosts and Clusters" view, and remove a the datastore from the Summary tab of that host.

My Projects: Virtualization - part 2: P2V

Here is the next post in "My Projects" series, and the second one about Virtualization.

After migration from XEN was finished, we started to migrate all existing physical machines to virtual. Generally, this was one of the most light project for the last couple of years. First of all - VMware Converter 4.0 has been released, so all CentOS-es were migrated smoothly, without an issue. Second - while migrating VMs from XEN, I've collected so much experience, that some tasks which seemed very complicated before, now was a kind of obvious for me (e.g. like playing with LVM volumes).

What we had:

A bunch of  physical servers (IBM) with a range of OSes installed (mainly RH-based).

What was done:

• Most of servers were migrated using VMware Converter.
• Some of those old ones were migrated manually as in previous project.
• Some servers were reinstalled as VMs and just services were migrated.

Result:

The result is pretty obvious for this kind of migration. However I'll provide a couple of benefits:

• Increased reliability. Very small  system outage after H/W failure (That even happened twice. HA automatically migrated all VMs to the another ESXi host.)
• Uninterrupted maintenance. I just migrate all VMs to another host in cluster during the upgrade.
• Energy savings. I cannot provide exact quotas, but I was really surprised when review last report. We save really a lot.
• Convenience. Add/remove some disk space/RAM/vCPUs is just several mouse clicks now.

My Projects: Virtualization - part 1: XEN to ESX

Here is the next post in "My Projects" series, and this time it is about the Virtualization.

Well, if think about that, I used virtualization for a very long time. First time I used jailed environments in FreeBSD about 10 years ago. After that it was VMware workstation server used mainly for home or some tests. But my first serious Project connected with virtualization took place just three years ago.

What we had:

When I moved to a main R&D center in Poland they had two or three XEN hosts for virtualization, and two new ESX servers with Standard license connected to a vCenter Server. Appreciating all benefits of using ESX it was decided to move all old services from the XEN-based virtualization to the ESX-based. In order to complete this task two more Standard licenses was bought and I was assigned to proceed with migration of 10-20 (don't really remember) of VMs.

The funny thing is that VMware Converter 4.0 has been released just in a few weeks after my project has been finished, but I don't feel sorry about that, because I've got the invaluable experience. Moreover, some of the migrated VMs couldn't be migrated with Converter because they were very old, like Fedora Core 1 for example.

Process:

Maybe, the way I did this migration looks too tricky, but it worked and I've migrated few tens of VMs from XEN. Most of guests in XEN had a single disk with /boot and / (root) primary partitions without LVM. Generally, in case with CentOS 5.x the migration looks like that:
Important note: This is just an example without the full description of commands used. In your case procedure can be completely different and single mistake can cause a real harm to your system!

• Create a VM on ESX
• Boot from newly created VM with CentOS CD/ISO in rescue mode with network enabled
• Create all necessary partitions and mount them (e.g. /mnt/newsys)
• On source VM allow ssh for root, or configure rsyncd
• Copy all data from source (e.g. 192.168.1.10) to destination by executing rsync on a new VM:
  # rsync -ah --progress --delete --exclude="dev/" --exclude="proc/" --exclude="sys/" --exclude="tmp/" //192.168.1.10:/ /mnt/newsys/ 
• Mount /dev /sys and /proc to /mnt/newsys
• # chroot /mnt/newsys su -
• Now it's time to change some values in fstab (to use /dev/sda instead of /dev/xvda) and grub.conf
• After this is changed you must (in most cases) rebuild your initrd (mkinitrd tool) and reinstall grub (grub-install)
• Temporary change the IP and boot the system on new VM
• Stop all services on source guest and use rsync to copy the changes
• Turn off the old VM and change back the IP on the new VM.

In most cases it worked, but there was number of cases when it won't work by different reasons (e.g. Fedora Core 1). So, I also had another way to make the same thing. It was more complicated, however it always works:

• Create a VM on ESX with Virtual Disk of the same size or a bit more like it was on XEN
• Boot from CentOS CD in rescue mode with network enabled
• Start netcat to listen for a data an pipe it directly to the disk:
  # nc -l 2121 | dd bs=1M of=/dev/sda
• On XEN host stop the source VM (wasn't necessary in all cases, but is more safe)
• Send the contents of virtual disk to the remote VM booted in rescue mode like mentioned above:
  # dd if=/path/to/disk.img bs=1M|nc 123.123.123.123 2121
• After all data is copied you can open /dev/sda with fdisk and extend second partition (by removing and adding it again with a bigger size)
  # fdisk /dev/sda
  : d (delete partition)
  : 2 (choose 2nd partition)
  : n (new partition)
  p (let it be primary partition)
  : w (write changes and exit)
• After second partition raised you must use file-system tool to extend it (like resie2fs)
  # resize2fs /dev/sda2
• Next you can mount this partition somewhere (don't forget to mount /boot after /), mount system FSes like /dev /sys and /proc, and chroot there.
• Now it's time to change values in fstab (to use /dev/sda instead of /dev/xvda) and grub.conf
• After this is changed you must (in most cases) rebuild your initrd (mkinitrd tool) and reinstall grub (grub-install)
• Temporary change the IP and boot with the new system
• Stop all services on source guest and use rsync to copy the changes
• Turn off the old VM and change back the IP on the new VM.

Of course, now with VMware Converter this procedure is not more needed for a range of Linux servers, but I had pretty interesting experience.

Result:

When project was finished all XEN hosts were reinstalled with ESX Server and configured in a cluster with HA enabled.
In progress of migration the switching between old and new VM for most important services was about 1-2 minutes (sync differences and restart the networking services).
Some services was successfully migrated from old Linux versions to newest CentOS.
Was created a detailed procedure of migration, that allowed engineers in remote offices to complete migration as well.

IBM DS3000/DS4000/DS5000 vCenter Management plugin issue

Not long ago I have found that IBM have a plugin for their storage for the vCenter. Particularly I was interested to manage DS3400 and DS3524 via vSphere client console, so I've got a plugin from this page.

However, during setup of this plugin I've faced couple of issues.

First of all (this is minor actually, but I've spent 20 minutes to find out what is happening) make sure you've set the free TCP ports for Jetty service. In my case there was a conflict with the vCenter Update service. Common issue, nothing special... however...

Second issue is rather VMware specific, since I found a solution on some page related to another plugin for vCenter. So, the problem is that you have the following error when try to select this plugin in vSphere Client:

User is not authorized to use this plug-in.
Please contact the Administrator and ask for StorageAdmin.readwrite or SorageAdmin.readonly Privileges

However, you are for sure in the group with the full privileges... And actually this is the problem. It turns out that your account should have those privileges apart from any group, set on per-user basis.

So, as workaround, just add a separate permission for your account and restart the vSphere Client.

Preconfigured ESXi on USB flash drive

It was quite long after my last post. Too much work and almost no free time. However, I've learned few interesting things, and I'm going to share them. And the first one - about the ESXi deployment.

We have up to 20 ESX nodes in our offices worldwide. And sometimes it was a hard way to deploy them remotely, using local supporters help. Moreover, even in our datacenter in place where we're sitting, it's quite uncomfortable to provide some maintenance if you forgot to take warm closes.

I knew there is a possibility to install ESXi on a USB flash drive. But I was interested in pre-deployment. The way is happened to be very easy.
To deploy a clean installation, all you need, is just to:
a) download the ESXi iso, for example here.
b) extract a file, named imagedd.bz2
c) unbzip2 it :)
d) deploy it on your flash drive. I used dd under the Linux like following:

# dd if=imagedd of=/dev/sdb bs=1M

(please, check if your flash drive is under /dev/sd?. if you just use the string above, you can destroy the existing data! Use it only if you understand what it means and on your own risk!)

Done! Now you have a fresh bootable flash drive with ESXi unconfigured yet.
Next thing I did, is just connect it to some laptop (I had Lenovo G550) and boot up.You can use any PC or laptop with H/W Virtualization support. The only issue I had with this budget laptop using ESXi, is that the keyboard doesn't work. So I just have plugged an external one. Now, you can configure the network and make some configuration remotely, with vSphere client, that is not possible to do from the console.

The main problem we had, is that after you select several interfaces to be used for NIC teaming (or portchannel on the Cisco switch side), ESXi that by default it uses "Route based on the originating virtual port ID". And that doesn't work well with the Cisco default configuration, which requires to be set the "Route based on IP hash" (or src-dst-ip hash).

Important note, for those who faced that NIC teaming is not working correctly! In difference to ESX 4.0, which we had used before, in ESXi 4.1. we found, that it makes NIC teaming configuration of management interface not as a default (inherited from the virtual switch config), but configures it independently! This strange behavior cost us 2 hours before we found this. Because we checked many times the NIC Teaming configuration of vSwitch and had no idea that Management Network has it different.

Next I configured all DNS settings, all VM and VMkernel networks, NTP, etc... Then I closed vSphere client and made the final network configuration to be used in the datacenter. Shutdown.... Ready!

All I need after that, is just go to the server room (or send a flash drive to the remote office), plug flash drive to the server (In case of IBM x3650 M2 and M3 there is a special USB port an a RAID controller for that purpose), boot from USB (in case of the mentioned IBM servers - boot from HardDisk1), and select the network interfaces to be for the Management Network after ESXi boot.

That is the way I do the ESXi deployment now. If you have any suggestions - you're welcome.