Zenoss: filtering the icons on the Infrastructure tree

This is the next article about my small hacks of Zenoss.

This time I will show how I filter the events which are taken for represent event severities with the appropriate Device Class, Group, System and Location at the Infrastructure->Devices page . The main reason for this, is that I have added in zenoss all systems from ours "Server" networks. These servers could be in "Test", "Pre-production", "Maintenance" and even "Decommissioned" state. Thus, there is number of events we could ignore. Moreover, on the "Production" systems we could have some events which we already aware of. If not filter these events, the icons on the Infrastructure tree element does not show us the real state of our infrastructure, and we have almost all tree filled with Warning, Error or Critical icons.

Unfortunately, because of poor knowledge of Python, the filter is hardcoded. Nevertheless - it works :)

[zenoss@zenoss infos]$ pwd
/opt/zenoss/Products/Zuul/infos
[zenoss@zenoss infos]$  diff ./device.py.orig ./device.py -u
--- ./device.py.orig    2011-12-06 19:08:19.000000000 +0000
+++ ./device.py 2011-12-06 18:59:06.000000000 +0000
@@ -100,7 +100,10 @@
         f = getFacade('event')
         root = self.root.uid.split('/')[3]
         orgcol = ORGTYPES[root]
-        q = 'select %s, max(severity) from status group by %s' % (orgcol,
+       """
+       Added "where eventState = '0'" to suppress error icon in tree on acknowledged events
+       """
+        q = "select %s, max(severity) from status where eventState = '0' and prodState > '499' group by %s" % (orgcol,
                                                                   orgcol)
         result = {}
         for org, sev in f._run_query(q, ()):

You can see my tree with- (left) and without (right) the filter applied. (Yep, there are event some critical events in Production. Will solve them tomorrow... if lucky... )

 

Hope this will help.
Stay tuned :)

Subversion "fun"

Well, although git is seems to be better from subversion even for our needs, we still use and will be using subversion in order to provide a long-term support for our customers, those who might require some changes in some old projects to be made... Anyway, the story is not about that...


"It was a rainy Thursday". As we could rely on our monitoring system, nobody was expecting some sudden troubles and I was doing some usual stuff... Suddenly... Suddenly we receive several calls at one time. All users reported some problem with accessing the SVN server. In a few second monitoring system reported: 100% use of the / (root) partition. WTH?! There should be two thresholds warnings before 90% and 95%! Well, login in, check.. Well... subversion repos are on another partition, so it's some system stuff... Logs are normal size - logrotate is configured for that. Aha!.. Several gigs in /tmp.... in a single file... Well, # lsof|grep filenname and we have a guilty. It was the svnsync process (we put it on postcommit hook to keep remote mirror synchronized), fortunately with a path to the repository to sync.

Now, let's see what was the number of the last synchronized revision on remote server (let's call it N), and then take a look on the N+1 revision. A single file was committed, named other_repo_name.tar.gz. With size of few gigs...

First of all block the synchronization to remote mirrors and block all commits (exit 1 in the pre-commit hook. Don't forget to put some message for users, sth like `echo "Commits are blocked, please contact support for more details" 1>&2`). Seems like we will need to cut this commit from the DB. Now, when further risks are eliminated, let's call to talk with the user.

- Hi Mike (let's call him Mike), this is John Smith (let's call me John :)) from the IT dept. We noticed that you have committed a few gigs file to the XXXX repo, didn't you?
- Hi, yes I did.
- But do you know that we have other services to store archives, like _service1_ and _service2_? Instead, our SVN servers are used as a version control system.
- Yes I know, that's why I put there a copy of the repository YYYY (Note: YYYY is an another repo on the same server), but since it was too big in sources I have compressed it to not take too much space on the server...
- ... *confused... (describing users the basics of work with services like SVN is far from my responsibility). Well, Mike, I understand. Thank you for description. Just to inform you, XXXX repo will be not available for commits for few hours. Bye...


Next, I called to Mike's department boss. Not to claim, but just to describe the situation and to ask him to inform his team that repo will be unavailable for commits for few hours (dump/recover), and that one commit that has been made after the huge one will have to be re-committed gain.

The moral of the story: On the important servers keep /tmp on a separate partition! I do now :)

CentOS 6 rpm sign issue (V4 signature is used by default)

Well, the problem I'm going to write about is known and the appropriate bug-reports exists. But I didn't found the proper threads on at the first pages of search results while googling by the error messages. So, I hope, this post will help somebody to find out what is happening when "rpm --checksig" returns "Header V3 DSA signature: OK, key ID xxxxxxxx" when "signature: OK" is expected.

I faced the problem when was deploying a Spacewalk server. I choose CentOS 6.0 to be the OS on that server and I was simply following install instructions (many thanks to the Spacewalk community, it was really easy to install the server) until I tried to create a custom Software Channel for some individual packages.

In fact, instructions about how to sign RPMs is the same on most of the howto pages... let's remind them (simple version):

gpg --gen-key

gpg --export -a XXXXXXX > RPM-GPG-KEY-Mycompany

put to ~/.rpmmacros:

%_signature gpg
%_gpg_name XXXXXXXX

and then just use:

rpm --resign some-package.rpm

On the client system it's enough to get the public key and import it by rpm command:

rpm --import /path/to/RPM-GPG-KEY-Mycompany

This was a very simplified recipe without expected outputs, but if everything is going  well there shouldn't be any unexpected questions.

After all is done, rpm --checksign some-package.rpm should return something like that:
some-package.rpm: rsa sha1 (md5) pgp md5 OK

And it was like that on the CentOS 6 server... But when I tried to install the package via yum on CentOS 5.7, I've received the following error:

error: rpmts_HdrFromFdno: Header V4 RSA/SHA1 signature: BAD, key ID XXXXXXXX

And rpm -v --checksig some-package.rpm was returning:

    Header V4 RSA/SHA1 signature: BAD, key ID xxxxxxxx
    Header SHA1 digest: OK (835b77fb70d2a6075c428b9eb57bbfcdc2a0d1ce)
    V4 RSA/SHA1 signature: BAD, key ID xxxxxxxx
    MD5 digest: OK (ede2464b724b0bafef0db4a53c02c1d0)
 

More weird thing, is that when I sign it with the same key on CentOS 5.7 the rpm was OK.
It was my first time signing the RPMs, so I have spent some time while found out the difference from the package with proper signature:

$ rpm -v --checksig rpmforge-release-0.5.1-1.el5.rf.i386.rpm
    Header V3 DSA signature: OK, key ID 6b8d79e6
    Header SHA1 digest: OK (56871fe945ed2b2c868430b0002bb47dc129e981)
    MD5 digest: OK (69c4cbf8229ba4b319d58f99ddebddf3)
    V3 DSA signature: OK, key ID 6b8d79e6

So, with that insight I found an old bug with a description how to force GPG signature version to v3. To do that your ~/.rpmmacros should look like this:

%_signature gpg
%_gpg_name  XXXXXXXX
%__gpg_sign_cmd %{__gpg} \
    gpg --force-v3-sigs --digest-algo=sha1 --batch --no-verbose --no-armor \
    --passphrase-fd 3 --no-secmem-warning -u "%{_gpg_name}" \
    -sbo %{__signature_filename} %{__plaintext_filename}

Seems like the "rpm --resign" command  uses GPG signature V4 by default, despite to the following text in rpm manual page:

For compatibility with older versions of GPG, PGP, and rpm, only V3 OpenPGP signature packets should be configured.  Either DSA or RSA  verification algorithms can be used, but DSA is preferred.

Hope this helps...

Zenoss: hiding the page header

I didn't have time for the last couple month to write anything to this blog. So I decided to write some short article for now. And this time it will be again about Zenoss.


Quite long ago at Zenoss Community Forum I've posted a request for a portlet for Zenoss 3.0.* that shows nothing but the filtered events. As I have mentioned in my post, I've made a trick with the Site Window portlet in Zenoss 2.5.2, but in v.3.0.* this possibility disappeared. After some time, because of no response on my post, I've decided to try to do something by myself. Actually, I'm not a programmer, and am a completely newbie in python, but I was curios to make it work. And I did it!

Maybe my solution is not very good, but it is as it is.
So, I've changed a file:
/opt/zenoss/Products/ZenUI3/browser/templates/base-new.pt
diff -u:

@@ -35,6 +35,10 @@
             href="/++resource++zenui/css/zenoss.css" />
         <link rel="stylesheet" type="text/css"
             href="/++resource++zenui/css/zenui3.css" />
+        <tal:block tal:condition="python:request.get('noheader',False)">
+            <link rel="stylesheet" type="text/css"
+            href="/++resource++zenui/css/noheader.css" />
+        </tal:block>
         <tal:block tal:content="structure provider:js-security"/>
         <tal:block tal:content="structure provider:all-js"/>
         <script tal:define="container context/getPrimaryParent" tal:content="string:

And create another one:
/opt/zenoss/Products/ZenUI3/browser/resources/css/noheader.css:

/**
 * HEADER
 */
#header {
    height: 0px !important;
    visibility:hidden ; }

Now, when you add "?noheader=1" to your URL to any Zenoss page, the header will not be displayed.

Here is an example of the Site Window on my Dashboard with URL to the filtered Event View:

Zenoss - introduction

This is a first post about Zenoss - the Open Source IT Management System.
I need to introduce it first, because I'm sure that there will be far more than one article about that. I worked with Zenoss very close for almost 2 years and I have some interesting things to share, which I didn't put to the Zenoss community forums.

First of all - why Zenoss?
Some time ago I was not satisfied of the existing monitoring system. It was Nagios, and generally – it was very good for the main thing we want from it – the monitoring. But I wanted something more. I wanted an easy way to collect and view the performance data (we used cacti before), I wanted to have auto-discovery, to have the agent-less solution, to make it easy to configure for others (actually, I failed with that :)), to have the alerts escalation and something else. And I wanted this all ready just from the box and, of course, for free :).

I have tested several solutions before I stopped with Zenoss. I will not write a list of competitors and all their 'pluses' and 'minuses'. I will describe with what I had success with Zenoss and where I failed.

Advantages:
Core. Generally, with quite some effort, but it's possible to get any kind of data, analyze it in any possible way and proceed with any kind of reaction. And out from that I have all the other "pluses".

Performance data. I can take a value from any kind of the datapoint and put it to the RRD. Most popular for me are SNMP and COMMAND. Also, there are some types of datasources from ZenPacks (e.g. MYSQL or VMWare), or you can write your own. COMMAND datasourse uses any executable with any parameters and parse it's output for the variables. It is very flexible as for me, so I can provide monitoring for any kind of the device that has any kind of data available from network. How many time I've wrote a word "any"? :)

Reports. ... any ... any ... all ... any. I feel no need to describe it in details, you can find it in the documentation. Really useful thing for me is the dynamic Graph reports.

Support. Of course, there is no guarantied support for the Core (free) version. But there is quite useful documentation, great videos and quite responsive community on forums.

... At that moment I have realized, that it will take too long for me to describe all "pluses", so I just list some of them:

• Very advanced and flexible Event Management
• Flexible Monitoring Templates management (and templates itself is a big plus also)
• Customizable Dashboard
• Big set of ZenPacks
• Distributed Collectors
• SNMP traps, syslog, Windows eventlog receivers
• Processes, IP-services and Windows-services monitoring "from the box"
• Nice Interface (from version 3.*)
• Agent-less
• Easy to deploy
• etc. (many other standard things like backups management, event DB maintenance etc.)

And now some disadvantages:

• Interface (version 2.*). Although it was significantly changed in version 3.0, this needs to be described, because it was one of my biggest problems with this project. When I was investigating Zenoss, I get used to the interface and it seemed OK to me. And it was described as one of the advantages, because I was able to see what I need to see from Zenoss, forgetting that the information that Zenoss provides is different from what people used to see from Nagios. In fact Nagios provides information about the state of the monitored devices&amp;components, but in Zenoss, the source for the state is an event. I have understood it very quickly, but I didn't describe it properly to a team and it was confusing to them, because instead of the state "OK" they should look for some events in the completely different Interface.
• Performance. Well, Zenoss is quite greedy for the resources. The "bottle neck" for me is a MySQL DB performance on SELECT operations. It produces extra high I/O when getting some data from the Events History.
• Complexity. Zenoss became not just a Monitoring System, as I planned from the very beginning, but it is a real IT Management System now with all those features and procedures developed and configured. And it requires a bit more than a superficial knowledge of the interface. I never wrote a code with an OOP language before. But to make Zenoss do what I want I had no other chose but to learn some Python.

There were few other disadvantages, but they became to a "must/good to have" list, and then most of them were solved.
Actually, next posts for Zenoss subject will describe how I solve some of those tasks.

Stay tuned! ;)

IBM Bootable Media Creator (BoMC) - "must have" in IBM-based environments

- Hi, John! I have a new task for you. We have received a "brand new server". But it was retained in the dealer's warehouse for 3 years, so please, could you go to our very-cold-server-room and make all firmwares upgrade?
- Of course Boss! It will take just 4 hours, while I try to connect something to boot from (since there is no CDROM or FDD inside), and then reboot server for seven times after each peace of FW upgrade. And, by the way, I need to check 5-dimensions compatibility matrix to download the proper upgrade... And that's it! Hope we will not need this very-expensive-emergency-on-site support to repair the server this time.

Sounds familiar? I remember those times, when you need to check the floppy-diskettes consistency before upgrade, because a single read-error can lead to an unrecoverable HW failure. Of course, most of the equipment manufacturers provides us with FW upgrades for any type of upgrades - bootable CD images, images for USB-drives, for most popular OSes, etc.. And, probably, most of IBM server maintainers are aware of the very useful tool - Bootable Media Creator (BoMC).

Well, there is no need for detailed description here, since it can be found on the official page. In my short free interpretation - You can easily deploy FW upgrades for IBM Server x and BladeCenter with images created by the BOMC tool.

Last year, when I have some unpredictable IBM server behavior, in about 90% of cases it was enough to create a fresh CD with the FW updates set, then boot once, and problem just disappears.
Hope, this help you.

How to preserve eth0 interface name for cloned or migrated CentOS or Ubuntu?

I'm sure that many people faced this nasty thing: after you clone or reconfigure a Virtual Machine, OS reconfigures the interface because it has another MAC. I will not say that this is bad behavior, but what can you do, if you need to preserve the eth0 name and configuration?

In CentOS it's very simple - all you need is just to remove a string like "HWADDR=00:11:22:aa:bb:cc" in file /etc/sysconfig/network-scripts/ifcfg-eth0. That's it! Now even after you clone this machine or in any other way change the ethernet card - config will not be changed. Of course, in some cases there is a risk to face an IP conflict, but you're good admin, right :)?

Well, I don't use Ubuntu for server often, so I didn't look for a persistent solution. But what I do time to time, is just removing all strings from file /etc/udev/rules.d/70-persistent-net.rules right before cloning. It makes Ubuntu to forget, that some ethX name is already assigned for some MAC address.

Preconfigured ESXi on USB flash drive

It was quite long after my last post. Too much work and almost no free time. However, I've learned few interesting things, and I'm going to share them. And the first one - about the ESXi deployment.

We have up to 20 ESX nodes in our offices worldwide. And sometimes it was a hard way to deploy them remotely, using local supporters help. Moreover, even in our datacenter in place where we're sitting, it's quite uncomfortable to provide some maintenance if you forgot to take warm closes.

I knew there is a possibility to install ESXi on a USB flash drive. But I was interested in pre-deployment. The way is happened to be very easy.
To deploy a clean installation, all you need, is just to:
a) download the ESXi iso, for example here.
b) extract a file, named imagedd.bz2
c) unbzip2 it :)
d) deploy it on your flash drive. I used dd under the Linux like following:

# dd if=imagedd of=/dev/sdb bs=1M

(please, check if your flash drive is under /dev/sd?. if you just use the string above, you can destroy the existing data! Use it only if you understand what it means and on your own risk!)

Done! Now you have a fresh bootable flash drive with ESXi unconfigured yet.
Next thing I did, is just connect it to some laptop (I had Lenovo G550) and boot up.You can use any PC or laptop with H/W Virtualization support. The only issue I had with this budget laptop using ESXi, is that the keyboard doesn't work. So I just have plugged an external one. Now, you can configure the network and make some configuration remotely, with vSphere client, that is not possible to do from the console.

The main problem we had, is that after you select several interfaces to be used for NIC teaming (or portchannel on the Cisco switch side), ESXi that by default it uses "Route based on the originating virtual port ID". And that doesn't work well with the Cisco default configuration, which requires to be set the "Route based on IP hash" (or src-dst-ip hash).

Important note, for those who faced that NIC teaming is not working correctly! In difference to ESX 4.0, which we had used before, in ESXi 4.1. we found, that it makes NIC teaming configuration of management interface not as a default (inherited from the virtual switch config), but configures it independently! This strange behavior cost us 2 hours before we found this. Because we checked many times the NIC Teaming configuration of vSwitch and had no idea that Management Network has it different.

Next I configured all DNS settings, all VM and VMkernel networks, NTP, etc... Then I closed vSphere client and made the final network configuration to be used in the datacenter. Shutdown.... Ready!

All I need after that, is just go to the server room (or send a flash drive to the remote office), plug flash drive to the server (In case of IBM x3650 M2 and M3 there is a special USB port an a RAID controller for that purpose), boot from USB (in case of the mentioned IBM servers - boot from HardDisk1), and select the network interfaces to be for the Management Network after ESXi boot.

That is the way I do the ESXi deployment now. If you have any suggestions - you're welcome.