Help Developing Zenpacks

Creating your first Zenpack? Me too.

I am finding these resources very helpful:

  1. A Zenoss Labs webcast on developing Zenpacks. It is in multiple parts on Youtube:

    They are pretty dry, so have coffee on hand. You can always pause if you need a break.

  2. The accompanying documentation at
  3. The Extending Zenoss wiki article links to lots of good info.
  4. Here’s an article on using git for version control of your zenpack.

On improvement and IT Operational Excellence

Way back in the early ’90s, back before I had an IT career, I did a temp job as a telephonist in a call center at a bank. My job was to answer calls for a promotional phone line that nobody called. I ended up doing a lot of reading and a lot of watching.

Telephone banking was pretty new back then and this particular bank was regarded by the industry as the best. As a result of this, they had many visits from folks wanting to know how they did it. They would do tours and answer lots of questions. They seemed very forthcoming. In fact, to my colleague and I, they seemed a little too forthcoming.

One day the call center manager had a quiet few moments, so she came to talk to us and see how we were doing. My colleague asked about the tours, and asked why she was being so nice and answering all their questions. She smiled and agreed with us that there is a secret to success in this venture, that they do share a lot of information with their competitors, but not the stuff that makes a difference. Obviously the key to their success was not technology – they all had the same stuff. It was operations. Just like now, operations is the differentiator but back then it was a trade secret.

Now lets come back to today and talk about IT operations. The world has changed. No more is hanging on to knowledge considered a key to cornering the market. This means that modern Internet companies are much more willing to share their ideas. There is lots of stuff out there about how the big successful shops do stuff, they share their tools, they talk about their operational practices. It is great stuff. I soak it all up, have learned a ton and I am truly thankful to them for sharing. Here’s the thing though, very often we actually miss the real secret. It worked for them because THEY FIGURED IT OUT FOR THEMSELVES.

So we need to work out for ourselves what works and what doesn’t. It’s only right and proper. We can look at what the big boys do, but we must constantly remind ourselves that what worked for them probably is not necessarily going to work for you. We understand that already. We call it a cargo cult.

So, what do we do? Here is my suggestion:

1. Understand the purpose of your organization. Your company is there for a reason. What is it? This is more than the what and the how, this is also (and especially) the why. You need to fully internalize the purpose of your company, and your division, and your team.

2. Understand what you do. How does what you do add value to the business? How do you help other teams add value? How do they help you? What works really well? Why? What doesn’t? Why not?

3. What small thing can you do that makes the company fulfill its purpose better? That does not necessarily mean you have to do all your stuff better. In some cases that may be true, in other cases it may be better to take a hit so another team can do better. If the organization is better as a result then it is a win.

4. How do you know if you are making progress? Well, by measuring something that roughly approximates the purpose of the organization. If this measure goes up then keep doing what you are doing. If it goes down then stop.

Go back to step 2 and repeat.

This is not new stuff. These ideas have been around for a long time. I claim no credit here. I distilled this from a long list of people and practices. I suspect that all the management fads of the last few decades where developed this way. Once that company gained success, instead of others doing the steps above, they just took the result and then we get stuff like Management By Objective.

On Drum Rugs

So, I am packing up for a gig tomorrow in Dickinson. I will be playing for Jessie Veeder and Frog Holler String Band at ‘Blues and Brews’. I have packed away all my stuff in to cases and I am down to my drum rug. I thought I might take this moment to talk a little about drum rugs as I think they are very important and often overlooked.

First, you really should have a drum rug. There are lots of reason why – here’s a few:

  • Very often you will not be performing on a carpeted floor.
  • You can use it to claim the space you need to set up – especially when space is a premium.
  • Even at home, playing on a rug saves the nice carpet underneath from strain from the bass drum spikes, etc.
  • All the wood chips from your drumsticks go on your rug and not in the nice carpet.
  • If you need to move the kit a foot or two then you can drag the rug and save some effort.

When it comes to getting a drum rug, you can get specially made rugs from the music store. The ones I have seen are pretty flimsy  – no better than a thick piece of cloth – and expensive. I have always just gone to my local hardware store and got a $20 rug, or off-cut of carpet. Even if it is a cheap rug, it is still better than the one you get at the music store, and remember all the wood chips and spikes in your bass drum? Remember all the drinks that get spilled on stage, and how about the state of that floor in that dive bar you play in once a month? Yeah! The cheapo rug from the hardware store is just fine.

Lastly, here are a few other random tips I have learned from bitter experience:

  • ALWAYS make sure the drum stool fits on the rug. Having your stool slip away from the rest of the kit is at least off-putting, and can be painful – as I found out once when I fell off the back of a stage mid song.
  • I have two sizes of tug. One is a 4 feet by 6 feet for small kits and small stages. The other is 5 by 7.
  • If you like your setup to be exactly the same each time you set it up – especially if’s complicated, then consider taping out where each foot goes. That plus memory locks should get you pretty close each time.


The picture above shows what I have marked out on my rug. You will notice the snare stand and stool are not there. I decided against it because they are always moving around, mostly because I have never come to a firm decision where the best place for them is. You will also notice that they are not centered on the rug. I noticed that too and might fix that one day.

Close Mic’ed Drums – The Obvious Lessons

The other big part of my life (family and computers being the other two) is music; specifically drums and percussion. Up until now, this blog has focused almost entirely on computers and IT, I am going to include more drums now.

drums from above

A few weeks ago, I got a set of microphones and all the various accouterments. I am finding more and more that bits, or all of the drum kit is, or should have been mic’ed up when playing live, so it seemed like a good investment for both gigging, and for just learning the ins and outs of using microphones with drums. I have read and heard from many sources that getting drums to sound good with microphones is a non-trivial task. Getting drums to sound good on their own is hard, I thought, adding mic’s couldn’t be much harder. Well, it certainly adds an extra level.

I am  still getting to grips with all the new aspects, like mic placement, using a mixer, and even stuff like stands and cables all over the place. Here is a list of things I noticed right away after the initial setup:

floot tom microphone

  1. Microphone placement is HUGELY important. Just a small angle change is the difference between a boomy, ringy drum and a crisp sharp sound.
  2. When close-mic’ing, a few millimeters closer to a drum can make a huge difference to the amount of gain, and the tone you get.
  3. When played acoustic only (i.e. no microphones) then if a drum is slightly out of tune, only the well trained ear will notice. You close mic it and everyone notices.

You are probably reading this and nodding your head sagely and saying to yourself that this is common sense and that you could have told me that for nothing. Here’s the thing, you need to learn this the hard way. I have heard these things many times too, and I was fully expecting them. That still doesn’t stop you from being a little shocked at just how apparent these things are when you experience if for yourself. Up until it bites you, it is just an intellectual thing. Once you experience it, and learn how to deal with it then it becomes part of your craft.

The next stage is then learning how to set levels, tone and mix your drums. I’m not going to give any tips on that yet – I still don’t know what I am doing. I will give some general ideas that I am finding are helping me a lot.

  • Give yourself lots of time. Do not expect to get a new set of microphones, and then use them in a gig that night and sound awesome. Set them up in your practice room and play with all the things.
  • Try to isolate  yourself from the ambient noise and listen just to the microphones. I have a pair of IEM;s and a set of isolation headphones – both of which have a 20-30 dB ambient noise drop. I have found plugging the IEM’s in to the desk and wearing the headphones on top of the IEM’s give sufficient isolation to hear just what the microphones are picking up.
  • Play, twiddle, play, twiddle, and repeat till your drums sound how you like them. The twiddling will probably include a bit of changing mic positions, tuning, and changing settings on the mixer.

Tivoli Directory Server – Performance monitoring with Zenoss

If you are trying to collect performance data from IBM’s Tivoli Directory Server, and you do not have Tivoli Directory Integrator installed, then you can still monitor some performance metrics with Zenoss.

The big Aha! moment for me was when I read that you can query some useful metrics with an LDAP client.

ldapsearch -h $LDAP_HOST -x -D "$LDAP_ADMIN_DN" -w $LDAP_ADMIN_PASSWORD -s base -b cn=monitor objectclass=*
# extended LDIF
# LDAPv3
# base <cn=monitor> with scope baseObject
# filter: objectclass=*
# requesting: ALL
version: IBM Tivoli Directory (SSL), 6.1
totalconnections: 50912
total_ssl_connections: 0
total_tls_connections: 0
currentconnections: 67
maxconnections: 1024
writewaiters: 0
readwaiters: 0
opsinitiated: 712582
livethreads: 1
opscompleted: 712581
entriessent: 620628
searchesrequested: 585265
searchescompleted: 585264
bindsrequested: 50917
bindscompleted: 50917
unbindsrequested: 50791

So we no longer need to use IBM’s SNMP listener therefore saving some time and maybe even some money. Zenoss allows you to run scripts and  so long as the script returns stuff in the right format Zenoss can graph them. Here’s how I did it.

1. The script

First we need a script to go get the data from TDS. As shown above, it is really only a simple LDAP search, but the output need to conform to the Nagios plugin standard. So here is my script. Feel free to use and improve upon it.


# Robert Hart July 2010
# Script to collect performance metrics from IBM Tivoli Directory Server 6.1
# For reference:

# List of attributes to get.
# format: space seperated - attribute,UOM
ATTRIBUTES="bindsrequested,c currentconnections,"

# Stuff to tweak
LDAP_ADMIN_DN=<administrator distinguished name>
CMD="ldapsearch -h $LDAP_HOST -x -D "$LDAP_ADMIN_DN" -w $LDAP_ADMIN_PASSWORD -s base -b cn=monitor objectclass=*"
# if needed, set to a real file

# this parses the attribute value out of the ldpsearch output.
# usage: parse attribute,UOM
function parse
 echo "parsing $1" > $DEBUG
 ATT=`echo $1 | awk -F "," '{print$1}'`
 UOM=`echo $1 | awk -F "," '{print$2}'`
 KEY=`grep "^$ATT" $TMP_FILE | awk '{print $1}' | sed -e 's/://'`
 VALUE=`grep "^$ATT" $TMP_FILE | awk '{print $2}'`
 if [ -z $UOM ]

# Lets do some work

 echo "command ran" > $DEBUG
 exit 2

 if parse $ATTRIBUTE
 echo "parsed" > $DEBUG
 exit 1

echo "tds |$OUTPUT"
exit 0

Save this script as a file on the Zenoss server, make the zenoss user the owner and give it execute permissions.

You should be able to test the script and get a result:

$ ./
tds |currentconnections=69;;;; bindsrequested=50880[c];;;;

2. Set up the template in Zenoss

First I created an device class because we have a couple of LDAP servers, and so devices inheriting from the device class is the more efficient way to do this.

I also set the LDAP monitor zProperties and bound the LDAP monitor template so we could graph LDAP response times too.

In the templates tab for the device class, I pulled down the menu in the Available Performance Templates section and selected “Add Template…”. Once you have given it a name, then you end up in a page where you can add a data source. In the Data Sources section pull down the menu and select “Add Datasource…”. Give it a name, and set the source to COMMAND. Make sure you set the parser to Nagios, and make sure you pass the device name to your command, eg. /opt/zenoss/scripts/ ${devname}. Click Save, and then add DataPoints at the bottom.

When you create the DataPoints, remember to set the correct Type. Since most of the metrics in TDS zero themselves when you restart the server, then COUNTER is probably the most appropriate.

Once you have done that then you can go back to the template and add graph definitions. Then you can attach the appropriate data point to the graph.

3. See your Graph Loveliness

Lastly you need to bind your new template to the device class. You do that in the templates tab again. Remember to hold down the control key or you will deselect all the other templates in use here.

If you have not already done so, you can put a device in to that class and odel. You should start to see data for the metrics you are collecting.


Monitoring using zencommands

Nagios Plugin Output

And thanks to Dan for his help too.

More details on how to integrate Ubuntu 10.04 into a Windows Domain

I notice that my notes from a couple of days ago gets a lot of hits, so I  feel inclined to be a bit more detailed about how to integrate Ubuntu Lucid Lynx in to a windows domain.

Setting up Kerberos

This is worth doing regardless of whether or not you set up PAM. You can see why later.

1. Install the following packages:

sudo apt-get install krb5-config krb5-user

During the installation process, it will ask you for your realm. Enter the Realm for your Windows Domain (talk to your Active Directory administrator if you don’t know)

2. Edit /etc/krb5.conf

You will need to add a stanza for your realm in /etc/krb5.conf. something like this:

<REALM> = {
 kdc = <active directory server>
 admin_server = <active directory server>
 default_domain  = <>

3. Test

kinit <user>

The user should be a windows domain user. When challenged input the windows password for that account. Look for the ticket with the klist command.

Setting up PAM

If you want to sign in to your desktop / server using your Windows network credentials, then do follow these steps.

1. Prerequisite -This is your get out of jail free card

Set up the root account. If you mess this up and you need to fix it, then you need to be able sign in as root. I strongly recommend you do this so that you can.

sudo passwd root

and then test:

su -

Glad we got that done – lets move on.

2. Install the PAM module

sudo apt-get install libpam-krb5

Ubuntu sets up PAM for you, so that should be it.

3. Test

Try to log in to the computer with your windows credentials. There must be a local account already created and the user name and must be the same as the domain user name. It should just work.  Run klist and you should also see a ticket. Nice!



If you have kerborized web applications, or spnego enabled sites, then you can configure Firefox to use your kerberos ticket to negotiate for you and logging you in unchallenged. You need to type the following in to the location bar (preferably in another tab – you don;t want to lose this page just yet)


Say that you will be careful and make sure the following attributes are set to true:

  • network.automatic-ntlm-auth.allow-proxies
  • network.negotiate-auth.allow-proxies
  • network.negotiate-auth.using-native-gsslib

Set the following to your local DNS sub-domain. This defines the scope of the trust for sites it will try to negotiate with:

  • network.automatic-ntlm-auth.trusted-uris
  • network.negotiate-auth.trusted-uris

Next time you attach to such a web site, then you will get logged in.


If your company uses the Microsoft Office Communicator Suite for IM, then you too can join the conversaton with the Sipe plugin. Install it with this commend:

sudo apt-get install pidgin-sipe

Once you have restarted Pidgin, then you can add an account. Here are some guidelines:

  • Protocol: Office Communicator
  • Username: Exchange email address
  • Login: <DOMAIN>\<User>
  • Password: <domain password>
  • In the advanced tab, set the server to your IM server.

Kerberos seemingly works but the version that is here has not really worked for me. Give it a try by blanking out the password and checking the kerberos box and see how you do.

Mounting Windows File Shares

Once you have a kerberos ticket, then you can mount file shares without providing a user name or password. You can use the connect to server form in the Places menu. You set the service type to be windows server, set the server to the fully qualified domain name of the windows file server and enter the share name. You do not need to put a user name of domain in. Bookmark if you like. The share should open up in nautilus with no further prompting.


I tried the evolution-mapi plugin which implements the Exchange MAPI protocol. It works but I found it sluggish and still very buggy. I would wait a little longer for anything more serious than testing.

That’s all folks!

If you have any other tricks that I have not mentioned then let us know.

Some notes on Ubuntu 10.04

Ubuntu 10.04, the Lucid Lynx is now released to the wild. There is lots of stuff out there talking about all the cool stuff that is included. I am going to talk about some of the stuff that you probably won’t see in the reviews.


A fairly niche subject important to those in a Kerberos environment, or who want better integration in a Windows Domain. In Ubuntu 10.04, setting up a kerberos client just got a lot easier.

When you install the krb5-config package, it will ask you some questions abot the realm you are in, etc. It does not do everything, but it does most of the work. I still had to edit /etc/krb5.conf to add in the hostname of the KDC, etc.

Installing libpam-krb5 does the right things to configure PAM. You can start to use it straight away, and it just works. For me, it also creates a kerberos ticket for you, which I don’t think it did before. This, to me, is a big deal. It means that I can use firefox to go into kerborized, and spnego enabled web sites without having to manually create a ticket before-hand. Same with kerborized ssh servers, and pidgin-sipe.

I tried libpam-ccreds too and it also just worked. Again, no messing with PAM configurations.

Connecting Pidgin to Office Communication Services

If you are in a Windows Domain and need to IM with your colleagues who are hanging out on Office Communicator, then pidgin-sipe does the job very well. If you hover over a buddy icon, then you can see what is on their calendar now. Pidgin also then sets your status according to what is on your calendar, so if you are scheduled for a meeting then it will set your status to busy at that time.

Mounting Windows File Shares

Once you have your kerberos ticket, then you can mount cifs file shares in the domain by running:

gvfs-mount smb://server-fqnd/share

The share is then set up in GNOME and you are not challenged for credentials. That means that with libpam-krb5 and a login script, you can have all your Windows shares auto-magically mounted when you log in. Nice!