vCenter Operations Manager – Monitoring vAPP Health Status

Overview

Need to monitor a vCOPS vAPP’s general Health so you are aware of problems with vCOPS itself?  Well, hopfully this post will help.

There are a number of options for monitoring vCOPS, self-monitoring, using vCenter or using an external monitoring system (like Xymon or Nagios).

Self Monitoring Option

I’m not a fan of the self-monitoring option as it just doesn’t give you the peace of mind that monitoring from an external system will give you, however, its pretty simple to setup and should alert you to a lot of issues with its internal operation.  There is a VMware KB on this subject.

Pre-requisites

You need to configure SMTP before you can create the Notification Rules.  If you haven’t already got this setup then login to the Admin UI (https://myserver/admin) and click the SMTP/SNMP tab, fill in the details and click Update.

Configure

Now login to the main enterprise UI (https://myserver/vcops-vsphere) and click on Notifications then add the rule as per the below example.

vCenter Monitoring Option

My Analytics VM generally runs with around 75% Memory usage and around 35% CPU Usage.  When I have had issues that has caused the Collector services to stop then Memory and CPU usually drops off a lot.  So in theory, if the usage drops way below these values for a significant period of time then there is a chance the collection and/or DB has stopped working.  All you would need to do is adjust your alerting based on how your Analytics VM performs.

I created 2 alerts, a CPU check and a RAM check.

Login to the vCenter client and find the Analytics VM and click the Alarms tab and then the Definitions view.  Right-click and create a new alarm.  Note: you will need to access this VM via the Hosts and Clusters view to select it directly.

Click the Triggers tab and set the conditions.

Now click the Actions tab and fill in the details.  I only want it to alert every hour for a Critical (so I actually take notice :)).

External Monitoring

The above options are OK, but not great.  Ideally you should be able to alert from a 3rd party system like Xymon or Nagios.  Unfortunately the vCOPS vAPP consists of 2 SUSE Linux Enterprise 11 VMs that have been installed with only minimum packages.  Also (at the time of writing) Xymon and Nagios do not have custom built rpms that you can just drop into the vAPPs.

So we had to build our own using source files which was a bit of a pain as the vCOPS vAPPs are minimal builds with no repositories enabled (probably to stop people like me messing with it too much :)).  The other thing we did was build these binaries on the vCOPS VMs running in our Lab, which we then copied over to the vCOPS VMs running in Production to try and limit the amount of changes we were making to the live machines.  I also made sure that the live vCOPs VMs were properly backed up before implementing this.

Also, as we are using Xymon, the rest of this article will focus on the Xymon client build.  I expect/hope a similar process would work for the Nagios client too.

Warning

Using the following instructions may well lead to an invalidation of your support agreement for vCOPs!  Use at your own discretion.

Pre-Requisites

First you will need to download the latest Xymon tarball from the SourceForge Xymon page and copy it to the UI VM.  Once its up the you need to install the following packages (gcc, gcc-c++ and make).

You will also need to configure the vAPP VM with the Suse Repo using Yast.  An up-to-date list can be found here:-

http://en.opensuse.org/Package_repositories#Oss

Repo setup (using Yast)

For a good explanation of Yast and SUSE repos take a look here.

http://en.opensuse.org/Portal:YaST

http://en.opensuse.org/YaST_Software_Repositories

Start Yast by running the yast command from the command line of the VM, you will be presented with the Yast menu.  Now tab over to Software Repositories and hit enter.

Tab down to Add and hit enter.

Choose Specifiy URL and hit enter.

Give the Repo a name and enter the repo url you have selected (make sure you choose the one that corresponds with the version of the SLES you are running, in this case it is 11.4).

Now tab to Import and hit enter.

Yast will now proceed to add the repo cache files.

Now that the Yast repo is configured you can install the packages required to build the Xymon client.  We’re using the zypper command to do this:-

Create a local user for the Xymon service (auto create the /home/xymon directory):-

Now unzip the tarball, then configure and install Xymon (client mode):-

Now run make:-

run make install:-

Now start the service to make sure it works:-

Install to Production

Now the binaries are built I just scp’d the /home/xymon directory to the Production UI and Analytics VMs.  Next thing to do is create a xymon user with the right directory and then set the correct permissions:-

As the UI and Analytics VMs may have incorrect names for your Xymon install (the Analytics VM is just called localhost), then it’s best to change the names.  This is best done by editting the runclient.sh script.

Now change the MACHINEDOTS variable from uname -n to something more sensible.  At this point you can run the Xymon client:-

Now run make the service start on boot:-

At this point Xymon was able to report on CPU, Memory, Disk Usage and Service status from an in-guest point of view.

I now have Xymon configured to verfiy that the following services are running:-

UI VM

  • OpenVPN
  • Postgres
  • java service (grepping for DALIVE_HOME=/usr/lib/vmware-vcops/tomcat-enterprise/webapps/vcops-custom)
          • 2 x java service (grepping for DALIVE_HOME=/usr/lib/vmware-vcops/tomcat/webapps/vcops-vsphere)

Analytics VM

  • OpenVPN
  • Postgres
  • java service (grepping for com.integrien.alive.collector.CollectorMain)
  • java service (grepping for com.integrien.analytics.AnalyticsMain)
  • java service (grepping for org.apache.activemq.console.Main)

Leave a Reply

Your email address will not be published. Required fields are marked *