• Home
• Archives
• Contact Me
• Links
• Who I am
• WPG2
• Humor

Using Groovy for monitoring via JMX

Where I work we had a desire to be able to monitor our application as well as various JVM stats. Currently we use a groovy script to scan the logs every 15 minutes and report on errors if a certain threshold is passed. In our case it is currently 100 errors in that time frame. We also have a heartbeat monitor set up for the servers and we can see the load, session counts, etc. via a customized version of linux’s top.

However, we wanted to be able to see more information. Things like blocked threads, connection pool reaching saturation, spikes in thread creation, etc. Naturally JMX is the appropriate mechanism to achieve all of that, but we wanted a means to automate the process. To script it.

Enter Groovy

I had written a few groovy scripts already, such as the log scanner, so I figured why not use it for JMX stuff as well. While groovy has GroovyMBean object, most of the work was done using the basic JMX api, which wasn’t too bad once you got the hang of it.

The Basics

The first thing to do of course is get a connection to the JMX server. In our case we use an array of URLs to connect to and pull the stats for each in turn.

Here is the connection part:

import javax.management.ObjectName
import javax.management.remote.JMXConnectorFactory as JmxFactory
import javax.management.remote.JMXServiceURL as JmxUrl
import javax.management.MBeanServerConnection
import javax.management.Query

def serverUrls = ['array of connection urls here']

serverUrls.each { serverUrl->
def matcher = serverUrl =~ /.*(web4\d).*/

def box = matcher[0][1]
  new File(box + '_stats.txt').withWriter { file->
    def server = JmxFactory.connect(new JmxUrl(serverUrl)).mBeanServerConnection

Notice the regex and File object. In our case I’m taking the info and dumping it to a text file, one for each server, and another process reads that file and processes it.

Monitoring Tomcat

The first thing we wanted to do was look at tomcat itself. Namely how many threads are waiting, blocked and running. If there are any blocked threads, then we dump out the thread info for it. So first let us get the Tomcat JMX object:

    query = new ObjectName('Catalina:type=Manager,path=/,host=XYZ')
    def attr = server.getAttribute(query,"activeSessions")

Naturally you would want to replace “XYZ” for your actual host name. From there, we can start extracting some information:

file.writeLine '#'
    file.writeLine '# Tomcat info'
    file.writeLine '#'
    file.writeLine('activeSessions=' + attr)
    file.writeLine('expiredSessions=' + server.getAttribute(query,"expiredSessions") )
    file.writeLine('maxActiveSessions=' + server.getAttribute(query,"maxActiveSessions") )

That takes care of some session stuff, now we dive into the threads. First we get a hold of the Threading JMX object and set up some variables.

def threadingObject = new ObjectName('java.lang:type=Threading')
def threadIds = server.getAttribute( threadingObject, 'AllThreadIds' )

def threadCount = 0
def blockedThreads = new ArrayList()
def waitingThreads = 0
def runningThreads = 0
def timedWaiting = 0

Now we loop through all of the thread ids and look specifically for the HTTP threads, which have a name like “http-8080-exec” or “http-8443-exec”:

def sig = ['long']
    threadIds.each { id->
      def params = new Object[1]
      params[0] = id.toLong()
      def thread =
        server.invoke( threadingObject, 'getThreadInfo',params, sig.toArray( new String[1]) )

      if( thread.get('threadName').matches("http-\\d\\d\\d\\d-e.*") ) {
        threadCount++
        if( thread.get('threadState') == 'TIMED_WAITING' )
          timedWaiting++
        else if( thread.get('threadState') == 'BLOCKED' )
          blockedThreads.add( thread.get('threadName') )
        else if( thread.get('threadState') == 'RUNNABLE' )
          runningThreads++
        else if( thread.get('threadState') =='WAITING' )
          waitingThreads++
      }
    }

Now we write out that info and if there were any blocked threads, we dump out the thread name. From that we can hook into the JVM via JConsole and examine the thread, look for deadlocks, etc. Of course we could probably dump out some of that stuff here too.

file.writeLine "HTTP Threads=" + threadCount
    file.writeLine "Waiting Threads=" + waitingThreads
    file.writeLine "Timed Waiting Threads=" + timedWaiting
    file.writeLine "Running threads=" + runningThreads
    file.writeLine "Blocked Thread Count=" + blockedThreads.size()

    if( blockedThreads.size() > 0 ) {
       file.write( "Blocked Threads=" )
       blockedThreads.each{ t->
         file.write t+','
       }
       file.writeLine ''
    }

Here we are looking at system level stuff such as the uptime:

//
// System stuff
//
file.writeLine '#'
file.writeLine '# System properties'
file.writeLine '#  - uptime is in milliseconds'
file.writeLine '# - ' + (server.getAttribute( new ObjectName('java.lang:type=Runtime'), 'Uptime' )/1000/60/60) + " hours"
file.writeLine 'cpuTime=' + server.getAttribute( new ObjectName('java.lang:type=OperatingSystem'), 'ProcessCpuTime' )
file.writeLine 'uptime=' + server.getAttribute( new ObjectName('java.lang:type=Runtime'), 'Uptime' )

Summary

As you can see, getting info via JMX is fairly easy and straightforward. There is other stuff we look at, such as our connection pool info. We use c3p0 for out pooling and it has a JMX component making that fairly easy as well.

As for processing time, it takes about 7 seconds or so to connect to our 5 servers and write out all of this data, so it is pretty quick. If you ran this locally on the machines it might be a second or so quicker since we are connecting remotely.

For the future, we are planning to add some JMX stuff to our actual application so we can monitor that as well. On the list are cache hits, requests per second, etc.

All together, it took me about 45 minutes to write this script and most of that was because I wasn’t familiar with the JMX api at all. After that I used JConsole to get the actual object names and property names, which was a big help.

Don’t miss anything, subscribe!

Did you enjoy this post? Why not leave a comment below and continue the conversation, or subscribe to my feed and get articles like this delivered automatically to your feed reader.

Copyright Thinking Outloud | Theme by DailyBlogTips