New and Used Mac Minis available on MacMini.com

Where I work we had a desire to be able to monitor our application as well as various JVM stats. Currently we use a groovy script to scan the logs every 15 minutes and report on errors if a certain threshold is passed. In our case it is currently 100 errors in that time frame. We also have a heartbeat monitor set up for the servers and we can see the load, session counts, etc. via a customized version of linux’s top.

However, we wanted to be able to see more information. Things like blocked threads, connection pool reaching saturation, spikes in thread creation, etc. Naturally JMX is the appropriate mechanism to achieve all of that, but we wanted a means to automate the process. To script it.

Enter Groovy

I had written a few groovy scripts already, such as the log scanner, so I figured why not use it for JMX stuff as well. While groovy has GroovyMBean object, most of the work was done using the basic JMX api, which wasn’t too bad once you got the hang of it.

The Basics

The first thing to do of course is get a connection to the JMX server. In our case we use an array of URLs to connect to and pull the stats for each in turn.

Here is the connection part:

import javax.management.ObjectName
import javax.management.remote.JMXConnectorFactory as JmxFactory
import javax.management.remote.JMXServiceURL as JmxUrl
import javax.management.MBeanServerConnection
import javax.management.Query

def box = matcher[0][1]
new File(box + ‘_stats.txt’).withWriter { file->
def server = JmxFactory.connect(new JmxUrl(serverUrl)).mBeanServerConnection


Notice the regex and File object. In our case I’m taking the info and dumping it to a text file, one for each server, and another process reads that file and processes it.

Monitoring Tomcat

The first thing we wanted to do was look at tomcat itself. Namely how many threads are waiting, blocked and running. If there are any blocked threads, then we dump out the thread info for it. So first let us get the Tomcat JMX object:
code>
query = new ObjectName(’Catalina:type=Manager,path=/,host=XYZ’)
def attr = server.getAttribute(query,”activeSessions”)

Naturally you would want to replace “XYZ” for your actual host name. From there, we can start extracting some information:


file.writeLine '#'
file.writeLine '# Tomcat info'
file.writeLine '#'
file.writeLine('activeSessions=' + attr)
file.writeLine('expiredSessions=' + server.getAttribute(query,"expiredSessions") )
file.writeLine('maxActiveSessions=' + server.getAttribute(query,"maxActiveSessions") )


That takes care of some session stuff, now we dive into the threads. First we get a hold of the Threading JMX object and set up some variables.

def threadingObject = new ObjectName('java.lang:type=Threading')
def threadIds = server.getAttribute( threadingObject, 'AllThreadIds' )

def threadCount = 0
def blockedThreads = new ArrayList()
def waitingThreads = 0
def runningThreads = 0
def timedWaiting = 0

Now we loop through all of the thread ids and look specifically for the HTTP threads, which have a name like “http-8080-exec” or “http-8443-exec”:

def sig = ['long']
    threadIds.each { id->
      def params = new Object[1]
      params[0] = id.toLong()
      def thread =
        server.invoke( threadingObject, 'getThreadInfo',params, sig.toArray( new String[1]) )

      if( thread.get('threadName').matches("http-\\d\\d\\d\\d-e.*") ) {
        threadCount++
        if( thread.get('threadState') == 'TIMED_WAITING' )
          timedWaiting++
        else if( thread.get('threadState') == 'BLOCKED' )
          blockedThreads.add( thread.get('threadName') )
        else if( thread.get('threadState') == 'RUNNABLE' )
          runningThreads++
        else if( thread.get('threadState') =='WAITING' )
          waitingThreads++
      }
    }

Now we write out that info and if there were any blocked threads, we dump out the thread name. From that we can hook into the JVM via JConsole and examine the thread, look for deadlocks, etc. Of course we could probably dump out some of that stuff here too.

file.writeLine "HTTP Threads=" + threadCount
    file.writeLine "Waiting Threads=" + waitingThreads
    file.writeLine "Timed Waiting Threads=" + timedWaiting
    file.writeLine "Running threads=" + runningThreads
    file.writeLine "Blocked Thread Count=" + blockedThreads.size()

    if( blockedThreads.size() > 0 ) {
       file.write( "Blocked Threads=" )
       blockedThreads.each{ t->
         file.write t+','
       }
       file.writeLine ''
    }

Here we are looking at system level stuff such as the uptime:

//
// System stuff
//
file.writeLine '#'
file.writeLine '# System properties'
file.writeLine '#  - uptime is in milliseconds'
file.writeLine '# - ' + (server.getAttribute( new ObjectName('java.lang:type=Runtime'), 'Uptime' )/1000/60/60) + " hours"
file.writeLine 'cpuTime=' + server.getAttribute( new ObjectName('java.lang:type=OperatingSystem'), 'ProcessCpuTime' )
file.writeLine 'uptime=' + server.getAttribute( new ObjectName('java.lang:type=Runtime'), 'Uptime' )

Summary

As you can see, getting info via JMX is fairly easy and straightforward. There is other stuff we look at, such as our connection pool info. We use c3p0 for out pooling and it has a JMX component making that fairly easy as well.

As for processing time, it takes about 7 seconds or so to connect to our 5 servers and write out all of this data, so it is pretty quick. If you ran this locally on the machines it might be a second or so quicker since we are connecting remotely.

For the future, we are planning to add some JMX stuff to our actual application so we can monitor that as well. On the list are cache hits, requests per second, etc.

All together, it took me about 45 minutes to write this script and most of that was because I wasn’t familiar with the JMX api at all. After that I used JConsole to get the actual object names and property names, which was a big help.

I’m a user of Google’s personal home page, and it just went through a big update. The tabs on the top are now on the left and even nicer is that gmail is now integrated in, so you no longer have to go to gmail to open mail and reply to them.

Very nice.

I’ve been working on FiBiRi.com on and off for about two months and I finally got it live. It is still kind of in alpha stage, but the functionality is largely there. This is my first serious application written using Grails, and I must say I’m hooked.

Of course I’ve dabbled a bit here and there, but there is nothing like a real app to learn a new framework with. I had some minor pains but nothing too bad. All told I have about 120 to 160 hours in it which isn’t too shabby. A good portion of that time was learning grails and groovy so if I had to do again I could probably shave a good 20 or so hours off of that development time.

Things to do

There is still a lot content left to add along with the monetization stuff. I still have a few functionality items to add, mostly to do with the itineraries but I’m pretty happy with it. There is actually almost as many hours in gathering the content as there is in development time. Needless to say that ratio will only get wider as more content is added. It is also only focused on Las Vegas, but is set up so that any city or destination can be added so there is room to grow.

Technical details

There are 23 controllers, 18 domain classes, two services and two custom tag libraries along with 60 or so view pages (mostly admin). It is running in tomcat 6 behind nginx on a small Linode.com box with MySQL as the database.

I could have run it straight on tomcat, but chose to front it with nginx for a couple of reasons. One because I wanted to try nginx (I love it…) and two because I can later do some fancy caching behind it using memcached.

If anyone has any suggestions, feel free to let me know!

It just plain looks really, really bad when you have a public facing site and you don’t trap errors. Here is from Staples.com today (1pm PST):

There has been a lot of buzz about Amazon’s SimpleDB and other related projects like CouchDB, Google’s Datastore and Thrudb. Personally I think they will all find a niche in modern web applications, but what I find amusing are the nay-sayers that talk about the faults and even saying they aren’t even databases.

Come on, not a database? What are these people smoking? Since when does database == relational ? So they don’t support all transactions, aggregate functions or other features that are common to RDBMs. Did it ever occur to these people that not all applications need these features?

Scalability

The prominent feature for these document style databases is scalability. One guy equates scalability to the number of rows in the database. Uhh, maybe someone should explain to him what scalability means. Who cares if you can put millions of rows in your db if you only have 100 users? What about say a few hundred thousand rows but millions of users? It isn’t necessarily the number of rows you can store but how quickly you can get data out to your users.

Where are my joins?

Others go on and on about how you can’t do things like join data across tables. There are many big sites that actually store data in multiple places. Yep, denormalizing (gasp!) their data in the name of speed. Sharding also fits into this category. Although generally sharding happens on a traditional RDBMS, the effect is the same, spreading data out for scalability sake.

It all comes down to the right tool for the job. Until recently RDMBS were pretty much the only tool in the toolbox, whether you needed all the features or not. Now we have more options to choose from and I’ll bet that many choose the alternate route. All the nay-sayers are just scared.

So I read today where LinkedIN uses ActiveMQ as part of their architecture. I haven’t used ActiveMQ personally, but we use it for a project where I work and the guy that set it up absolutely hates it now.

We were looking to use a message server for a new project and started kicking around ideas. The primary requirement was multiple language support. New stuff is largely being done in Java but we also have a really large perl codebase and some backend stuff is still be done in perl. Naturally that made ActiveMQ a front runner because of its open language support. Once he got wind of the talks it was quickly shelved with “over my dead body”.

Turns out that despite the great documentation and its visibility, it doesn’t work quite as speedy as one would think. After fixing several bugs himself (out of order messages being one that I recall), the one glaring problem is performance. Seems that if a subscriber is a bit slow, the whole server becomes slow even on the receiving end. Well, that is just counter productive in my book. The whole idea of a message system is to eliminate these kinds of bottlenecks yes?

We then tried openAMQ (a C-based server) and that shelved too because it kept dropping messages. For the time being we settled on Websphere MQ which just makes me want to puke. I know IBM has had a message server forever, but I have yet to see a piece of software they’ve written that was worth a shit.

Hopefully we can find another solution before big blue takes a firm hold on us. Any suggestions?

Any really good experiences with ActiveMQ?

I’ve written a little bit about grails before, but until now I haven’t had a real application to build to really put it through it’s paces. At first I was going to build this application with Rails for the same reason, even deploy it on Glassfish with its RoR support. I don’t know though, something about an application that requires a new instance for every request just seems wrong to me. Seems like such a flawed architecture that I just couldn’t bring myself to do it. Yea, there are other Ruby based web frameworks and I might try one at some time, but not this time.

I also have to say that I generally don’t generally like any kind of black-box magic frameworks. Code generation and such tend to make me cringe, but what the hell I was going to give it a shot. Needless to say so far I’m pretty darn impressed. While I’m not quite finished it is moving along very rapidly, and adding features is pretty damn easy. I would say I’ve put in about 50 hours or so over three weeks and I am at about 85% finished.

What I like

This is ironic part. I really hate tools like Hibernate that force you into a specific format. Most of this stems from my corporate application experience where databases are far more complex than what ORMs like Hibernate can handle. However in this case the GORM has been a wonderful time saver. Setting up relationships has been fairly easy, although I had a few head-scratching moments that weren’t quite textbook and the docs weren’t very clear on.

I also really like the URL mapping stuff. I’m sure Rails has a good URL mapping engine as well, but I really like being able to customize these as I see fit. I still get a bit confused on what return/render method to use in the controllers, but it hasn’t been too bad.

What I don’t like

About my only gripe so far is with the BootStrap class. This is a great idea and wonderful for development, but man it can be a pain. Actually the gripe isn’t so much as the BootStrap itself but with GORM. Go figure.

There are cases where I have a three level deep object graph and populating it isn’t as straightforward as it could have been. Most of my problems have been when I populate the leaf object and then I need to get that leaf object later for some other reason. The getBy and findBy methods will sometime return a null object, even when I know the parameters are good. I end up having to retrieve its parent and then getting the child out of there. I would say I’ve spent about 20%-25% of my time just trying to get my simple test data in. I could very well be doing something wrong, but the docs haven’t shown me anything different and what seems intuitive to me apparently isn’t. Oh well.

I haven’t been doing anything nearly as fancy as Glen Smith but isn’t a “hello world” app either. My current stats are:

16 controllers with 864 LOC
12 domain classes with 245 LOC
1 taglib with 22 LOC

I know I’m going to have to write at least one new tag lib for paging, as the provided paging taglib won’t work for me. A shame because I really need it. I should be done with domain classes but I’ll probably add a couple more controllers.

All in all I’m pretty happy with Grails and I look forward to getting this thing public.

Get the latest celebrity news from around the web here!

In my last post I wrote about some strange threading behavior in tomcat 6. After a couple weeks of struggling between iBatis, our app, our connection pool and even trying the NIO connector we finally narrowed it down to the connection pool.

For some unknown reason, DBCP wasn’t taking the connections back into the pool. The result was the thread that was using them would hang and eventually tomcat would hang. How on earth DBCP has lived as long as it has with this problem is beyond me. We tried everything, even manually pulling the connection out of iBatis and closing it. Nothing worked. I could get DBCP to lock up with a connection pool of 150 and a user load of 50. Go figure.

c3p0 to the rescue

Thankfully there is another popular connection pool out there and it is c3p0. We swapped it in nice and easy (thanks Spring!) and presto, all our troubles went away. I could even overload the app (more concurrent users than connections available) and it wouldn’t die. It got slow, but it wouldn’t die.

Thanks to my finding, we swapped out DBCP as the pool that ActiveMQ was using and solved a few problems there as well.

Now our bottleneck is the communication between the application and our search engine, SOLR. That is to be expected somewhat as it opens an HTTP connection. We use SOLR for more than search however, it is also our navigation engine by means of faceting so needless to say we make a lot of calls to it. A little caching will solve that problem though and should be fairly easy to implement.

What are the lessons learned?

Don’t think any piece of the puzzle is immune to being the problem. My initial thoughts were that since DBCP has been around so long it couldn’t possibly be the problem. I spent most of my time in iBatis since I was new to it and thought maybe we were just doing something wrong.

In the end we were able to run load tests that were several times the capacity that our old (current) site sees. A good sign indeed.

Two tools helped more than anything else. JMeter for our load testing and JConsole, the built in JMX application for monitoring. With JConsole it was easy to see what threads were being locked, what they were waiting on and how many threads were running. Very, very handy indeed.

$5 a month Ruby on Rails hosting. No kidding! Unlimited domains, 7.5 TB of bandwidth

We are having some threading issues while load testing our production tomcat servers. When I fire up JMeter and hit it with about 50 users, tomcat grabs a ton of threads. Eventually this causes some instability and tomcat hangs.

The weird thing is I can’t duplicate it locally. I hit my local running instance and it might grab a few more threads which is to be expected, but not skyrocket up. At first I thought it was a problem with iBatis which had a known threading issue. But after upgrading to the latest beta it hasn’t solved anything. I also switched to using the session API in iBatis as well to no avail.

My gut tells me it isn’t directly tomcat’s fault since I can’t reproduce it and I don’t want to give up on tomcat just yet. However this problem is really keeping us from ramping up traffic on our beta site which of course doesn’t make the brass too thrilled.

In both cases tomcat is configured to use the APR connector and maxThreads are set at about 150. The only real difference in the configs is that production is in a cluster and is using large memory page configurations. Our production boxes have 16gb of RAM and 8 cores, so it the hardware has a lot of room to go.

Anyone have any thoughts about what is causing this?

$5 a month Ruby on Rails hosting. No kidding! Unlimited domains, 7.5 TB of bandwidth