Strange tomcat threading behavior
We are having some threading issues while load testing our production tomcat servers. When I fire up JMeter and hit it with about 50 users, tomcat grabs a ton of threads. Eventually this causes some instability and tomcat hangs.
The weird thing is I can’t duplicate it locally. I hit my local running instance and it might grab a few more threads which is to be expected, but not skyrocket up. At first I thought it was a problem with iBatis which had a known threading issue. But after upgrading to the latest beta it hasn’t solved anything. I also switched to using the session API in iBatis as well to no avail.
My gut tells me it isn’t directly tomcat’s fault since I can’t reproduce it and I don’t want to give up on tomcat just yet. However this problem is really keeping us from ramping up traffic on our beta site which of course doesn’t make the brass too thrilled.
In both cases tomcat is configured to use the APR connector and maxThreads are set at about 150. The only real difference in the configs is that production is in a cluster and is using large memory page configurations. Our production boxes have 16gb of RAM and 8 cores, so it the hardware has a lot of room to go.
Anyone have any thoughts about what is causing this?
Don't miss anything, subscribe!
Did you enjoy this post? Why not leave a comment below and continue the conversation, or subscribe to my feed and get articles like this delivered automatically to your feed reader.
Comments
Just for grins did you run JMeter against a single production instance with clustering disabled? We have a layer in front of our Tomcat servers that handles sticky session to bind the user to a tomcat instance. I do have a memcached session manager written for the SunOne Java Webserver for such situations where we can’t do sticky stuff.
When you say “cluster” do you mean that the Session clustering mechanism is turned on? We have sort of the same setup only without the session clustering turned on and it seems to work fine under real world loads. Tomcat 6.0.14, Java 1.6.0_04 64-bit and the APR connector.