Will document databases make an impact?
There has been a lot of buzz about Amazon’s SimpleDB and other related projects like CouchDB, Google’s Datastore and Thrudb. Personally I think they will all find a niche in modern web applications, but what I find amusing are the nay-sayers that talk about the faults and even saying they aren’t even databases.
Come on, not a database? What are these people smoking? Since when does database == relational ? So they don’t support all transactions, aggregate functions or other features that are common to RDBMs. Did it ever occur to these people that not all applications need these features?
Scalability
The prominent feature for these document style databases is scalability. One guy equates scalability to the number of rows in the database. Uhh, maybe someone should explain to him what scalability means. Who cares if you can put millions of rows in your db if you only have 100 users? What about say a few hundred thousand rows but millions of users? It isn’t necessarily the number of rows you can store but how quickly you can get data out to your users.
Where are my joins?
Others go on and on about how you can’t do things like join data across tables. There are many big sites that actually store data in multiple places. Yep, denormalizing (gasp!) their data in the name of speed. Sharding also fits into this category. Although generally sharding happens on a traditional RDBMS, the effect is the same, spreading data out for scalability sake.
It all comes down to the right tool for the job. Until recently RDMBS were pretty much the only tool in the toolbox, whether you needed all the features or not. Now we have more options to choose from and I’ll bet that many choose the alternate route. All the nay-sayers are just scared.
Don't miss anything, subscribe!
Do you use ActiveMQ?
So I read today where LinkedIN uses ActiveMQ as part of their architecture. I haven’t used ActiveMQ personally, but we use it for a project where I work and the guy that set it up absolutely hates it now.
We were looking to use a message server for a new project and started kicking around ideas. The primary requirement was multiple language support. New stuff is largely being done in Java but we also have a really large perl codebase and some backend stuff is still be done in perl. Naturally that made ActiveMQ a front runner because of its open language support. Once he got wind of the talks it was quickly shelved with “over my dead body”.
Turns out that despite the great documentation and its visibility, it doesn’t work quite as speedy as one would think. After fixing several bugs himself (out of order messages being one that I recall), the one glaring problem is performance. Seems that if a subscriber is a bit slow, the whole server becomes slow even on the receiving end. Well, that is just counter productive in my book. The whole idea of a message system is to eliminate these kinds of bottlenecks yes?
We then tried openAMQ (a C-based server) and that shelved too because it kept dropping messages. For the time being we settled on Websphere MQ which just makes me want to puke. I know IBM has had a message server forever, but I have yet to see a piece of software they’ve written that was worth a shit.
Hopefully we can find another solution before big blue takes a firm hold on us. Any suggestions?
Any really good experiences with ActiveMQ?
Don't miss anything, subscribe!
First real-world experience with Grails
I’ve written a little bit about grails before, but until now I haven’t had a real application to build to really put it through it’s paces. At first I was going to build this application with Rails for the same reason, even deploy it on Glassfish with its RoR support. I don’t know though, something about an application that requires a new instance for every request just seems wrong to me. Seems like such a flawed architecture that I just couldn’t bring myself to do it. Yea, there are other Ruby based web frameworks and I might try one at some time, but not this time.
I also have to say that I generally don’t generally like any kind of black-box magic frameworks. Code generation and such tend to make me cringe, but what the hell I was going to give it a shot. Needless to say so far I’m pretty darn impressed. While I’m not quite finished it is moving along very rapidly, and adding features is pretty damn easy. I would say I’ve put in about 50 hours or so over three weeks and I am at about 85% finished.
What I like
This is ironic part. I really hate tools like Hibernate that force you into a specific format. Most of this stems from my corporate application experience where databases are far more complex than what ORMs like Hibernate can handle. However in this case the GORM has been a wonderful time saver. Setting up relationships has been fairly easy, although I had a few head-scratching moments that weren’t quite textbook and the docs weren’t very clear on.
I also really like the URL mapping stuff. I’m sure Rails has a good URL mapping engine as well, but I really like being able to customize these as I see fit. I still get a bit confused on what return/render method to use in the controllers, but it hasn’t been too bad.
What I don’t like
About my only gripe so far is with the BootStrap class. This is a great idea and wonderful for development, but man it can be a pain. Actually the gripe isn’t so much as the BootStrap itself but with GORM. Go figure.
There are cases where I have a three level deep object graph and populating it isn’t as straightforward as it could have been. Most of my problems have been when I populate the leaf object and then I need to get that leaf object later for some other reason. The getBy and findBy methods will sometime return a null object, even when I know the parameters are good. I end up having to retrieve its parent and then getting the child out of there. I would say I’ve spent about 20%-25% of my time just trying to get my simple test data in. I could very well be doing something wrong, but the docs haven’t shown me anything different and what seems intuitive to me apparently isn’t. Oh well.
I haven’t been doing anything nearly as fancy as Glen Smith but isn’t a “hello world” app either. My current stats are:
16 controllers with 864 LOC
12 domain classes with 245 LOC
1 taglib with 22 LOC
I know I’m going to have to write at least one new tag lib for paging, as the provided paging taglib won’t work for me. A shame because I really need it. I should be done with domain classes but I’ll probably add a couple more controllers.
All in all I’m pretty happy with Grails and I look forward to getting this thing public.
Don't miss anything, subscribe!
Lessons in scaling
In my last post I wrote about some strange threading behavior in tomcat 6. After a couple weeks of struggling between iBatis, our app, our connection pool and even trying the NIO connector we finally narrowed it down to the connection pool.
For some unknown reason, DBCP wasn’t taking the connections back into the pool. The result was the thread that was using them would hang and eventually tomcat would hang. How on earth DBCP has lived as long as it has with this problem is beyond me. We tried everything, even manually pulling the connection out of iBatis and closing it. Nothing worked. I could get DBCP to lock up with a connection pool of 150 and a user load of 50. Go figure.
c3p0 to the rescue
Thankfully there is another popular connection pool out there and it is c3p0. We swapped it in nice and easy (thanks Spring!) and presto, all our troubles went away. I could even overload the app (more concurrent users than connections available) and it wouldn’t die. It got slow, but it wouldn’t die.
Thanks to my finding, we swapped out DBCP as the pool that ActiveMQ was using and solved a few problems there as well.
Now our bottleneck is the communication between the application and our search engine, SOLR. That is to be expected somewhat as it opens an HTTP connection. We use SOLR for more than search however, it is also our navigation engine by means of faceting so needless to say we make a lot of calls to it. A little caching will solve that problem though and should be fairly easy to implement.
What are the lessons learned?
Don’t think any piece of the puzzle is immune to being the problem. My initial thoughts were that since DBCP has been around so long it couldn’t possibly be the problem. I spent most of my time in iBatis since I was new to it and thought maybe we were just doing something wrong.
In the end we were able to run load tests that were several times the capacity that our old (current) site sees. A good sign indeed.
Two tools helped more than anything else. JMeter for our load testing and JConsole, the built in JMX application for monitoring. With JConsole it was easy to see what threads were being locked, what they were waiting on and how many threads were running. Very, very handy indeed.
Don't miss anything, subscribe!
Strange tomcat threading behavior
We are having some threading issues while load testing our production tomcat servers. When I fire up JMeter and hit it with about 50 users, tomcat grabs a ton of threads. Eventually this causes some instability and tomcat hangs.
The weird thing is I can’t duplicate it locally. I hit my local running instance and it might grab a few more threads which is to be expected, but not skyrocket up. At first I thought it was a problem with iBatis which had a known threading issue. But after upgrading to the latest beta it hasn’t solved anything. I also switched to using the session API in iBatis as well to no avail.
My gut tells me it isn’t directly tomcat’s fault since I can’t reproduce it and I don’t want to give up on tomcat just yet. However this problem is really keeping us from ramping up traffic on our beta site which of course doesn’t make the brass too thrilled.
In both cases tomcat is configured to use the APR connector and maxThreads are set at about 150. The only real difference in the configs is that production is in a cluster and is using large memory page configurations. Our production boxes have 16gb of RAM and 8 cores, so it the hardware has a lot of room to go.
Anyone have any thoughts about what is causing this?
Don't miss anything, subscribe!