Finding and Fixing Slow and Leaky Requests this MLK

Happy MLK! For this year’s MLK Spongecell has released a new ultra-beta Boxed Calendar. It’s so ultra-beta we haven’t even mentioned it on our main blog yet.

If The Reverend had been a system administrator he definitely would not have allowed slow or leaky requests on his system. Spongecell was recently experiencing a very bad leaky request. Luckily we were logging PIDS to support Rawk so simply watching our mongrels and seeing when a process exploded let me know when and which process was bad. I looked at the logs and saw a few requests for that process at that time. I ran those requests on our staging server and very quickly saw a bad sql query. Turns out some new code found some very old code and the mix wasn’t pleasant.

I’ve also been spending a little bit of time optimizing requests that Rawk reports have shown our server spends a lot of time on. For one of these MySql had decided to start using the wrong index on a paginated join. Thanks to Blythe, our resident ActiveRecord expert, we were able to change the has_many relationship to use an index hint. Here’s what the code looked like for User:

has_many(:email_addresses, :foreign_key => :user_id, :index_hint => :fk_ea_user

Engine Yard also put in fair scheduling into our Nginx setup. By default nginx will distribute requests round-robin which is bad if one request is slow. When nginx sends to that slow mongrel the request will queue and wait for completion. With fair scheduling Nginx sends to available mongrels first thus less waiting.

I found this picture on the internet. “Leaky pipes are the biggest wasters.”



Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: