Happy MLK! For this year’s MLK Spongecell has released a new ultra-beta Boxed Calendar. It’s so ultra-beta we haven’t even mentioned it on our main blog yet.
If The Reverend had been a system administrator he definitely would not have allowed slow or leaky requests on his system. Spongecell was recently experiencing a very bad leaky request. Luckily we were logging PIDS to support Rawk so simply watching our mongrels and seeing when a process exploded let me know when and which process was bad. I looked at the logs and saw a few requests for that process at that time. I ran those requests on our staging server and very quickly saw a bad sql query. Turns out some new code found some very old code and the mix wasn’t pleasant.
I’ve also been spending a little bit of time optimizing requests that Rawk reports have shown our server spends a lot of time on. For one of these MySql had decided to start using the wrong index on a paginated join. Thanks to Blythe, our resident ActiveRecord expert, we were able to change the has_many relationship to use an index hint. Here’s what the code looked like for User:
has_many(:email_addresses, :foreign_key => :user_id, :index_hint => :fk_ea_user …
Engine Yard also put in fair scheduling into our Nginx setup. By default nginx will distribute requests round-robin which is bad if one request is slow. When nginx sends to that slow mongrel the request will queue and wait for completion. With fair scheduling Nginx sends to available mongrels first thus less waiting.
I found this picture on the internet. “Leaky pipes are the biggest wasters.”