My co-worker Sam Steingold and I have a new article at the Magnetic engineering blog about how we predict clicks in real-time online advertising.

Magnetic's real-time bidding system, written in pure Python, needs to keep up with a tremendous volume of incoming requests. On an ordinary weekday, our application handles about 300,000 requests per second at peak volumes, and responds in under 10 milliseconds. It should be obvious that at this scale optimizing the performance of the hottest sections of our code is of utmost importance. This is the story of the evolution of one such hot section over several performance-improving revisions.

A good test suite is a developer's best friend -- it tells you what your code does and what it's supposed to do. It's your second set of eyes as you're working, and your safety net before you go to production.

By contrast, a bad test suite stands in the way of progress -- whenever you make a small change, suddenly fifty tests are failing, and it's not clear how or why the cases are related to your change.

It's been real quiet around here. Like, tumbleweed-rolling-by quiet.

Today I managed to get myself a little bit stuck with Heroku's otherwise seamless Python support. In an earlier revision of my app, I had used -e to install an editable version of a package (the version of the package I needed had not yet been released to PyPI, so I pointed pip at Github). Since then, the version I need has been released.

So, update requirements.txt, then git push heroku master, and all is well, right? Not so fast...