"Actually I think I/O was my primary bottleneck. Once I addressed that I started hitting CPU/memory constraints on Dynos....At one point I was running 20 dynos trying to get enough CPU for worker tasks to actually keep up"
Something doesn't make sense. If you "addressed" your I/O problem, your CPUs were therefore all busy doing something much, much slower than a disk read/write, in software (which would have to be both obvious and unbelievably horrible). If that's true, something pathological was going on in your code. I'm going to assume that you would have noticed it -- swapping, for example.
So let's go back to I/O: if your database was slow, you might observe something superficially similar to what you've described: throwing lots of extra CPUs at the problem would result in lots of blocked request threads, and appear that your dynos were all pegged. The exact symptoms would depend on your database connection code, and your monitoring tools. But in no case would throwing more dynos at a slow database make sense, so I'm going to assume that you didn't do that on purpose (right?)
Given the above, I still can't meet you at the conclusion that abandoning Heroku was the magic bullet for your problems. There's not enough information, and it doesn't add up. My money is on one or more of the following: DB cache misses (i.e. not enough cache); a heavy DB write load; frequent, small writes to an indexed table; or pathological memory usage on your web nodes. And if it turns out that the cause is due to I/O, you've only bought yourself a temporary respite by moving off Heroku. Eventually, you'll get big enough that the problem will re-emerge, even though your homebuilt servers are 10% faster (or whatever).
EDIT: Aha! Your comment in another thread actually explains your problem: you were swapping your web nodes by using more than 500MB RAM (http://news.ycombinator.com/item?id=4458657).
It would take me more than one blog post to describe the architecture that powers Sentry, the various areas that have and can have bottlenecks (some more obvious than others).
More importantly, this a few months ago I made the switch, and I don't remember the specifics of the order of events. I can assure you though that I know a little something about a little something, and I wasnt imagining problems.
(Replied to the wrong post originally, I fail at HN)
"I can assure you though that I know a little something about something, and I wasn't imagining problems."
Since you've made it clear in another thread that you were actually running out of RAM on your dynos, I imagine you were running into trouble. There's no need to be snide about it.
Bottom line: you hit an arbitrary limit in the platform. If heroku had high-memory dynos, the calculus would be different. In the future, instead of arguing that your homebrew system is better than "the cloud", you could just present the actual justification for your choice.
Something doesn't make sense. If you "addressed" your I/O problem, your CPUs were therefore all busy doing something much, much slower than a disk read/write, in software (which would have to be both obvious and unbelievably horrible). If that's true, something pathological was going on in your code. I'm going to assume that you would have noticed it -- swapping, for example.
So let's go back to I/O: if your database was slow, you might observe something superficially similar to what you've described: throwing lots of extra CPUs at the problem would result in lots of blocked request threads, and appear that your dynos were all pegged. The exact symptoms would depend on your database connection code, and your monitoring tools. But in no case would throwing more dynos at a slow database make sense, so I'm going to assume that you didn't do that on purpose (right?)
Given the above, I still can't meet you at the conclusion that abandoning Heroku was the magic bullet for your problems. There's not enough information, and it doesn't add up. My money is on one or more of the following: DB cache misses (i.e. not enough cache); a heavy DB write load; frequent, small writes to an indexed table; or pathological memory usage on your web nodes. And if it turns out that the cause is due to I/O, you've only bought yourself a temporary respite by moving off Heroku. Eventually, you'll get big enough that the problem will re-emerge, even though your homebuilt servers are 10% faster (or whatever).
EDIT: Aha! Your comment in another thread actually explains your problem: you were swapping your web nodes by using more than 500MB RAM (http://news.ycombinator.com/item?id=4458657).