Heh, I knew it. You need to prune your own following list regularly or become less and less visible. I suspect (but have yet to check) that they also weight visibility in terms of historical follower growth.

That's why you see so many trolls with very low follower counts; it's more effective to make/purchase a new firstname-bunchanumbers account and poop in people's replies than to let Twitter decide placement based on historical factors.