(Commence eyerolls from people who've heard me play this card before) Before - Matasano - I - led - a - project - that - bucketed - and - baselined - entire - tier - one - ISP - backbones - that's - now - deployed - at - every - ISP - and...
The "gigabytes of data" thing doesn't mean anything on its own; you have to consider what you're doing with the data, and what the operating constraints are. I don't think this argument is very compelling. For instance, could you retrofit the ircd source code (leaving everything in IRC protocol) to handle Twitter's work set?
I'm sure Twitter could be optimized quite a bit from where it is today, but it isn't one Rails neophyte's side project anymore; there are very smart people with lots of resources working on it, and I'm inclined to believe those people when they say parts of the problem are Hard. Haven't you ever been surprised by how Hard a simple-looking problem turned out to be?
Haven't you ever been surprised by how Hard a simple-looking problem turned out to be?
I find computing problems tend to become hard based on the constraints not the bassic problem. A friend of mine needed to create a simple counting program that sent the results back over the network. The problem was he needed to do it in less than 340 bytes of ram, on a 8khz chip with 2kbites of permanate storage. AND he needed to validate the memory so the program could sit on a chip in the field for 10 years without problems even as the memory got randomly fliped.
That's a hard problem that took a few months but hey it works just fine. I expect Twitter to have a few anoying problems to track down, but they are long past the point where they have no budget and no time to solve them. Worst case they change some of their constraints and move on.
PS: My problem is not that twitter is trying to fix somthing dificult and limping by my problem is they have been doing it for as long as they have.
The "gigabytes of data" thing doesn't mean anything on its own; you have to consider what you're doing with the data, and what the operating constraints are. I don't think this argument is very compelling. For instance, could you retrofit the ircd source code (leaving everything in IRC protocol) to handle Twitter's work set?
I'm sure Twitter could be optimized quite a bit from where it is today, but it isn't one Rails neophyte's side project anymore; there are very smart people with lots of resources working on it, and I'm inclined to believe those people when they say parts of the problem are Hard. Haven't you ever been surprised by how Hard a simple-looking problem turned out to be?