Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

It just goes to show the difference between best practices in cloud computing, and what everyone ends up doing in reality, including well known industry names.


> It just goes to show the difference between best practices in cloud computing, and what everyone ends up doing in reality,

Well, inter-region DR/HA is a expensive thing to ensure (whether on salaries, infra or both), specially when you are in AWS.


Eh, the "best practices" that would've prevented this aren't trivial to implement and are definitely far beyond what most engineering teams are capable of, in my experience. It depends on your risk profile. When we had cloud outages at the freemium game company I worked at, we just shrugged and waited for the systems to come back online - nobody dying because they couldn't play a word puzzle. But I've also had management come down and ask what it would take to prevent issues like that from happening again, and then pretend they never asked once it was clear how much engineering effort it would take. I've yet to meet a product manager that would shred their entire roadmap for 6-18 months just to get at an extra 9 of reliability, but I also don't work in industries where that's super important.


Indeed, yet one would expect AWS to lead by example, including all of those that are only using a single region.


Like any company over a handful of years old, I'm sure they have super old, super critical systems running they dare not touch for fear of torching the entire business. For all we know they were trying to update one of those systems to be more resilient last night and things went south.


So many high profile companies with old deployments stuck in a single region, then.


Best practice does not include plan for when AWS going down. Netflix does not plan for it and they have a very strong eng org.


Did they stop their Chaos Gorilla, which simulates a region outage?


It was only one region.


Does AWS follow its own Well-Architected Framework!?




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: