A few of my teammates at SendGrid have been following the situation, and we definitely feel for the engineers who are scrambling to fix the problem. It's never fun to get paged, especially when the trust of your customers on the line.
Some folks on the email thread were personally involved in handling major outages in the early days. We've had to learn a lot of hard lessons since then. Even when everything seems like it's going fine ("wow, we're growing so fast, good problems!"), scaling issues could be right around the corner.
Anyone with a large enough installation of Postgres could've had the wraparound issue we're seeing right now. That's why it's important to monitor for what could go wrong, detect these issues early, and provide customers with rapid communication so they can plan around it.
Sending our best wishes to the MailChimp engineering teams working on the problem right now. Good luck, you've got this!
Some folks on the email thread were personally involved in handling major outages in the early days. We've had to learn a lot of hard lessons since then. Even when everything seems like it's going fine ("wow, we're growing so fast, good problems!"), scaling issues could be right around the corner.
Anyone with a large enough installation of Postgres could've had the wraparound issue we're seeing right now. That's why it's important to monitor for what could go wrong, detect these issues early, and provide customers with rapid communication so they can plan around it.
Sending our best wishes to the MailChimp engineering teams working on the problem right now. Good luck, you've got this!