This was pretty cool to read back in 2021. They also had a previous post in 2018: https://openai.com/index/scaling-kubernetes-to-2500-nodes/ where they had previously hit this limit.

Cool to see infra solutions at OpenAI. I wonder if these are still powering existing solutions. Ben, one of the authors, seems to have left OpenAI.