I work at a global corporation with 50,000 employees. Even though I've never been at Google I felt every pain point this video was getting at because our company is trying to implement all of this stuff right now.
"Oh you want to go to production? Here's a list from A-XX stating what you need to accomplish that." Thing is I thought they actually handled this gracefully when I started because lots of requirements were tiered with various criteria you had to meet to move up (mostly for brownie points).
But then one day the Tech Execs lose their minds and decide "everything needs to meet all criteria for every single process." You want to create an S3 bucket to store data? That will be a week of submitting paperwork and another month of meetings and approvals from various teams you've never heard of. Plus you have to register your schema, implement data quality checks, unit tests, regression tests, get a PR and CO approved for your central config change, remediate any CVEs in the tooling that you used, and build all of this using our in-house CI/CD platform we created because we're just soooo special. Now you're allowed to launch. Oh wait, NO because we've put the entire corporation on hold from launching new systems for the last calendar year because we're still trying to agree on the final process everyone needs to follow to go to production.
It's surreal how universally so many orgs makes the same mistake of trying to throw more and more process at problems.
> It's surreal how universally so many orgs makes the same mistake of trying to throw more and more process at problems.
It's hard to find the right balance. You want a bit of process, but not too much.
But it's one of the hardest problems in the existence of humanity and whoever solves it should probably get all the Nobel prizes available (including peace and chemistry!).
In my previous role, the secdevops groups (matrixed teams) were building custom terraform modules for our devs to use in order to easily deploy compliant AWS infrastructure - and devs could only deploy via terraform/CI-CD. While TF specifically states that custom modules are not meant to be used as wrappers, I thought it was a clever way to try getting security "out of the way" while still enforcing best practices.
"We do not recommend writing modules that are just thin wrappers around single other resource types. If you have trouble finding a name for your module that isn't the same as the main resource type inside it, that may be a sign that your module is not creating any new abstraction and so the module is adding unnecessary complexity. Just use the resource type directly in the calling module instead."
If you write different versions of the terraform modules that do some corporate specific magic, I think that would be okay under this rule. It's when you're writing a module that doesn't do any useful magic that they want you to stop and think.
Well we're small, but our development is currently starting to build new products and new extensions to their products. I'm pretty happy that everyone is pretty much onboard with our situation #1.
There are a few hard requirements, but most of the requirements we as operations put up are tied to the guaranteed service level agreement to the customer and possibly overall user count.
If there is just an entirely lax service level agreement there might be no need to invest time in clustering non-trivial applications, or implementing more monitoring than a simple HTTP check. On the other hand, if you're selling some 99.95 24/7 with penalties to a customer, the list of must-dos suddenly grows a lot.
The nice thing of approaching it like this: It allows a gradual increase in operational rigidity and robustness. A product team doesn't hit a wall of requirements for their first productive customer. They rather have to incorporate more requirements as the service becomes more successful. Or they don't if the idea doesn't work.
Literally every point you made applies also at my workplace. The optimist in me hopes we work at the same place, but I fear that your last statement might just be the truth :-)
"Oh you want to go to production? Here's a list from A-XX stating what you need to accomplish that." Thing is I thought they actually handled this gracefully when I started because lots of requirements were tiered with various criteria you had to meet to move up (mostly for brownie points).
But then one day the Tech Execs lose their minds and decide "everything needs to meet all criteria for every single process." You want to create an S3 bucket to store data? That will be a week of submitting paperwork and another month of meetings and approvals from various teams you've never heard of. Plus you have to register your schema, implement data quality checks, unit tests, regression tests, get a PR and CO approved for your central config change, remediate any CVEs in the tooling that you used, and build all of this using our in-house CI/CD platform we created because we're just soooo special. Now you're allowed to launch. Oh wait, NO because we've put the entire corporation on hold from launching new systems for the last calendar year because we're still trying to agree on the final process everyone needs to follow to go to production.
It's surreal how universally so many orgs makes the same mistake of trying to throw more and more process at problems.