Not trampling, just holding at arms length. Recognising the reality of the economics of most projects. Unless you live in a completely deterministic world, devoid of human fallibility, or perhaps omnipotent, or simply have unlimited time or resource, at some point you are going to have to admit you just don't know, and you are managing the percentages. Delusional overconfidence is more close to magical thinking than recognising the reality that complex systems will fail in surprising ways, and that it costs ever increasing resource to reduce the risks with ever diminishing returns, until you are forced to stop. The ruleset the computer abides by probably represents a fraction of the factors affecting success. If you feel you have never got to a point where there is an element of faith involved in your choice, then Im envious.
Thank you for your thoughtful reply; you deserve a serious answer.
I accept that, occasionally, my systems will fail. My data will be lost. No recovery will be possible, and the damage will be permanent and lasting. I do not hope to succeed in those times, but expect to be scarred. I will fail.
Given that I will fail, I'd like to understand how I fail. I'd like to understand why I fail. I'd like to measure how often I fail, how short I fall of success, and the root causes of my failure. I'd like to know when failure is about to happen or is happening.
By writing down what I do, I know how to fail. I can write down what I do when I don't fail, too. I can write while I do, and I can read what I wrote to do it again. I can let somebody else read and do what I have done.
I can expect to fail sometimes. I can expect to not fail sometimes. I expect failures based on causes, not based on self-blame. I expect to not fail most of the time, and only fail at certain times when something has caused me to fail.
I can fail less in the future. My actions today can change how I fail in the future. I can plan to fail, or intentionally fail, or sometimes fail less. I act intentionally.
I haven't failed in a while. The last time I failed, I looked at why I failed and I did what was necessary to try to recover and fail less.
This is how SRE works. This isn't overconfidence; this is fault-tolerance. It's not easy, but it works.
To respond to your point about faith, I have plenty of faith, just not hope that my faith will be able to prevent me from failing.
hmm, yes, this was always going to be a contentious point, and highly subjective. I think the main issue here is it was in a large part also a very human impact, which is much harder to measure. I think it was fair to say that our build system at the time was more rigid and testing fully both flows not within our grasp. I discussed this point about 'hope' with a colleague, I think I agree with his conclusion: "Google actually designs some of the chips it uses! And runs its own power stations - I think if anyone could legitimately say "Hope is not a Strategy" it might be them."
Regarding Gazelle, yes, the CI would break early if it detected a none porcelain repo after running Gazelle itself. I think others may have configured their editors, but for me the occasional 30+s run time (and as mentioned in some workflows 3-4 mins) on file save - plus the already creaking VSCode plugins was too much to bear.
To give a tool a fair chance I think it has to be used daily to really grok the pleasure / pain. We were aware of some of the problems, but perhaps overly confident about managing them, and underestimated the frustration and lack of agency. Possibly we let it embed too far before we decided we have tried as hard as could be expected. Around 6 months in, once a number of us formally proposed that it be reverted, we took the bold decision that we should persevere no further. Having worked in C codebases in the past - I can imagine it is almost a miraculous improvement there.
I probably should have made the context clearer earlier in the post - later on I mention :
"To set the scene, our codebase is mainly Go, including vendor directories, and a considerable amount of data compiled in, we have 2.5M lines of code, a full build from a clean clone on one of our Jenkins slaves takes 1 minute.
We had between 8 and 20 engineers working on that codebase. Importantly we all develop on Mac, but run in production in linux."
I would contribute to a federated non-repudiable system.
Moderation can be just as auditable. (Perhaps with some emergency actions that then require a follow up consensus to make permanent)
On some occasions the original content must be purged, but the signatures can remain. Communal censorship is vastly different from changing the content.
