You rather generously assume engineers are in touch with their systems.
Even before layoffs many teams just maintained things org has long lost coherent knowledge of
After layoffs and typical org knowledge churn - you can either rewrite it (but how? Product team responsible for original implement requirements is long gone too) or recoup (reverse document) some of that lost knowledge with AI and actually learn
Our experience is very similar except we didn't really have a review process before, and now LLMs find bugs before PRs get merged in main.
We had 5x-100x speedups in some legacy but important pipelines, with no regressions (validated after extensively by humans).
It's not that the code was actively bad. It's just only 1-5% people in the local SWE market would be able to write code that runs so fast and efficient and benchmark it correctly.
We found a subtle correctness bug that was in production for half of the decade (both GPT-5 and Claude Opus were able to find it), confirmed by human after.
And we keep finding subtle bugs that have been introduced by humans before (despite the human reviews, the particular domain is just difficult no matter how many docs and comments and tests one writes)
I am convinced human reviews are overhyped in the industry. We've done it in my company since we started it, and bugs keep happening. People are just terrible at spotting them in the middle of 100 lines of correct code.
Machines, OTOH, are very good at it. I am currently trying to make the code review experience better for humans by not just having the AI review the code, but interact with the human, pointing out potential problems, bad patterns, perhaps hiding some code (e.g. renamings, formatting changes).
Developers still want to review the code, despite provably being bad at spotting bugs, because they want to actually keep knowledge of what's being modified in the code base, so I think this is the best approach.
Maybe the humans are just overwhelmed by the amount of poorly readable AI code you're throwing at them? Maybe they'd be better at reviewing if the code was written by somebody who had put thought into the code instead?
Like we had done for the 10 years prior? Don’t think so. BTW the ai code is as readable as the human’s. Never had to call out people on the AI code being unreadable.
I have not had the same experience. In the PRs I have read, AI accomplishes in 300 very verbose lines what a competent human could in like 60, quintupling cognitive load to review.
But that's so easy to fix! I can't believe people complain about stuff like this. The person making the PR could've told the AI "hey you can easily reduce the amount of code by using this technique" or whatever... or saying you prefer concise code than verbose code in the system prompt so the AI behaves how you want it to... If you can't do any of that, and I can guarantee that if you did you wouldn't have this problem, then you're not really trying.
Not really, they're only as good as their context and they do miss and forget important things. It doesn't matter how often, because they do, and they will tell you with 100% confidence and with every synonym of "sure" that they caught it all. That's the issue.
I am very confident that these tools are better than the median programmer at code review now. They are certainly much more diligent. An actually useful standard to compare them to is human review, and for technical problems, they definitely pass it. That said, they’re still not great at giving design feedback.
But GPT-5 Pro, and to a certain extent GPT-5 Codex, can spot complex bugs like race conditions, or subtly incorrect logic like memory misuse in C, remarkably well. It is a shame GPT-5 Pro is locked behind a $200/month subscription, which means most people do not understand just how good the frontier models are at this type of task now.
Its not scaffolding if the intelligence itself is adding it. Humans can make their own diagrams ajd maps to help them, LLM agentsbneed humans to scaffold for them, thats the setup for the bitter lesson
I deployed lots of high performance, clean, well documented etc code generated by Claude or o3. I reviewed it wrt requirements, added tests and so on. Even with that in mind it allowed me to work 3x faster.
But it required conscious effort on my part to point out issues and inefficiencies on LLMs part.
It is a collaborative type of work where LLMs shine (even in so called agentic flows)
They have different uses. The reasoning models aren't good at multi-turn conversations.
"GPT-4.5" is the best at conversations IMO, but it's slow. It's a lot lazier than o4 though; it likes giving brief overview answers when you want specifics.
Even before layoffs many teams just maintained things org has long lost coherent knowledge of
After layoffs and typical org knowledge churn - you can either rewrite it (but how? Product team responsible for original implement requirements is long gone too) or recoup (reverse document) some of that lost knowledge with AI and actually learn
reply