I completely agree. That observation is actually what led us to build StateGuard. We expected model failures to dominate, but after reproducing a few dozen real-world issues, most of them turned out to be contract mismatches between APIs, providers, schemas, tool outputs, and application logic.
We're trying to understand which of those failures developers would actually want repaired automatically versus simply surfaced through observability.
This is a really interesting class of failure. It feels like we're going to see more cases where the "human view" and the "LLM view" of the same data diverge. Have you run into similar issues outside of MCP as well, or is this mostly specific to terminal-based tool interactions?
So far I have more feedback from PR authors about them using it, and I have been dogfooding it myself. I need help validating it in teams. It reliably surfaces some trade-offs that I would otherwise miss (that are not in the diff). The reviewer angle is the more interesting one, but I don't have clear validation yet. If you try it on a repo with real reviews, I'd be happy to hear from you
Nice work. I'm curious what made you choose this approach over the more conventional alternatives. Was there a specific pain point that pushed you in this direction?
I guess, every time I work with yaml pipelines I end up refactoring things out to “normal” programming languages, so all these yaml harness looks really weird and unnecessary. As for the current project state mostly I am building community, people who would like to use the tool somehow …
We're trying to understand which of those failures developers would actually want repaired automatically versus simply surfaced through observability.