Maybe I missed in when scanning the paper, but did they compare to a simple self modifying harness, e.g. instructing Pi to update some code or skill docs based on the results?
Turning 100+ page SOP documents into executable policies using DeepClause and DML.
Building on the self-modifying harness concept we show here how to use DeepClause to build more reliable agents for outsourcing work defined via SOPs and policies. Instead of taking a big markdown file and putting it into the system prompt, we let a coding agent convert policies into small executable logic programs in a DSL called “DML”. The “leaves” of these programs can either be deterministic rules or LLM-driven agent loops. DML programs run safely inside the WASM build of SWI-Prolog.
Great work! If anyone is looking for a way to integrate something like this into their own harness or the pi coding agent, then you might be interested in DeepClause [0]. It comes with a Prolog-like language implemented on top of SWI-Prolog (WASM Version). The purpose of the project is to allow for broad experimentation around the intersection of LLMs/Agents and GOFAI. So you could use it to build memory systems like OP did, create executable specs, define graphs and loops for agents and subagents... It also comes with a pi extension that greatly simplifies getting started with it.
Opposed to OP, DeepClause uses Prolog semantics, so running some more complex queries on knowledgebases might cause some issues (which is the use case where a Datalog might be more useful). For smaller scales it should be fine though.
I'm not sure if this is the right direction, but it's certainly momentarily helpful. I think the right direction would be to enable the model itself do dynamic program analysis, deterministically and dynamically via runtime-inference.
btw. your comment is grayed out, not sure what it means. However, thanks for sharing, I'll look into it.
I’ve been working on DeepClause [0], an agentic harness, the core of which is based on Prolog. Every interaction (conversation turn, plan, execute plan, etc.) triggers a small logic program that orchestrates one or more agent loops or prompts. All these programs are completely hackable, so you can easily adjust the core logic and behavior of your agents in a reliable manner. Also, instead of using markdown specs, you can “compile” your markdown into Prolog code. My hope is that this might unlock new possibilities for spec driven development.
It’s been a lot of fun and I am somewhat proud of Prolog/Typescript integration layer built on SWI Prolog’s WASM version.
Other than that, I am not sure how and if I will continue with it. Feedback welcome!
Indeed, and maybe that's all there is to it. Still, I'd hope we will eventually better understand what's exactly happening in the wake of many repeated applications of f().
For my own (rather idiosyncratic) harness I've been experimenting [0] with "compiling" long markdown specifications into small executable logic programs. It's too early to tell for sure, but I believe that this approach does have its merits when you want some guarantees about how your agents behave for longer tasks.
Interesting idea. I’ve been noodling about something similar myself for a few months, but I haven’t moved forward with testing it. What sort of outcomes are you seeing with it? IMO, we’re never going to get to AGI without fusing “soft” AI decision making with “hard” logic and symbolic algorithmic reasoning. Humans don’t realize this most of the time, but we routinely use them all.
Sorry, did not notice your comment until just now.
So far I am observing two things:
1. For smaller models, performance on Benchmarks such as DeepPlanning does increase significantly.
2. Context hygiene for sub agents becomes much simpler, since that can be expressed relatively concise and the mechanics are handled by the runtime automatically.
Still looking for a good test cases to study possible advantages, but running reliable benchmarks does take time and money...
Are you the Deepclause author? I tried it yesterday and my first reaction was that it was slow. Perhaps I’m doing something wrong, however. Running against the same model in Pi was lightspeed in comparison. Second reaction is the prompt editor needs to handle more than a single line of text and it needs Emacs editing key bindings like Pi. I’m happy to do some testing on it and provide further feedback. What’s the best way to submit that? GitHub issues?
Yes, I am the author and thanks so much for trying! Please do submit a github issue. My first suspicion about the speed is that maybe an inner loop is taking too many turns until the model finally realizes that a task is finished (so that in turn the runtime knows whether the predicate failed or not and can continue execution accordingly). Happy to take a closer look!
The point about multiline prompts is very valid obviously, that's on the todo list.
reply