> AI-written emails where the commenter would rather see the prompt

this a thousand times this

LLMs are information transformers. if you're trying to take some rough idea and blow it out into something that "feels" substantive, you're just combining your high-signal idea / prompt with a bunch of meaningless bloat and noise from the model weights.

they seem to be able to do the call-stack diff stuff pretty well still

so wait is the finding that most of those skills reduce pass rates against SCB? wild

Yep that was interesting but imho not completely unexpected. My original hypothesis was that these agent skills are castles built on sand and that the models would quickly adapt and make them redundant. Also the skills use up context and I think a lot of them partially focus on investigating a large complex codebase properly... which is wasted on greenfield projects.

why would that worry you?

Wasting a lot of power.

agree, i think the implication is that low quality code is harder to change in the future

somebody get this man a curl-pipe-bash stat

lol

oh i really like the idea of flipping around the order of checkpoints and comparing results. Could be an interesting way to increase/decrease difficulty even

i will look into how easy it would be to zip up some subset of the results without leaking anything...probably doable

Yeah I would hold that models don’t know how to simplify because most rl/benchmarks doesn’t penalize complexity

Yeah the main reason I skipped fable was because we have a ZDR with anthropic and I didn’t feel like spinning up another account to circumvent that. Next run will have fable and sol

my issue with frontier code is that it uses a model judge for quality whereas slop code bench forces a model to grapple with its own garbage code in order to receive a functionality reward