> The full benchmark is hard in a way that matters to me: the agents get the main behavior right much more often than they keep all tested behavior working as requirements change
So, it's like... software development? 90% of bugs come from unintended consequences of changes after the fact. There's a couple decades of design lessons people had to learn to keep this number low. The agents don't follow those lessons.
Good programmers rely on structure and reinforcement to prevent themselves from introducing bugs. Today that structure is a combination of software frameworks, SDLC gates, and system design patterns. For example, most any Python programmer can reliably churn out a Django app as a Microservice and be fairly sure it won't be that hard to make, or result in tons of unexpected bugs. The reason isn't that it's some perfect software design. It's that prescribed methods in themselves are bumper lanes, keeping you on track. A few of you may remember what programming was like before popular open source frameworks, or maybe you've worked on proprietary big-iron mega-projects. Left entirely to one's own devices, programmers can make a mess of things. Give them a formal structure and suddenly there's less to go wrong.
I wouldn't sit around waiting for models to become uber-programmers. Instead I'd treat 'em like humans, and enforce the structures we know have better results. You can try to impose good design as skills (ciembor/agent-rules-books) but the model will still do things its own way. That's why coding harnesses need to intervene and explicitly enforce the structure/framework/bumper lanes. Basically we need 1,000 different Pi extensions that will intervene as the agent codes in order to enforce the correct design. Otherwise we will forever be chasing some "perfect coding rlhf", and still the model will blunder into corners.
Mentions
> The full benchmark is hard in a way that matters to me: the agents get the main behavior right much more often than they keep all tested behavior working as requirements change So, it's like... software development? 90% of bugs come from unintended consequences of changes after the fact. There's a couple decades of design lessons people had to learn to keep this number low. The agents don't follow those lessons. Good programmers rely on structure and reinforcement to prevent themselves from introducing bugs. Today that structure is a combination of software frameworks, SDLC gates, and system design patterns. For example, most any Python programmer can reliably churn out a Django app as a Microservice and be fairly sure it won't be that hard to make, or result in tons of unexpected bugs. The reason isn't that it's some perfect software design. It's that prescribed methods in themselves are bumper lanes, keeping you on track. A few of you may remember what programming was like before popular open source frameworks, or maybe you've worked on proprietary big-iron mega-projects. Left entirely to one's own devices, programmers can make a mess of things. Give them a formal structure and suddenly there's less to go wrong. I wouldn't sit around waiting for models to become uber-programmers. Instead I'd treat 'em like humans, and enforce the structures we know have better results. You can try to impose good design as skills (ciembor/agent-rules-books) but the model will still do things its own way. That's why coding harnesses need to intervene and explicitly enforce the structure/framework/bumper lanes. Basically we need 1,000 different Pi extensions that will intervene as the agent codes in order to enforce the correct design. Otherwise we will forever be chasing some "perfect coding rlhf", and still the model will blunder into corners.
手里的AI编码工具想直接升级成编程大师？有个叫AGENTS Book Rules的开源项目值得一看。🚨 13本经典编程书的核心原则，如今可以直接喂给AI了。 从《代码整洁之道》到《领域驱动设计》，再到《重构》——这些必读书的精髓，统统整理成了开箱即用的规则文件。 🔗 GitHub：ciembor/agent-rules-books 每本书都配了三个版本： 完整版：理论控首选，底层逻辑拉满 精简版：日常搬砖，拿来就用 极简版：上下文窗口紧张时救命用 按任务类型分得明明白白： 日常代码质量 → 稳 架构设计 → 有章法 遗留代码处理 → 拆解到位 Claude Code、Codex、Cursor都能直接用。规则文件复制进项目，AI秒变资深老鸟。 别等了，不然你的代码还活在石器时代。