AgentSWE (BAAI / VectorSpaceLab) asks coding agents to create, edit, or optimize agents from a brief. builders get a few illustrative cases. acceptance freezes the delivery and runs held out cases covering clauses the demos left out
arxiv.org
VectorSpaceLab/AgentSWE
7 views
Play it free: neuralloot.github.io
Paper (AgentSWE, arXiv 2610.13361): arxiv.org
Code: VectorSpaceLab/AgentSWE
1 like · 25 views
Can coding agents actually create, edit, or optimize usable agents?
Current frontier models score poorly. Let em cook.
arxiv.org
VectorSpaceLab/AgentSWE
1 repost · 8 views
Paper: paperswithcode.co
Code: VectorSpaceLab/AgentSWE
Finding: builders show working agents, then deliver hollow runtimes and unsupported completion claims.
1 like · 350 views
AgentSWE: Can Coding Agents Build the Agent You Actually Want?
Jiahao Wang, Hongjin Qian, Yuyang Hu, Jiajun Zhang, Zheng Liu
arxiv.org [𝚌𝚜.𝚂𝙴 𝚌𝚜.𝙰𝙸 𝚌𝚜.𝙻𝙶]
💬Code: VectorSpaceLab/AgentSWE t.co
35 views
AgentSWE: Can Coding Agents Build the Agent You Actually Want?
As the production of software passes into the hands of software itself, coding agents are increasingly commissioned to build agents fitted to their users' own workflows. This delegation carries a quiet assumption, that the delivered agent will faithfully serve the requirement it was built for, rather than merely appear to. No existing benchmark tests that assumption. We introduce AgentSWE, a benchmark for agent software engineering spanning three stages of the agent development lifecycle: Creation of a complete agent from a natural-language requirement, Editing of production agent codebases, and Optimization toward a measurable target. Each of its 25 tasks is framed as a commission rather than an exam: the builder iterates against a handful of illustrative cases, while acceptance exercises the frozen submission on held-out cases designed to span the full requirement, grading behavior against requirement-derived rubrics under hard pass/fail gates or, for Optimization, against the source benchmark's own objective. Evaluating a range of coding agents across models and harnesses, we find that none comes close to delivering what a requirement asks for: the gap between what a builder demonstrates and what it delivers is the norm, not the corner case. An accompanying empirical analysis of the failure trajectories identifies recurring failure modes, among them overfitting to the visible cases, hollow runtimes behind code that looks complete, and unsupported claims of completion, each a signal by which coding agents can learn to build the agents they are asked for.
Comments: Code: VectorSpaceLab/AgentSWE
Mentions
AgentSWE (BAAI / VectorSpaceLab) asks coding agents to create, edit, or optimize agents from a brief. builders get a few illustrative cases. acceptance freezes the delivery and runs held out cases covering clauses the demos left out arxiv.org VectorSpaceLab/AgentSWE
Play it free: neuralloot.github.io Paper (AgentSWE, arXiv 2610.13361): arxiv.org Code: VectorSpaceLab/AgentSWE
Can coding agents actually create, edit, or optimize usable agents? Current frontier models score poorly. Let em cook. arxiv.org VectorSpaceLab/AgentSWE
Paper: paperswithcode.co Code: VectorSpaceLab/AgentSWE Finding: builders show working agents, then deliver hollow runtimes and unsupported completion claims.
AgentSWE: Can Coding Agents Build the Agent You Actually Want? Jiahao Wang, Hongjin Qian, Yuyang Hu, Jiajun Zhang, Zheng Liu arxiv.org [𝚌𝚜.𝚂𝙴 𝚌𝚜.𝙰𝙸 𝚌𝚜.𝙻𝙶] 💬Code: VectorSpaceLab/AgentSWE t.co
AgentSWE: Can Coding Agents Build the Agent You Actually Want? As the production of software passes into the hands of software itself, coding agents are increasingly commissioned to build agents fitted to their users' own workflows. This delegation carries a quiet assumption, that the delivered agent will faithfully serve the requirement it was built for, rather than merely appear to. No existing benchmark tests that assumption. We introduce AgentSWE, a benchmark for agent software engineering spanning three stages of the agent development lifecycle: Creation of a complete agent from a natural-language requirement, Editing of production agent codebases, and Optimization toward a measurable target. Each of its 25 tasks is framed as a commission rather than an exam: the builder iterates against a handful of illustrative cases, while acceptance exercises the frozen submission on held-out cases designed to span the full requirement, grading behavior against requirement-derived rubrics under hard pass/fail gates or, for Optimization, against the source benchmark's own objective. Evaluating a range of coding agents across models and harnesses, we find that none comes close to delivering what a requirement asks for: the gap between what a builder demonstrates and what it delivers is the norm, not the corner case. An accompanying empirical analysis of the failure trajectories identifies recurring failure modes, among them overfitting to the visible cases, hollow runtimes behind code that looks complete, and unsupported claims of completion, each a signal by which coding agents can learn to build the agents they are asked for. Comments: Code: VectorSpaceLab/AgentSWE