Medium Beat XHigh for Opus 5.5 on SlopCodeBench: Cost per Pass, the Noise Floor, and an Effort A/B for Your Repo
> Attributed compile + two small offline scripts
> Primary source: Dex Horthy (with Gabe Orlanski), SlopCodeBench update 10/2026 - Opus 5.5 update, Astra improves by 1.5% over gpt-5.5, xhigh considered harmful (HumanLayer, 2026-10-15).
> Benchmark paper: Gabriel Orlanski et al., SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks (arXiv 2603.24755).
> Second data point: paddo, The Default Was Right: Opus 5.5, Sonnet 5.5 and GPT-6 Sol Effort Levels on Real Work (2026-09-29).
> Discussion: Hacker News thread (283 points on 2026-10-21).
> I did not run SlopCodeBench or any coding agent. Everything below re-analyzes the published numbers offline, and every number is credited to its source.
---
HumanLayer's October SlopCodeBench (SCB) write-up made the Hacker News front page this week with a provocative subtitle: "xhigh considered harmful." The finding behind it: Opus 5.5 at medium effort got 28 of 196 checkpoints strictly passing, and at xhigh it got 25, at 4.22× the estimated cost ($193.47 vs $817.04). GPT-6 SOL went 30 → 26 at 2.11×. Only GPT-6 Astra improved, 29 → 32, at 2.91×.
To his credit, Horthy hedges right in the post: one campaign per setting, so "this doesn't prove that more reasoning hurts coding." The HN thread split along familiar lines: "I leave it on max," "medium is my sweet spot," and "run it five times and put a confidence interval on it." This post takes the last commenter's side. I ask what the numbers can support, re-rank the leaderboard by cost per pass, and give you a small script to run the same A/B on your own repo.
What SCB measures (and why it's harder than SWE-style benchmarks)
SCB gives an agent a problem in stages. It builds a CLI or API from a spec, then gets new requirements and must extend its own earlier code, with no fresh reference solution and no visible tests. The paper has 36 problems and 196 checkpoints. Hidden tests come in four groups: Core, Error, Functionality, and Regression (all earlier checkpoints' tests).
- Strict pass: every test category passes at that checkpoint, Regression included.
- Core pass: only the core tests for that checkpoint pass.
That gap is the whole story. In Horthy's run, the models score 65.8-74.0% on core but only 12.2-16.3% strict. Agents build the new feature far more often than they keep the old features working. The paper also found structural erosion rising in 77% of trajectories and verbosity in 75.5%, and agent code was 2.3× more verbose than a panel of 473 human Python repos.
Setup for anyone reproducing it (from the post): pinned v1.0 catalog, the stock just-solve prompt, seed 42, the any-case pass policy, Docker with Python 3.12 and uv. Problems live at gabeorlanski/scb-problems.
Run 1: is 28 vs 25 a signal?
scb_effort_math.py takes the published table and runs standard proportion tests, stdlib only. One caveat applies to everything below: the tests treat the 196 checkpoints as independent coin flips. They aren't. Checkpoints in one problem share code, and a single regression can wipe out every later strict pass. So the intervals I report are too narrow, which means the real picture is even noisier.
From my run on this box (scb_effort_math.py, output trimmed):
Opus 5.5 strict 28->25 (-3), cost x4.22 (+$623.57); no extra strict passes bought
Fisher exact two-sided p (unpaired, indep. approx) = 0.768
paired best case: 3 discordant all one way -> exact sign-test p = 0.250
GPT-6 SOL strict 30->26 (-4), cost x2.11 (+$271.63)
paired best case: 4 discordant all one way -> exact sign-test p = 0.125
GPT-6 Astra strict 29->32 (+3), cost x2.91 (+$497.57); $165.86 per extra strict pass
paired best case: 3 discordant all one way -> exact sign-test p = 0.250
The "paired best case" line matters most. Medium and xhigh ran on the same 196 checkpoints, so the right test only looks at checkpoints where the two settings disagree. A net gap of 3 needs at least 3 disagreements. Even if all 3 went the same way, the exact two-sided p-value is 2 × 0.5³ = 0.25. No arrangement of these results can make the gap significant. The 95% Wilson interval for Opus Medium's strict rate (independence approximation) is 10.1%-19.9%, and all ten campaigns' strict counts (24-32) fall inside it.
How much data would settle it? Telling 14.29% from 12.76% at α = 0.05 with 80% power takes roughly 7,800 independent checkpoints per setting, about 40 full SCB campaigns each. Nobody is paying for that.
The honest reading: on strict passes, SCB can't tell medium from xhigh. "Harmful" overstates it, and so does "xhigh is better."
Run 2: cost per pass is where the signal is
Cost doesn't have that noise problem. The gaps are 2-4×, not ±3 checkpoints. Divide each campaign's estimated cost by its strict passes (same script, trimmed columns):
campaign strict $ $/strict
GPT-6.1 SOL Medium 29/196 178.51 6.16
Opus 5.5 Medium 28/196 193.47 6.91
GPT-6 SOL Medium 30/196 245.69 8.19
GPT-6 Astra Medium 29/196 261.05 9.00
GLM 5.3 XHigh 24/196 365.13 15.21
GPT-6 SOL XHigh 26/196 517.32 19.90
GPT-6 Astra XHigh 32/196 758.62 23.71
GPT-5.6 SOL XHigh 28/196 753.30 26.90
Opus 5.5 XHigh 25/196 817.04 32.68
Fable 5.1 XHigh 26/196 1550.07 59.62
All four medium campaigns are cheaper per strict pass than all six xhigh campaigns. Astra XHigh is the one setting that bought extra passes, at about $166 per extra checkpoint over Astra Medium. That price might be worth paying on a hard migration. It's a bad default for everyday edits.
paddo's independent bench points the same way. It used six real merged changes in a production TypeScript monorepo, 20 runs per level, graded by the original fix's tests. Opus 5.5 got 13/20 clean runs at medium, 12/20 at xhigh, and 11/20 at max, for $44.10, $139.79, and $307.16. His one exception: Sonnet 5.5 at max hit 13/20 versus 11/20 at medium, for about 35× the cost. My script puts that gap at Fisher p ≈ 0.75. paddo also warns that his Claude effort levels are inferred, since the requests don't record the setting.
Two benchmarks built differently, both noisy on pass counts, both clear on cost. Treat effort as a spending knob, not a quality knob. paddo's phrasing is "a spend dial."
Run 3: an effort A/B on your own tasks
Use your own repo as the benchmark. Pick 10-20 merged changes that each have tests, replay each change from its parent commit at two effort levels, and record whether the full suite passes. Two key facts from the docs (I did not run either command):
# Claude Code: per-session flag (also /effort, or CLAUDE_CODE_EFFORT_LEVEL)
claude -p "$(cat ticket.md)" --effort medium
# Codex CLI: per-run config override (persistent key lives in ~/.codex/config.toml)
codex exec -c model_reasoning_effort='"medium"' "$(cat ticket.md)"
Log one row per run, task_id,effort,run,passed,broke_existing,cost_usd, and pair the runs by task:
python effort_ab.py results.csv medium xhigh
effort_ab.py reports strict pass rate, regressions, cost per strict pass, and an exact paired sign test on the tasks where the two settings disagree. Its rule is deliberately conservative: switch to the pricier setting only if it wins on paired disagreements with p < 0.10. Otherwise keep the cheaper one as the default and save the pricier one for retries on failures, which is also paddo's conclusion. I smoke-tested the script only on clearly labeled synthetic data (demo_results_SYNTHETIC.csv). Those numbers say nothing about any model, so they aren't quoted here.
Three habits from the HN thread are worth stealing either way. Run a formatter and dead-code warnings as hard gates, because several commenters say Astra/SOL output comes back with odd or missing line breaks. Make the agent run the full suite before it finishes, since regressions are what separate core from strict. And read the diff size, because paddo saw Opus's diff on one change grow from about 480 added lines at medium to about 830 at max.
What I didn't test
- Any model, any harness, any effort level. No SCB campaign, no paddo replay.
- Whether SCB's slop metrics favor xhigh. Horthy reports Opus XHigh's flagged-code share at 77.75% versus 94.25% for Medium, and he notes the linters may not cover newer models' patterns. I didn't touch the quality data.
- The claude -p ... --effort and codex exec -c ... lines are taken from current docs, not executed.
- high and max aren't in the SCB effort comparison, which several HN commenters pointed out.
Why this matters
When most of your agent runs fail on regressions, more thinking mostly buys more tokens and bigger diffs. Default to medium, make the agent run the full test suite before it finishes, and escalate effort only as a retry. If you want to know whether xhigh helps on your codebase, a paired test on 20 of your own merged changes will tell you more than any leaderboard headline.
---
Written by YongBo Yu, AI engineer in Toronto. More at yongbo-yu.vercel.app and GitHub.
Mentions
Opus 5.5, an AI coding model, cost over four times as much at its higher effort setting but passed fewer tests in October’s SlopCodeBench benchmark. Test passes alone make that extra spending look like a loss, but automated checks found fewer maintenance problems in the code. 1/5
Check this out 🤯 humanlayer/12-factor-agents Principles for building reliable, maintainable LLM applications.
skills: Claude Code skills from HumanLayer (5.5k stars) What it does All text below is quoted directly from the project README. >Claude Code skills from HumanLayer Available Skills: - rpi: Researches a change, works through design choices with you, and implements it from local documents. - show-me: Explains the current topic with concise diagrams, code-shape sketches, and focused HTML artifacts. - visual-pr: Creates or updates a pull request with a concise visual outline that helps reviewers understand the change. - improve-claude-md: Rewrites your CLAUDE.md using <important if> blocks to improve instruction adherence. - narrow-react-prop-types: Narrows React component prop types to match live code paths instead of Storybook, test, or mock-only states. - build-iterated-agentic-loop: Builds a repo-local skill plus an iterated coding-agent GitHub Actions workflow, prompt, memory file, and reference templates. >npx skills add humanlayer/skills --skill SKILLNAME Repository humanlayer/skills Repository stats |Metric|Value| |:-|:-| |Stars|5,492| |Forks|182| |Last release|None| |Languages|TypeScript 100%| |Contributors|4| |AI contributor accounts (Claude / Codex / Cursor / Copilot)|None| |AI co-authored commits|Claude: 4| Main contributor |Field|Value| |:-|:-| |Name|Dex| |GitHub|@dexhorthy| |Commits to this repo|23 (top contributor)| |Followers|2.2k| ────────────────────────────── 📌 How many great GitHub repos have you discovered and completely forgotten about? Don't let another useful tool disappear into your Reddit feed. Get the community's best repos, organized by category, plus key takeaways from real developer discussions, delivered to your inbox twice a week. 📬 Never lose track of great repos again. Subscribe free.
I think this is where we may have to agree to disagree, because this not something that I'd do outside of toy projects. Not necessarily because I'm worried about the correctness of a single MR (in fact, it may very well be more likely to be correct than a human-authored and -reviewed one), but because I'm concerned about the accrual of technical debt. Even if each individual MR is correct, the trajectory for code quality can trend downwards. The recent update to SlopCodeBench that includes Astra and Opus 5.5 (including the insight that higher reasoning levels may not improve the situation) seems to match my intuition. [1] [1] github.com/humanlayer/advanced-context-engineerin…
Humanlayer: Way to get AI coding agents in complex codebases
Top 3 GitHub Repos for AI Builders 2024 #Shorts Top 3 GitHub Repos for AI Builders 2024 Repos in this video: rohitg00/agentmemory humanlayer/12-factor-agents zai-org/Open-AutoGLM New GitHub finds every day. #Shorts #github #opensource #devtools #coding #ai
看了一下 SlopCodeBench 的作者分享 很难想象 Opus 5.5 会不如 6 Sol 🤔 感觉只有刚出的那两天是比较工作的 最近 Sol 跌落神坛得太快了 一开始我也比较 Opus 吹，但最近两天看了看比较复杂 feature 的 代码又觉得也没有特别优秀 github.com/humanlayer/advanced-context-engineerin…
Medium Beat XHigh for Opus 5.5 on SlopCodeBench: Cost per Pass, the Noise Floor, and an Effort A/B for Your Repo > Attributed compile + two small offline scripts > Primary source: Dex Horthy (with Gabe Orlanski), SlopCodeBench update 10/2026 - Opus 5.5 update, Astra improves by 1.5% over gpt-5.5, xhigh considered harmful (HumanLayer, 2026-10-15). > Benchmark paper: Gabriel Orlanski et al., SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks (arXiv 2603.24755). > Second data point: paddo, The Default Was Right: Opus 5.5, Sonnet 5.5 and GPT-6 Sol Effort Levels on Real Work (2026-09-29). > Discussion: Hacker News thread (283 points on 2026-10-21). > I did not run SlopCodeBench or any coding agent. Everything below re-analyzes the published numbers offline, and every number is credited to its source. --- HumanLayer's October SlopCodeBench (SCB) write-up made the Hacker News front page this week with a provocative subtitle: "xhigh considered harmful." The finding behind it: Opus 5.5 at medium effort got 28 of 196 checkpoints strictly passing, and at xhigh it got 25, at 4.22× the estimated cost ($193.47 vs $817.04). GPT-6 SOL went 30 → 26 at 2.11×. Only GPT-6 Astra improved, 29 → 32, at 2.91×. To his credit, Horthy hedges right in the post: one campaign per setting, so "this doesn't prove that more reasoning hurts coding." The HN thread split along familiar lines: "I leave it on max," "medium is my sweet spot," and "run it five times and put a confidence interval on it." This post takes the last commenter's side. I ask what the numbers can support, re-rank the leaderboard by cost per pass, and give you a small script to run the same A/B on your own repo. What SCB measures (and why it's harder than SWE-style benchmarks) SCB gives an agent a problem in stages. It builds a CLI or API from a spec, then gets new requirements and must extend its own earlier code, with no fresh reference solution and no visible tests. The paper has 36 problems and 196 checkpoints. Hidden tests come in four groups: Core, Error, Functionality, and Regression (all earlier checkpoints' tests). - Strict pass: every test category passes at that checkpoint, Regression included. - Core pass: only the core tests for that checkpoint pass. That gap is the whole story. In Horthy's run, the models score 65.8-74.0% on core but only 12.2-16.3% strict. Agents build the new feature far more often than they keep the old features working. The paper also found structural erosion rising in 77% of trajectories and verbosity in 75.5%, and agent code was 2.3× more verbose than a panel of 473 human Python repos. Setup for anyone reproducing it (from the post): pinned v1.0 catalog, the stock just-solve prompt, seed 42, the any-case pass policy, Docker with Python 3.12 and uv. Problems live at gabeorlanski/scb-problems. Run 1: is 28 vs 25 a signal? scb_effort_math.py takes the published table and runs standard proportion tests, stdlib only. One caveat applies to everything below: the tests treat the 196 checkpoints as independent coin flips. They aren't. Checkpoints in one problem share code, and a single regression can wipe out every later strict pass. So the intervals I report are too narrow, which means the real picture is even noisier. From my run on this box (scb_effort_math.py, output trimmed): Opus 5.5 strict 28->25 (-3), cost x4.22 (+$623.57); no extra strict passes bought Fisher exact two-sided p (unpaired, indep. approx) = 0.768 paired best case: 3 discordant all one way -> exact sign-test p = 0.250 GPT-6 SOL strict 30->26 (-4), cost x2.11 (+$271.63) paired best case: 4 discordant all one way -> exact sign-test p = 0.125 GPT-6 Astra strict 29->32 (+3), cost x2.91 (+$497.57); $165.86 per extra strict pass paired best case: 3 discordant all one way -> exact sign-test p = 0.250 The "paired best case" line matters most. Medium and xhigh ran on the same 196 checkpoints, so the right test only looks at checkpoints where the two settings disagree. A net gap of 3 needs at least 3 disagreements. Even if all 3 went the same way, the exact two-sided p-value is 2 × 0.5³ = 0.25. No arrangement of these results can make the gap significant. The 95% Wilson interval for Opus Medium's strict rate (independence approximation) is 10.1%-19.9%, and all ten campaigns' strict counts (24-32) fall inside it. How much data would settle it? Telling 14.29% from 12.76% at α = 0.05 with 80% power takes roughly 7,800 independent checkpoints per setting, about 40 full SCB campaigns each. Nobody is paying for that. The honest reading: on strict passes, SCB can't tell medium from xhigh. "Harmful" overstates it, and so does "xhigh is better." Run 2: cost per pass is where the signal is Cost doesn't have that noise problem. The gaps are 2-4×, not ±3 checkpoints. Divide each campaign's estimated cost by its strict passes (same script, trimmed columns): campaign strict $ $/strict GPT-6.1 SOL Medium 29/196 178.51 6.16 Opus 5.5 Medium 28/196 193.47 6.91 GPT-6 SOL Medium 30/196 245.69 8.19 GPT-6 Astra Medium 29/196 261.05 9.00 GLM 5.3 XHigh 24/196 365.13 15.21 GPT-6 SOL XHigh 26/196 517.32 19.90 GPT-6 Astra XHigh 32/196 758.62 23.71 GPT-5.6 SOL XHigh 28/196 753.30 26.90 Opus 5.5 XHigh 25/196 817.04 32.68 Fable 5.1 XHigh 26/196 1550.07 59.62 All four medium campaigns are cheaper per strict pass than all six xhigh campaigns. Astra XHigh is the one setting that bought extra passes, at about $166 per extra checkpoint over Astra Medium. That price might be worth paying on a hard migration. It's a bad default for everyday edits. paddo's independent bench points the same way. It used six real merged changes in a production TypeScript monorepo, 20 runs per level, graded by the original fix's tests. Opus 5.5 got 13/20 clean runs at medium, 12/20 at xhigh, and 11/20 at max, for $44.10, $139.79, and $307.16. His one exception: Sonnet 5.5 at max hit 13/20 versus 11/20 at medium, for about 35× the cost. My script puts that gap at Fisher p ≈ 0.75. paddo also warns that his Claude effort levels are inferred, since the requests don't record the setting. Two benchmarks built differently, both noisy on pass counts, both clear on cost. Treat effort as a spending knob, not a quality knob. paddo's phrasing is "a spend dial." Run 3: an effort A/B on your own tasks Use your own repo as the benchmark. Pick 10-20 merged changes that each have tests, replay each change from its parent commit at two effort levels, and record whether the full suite passes. Two key facts from the docs (I did not run either command): # Claude Code: per-session flag (also /effort, or CLAUDE_CODE_EFFORT_LEVEL) claude -p "$(cat ticket.md)" --effort medium # Codex CLI: per-run config override (persistent key lives in ~/.codex/config.toml) codex exec -c model_reasoning_effort='"medium"' "$(cat ticket.md)" Log one row per run, task_id,effort,run,passed,broke_existing,cost_usd, and pair the runs by task: python effort_ab.py results.csv medium xhigh effort_ab.py reports strict pass rate, regressions, cost per strict pass, and an exact paired sign test on the tasks where the two settings disagree. Its rule is deliberately conservative: switch to the pricier setting only if it wins on paired disagreements with p < 0.10. Otherwise keep the cheaper one as the default and save the pricier one for retries on failures, which is also paddo's conclusion. I smoke-tested the script only on clearly labeled synthetic data (demo_results_SYNTHETIC.csv). Those numbers say nothing about any model, so they aren't quoted here. Three habits from the HN thread are worth stealing either way. Run a formatter and dead-code warnings as hard gates, because several commenters say Astra/SOL output comes back with odd or missing line breaks. Make the agent run the full suite before it finishes, since regressions are what separate core from strict. And read the diff size, because paddo saw Opus's diff on one change grow from about 480 added lines at medium to about 830 at max. What I didn't test - Any model, any harness, any effort level. No SCB campaign, no paddo replay. - Whether SCB's slop metrics favor xhigh. Horthy reports Opus XHigh's flagged-code share at 77.75% versus 94.25% for Medium, and he notes the linters may not cover newer models' patterns. I didn't touch the quality data. - The claude -p ... --effort and codex exec -c ... lines are taken from current docs, not executed. - high and max aren't in the SCB effort comparison, which several HN commenters pointed out. Why this matters When most of your agent runs fail on regressions, more thinking mostly buys more tokens and bigger diffs. Default to medium, make the agent run the full test suite before it finishes, and escalate effort only as a retry. If you want to know whether xhigh helps on your codebase, a paired test on 20 of your own merged changes will tell you more than any leaderboard headline. --- Written by YongBo Yu, AI engineer in Toronto. More at yongbo-yu.vercel.app and GitHub.
SlopCodeBench 10/2026 – Opus 5.5, Astra wins overall, xhigh considered harmful github.com/humanlayer/advanced-context-engineerin… (news.ycombinator.com)
SlopCodeBench 10/2026 – Opus 5.5, Astra wins overall, xhigh considered harmful github.com/humanlayer/advanced-context-engineerin… (news.ycombinator.com)