Review AI-generated SQL before you trust the number: a checklist with a runnable fixture
!A dark code editor with lines of colorful source code
Photo by Chris Ried on Unsplash (Unsplash License).
A query that runs is not a query that's right. Most of us know this, and few of us act on it when an assistant hands back a query in four seconds and the number looks about right.
This is the review I run on AI-written SQL before its number goes anywhere near a sign-off. Eight checks, each written as a query that returns rows only when something is wrong. It comes with a tiny fictional fixture you can paste into SQLite, where a perfectly plausible revenue query returns 1,820 and the reviewed one returns 825. Same tables, same question, same month.
Nothing in the bad query is exotic. That's sort of the point.
Why the "it ran" signal got weaker, not stronger
The text-to-SQL benchmarks look great right now, and they're worth reading closely before you lean on them.
On the BIRD leaderboard, the top test execution accuracy as of 22 October 2026 is 83.40% (AskData, from AT&T's CDO team), with GrainSQL at 82.95% and Huawei's DataGallery at 82.39%. The human baseline on the same page, from data engineers and database students, is 92.96%. So the best system still misses roughly one question in six, and that's on a curated benchmark with clean schemas.
Numbers like these are easy to misread. Spider 2.0 uses enterprise-style workflows on Snowflake and BigQuery, and its top Spider 2.0-Snow entries sit around 96.70%, while the top Spider 2.0-Lite score is 76.23%. Same family of tasks, very different numbers. The Spider 2.0 repo also has a long changelog of evaluation fixes and ambiguity cleanups, and as of August 2026 a suspended Snowflake evaluation account was disrupting Snow access.
Then there's the paper that made me stop quoting leaderboard numbers at all. In Pervasive Annotation Errors Break Text-to-SQL Benchmarks and Leaderboards (Jin, Choi, Zhu and Kang, January 2026), expert review found annotation error rates of 52.8% in BIRD Mini-Dev and 62.8% in Spider 2.0-Snow. When they re-scored 16 open-source agents on a corrected BIRD Dev subset, relative performance moved by -7% to +31% and rankings shifted by up to nine places. BIRD has since shipped a 2026 Mini-Dev update with corrected annotations (28 October 2026), which is the right response. But think about what it means: the people grading the SQL got it wrong more than half the time on some subsets. The "gold" queries had the same class of mistakes I'm about to show you.
The vendors say it plainly too. Google's Gemini in BigQuery docs say the model "can generate output that seems plausible but is factually incorrect," recommend you "validate all output," and note it "might suggest different syntax each time that you enter the same prompt." That last line matters for review. If you regenerate, you're reviewing a new query.
And developers know it. In the Stack Overflow 2026 Developer Survey, the most common answer to "How much do you trust the output from AI tools or AI agents as part of your workflow?" was "I trust it when I can easily verify the output," at 48.0% of 14,304 respondents. Only 6.6% said they trust it for many tasks including important work decisions. (The survey announcement has the broader picture: more than 30,000 people took it.)
A revenue number in a steering-committee deck is an important work decision. So for an analyst the question is very practical: what does "easily verify" look like for SQL?
The take you can argue with
Here's my position. The person who reviews AI-written SQL before sign-off should be the person who owns the metric definition, not the strongest SQL writer in the room. On most teams that's the BA or the analyst, not the engineer.
The reason is in the fixture below. None of the five bugs is a syntax problem. Every one is a definition problem: which statuses count, whose calendar, which customers, what one row means. An engineer who writes beautiful SQL but wasn't in the room when finance said "test accounts don't count" will approve the query, and they'll approve it faster if it looks tidy.
You can disagree with me on this one. A fair counter is that BAs miss performance and correctness issues an engineer catches right away, like a NOT IN against a nullable column. True, and that's why the checklist includes those. But I'd still rather teach the definition owner eight mechanical checks than teach the strongest SQL writer every one of the business rules.
The fixture: Northstar Systems, October revenue
Northstar Systems is a fictional B2B software company, and every row below is invented. Here's the question someone typed into an assistant:
> "What was Northstar's revenue from completed orders in October 2026, and how many orders?"
Before any SQL, write the definition down. A reader taught me this habit: name the grain first, "one row = one order line" (ontomato's comment). Shivam Pandit makes the same point at metric level in Defining revenue for an SQL agent, one row at a time: a definition an agent can use has to state grain, statuses, join behaviour and what an empty result means.
So here's Northstar's:
| Part | Definition |
|---|---|
| Grain of the source | order_lines: one row = one order line |
| Grain of the answer | one row = one order, then one total |
| Revenue | sum of qty * unit_price over lines |
| Statuses | completed only. Not refunded, pending or cancelled |
| Calendar | October in America/New_York. Timestamps are stored in UTC |
| Exclusions | customers flagged is_test = 1 |
| Empty month | report 0 only if the month's data is loaded; otherwise "no data" |
Now the schema and data. It runs as-is in the sqlite3 shell or any SQLite client.
-- Northstar Systems (fictional). All data invented.
CREATE TABLE customers (customer_id INTEGER PRIMARY KEY, name TEXT, is_test INTEGER NOT NULL DEFAULT 0);
CREATE TABLE orders (order_id INTEGER PRIMARY KEY, customer_id INTEGER, status TEXT, created_at_utc TEXT);
CREATE TABLE order_lines (line_id INTEGER PRIMARY KEY, order_id INTEGER, sku TEXT, qty INTEGER, unit_price INTEGER);
CREATE TABLE payments (payment_id INTEGER PRIMARY KEY, order_id INTEGER, amount INTEGER, paid_at_utc TEXT);
CREATE TABLE support_tickets (ticket_id INTEGER PRIMARY KEY, customer_id INTEGER, opened_at_utc TEXT);
INSERT INTO customers VALUES
(1,'Alder Clinic',0),(2,'Birch Freight',0),(3,'Cedar Labs',0),(4,'Dogwood Foods',0),(5,'QA Test Account',1);
INSERT INTO orders VALUES
(1001,1,'completed','2026-10-01 02:30:00'),
(1002,2,'completed','2026-10-05 15:00:00'),
(1003,3,'completed','2026-10-12 18:20:00'),
(1004,1,'cancelled','2026-10-14 09:00:00'),
(1005,4,'refunded', '2026-10-20 13:45:00'),
(1006,2,'completed','2026-10-31 21:15:00'),
(1007,5,'completed','2026-10-22 16:00:00'),
(1008,3,'completed','2026-11-01 02:10:00'),
(1009,4,'pending', '2026-10-25 11:30:00');
INSERT INTO order_lines (order_id,sku,qty,unit_price) VALUES
(1001,'NS-BASE',1,120),
(1002,'NS-PRO',1,200),(1002,'NS-ADDON',1,50),
(1003,'NS-PRO',1,300),(1003,'NS-ADDON',1,80),(1003,'NS-SETUP',1,20),
(1004,'NS-BASE',1,90),
(1005,'NS-PRO',1,150),
(1006,'NS-BASE',1,60),(1006,'NS-ADDON',1,40),
(1007,'NS-ENT',1,500),
(1008,'NS-BASE',1,75),
(1009,'NS-PRO',1,110);
INSERT INTO payments (order_id,amount,paid_at_utc) VALUES
(1001,120,'2026-10-01 02:31:00'),
(1002,250,'2026-10-05 15:02:00'),
(1003,200,'2026-10-12 18:25:00'),(1003,200,'2026-10-26 10:00:00'),
(1005,150,'2026-10-20 13:50:00'),
(1006,100,'2026-10-31 21:16:00'),
(1007,500,'2026-10-22 16:01:00'),
(1008,75,'2026-11-01 02:12:00');
INSERT INTO support_tickets (customer_id,opened_at_utc) VALUES
(1,'2026-10-03 14:00:00'),(3,'2026-10-15 10:00:00'),(NULL,'2026-10-18 09:00:00');
Three rows are planted on purpose. Order 1001 is 2:30 a.m. UTC on 1 October, which is 10:30 p.m. on 30 September in New York. Order 1008 is the reverse: November in UTC, October in New York. Order 1003 was paid in two instalments.
The query that looks fine
This is an illustrative example I wrote myself, in the shape I see assistants produce for this kind of prompt. I'm not claiming any specific tool wrote it. Read it once and note what you'd flag before scrolling.
SELECT COUNT(DISTINCT o.order_id) AS orders,
SUM(l.qty * l.unit_price) AS revenue
FROM orders o
JOIN order_lines l ON l.order_id = o.order_id
JOIN payments p ON p.order_id = o.order_id
WHERE o.status != 'cancelled'
AND o.created_at_utc BETWEEN '2026-10-01' AND '2026-10-31';
It looks tidy, and the aliases are sensible. Joining payments even feels responsible, as if it's making sure we only count money that actually came in. Here's the output (SQLite 3.53.1, run through Python's sqlite3 module):
orders | revenue
5 | 1820
The reviewed query
WITH order_totals AS ( -- grain: one row = one order
SELECT order_id, SUM(qty * unit_price) AS order_revenue
FROM order_lines
GROUP BY order_id
)
SELECT COUNT(*) AS orders,
COALESCE(SUM(t.order_revenue), 0) AS revenue
FROM orders o
JOIN order_totals t ON t.order_id = o.order_id
JOIN customers c ON c.customer_id = o.customer_id
WHERE o.status = 'completed'
AND c.is_test = 0
-- October in America/New_York (EDT, UTC-4 all month), half-open range
AND o.created_at_utc >= '2026-10-01 04:00:00'
AND o.created_at_utc < '2026-11-01 04:00:00';
orders | revenue
4 | 825
Side by side:
| | Illustrative AI-style query | Reviewed query |
|---|---|---|
| Orders | 5 | 4 |
| Revenue | 1,820 | 825 |
| Which orders | 1001, 1002, 1003 (counted twice), 1005, 1007 | 1002, 1003, 1006, 1008 |
The flawed number is 121% too high. Look at the order count, though: five versus four. If you sanity-checked the count against a CRM export, you'd shrug and approve it. COUNT(DISTINCT ...) quietly absorbed the duplication that SUM didn't.
That's the pattern Shivam's post names: "Distinct values are not distinct orders." I'd add a corollary. A DISTINCT you didn't ask for is a confession that someone saw duplicates and chose to hide them instead of explaining them.
Where the 995 went: a fix ladder
One fix at a time, real output:
step | revenue | orders
1 flawed as written | 1820 | 5
2 drop payments join | 1530 | 6
3 status = completed | 1270 | 4
4 ET half-open range | 1325 | 5
5 exclude test accounts | 825 | 4
Step 2 is the interesting one. Removing the payments join takes out 400 of double-counted revenue from order 1003, but the order count goes up. The inner join to payments had been silently dropping order 1009, the pending order with no payment. Two bugs were partly cancelling each other out. That happens more than you'd think, and it's why I don't trust "the total looks roughly right" as evidence of anything.
Step 4 also goes up: the BETWEEN with bare dates excluded order 1006 at 9:15 p.m. UTC on 31 October (because '2026-10-31 21:15:00' sorts after '2026-10-31'), and the UTC calendar put 1008 in November and 1001 in October. Fixing the calendar removes one order and adds two.
!A monitor showing code in an editor, slightly out of focus
Photo by Ilya Pavlov on Unsplash (Unsplash License).
The checklist: 8 checks, each one a query that fails loudly
Manuel Diaz left a comment on one of my Dev.to posts that changed how I write these: make each check a query that returns rows only when something is wrong. Zero rows is a pass, and anything else is a conversation. Here's the review in that form.
1. Grain: does the join multiply rows?
Ask: after all the joins, how many rows does each order have, compared with how many lines it really has?
SELECT o.order_id, COUNT(*) AS joined_rows,
(SELECT COUNT(*) FROM order_lines x WHERE x.order_id = o.order_id) AS line_rows
FROM orders o
JOIN order_lines l ON l.order_id = o.order_id
JOIN payments p ON p.order_id = o.order_id
GROUP BY o.order_id
HAVING COUNT(*) > (SELECT COUNT(*) FROM order_lines x WHERE x.order_id = o.order_id);
order_id | joined_rows | line_rows
1003 | 6 | 3
One row back means fail. Order 1003 has three lines and two payments, so the join gives six rows. The usual sources of fan-out are payments, shipments, status history, any table with "events" in its name, and slowly changing dimensions joined without a date condition. The fix is to aggregate to the answer's grain before joining, which is exactly what the CTE in the reviewed query does.
2. Every inner join: what does it drop?
Fan-out inflates; inner joins deflate. For each JOIN, count the left-side rows that have no match. If the number isn't zero, someone has to decide whether those rows should be gone. The payments join dropped order 1009, which happened to be right for the wrong reason.
3. Status filters: list the values, don't guess them
SELECT COALESCE(status,'<NULL>') AS status, COUNT(*) AS n
FROM orders GROUP BY status ORDER BY n DESC;
status | n
completed | 6
refunded | 1
pending | 1
cancelled | 1
status != 'cancelled' is an exclusion list, and exclusion lists rot. Every new status (refunded, on_hold, chargeback) silently joins the revenue. Prefer an inclusion list, status IN ('completed'), that maps to the written definition. Also note that != drops rows where status is NULL, which is a separate decision nobody made.
4. Date boundaries and time zones
SELECT order_id, created_at_utc,
datetime(created_at_utc, '-4 hours') AS created_at_et
FROM orders
WHERE strftime('%m', created_at_utc) <> strftime('%m', datetime(created_at_utc, '-4 hours'))
OR created_at_utc LIKE '2026-10-31 %';
order_id | created_at_utc | created_at_et
1001 | 2026-10-01 02:30:00 | 2026-09-30 22:30:00
1006 | 2026-10-31 21:15:00 | 2026-10-31 17:15:00
1008 | 2026-11-01 02:10:00 | 2026-10-31 22:10:00
Three rows, three questions. What zone are the timestamps stored in? What zone does the business mean by "October"? Does the range include the whole last day? Use half-open ranges (>= start AND < next_start), never BETWEEN on timestamps.
Daylight saving time adds a trap, and this year it sits right on the boundary. US DST ended at 2 a.m. on Sunday, 1 November 2026, so midnight on 1 November in New York was still UTC-4, which is 04:00 UTC. Midnight on 1 December is UTC-5, 05:00 UTC. A hardcoded -4 hours is only right for part of the year. In a warehouse, use its real time-zone conversion function (CONVERT_TIMEZONE, AT TIME ZONE, DATETIME(ts, 'America/New_York')) and check what it does at the switch.
Alex Shev's comment on boundary records is the habit to copy here (comment): put a row a minute either side of every boundary into your test data, deliberately.
5. NULLs in aggregates and in NOT IN
Two different problems, both quiet.
Aggregates skip NULLs, and COUNT(*) doesn't:
SELECT COUNT(*) AS all_rows, COUNT(customer_id) AS non_null_ids,
COUNT(DISTINCT customer_id) AS distinct_ids
FROM support_tickets;
all_rows | non_null_ids | distinct_ids
3 | 2 | 2
If an assistant writes AVG(discount) and a NULL discount means "no discount," the average covers discounted rows only. That might be the definition, but usually it's not. Ask what NULL means in that column, then choose between AVG(x) and AVG(COALESCE(x, 0)) on purpose.
NOT IN is worse. Ask for "customers with no support tickets":
SELECT customer_id, name FROM customers
WHERE customer_id NOT IN (SELECT customer_id FROM support_tickets);
customer_id | name
(0 rows)
Zero. A single ticket with a NULL customer_id makes NOT IN evaluate to unknown for every customer, so nothing comes back, and there's no error. The same question with NOT EXISTS:
SELECT c.customer_id, c.name FROM customers c
WHERE NOT EXISTS (SELECT 1 FROM support_tickets t WHERE t.customer_id = c.customer_id);
customer_id | name
2 | Birch Freight
4 | Dogwood Foods
5 | QA Test Account
And notice the test account turned up again. Every check leaks into the others.
6. The empty set: what should "nothing" return?
SELECT SUM(l.qty*l.unit_price) AS raw_sum,
COALESCE(SUM(l.qty*l.unit_price),0) AS coalesced,
COUNT(*) AS rows_matched
FROM orders o JOIN order_lines l ON l.order_id=o.order_id
WHERE o.created_at_utc >= '2026-12-01 05:00:00' AND o.created_at_utc < '2027-01-01 05:00:00';
raw_sum | coalesced | rows_matched
NULL | 0 | 0
SUM over nothing is NULL. Assistants often wrap it in COALESCE(..., 0) because a 0 looks friendlier on a dashboard. But "zero revenue in December" and "December hasn't loaded yet" are different statements, and the coalesced query can't tell them apart. My rule: return the row count next to the total, and let the definition decide what an empty month displays. I used COALESCE in the reviewed query above because Northstar's definition says October is loaded. That's a claim you should check with a control total from the source system, not assume.
7. Silent DISTINCT
Search the query for DISTINCT. For each one, ask what duplicate it was hiding and why that duplicate exists. If nobody can answer, remove the DISTINCT and run check 1. In the fixture, COUNT(DISTINCT o.order_id) gave a believable order count on top of a revenue figure that was more than double.
SELECT DISTINCT * at the top of a query and UNION (instead of UNION ALL) belong in the same bucket. Both deduplicate without telling you what got merged.
8. Hallucinated columns, and the SQLite trap
Assistants invent plausible column names: region, order_total, is_active. Usually the database throws an error and you're fine:
SELECT order_id, region FROM orders LIMIT 2;
Error: no such column: region
But wrap that invented name in double quotes, as some generators do by habit:
SELECT order_id, "region" FROM orders LIMIT 2;
order_id | "region"
1001 | region
1002 | region
No error. In this SQLite build, a double-quoted identifier that doesn't match a column falls back to a string literal, so every row gets the word "region." Group by it and you get one tidy bucket. Whether this happens depends on how your SQLite build is compiled and configured (look up its "double-quoted string literals" setting); the library build I ran here allows it. Other engines may behave differently, but the habit still matters: for every column the query uses, confirm it exists in the schema and means what the query assumes. A column that exists but means something else, like created_at versus booked_at, is the harder version to catch.
The review card
If you want one thing to copy, it's this. Paste it into the PR description or the ticket next to any AI-drafted query.
| # | Check | Pass looks like | Ask the AI or the author |
|---|---|---|---|
| 0 | Definition written | Grain, statuses, calendar, exclusions, empty-set rule are on paper | "Which line of the definition does each WHERE clause implement?" |
| 1 | Grain / fan-out | Grain check returns 0 rows | "What is one row at each step?" |
| 2 | Inner joins | Dropped-row count is 0 or explained | "Which rows does this join remove?" |
| 3 | Status filter | Inclusion list matching the definition | "List every status value and say why it's in or out." |
| 4 | Dates and zones | Half-open range, stated zone, boundary rows checked | "What happens to a row at 11:59 p.m. on the last day?" |
| 5 | NULLs | NULL meaning stated; NOT EXISTS instead of NOT IN | "Which columns in this query can be NULL?" |
| 6 | Empty set | Row count returned with total | "What does this return for a month with no data?" |
| 7 | DISTINCT / UNION | Each one justified | "What duplicate is this removing?" |
| 8 | Columns exist and mean it | Every column matched to the data dictionary | "Show me where this column is defined." |
| 9 | Control total | Matches the source system within an agreed tolerance | (not a question for the AI) |
Row 9 isn't SQL review in the strict sense. It's Alex Shev's control-total habit, and it's the only check that catches a query that's internally consistent and still wrong because the data underneath is incomplete.
I keep a longer set of these as copy-paste queries in data-validation-checks.sql in my BA Toolkit, and the reconciliation walk-through is in examples/revenue-reconciliation.
What the sharpest commenter will say
"Just give the model the schema and the definition and it'll get it right." Partly. Giving an assistant the definition table above would probably fix the status and test-account bugs. It's the single best thing you can do on the generation side, and it's what the r/LLMDevs commenters meant by a semantic layer. But it doesn't remove the review. The definition can be incomplete (ours says nothing about NULL statuses), and Google's own docs warn the same prompt can give different SQL each time. You'd be reviewing a better first draft, which is nice, not skipping the review.
"Ask the AI to review its own query." I do this, and it does catch things. I treat it like a linter: helpful for the mechanical items (NOT IN, BETWEEN on timestamps), weak on definition items, because the model doesn't know that finance excludes test accounts unless you tell it, and if you told it, it should have used that the first time.
"Benchmarks say 83%, so it's fine most of the time." Benchmarks score whether the result set matches a reference answer. The annotation-error paper suggests the references themselves were wrong a lot of the time. And your schema isn't a benchmark schema. A per-query review is the only number that applies to your query.
When to rerun the review: whenever the query is regenerated, the source schema changes, a new status value appears, the reporting period crosses a DST change or year end, or the definition changes. A reviewed query is reviewed against a definition and a snapshot of data. Change either and the review expires.
What I didn't test
Honest limits of this fixture:
- It's SQLite, with timestamps stored as text. Warehouses store real timestamp types, and time-zone functions differ between engines. Check 4's -4 hours is a teaching shortcut, not a production pattern.
- There's no NULL status row in the data, so I haven't shown != dropping one. Add (1010, 2, NULL, '2026-10-15 12:00:00') to the orders and see what each query does.
- No currency, no tax, no partial refunds, no line-level discounts. Real revenue definitions have all of these, and each one is another grain question.
- I didn't time anything. Some fixes (pre-aggregating in a CTE) change query plans on large tables.
- I wrote the flawed query. It's realistic, but it's mine, so it's a teaching example, not evidence about how often any assistant makes these mistakes.
What I'd do on Monday
If you're reviewing AI-drafted SQL starting this week, here's the order I'd go in.
1. Before opening the query, write the definition table: grain, statuses, calendar and zone, exclusions, empty-set rule. Five lines. If you can't fill it in, the problem isn't the SQL.
2. Build a ten-row fixture with one boundary row on each side of the period, one order with two payments, one test account, one NULL in a key you filter on. Fifteen minutes. Reuse it for every query on that metric.
3. Run the grain check and the status list first. They catch the most, fastest.
4. Ban BETWEEN on timestamps and NOT IN on subqueries in your team's review notes. These two are pure mechanics and easy to agree on.
5. Return the row count with every total. It costs one column and answers the empty-set question for free.
6. Get one control total from the source system for the period and write down the tolerance you accept.
7. Save the reviewed query with its definition next to it, so the next regeneration has something to be compared against.
If you're on the product side setting up an AI analytics tool, my AI agent test plan template is built for the next step: a golden set of questions with known answers, run more than once.
Watch: finding the gap when two numbers don't match
This is my short animated explainer on reconciling two numbers that should agree. It walks through definition, calendar and time zone, grain and fan-out, and orphan rows, which are most of the same failure modes from the reconciliation side.
{% embed youtube.com %}
Video: "Dashboard Numbers Don't Match? Find the Gap in SQL" by Prachit Parikh, on YouTube.
For the testing side of AI tools in general (golden sets, running each case three times, flagging unstable answers), there's AI Agent Testing for Business Analysts: Dev, UAT, Prod.
Discussion
A few questions I'd like real answers to:
1. Who signs off on AI-drafted SQL on your team: the person who wrote the prompt, an engineer, or the metric owner? Has that ever gone wrong?
2. What's a bug you've caught in generated SQL that isn't on this list? I suspect window functions with the wrong PARTITION BY belong here.
3. Should an empty period return 0, NULL, or no row at all? I've seen teams argue all three.
4. Has giving the assistant a written metric definition measurably reduced the fixes you make?
Comment prompt: paste the one-line check you'd add to the review card, written as a query that returns rows only when something's wrong. I'll test the good ones against the Northstar fixture and post the output in reply.
References
- BIRD benchmark leaderboard and news (top test EX 83.40%, human 92.96%, Mini-Dev 2026 update): bird-bench.github.io
- Spider 2.0 leaderboard (Snow and Lite top scores): spider2-sql.github.io
- Spider 2.0 GitHub repository and changelog: xlang-ai/Spider2
- Jin, Choi, Zhu, Kang, "Pervasive Annotation Errors Break Text-to-SQL Benchmarks and Leaderboards," arXiv:2601.08778, January 2026: arxiv.org
- Google Cloud, "Write queries with Gemini assistance" (BigQuery): docs.cloud.google.com
- Stack Overflow 2026 Developer Survey, full results: survey.stackoverflow.co
- Stack Overflow blog, "The results of the 2026 Developer Survey are here," 6 October 2026: stackoverflow.blog
Reader lessons credited above: ontomato, Shivam Pandit, Alex Shev, and Manuel Diaz.
Related posts of mine: 7 SQL checks I run before UAT sign-off, Data mapping document template, and AI agents can write the code now. The bottleneck is the requirements. More of my work is on my portfolio page.
This article was created with the help of AI. Every query and output above was run against the Northstar fixture in SQLite and checked before publishing.
About me
I'm Prachit Parikh, a business analyst. I turn what stakeholders ask for into requirements a team can test, check the data with SQL, and see changes through UAT. I keep free templates, including the SQL data validation checks behind this review card, in my BA Toolkit on GitHub, and I make short animated explainers on YouTube. Besides Dev.to, I write on Substack, Medium, Hashnode, Blogger and WordPress, and I answer questions on Quora. You can also find me on LinkedIn, Bluesky, Kaggle, Credly, Crunchbase and Gravatar. Everything is collected on my website and my profile hub.
Why computer use benchmarks are so hard to run
Let's say you built a computer use agent and want to know how good it is. The usual way is OSWorld, a popular benchmark of 108 tasks on a real desktop. It is easy to start and hard to get right. I read about 200 GitHub issues to see what goes wrong. Here are the 10 problems you will face, and how I would handle each.
The first hard part is the machines. A coding benchmark gets by with a container and a shell. A computer use benchmark needs a whole desktop for every attempt, with its own apps and websites, and the agent takes hundreds of steps per task.
The second is that one run may not be enough. Agents are not deterministic, so a single run can come out higher or lower just by luck. That is why it is good to run the benchmark a few times and average the results. Three runs of OSWorld 2.1 are 324 attempts and roughly 400 machine-hours.
All that machine time is a plain reason so few published scores get re-run. My last article found it on OSWorld 2.0:
{% embed x.com %}
Every September flagship number there is self-reported, and the maintainers' own board has not run those models yet.
Most teams run once: in Yutori's tracker of OSWorld-Verified results, 94 of the 111 entries that list a run count are a single run. Some do repeat: Anthropic averaged five runs for Claude Fable 5.1 on OSWorld 2.0, and Yutori reported three runs for its own Navigator n2.
The 400 is my estimate from Simular's numbers. Simular published every run of its Sai agent on OSWorld 2.1. It took 14 runs and reruns to get one clean result on every task, and that clean pass took about 139 machine-hours, with a median task of 46 minutes and the longest at 8.9 hours. Three passes like that come to roughly 400. OSWorld-Verified, with 369 shorter tasks, costs about the same: Simular reports 4 to 5 hours per full run on 30 parallel environments, close to one 2.1 pass in machine time.
!Chart titled One clean pass of OSWorld 2.1, task by task: every one of the 108 tasks as a dot on a time axis from 15 minutes to 8 hours, 69 under an hour and 39 over, median 46 minutes, mean 77 minutes, longest 8.9 hours.
Each of the ten can change the score, and they come in order: on the way from one desktop to a daily run, each fix exposes the next one.
The problems come from other people's runs
I did not run a campaign. Every problem below was reported by people who did, in issues and pull requests on OSWorld, SWE-bench and Terminal-Bench. Most are known, and many are already fixed. The "how I would handle it" parts are my design, and some of it comes from agent runtimes I built before.
The ten problems
| Problem | What it does to the number | What to check |
|---|---|---|
| 1. Capacity | The slowest tasks hold the run | Valid results per hour |
| 2. Preparation | Setup time hides inside task time | Each readiness step |
| 3. Rate limits | Throttled tasks end early | Calls per attempt against quota |
| 4. Isolation | Attempts change each other's results | Two attempts at once |
| 5. Runtime failures | Crashes look like agent failures | Timeouts and real health checks |
| 6. Retries | Actions run twice; best-of reads as mean | When the retry happened |
| 7. Grading | Infrastructure errors become zeros | Ran, graded, evidence kept |
| 8. Reproducibility | Same model, different number | The run manifest |
| 9. Observability | Stuck looks like slow | One ID per attempt |
| 10. Cost | Cheap runs that prove nothing | Cost per valid result |
What happens in one attempt
An attempt is one task run once, and it has four parts. The benchmark gives the task and the checks. The desktop gives the apps and the starting state. The agent decides what to do. The harness runs the loop between them: screenshot in, action out, until the agent stops. Then a grader checks what is left on the machine.
!Diagram titled One attempt: the agent harness sits between the prepared sandbox and the model endpoint, sending observations and requests and returning decisions and actions, about 318 tool calls per task, and records a step trace of every step in the sandbox; when the episode ends, the sandbox final state is handed to the grader, which produces the score and evidence along with the step trace.
In OSWorld 2.1 the desktop is a QEMU virtual machine inside a Docker container, with 4 GB of RAM and 4 virtual CPUs by default. The agent sees screenshots, and in the OSWorld 2.0 paper Claude Opus 4.7 averaged 318 tool calls per task.
Once one task works, you know the model can use the desktop. What you need to know next is how long all 324 attempts take and how many of their results you can trust.
1. Capacity: rent the desktops or run them yourself
On one desktop, the 324 attempts run one after another, and 400 machine-hours means 400 hours of waiting. Parallel desktops are the first answer, and they work on the wait, not the work: the total stays 400 machine-hours. On 40 desktops that is 10 hours if everything packs perfectly, and it never does, because the run ends when the slowest task ends. Simular reports 4 to 5 hours per OSWorld run on 30 parallel environments, sometimes 6, while the run waits for its last tasks.
!Chart titled Parallelism changes elapsed time, not total work: on a log scale, wall time for one update falls from 416 hours on 1 desktop to 27.0 hours on 16, 11.6 hours on 40 and 3.9 hours on 108, while a dashed line holds total work at 416 machine-hours at every width; 40 desktops need 160 GB of RAM and 160 virtual CPUs at the default size.
Parallel desktops need real hardware. Forty of them at the default size is 160 GB of guest RAM and 160 virtual CPUs before anything else runs on the host. Tianbao Xie, who leads OSWorld, described what AWS changed for them:
{% embed x.com %}
That hardware can move the score as well as the speed. Anthropic found that with strict per-task CPU and memory limits, up to 6% of Terminal-Bench tasks failed on pod errors unrelated to the model, and loosening the limits moved scores by up to 6 points. Their summary:
{% embed x.com %}
So capacity is a question of managing containers, one per desktop, and there are two ways to get them. One is to rent. The OSWorld team moved to AWS, and sandbox providers such as Cua now run the official OSWorld image on their own cloud. Their limits then become yours: one OSWorld user hit their AWS account's vCPU limit at five desktops.
The other is to run them on your own servers, which makes you the orchestrator. You plan per container: 4 GB of RAM and 4 virtual CPUs each, plus headroom, because tight limits are what cost those 6 points. On one server, the OSWorld team ran 8 or 16 desktops at a time.
Either way, I would size one container first, then add more in steps and watch completed, validly graded attempts per hour. If 32 desktops are already waiting on the model API, 64 mostly doubles the bill.
2. Preparation: boot desktops before the attempts need them
Once you have the machines, the next wait is getting each one ready for its task. Ready is four steps: the image is cached on the host, the VM has booted, control works (the screenshot server answers and the apps open), and the task's files and websites are in place. Each step can be slow on its own.
!Diagram titled Ready has four steps: 324 attempts wait in a queue for a warm pool of 10 desktops that are already cached, booted and control-verified; only the task setup happens per attempt, and a desktop goes back to the pool only after a verified reset.
The OSWorld team saw LibreOffice take a minute to open on first launch, because AWS loads disks restored from snapshots lazily. They cut the image from 50 GB to 25 GB, and first launch went to 5 seconds. Coding benchmarks have the same problem with bigger images. Epoch found SWE-bench's task images totaled 684 GiB and rebuilt them down to 67:
{% embed x.com %}
The usual answer is a warm pool: a few desktops booted and checked before they are needed. I built one for agent containers in another system, and in the case I measured a deploy went from about 60 seconds to under a second. For a benchmark, warm also has to mean clean, so a desktop goes back to the pool only after a verified reset. Ten clean desktops keep up with attempts as they arrive. Keeping all 324 warm in advance is a separate spending decision.
So I would gate each attempt on every readiness step, pin the image by digest, and cache it near the workers.
3. Rate limits: every service the run touches has a limit
With the desktops ready, the agents start calling the model. At 318 tool calls per task, 324 attempts come to about 100,000 tool calls driven by the model, most of them carrying a screenshot.
Providers cap that traffic. Princeton's HAL team ran 21,730 agent rollouts across nine benchmarks and found that parallel evaluation runs into rate limits at every provider. When they ran, Anthropic capped API spending at $5,000 a month by default, even on its highest usage tier, so larger runs needed special approval. The tiers have changed since, but the standard ones still have a monthly cap, and reaching it pauses the API with the same 429 error a rate limit returns.
The model API is one of several services a run depends on, and each one has its own limit. Nebius hit rate limits on PyPI, Docker Hub and the Ubuntu package archive while building SWE-bench task images, before any model was involved. Benchmarks that touch the live web meet bot detection, which HUD described in a walkthrough on browser evals:
{% embed x.com %}
A provider's content policy can stop a task outright. When Cua tried to run its benchmark on Muse, a KiCad task was flagged as a violation:
{% embed x.com %}
In OSWorld, the grader and the simulated user can call models as well as the agent, and all of them draw on the same quota unless you split it.
So I would do most of the work before the run starts. First, confirm with the provider the limits on the tier you will actually run on, requests and tokens per minute and the monthly spend cap, and ask for more before the campaign, not in the middle of it. Pull every image and package once into your own mirror, so the public registries see one download instead of one per desktop. Give live-web tasks proxies you control, and pin one IP to any task where a site checks for humans; the OSWorld team did both. And run each task once on the provider you plan to use, so a content policy flag shows up before the campaign instead of as a zero inside it.
During the run, I would put one admission controller in front of the shared model budget. It counts the grader and the simulated user against that budget, starts a desktop only when there is capacity for its traffic, and records a rate-limited attempt as an infrastructure failure to rerun, never as a zero.
4. Isolation: give each attempt its own world, then test it
With attempts running side by side, each one needs a clean world, and a fresh VM only cleans the desktop. A task can post a comment, create an account or upload a file, and those writes live on a server. The next attempt inherits them, and an attempt running at the same time can see them.
!Diagram titled One identity per attempt: one attempt ID runs like a line through the desktop, the website services, the grader, the browser, the harness and the evidence.
WebArena's tooling handles this by running tasks in a fixed order, and its maintainers note that dropping the order for more parallelism can cost 1 to 2 percent. OSWorld 2.0 took the other route: it ships 31 self-hosted websites and scopes each attempt's state by cookie, so attempts do not contaminate each other.
A clean world also means one without the answer in it. SWE-bench images once kept the repository's future commits, and some agents found the fix with git log. The maintainers rebuilt the images and measured the effect as small, about 5 in 10,000 trajectories.
So I would give each attempt one identity across desktop, browser and website, and test it directly: run two attempts at once, change each world differently, and check that neither sees the other.
5. Runtime failures: the desktop dies before the agent does
An attempt that runs for an hour or more outlasts some of the parts under it. One of them crashes, expires or hangs, and the attempt fails for a reason that is not the agent's.
A crash can come from the image itself. OSWorld's in-VM server leaked a display connection on every screenshot, so long tasks crashed after about 220 steps. The fix went into the code in May, but the published image is older than the fix. In the logs it looks like a network error at a random step, until someone counts the steps.
Clocks end attempts the same way. OSWorld 2.1's AWS provider can schedule a VM's termination after 300 minutes by default, while H Company's Holo4 notes list a per-task budget of 500 steps and 6 hours, an hour more than that VM lives. Tokens, sandboxes and leases all have lifetimes, and each has to be longer than the longest task.
A hang is harder to see, because nothing fails. An open pull request on OSWorld found that 10 of 14 controller calls into the VM had no timeout, so a desktop that stops answering leaves the call waiting forever. I built a watchdog for the same shape of problem in an agent runtime: the process looks alive, but its event loop has frozen and nothing answers.
So I would put a timeout on every call, check health by asking the desktop to do something real, and set every lifetime above the task budget.
6. Retries: mean of three and best of three are different scores
Once failures happen, the instinct is to retry. Before the agent's first action that is safe, because the desktop has not changed yet. After it, the desktop has state, and a retry can repeat something that already happened.
An open OSWorld pull request shows how that goes unnoticed. The desktop stops any command that runs past 120 seconds and reports an error, and the controller retried every error, including commands the desktop had already partly run. In that runner's logs, 357 of 23,244 actions ran twice. When the repeated command was a ctrl key press, the key stayed down for the rest of the task, and the trajectory looked normal. Harbor's desktop runtime has the rule in a code comment: replaying a click is not idempotent.
Ashton Chew's input daemon for computer use on Modal makes the same call: once a press may have reached the X server, it releases what it pressed, returns the error, and does not retry.
{% embed x.com %}
Retries at different layers multiply. A Harbor issue describes SDK and harness retries stacking until one hung call could take about 90 minutes.
Rerunning a whole task brings back the policy question from my last article, which run counts. Simular's published runs give 0.7674 if you count the first clean run per task and 0.7938 if you count the best one, 2.6 points apart from the same campaign.
So I would retry freely before the first action, start and log a new attempt after it, give every layer one shared deadline, and say in the report whether the score is a mean of three or a best of three.
7. Grading: check that it ran, was graded and kept its evidence
When the run ends, you have a folder of result files, each with a score. A zero in that folder can mean the agent failed. It can also mean the grader crashed, the output went missing, or the evidence never uploaded, and the file looks the same in every case.
Simular hit this while making its OSWorld 2.1 reference runs. A usage limit on the model key stopped tasks after one step, and they were logged as success with a score of 0. The average over the folder came out at 0.40 and, in their words, looked like a measurement. The step count was what gave it away.
!Diagram titled An infrastructure error is not a zero: result files pass through acceptance checks (did it run, was it graded, is the evidence there); all pass goes to an accepted score, including a real zero, any failure goes to an invalid attempt that keeps its cause, such as a usage limit, a grader crash or missing evidence.
Graders can write the same false zero themselves. The OSWorld maintainers have an open fix for 14 of the 108 tasks, whose graders turn a failed call to the judge model into a zero.
The grader also has to be out of the agent's reach. Some graders read the answer by running commands inside the VM the agent controlled. An engineer at H Company suggested loading the grading code in memory, so it never touches a disk the agent can read.
Teams that publish numbers decide before the run how infrastructure failures count. An engineer at poolside described their rule:
{% embed x.com %}
So I would record three things per attempt: did it run, was it graded, and is the evidence there. I would run the reference solutions before each campaign, because a grader that fails the right answer shows up there first. One team logged 240 SWE-bench rollouts at 0% and blamed their agent until the gold patches failed too. And I would report coverage, the share of attempts that were validly graded, next to the score.
8. Reproducibility: pin everything the number depends on
Once a run is reliable, you run it again after the next harness change. If the number moves, you need to know whether your change moved it.
The harness alone can do it. Anthropic re-ran older models after fixing a bug in its zoom tool and raising the token limit per turn, and wrote that it had been underreporting OSWorld across its model families. Claude Opus 4.7 went from 78.0% to 82.8% with no change to the model.
The rest of the setup moves too. Between OSWorld 2.0 and 2.1, 174 pull requests to the task files were merged. Simular ran the same agent on two desktop images and got 0.7766 on one and 0.7414 on the other. HAL notes that OpenRouter can serve different quantizations of a model under the same endpoint name, so the same name does not guarantee the same model.
So every run gets a manifest, with a line for each thing that can move the number:
- task and grader revisions
- image digest and website builds
- the model and the provider route that actually served it
- the judge model
- harness commit
- observation and action settings, screen size
- step and time budgets
- resources per desktop
- retry and repeat policy: mean or best of three
A manifest does not make a model deterministic. It tells you what stayed the same when the number moved.
9. Observability: you need to tell slow from stuck
Hours into a run, you want to know which attempts are working, which are slow and which are stuck. The first thing you reach for is a live view of one desktop. Terminal-Bench had one early:
{% embed x.com %}
A live view shows one screen. It cannot tell you whether twenty other attempts are stuck in preparation or waiting on the model. A Harbor user spent about two days on a stall like that. Trials sat silent until a 30-minute timeout. The cause was two crashed trials whose 4 GB sandboxes kept running and filled the account's 10 GB memory limit, so every new sandbox was refused and retried without a word.
You need the same visibility after the run, to explain a score. Simular found that OSWorld 2.1's rubric graders return only the total, so a partial score like 0.6969 cannot be explained or re-graded later.
So I would carry one attempt ID from allocation through model calls, desktop actions, grading and cleanup, record how long each phase took, and keep the evidence of failed attempts instead of deleting it.
10. Cost: count cost per valid result
Once the run works every day, it has a bill, and the model is only one line of it. The others are desktops, idle warm capacity, retries, failed preparation, storage, and anything left running after a crash.
Simular published its model spend for the OSWorld 2.1 reference campaign: $2,417.63 over 27,662 gateway calls, about $18 per clean attempt. Desktop hours are not in that number.
Leftovers add up quickly. OSWorld's Docker provider once left a 32 GB volume behind for every task, enough to fill a disk within an hour of a full run, until it was fixed.
At scale, cost decides the method. HAL spent about $40,000 on its rollouts and still had to rely on single runs for most evaluations. Some models they did not run at all:
{% embed x.com %}
So I would track total cost divided by validly graded attempts, shown next to coverage. A cheap run that failed early has few valid attempts, so its cost per result goes up. A run that skipped the hard tasks shows up as lower coverage.
A daily run has to hold all ten at once
Every fix in this article is ordinary on its own. The trouble is that each one exposes the next problem, so a daily run has to hold all ten at once. Here is what that takes: eight stations every attempt passes through, and one attempt ID from allocation to cleanup.
!Matrix titled What a daily run needs: eight stations, queue, admission, ready desktop, attempt, watchdog, grading, evidence and report, against the ten problems, with a dot where each station answers a problem, and one attempt ID from allocation to cleanup.
You can find each station somewhere today. Parallel desktops, warm pools, retries and dashboards exist in OSWorld's own runner, Harbor, HUD, Cua and the sandbox providers. One setup that holds all of them, on the benchmark's exact package, every day, is still rare. That setup is the hardest benchmark.
What to take away
A benchmark score is the output of an operation. Before you trust one, yours or a lab's, ask how it was run.
Save the table and the manifest. They work for SWE-bench and Terminal-Bench too.
I build harnesses that let agents use computers. If you want the full notes with every source, reply and I will send them.
---
Originally published on arthurkatcher.com.
1 reaction · 3 comments
Mentions
Review AI-generated SQL before you trust the number: a checklist with a runnable fixture !A dark code editor with lines of colorful source code Photo by Chris Ried on Unsplash (Unsplash License). A query that runs is not a query that's right. Most of us know this, and few of us act on it when an assistant hands back a query in four seconds and the number looks about right. This is the review I run on AI-written SQL before its number goes anywhere near a sign-off. Eight checks, each written as a query that returns rows only when something is wrong. It comes with a tiny fictional fixture you can paste into SQLite, where a perfectly plausible revenue query returns 1,820 and the reviewed one returns 825. Same tables, same question, same month. Nothing in the bad query is exotic. That's sort of the point. Why the "it ran" signal got weaker, not stronger The text-to-SQL benchmarks look great right now, and they're worth reading closely before you lean on them. On the BIRD leaderboard, the top test execution accuracy as of 22 October 2026 is 83.40% (AskData, from AT&T's CDO team), with GrainSQL at 82.95% and Huawei's DataGallery at 82.39%. The human baseline on the same page, from data engineers and database students, is 92.96%. So the best system still misses roughly one question in six, and that's on a curated benchmark with clean schemas. Numbers like these are easy to misread. Spider 2.0 uses enterprise-style workflows on Snowflake and BigQuery, and its top Spider 2.0-Snow entries sit around 96.70%, while the top Spider 2.0-Lite score is 76.23%. Same family of tasks, very different numbers. The Spider 2.0 repo also has a long changelog of evaluation fixes and ambiguity cleanups, and as of August 2026 a suspended Snowflake evaluation account was disrupting Snow access. Then there's the paper that made me stop quoting leaderboard numbers at all. In Pervasive Annotation Errors Break Text-to-SQL Benchmarks and Leaderboards (Jin, Choi, Zhu and Kang, January 2026), expert review found annotation error rates of 52.8% in BIRD Mini-Dev and 62.8% in Spider 2.0-Snow. When they re-scored 16 open-source agents on a corrected BIRD Dev subset, relative performance moved by -7% to +31% and rankings shifted by up to nine places. BIRD has since shipped a 2026 Mini-Dev update with corrected annotations (28 October 2026), which is the right response. But think about what it means: the people grading the SQL got it wrong more than half the time on some subsets. The "gold" queries had the same class of mistakes I'm about to show you. The vendors say it plainly too. Google's Gemini in BigQuery docs say the model "can generate output that seems plausible but is factually incorrect," recommend you "validate all output," and note it "might suggest different syntax each time that you enter the same prompt." That last line matters for review. If you regenerate, you're reviewing a new query. And developers know it. In the Stack Overflow 2026 Developer Survey, the most common answer to "How much do you trust the output from AI tools or AI agents as part of your workflow?" was "I trust it when I can easily verify the output," at 48.0% of 14,304 respondents. Only 6.6% said they trust it for many tasks including important work decisions. (The survey announcement has the broader picture: more than 30,000 people took it.) A revenue number in a steering-committee deck is an important work decision. So for an analyst the question is very practical: what does "easily verify" look like for SQL? The take you can argue with Here's my position. The person who reviews AI-written SQL before sign-off should be the person who owns the metric definition, not the strongest SQL writer in the room. On most teams that's the BA or the analyst, not the engineer. The reason is in the fixture below. None of the five bugs is a syntax problem. Every one is a definition problem: which statuses count, whose calendar, which customers, what one row means. An engineer who writes beautiful SQL but wasn't in the room when finance said "test accounts don't count" will approve the query, and they'll approve it faster if it looks tidy. You can disagree with me on this one. A fair counter is that BAs miss performance and correctness issues an engineer catches right away, like a NOT IN against a nullable column. True, and that's why the checklist includes those. But I'd still rather teach the definition owner eight mechanical checks than teach the strongest SQL writer every one of the business rules. The fixture: Northstar Systems, October revenue Northstar Systems is a fictional B2B software company, and every row below is invented. Here's the question someone typed into an assistant: > "What was Northstar's revenue from completed orders in October 2026, and how many orders?" Before any SQL, write the definition down. A reader taught me this habit: name the grain first, "one row = one order line" (ontomato's comment). Shivam Pandit makes the same point at metric level in Defining revenue for an SQL agent, one row at a time: a definition an agent can use has to state grain, statuses, join behaviour and what an empty result means. So here's Northstar's: | Part | Definition | |---|---| | Grain of the source | order_lines: one row = one order line | | Grain of the answer | one row = one order, then one total | | Revenue | sum of qty * unit_price over lines | | Statuses | completed only. Not refunded, pending or cancelled | | Calendar | October in America/New_York. Timestamps are stored in UTC | | Exclusions | customers flagged is_test = 1 | | Empty month | report 0 only if the month's data is loaded; otherwise "no data" | Now the schema and data. It runs as-is in the sqlite3 shell or any SQLite client. -- Northstar Systems (fictional). All data invented. CREATE TABLE customers (customer_id INTEGER PRIMARY KEY, name TEXT, is_test INTEGER NOT NULL DEFAULT 0); CREATE TABLE orders (order_id INTEGER PRIMARY KEY, customer_id INTEGER, status TEXT, created_at_utc TEXT); CREATE TABLE order_lines (line_id INTEGER PRIMARY KEY, order_id INTEGER, sku TEXT, qty INTEGER, unit_price INTEGER); CREATE TABLE payments (payment_id INTEGER PRIMARY KEY, order_id INTEGER, amount INTEGER, paid_at_utc TEXT); CREATE TABLE support_tickets (ticket_id INTEGER PRIMARY KEY, customer_id INTEGER, opened_at_utc TEXT); INSERT INTO customers VALUES (1,'Alder Clinic',0),(2,'Birch Freight',0),(3,'Cedar Labs',0),(4,'Dogwood Foods',0),(5,'QA Test Account',1); INSERT INTO orders VALUES (1001,1,'completed','2026-10-01 02:30:00'), (1002,2,'completed','2026-10-05 15:00:00'), (1003,3,'completed','2026-10-12 18:20:00'), (1004,1,'cancelled','2026-10-14 09:00:00'), (1005,4,'refunded', '2026-10-20 13:45:00'), (1006,2,'completed','2026-10-31 21:15:00'), (1007,5,'completed','2026-10-22 16:00:00'), (1008,3,'completed','2026-11-01 02:10:00'), (1009,4,'pending', '2026-10-25 11:30:00'); INSERT INTO order_lines (order_id,sku,qty,unit_price) VALUES (1001,'NS-BASE',1,120), (1002,'NS-PRO',1,200),(1002,'NS-ADDON',1,50), (1003,'NS-PRO',1,300),(1003,'NS-ADDON',1,80),(1003,'NS-SETUP',1,20), (1004,'NS-BASE',1,90), (1005,'NS-PRO',1,150), (1006,'NS-BASE',1,60),(1006,'NS-ADDON',1,40), (1007,'NS-ENT',1,500), (1008,'NS-BASE',1,75), (1009,'NS-PRO',1,110); INSERT INTO payments (order_id,amount,paid_at_utc) VALUES (1001,120,'2026-10-01 02:31:00'), (1002,250,'2026-10-05 15:02:00'), (1003,200,'2026-10-12 18:25:00'),(1003,200,'2026-10-26 10:00:00'), (1005,150,'2026-10-20 13:50:00'), (1006,100,'2026-10-31 21:16:00'), (1007,500,'2026-10-22 16:01:00'), (1008,75,'2026-11-01 02:12:00'); INSERT INTO support_tickets (customer_id,opened_at_utc) VALUES (1,'2026-10-03 14:00:00'),(3,'2026-10-15 10:00:00'),(NULL,'2026-10-18 09:00:00'); Three rows are planted on purpose. Order 1001 is 2:30 a.m. UTC on 1 October, which is 10:30 p.m. on 30 September in New York. Order 1008 is the reverse: November in UTC, October in New York. Order 1003 was paid in two instalments. The query that looks fine This is an illustrative example I wrote myself, in the shape I see assistants produce for this kind of prompt. I'm not claiming any specific tool wrote it. Read it once and note what you'd flag before scrolling. SELECT COUNT(DISTINCT o.order_id) AS orders, SUM(l.qty * l.unit_price) AS revenue FROM orders o JOIN order_lines l ON l.order_id = o.order_id JOIN payments p ON p.order_id = o.order_id WHERE o.status != 'cancelled' AND o.created_at_utc BETWEEN '2026-10-01' AND '2026-10-31'; It looks tidy, and the aliases are sensible. Joining payments even feels responsible, as if it's making sure we only count money that actually came in. Here's the output (SQLite 3.53.1, run through Python's sqlite3 module): orders | revenue 5 | 1820 The reviewed query WITH order_totals AS ( -- grain: one row = one order SELECT order_id, SUM(qty * unit_price) AS order_revenue FROM order_lines GROUP BY order_id ) SELECT COUNT(*) AS orders, COALESCE(SUM(t.order_revenue), 0) AS revenue FROM orders o JOIN order_totals t ON t.order_id = o.order_id JOIN customers c ON c.customer_id = o.customer_id WHERE o.status = 'completed' AND c.is_test = 0 -- October in America/New_York (EDT, UTC-4 all month), half-open range AND o.created_at_utc >= '2026-10-01 04:00:00' AND o.created_at_utc < '2026-11-01 04:00:00'; orders | revenue 4 | 825 Side by side: | | Illustrative AI-style query | Reviewed query | |---|---|---| | Orders | 5 | 4 | | Revenue | 1,820 | 825 | | Which orders | 1001, 1002, 1003 (counted twice), 1005, 1007 | 1002, 1003, 1006, 1008 | The flawed number is 121% too high. Look at the order count, though: five versus four. If you sanity-checked the count against a CRM export, you'd shrug and approve it. COUNT(DISTINCT ...) quietly absorbed the duplication that SUM didn't. That's the pattern Shivam's post names: "Distinct values are not distinct orders." I'd add a corollary. A DISTINCT you didn't ask for is a confession that someone saw duplicates and chose to hide them instead of explaining them. Where the 995 went: a fix ladder One fix at a time, real output: step | revenue | orders 1 flawed as written | 1820 | 5 2 drop payments join | 1530 | 6 3 status = completed | 1270 | 4 4 ET half-open range | 1325 | 5 5 exclude test accounts | 825 | 4 Step 2 is the interesting one. Removing the payments join takes out 400 of double-counted revenue from order 1003, but the order count goes up. The inner join to payments had been silently dropping order 1009, the pending order with no payment. Two bugs were partly cancelling each other out. That happens more than you'd think, and it's why I don't trust "the total looks roughly right" as evidence of anything. Step 4 also goes up: the BETWEEN with bare dates excluded order 1006 at 9:15 p.m. UTC on 31 October (because '2026-10-31 21:15:00' sorts after '2026-10-31'), and the UTC calendar put 1008 in November and 1001 in October. Fixing the calendar removes one order and adds two. !A monitor showing code in an editor, slightly out of focus Photo by Ilya Pavlov on Unsplash (Unsplash License). The checklist: 8 checks, each one a query that fails loudly Manuel Diaz left a comment on one of my Dev.to posts that changed how I write these: make each check a query that returns rows only when something is wrong. Zero rows is a pass, and anything else is a conversation. Here's the review in that form. 1. Grain: does the join multiply rows? Ask: after all the joins, how many rows does each order have, compared with how many lines it really has? SELECT o.order_id, COUNT(*) AS joined_rows, (SELECT COUNT(*) FROM order_lines x WHERE x.order_id = o.order_id) AS line_rows FROM orders o JOIN order_lines l ON l.order_id = o.order_id JOIN payments p ON p.order_id = o.order_id GROUP BY o.order_id HAVING COUNT(*) > (SELECT COUNT(*) FROM order_lines x WHERE x.order_id = o.order_id); order_id | joined_rows | line_rows 1003 | 6 | 3 One row back means fail. Order 1003 has three lines and two payments, so the join gives six rows. The usual sources of fan-out are payments, shipments, status history, any table with "events" in its name, and slowly changing dimensions joined without a date condition. The fix is to aggregate to the answer's grain before joining, which is exactly what the CTE in the reviewed query does. 2. Every inner join: what does it drop? Fan-out inflates; inner joins deflate. For each JOIN, count the left-side rows that have no match. If the number isn't zero, someone has to decide whether those rows should be gone. The payments join dropped order 1009, which happened to be right for the wrong reason. 3. Status filters: list the values, don't guess them SELECT COALESCE(status,'<NULL>') AS status, COUNT(*) AS n FROM orders GROUP BY status ORDER BY n DESC; status | n completed | 6 refunded | 1 pending | 1 cancelled | 1 status != 'cancelled' is an exclusion list, and exclusion lists rot. Every new status (refunded, on_hold, chargeback) silently joins the revenue. Prefer an inclusion list, status IN ('completed'), that maps to the written definition. Also note that != drops rows where status is NULL, which is a separate decision nobody made. 4. Date boundaries and time zones SELECT order_id, created_at_utc, datetime(created_at_utc, '-4 hours') AS created_at_et FROM orders WHERE strftime('%m', created_at_utc) <> strftime('%m', datetime(created_at_utc, '-4 hours')) OR created_at_utc LIKE '2026-10-31 %'; order_id | created_at_utc | created_at_et 1001 | 2026-10-01 02:30:00 | 2026-09-30 22:30:00 1006 | 2026-10-31 21:15:00 | 2026-10-31 17:15:00 1008 | 2026-11-01 02:10:00 | 2026-10-31 22:10:00 Three rows, three questions. What zone are the timestamps stored in? What zone does the business mean by "October"? Does the range include the whole last day? Use half-open ranges (>= start AND < next_start), never BETWEEN on timestamps. Daylight saving time adds a trap, and this year it sits right on the boundary. US DST ended at 2 a.m. on Sunday, 1 November 2026, so midnight on 1 November in New York was still UTC-4, which is 04:00 UTC. Midnight on 1 December is UTC-5, 05:00 UTC. A hardcoded -4 hours is only right for part of the year. In a warehouse, use its real time-zone conversion function (CONVERT_TIMEZONE, AT TIME ZONE, DATETIME(ts, 'America/New_York')) and check what it does at the switch. Alex Shev's comment on boundary records is the habit to copy here (comment): put a row a minute either side of every boundary into your test data, deliberately. 5. NULLs in aggregates and in NOT IN Two different problems, both quiet. Aggregates skip NULLs, and COUNT(*) doesn't: SELECT COUNT(*) AS all_rows, COUNT(customer_id) AS non_null_ids, COUNT(DISTINCT customer_id) AS distinct_ids FROM support_tickets; all_rows | non_null_ids | distinct_ids 3 | 2 | 2 If an assistant writes AVG(discount) and a NULL discount means "no discount," the average covers discounted rows only. That might be the definition, but usually it's not. Ask what NULL means in that column, then choose between AVG(x) and AVG(COALESCE(x, 0)) on purpose. NOT IN is worse. Ask for "customers with no support tickets": SELECT customer_id, name FROM customers WHERE customer_id NOT IN (SELECT customer_id FROM support_tickets); customer_id | name (0 rows) Zero. A single ticket with a NULL customer_id makes NOT IN evaluate to unknown for every customer, so nothing comes back, and there's no error. The same question with NOT EXISTS: SELECT c.customer_id, c.name FROM customers c WHERE NOT EXISTS (SELECT 1 FROM support_tickets t WHERE t.customer_id = c.customer_id); customer_id | name 2 | Birch Freight 4 | Dogwood Foods 5 | QA Test Account And notice the test account turned up again. Every check leaks into the others. 6. The empty set: what should "nothing" return? SELECT SUM(l.qty*l.unit_price) AS raw_sum, COALESCE(SUM(l.qty*l.unit_price),0) AS coalesced, COUNT(*) AS rows_matched FROM orders o JOIN order_lines l ON l.order_id=o.order_id WHERE o.created_at_utc >= '2026-12-01 05:00:00' AND o.created_at_utc < '2027-01-01 05:00:00'; raw_sum | coalesced | rows_matched NULL | 0 | 0 SUM over nothing is NULL. Assistants often wrap it in COALESCE(..., 0) because a 0 looks friendlier on a dashboard. But "zero revenue in December" and "December hasn't loaded yet" are different statements, and the coalesced query can't tell them apart. My rule: return the row count next to the total, and let the definition decide what an empty month displays. I used COALESCE in the reviewed query above because Northstar's definition says October is loaded. That's a claim you should check with a control total from the source system, not assume. 7. Silent DISTINCT Search the query for DISTINCT. For each one, ask what duplicate it was hiding and why that duplicate exists. If nobody can answer, remove the DISTINCT and run check 1. In the fixture, COUNT(DISTINCT o.order_id) gave a believable order count on top of a revenue figure that was more than double. SELECT DISTINCT * at the top of a query and UNION (instead of UNION ALL) belong in the same bucket. Both deduplicate without telling you what got merged. 8. Hallucinated columns, and the SQLite trap Assistants invent plausible column names: region, order_total, is_active. Usually the database throws an error and you're fine: SELECT order_id, region FROM orders LIMIT 2; Error: no such column: region But wrap that invented name in double quotes, as some generators do by habit: SELECT order_id, "region" FROM orders LIMIT 2; order_id | "region" 1001 | region 1002 | region No error. In this SQLite build, a double-quoted identifier that doesn't match a column falls back to a string literal, so every row gets the word "region." Group by it and you get one tidy bucket. Whether this happens depends on how your SQLite build is compiled and configured (look up its "double-quoted string literals" setting); the library build I ran here allows it. Other engines may behave differently, but the habit still matters: for every column the query uses, confirm it exists in the schema and means what the query assumes. A column that exists but means something else, like created_at versus booked_at, is the harder version to catch. The review card If you want one thing to copy, it's this. Paste it into the PR description or the ticket next to any AI-drafted query. | # | Check | Pass looks like | Ask the AI or the author | |---|---|---|---| | 0 | Definition written | Grain, statuses, calendar, exclusions, empty-set rule are on paper | "Which line of the definition does each WHERE clause implement?" | | 1 | Grain / fan-out | Grain check returns 0 rows | "What is one row at each step?" | | 2 | Inner joins | Dropped-row count is 0 or explained | "Which rows does this join remove?" | | 3 | Status filter | Inclusion list matching the definition | "List every status value and say why it's in or out." | | 4 | Dates and zones | Half-open range, stated zone, boundary rows checked | "What happens to a row at 11:59 p.m. on the last day?" | | 5 | NULLs | NULL meaning stated; NOT EXISTS instead of NOT IN | "Which columns in this query can be NULL?" | | 6 | Empty set | Row count returned with total | "What does this return for a month with no data?" | | 7 | DISTINCT / UNION | Each one justified | "What duplicate is this removing?" | | 8 | Columns exist and mean it | Every column matched to the data dictionary | "Show me where this column is defined." | | 9 | Control total | Matches the source system within an agreed tolerance | (not a question for the AI) | Row 9 isn't SQL review in the strict sense. It's Alex Shev's control-total habit, and it's the only check that catches a query that's internally consistent and still wrong because the data underneath is incomplete. I keep a longer set of these as copy-paste queries in data-validation-checks.sql in my BA Toolkit, and the reconciliation walk-through is in examples/revenue-reconciliation. What the sharpest commenter will say "Just give the model the schema and the definition and it'll get it right." Partly. Giving an assistant the definition table above would probably fix the status and test-account bugs. It's the single best thing you can do on the generation side, and it's what the r/LLMDevs commenters meant by a semantic layer. But it doesn't remove the review. The definition can be incomplete (ours says nothing about NULL statuses), and Google's own docs warn the same prompt can give different SQL each time. You'd be reviewing a better first draft, which is nice, not skipping the review. "Ask the AI to review its own query." I do this, and it does catch things. I treat it like a linter: helpful for the mechanical items (NOT IN, BETWEEN on timestamps), weak on definition items, because the model doesn't know that finance excludes test accounts unless you tell it, and if you told it, it should have used that the first time. "Benchmarks say 83%, so it's fine most of the time." Benchmarks score whether the result set matches a reference answer. The annotation-error paper suggests the references themselves were wrong a lot of the time. And your schema isn't a benchmark schema. A per-query review is the only number that applies to your query. When to rerun the review: whenever the query is regenerated, the source schema changes, a new status value appears, the reporting period crosses a DST change or year end, or the definition changes. A reviewed query is reviewed against a definition and a snapshot of data. Change either and the review expires. What I didn't test Honest limits of this fixture: - It's SQLite, with timestamps stored as text. Warehouses store real timestamp types, and time-zone functions differ between engines. Check 4's -4 hours is a teaching shortcut, not a production pattern. - There's no NULL status row in the data, so I haven't shown != dropping one. Add (1010, 2, NULL, '2026-10-15 12:00:00') to the orders and see what each query does. - No currency, no tax, no partial refunds, no line-level discounts. Real revenue definitions have all of these, and each one is another grain question. - I didn't time anything. Some fixes (pre-aggregating in a CTE) change query plans on large tables. - I wrote the flawed query. It's realistic, but it's mine, so it's a teaching example, not evidence about how often any assistant makes these mistakes. What I'd do on Monday If you're reviewing AI-drafted SQL starting this week, here's the order I'd go in. 1. Before opening the query, write the definition table: grain, statuses, calendar and zone, exclusions, empty-set rule. Five lines. If you can't fill it in, the problem isn't the SQL. 2. Build a ten-row fixture with one boundary row on each side of the period, one order with two payments, one test account, one NULL in a key you filter on. Fifteen minutes. Reuse it for every query on that metric. 3. Run the grain check and the status list first. They catch the most, fastest. 4. Ban BETWEEN on timestamps and NOT IN on subqueries in your team's review notes. These two are pure mechanics and easy to agree on. 5. Return the row count with every total. It costs one column and answers the empty-set question for free. 6. Get one control total from the source system for the period and write down the tolerance you accept. 7. Save the reviewed query with its definition next to it, so the next regeneration has something to be compared against. If you're on the product side setting up an AI analytics tool, my AI agent test plan template is built for the next step: a golden set of questions with known answers, run more than once. Watch: finding the gap when two numbers don't match This is my short animated explainer on reconciling two numbers that should agree. It walks through definition, calendar and time zone, grain and fan-out, and orphan rows, which are most of the same failure modes from the reconciliation side. {% embed youtube.com %} Video: "Dashboard Numbers Don't Match? Find the Gap in SQL" by Prachit Parikh, on YouTube. For the testing side of AI tools in general (golden sets, running each case three times, flagging unstable answers), there's AI Agent Testing for Business Analysts: Dev, UAT, Prod. Discussion A few questions I'd like real answers to: 1. Who signs off on AI-drafted SQL on your team: the person who wrote the prompt, an engineer, or the metric owner? Has that ever gone wrong? 2. What's a bug you've caught in generated SQL that isn't on this list? I suspect window functions with the wrong PARTITION BY belong here. 3. Should an empty period return 0, NULL, or no row at all? I've seen teams argue all three. 4. Has giving the assistant a written metric definition measurably reduced the fixes you make? Comment prompt: paste the one-line check you'd add to the review card, written as a query that returns rows only when something's wrong. I'll test the good ones against the Northstar fixture and post the output in reply. References - BIRD benchmark leaderboard and news (top test EX 83.40%, human 92.96%, Mini-Dev 2026 update): bird-bench.github.io - Spider 2.0 leaderboard (Snow and Lite top scores): spider2-sql.github.io - Spider 2.0 GitHub repository and changelog: xlang-ai/Spider2 - Jin, Choi, Zhu, Kang, "Pervasive Annotation Errors Break Text-to-SQL Benchmarks and Leaderboards," arXiv:2601.08778, January 2026: arxiv.org - Google Cloud, "Write queries with Gemini assistance" (BigQuery): docs.cloud.google.com - Stack Overflow 2026 Developer Survey, full results: survey.stackoverflow.co - Stack Overflow blog, "The results of the 2026 Developer Survey are here," 6 October 2026: stackoverflow.blog Reader lessons credited above: ontomato, Shivam Pandit, Alex Shev, and Manuel Diaz. Related posts of mine: 7 SQL checks I run before UAT sign-off, Data mapping document template, and AI agents can write the code now. The bottleneck is the requirements. More of my work is on my portfolio page. This article was created with the help of AI. Every query and output above was run against the Northstar fixture in SQLite and checked before publishing. About me I'm Prachit Parikh, a business analyst. I turn what stakeholders ask for into requirements a team can test, check the data with SQL, and see changes through UAT. I keep free templates, including the SQL data validation checks behind this review card, in my BA Toolkit on GitHub, and I make short animated explainers on YouTube. Besides Dev.to, I write on Substack, Medium, Hashnode, Blogger and WordPress, and I answer questions on Quora. You can also find me on LinkedIn, Bluesky, Kaggle, Credly, Crunchbase and Gravatar. Everything is collected on my website and my profile hub.
Why computer use benchmarks are so hard to run Let's say you built a computer use agent and want to know how good it is. The usual way is OSWorld, a popular benchmark of 108 tasks on a real desktop. It is easy to start and hard to get right. I read about 200 GitHub issues to see what goes wrong. Here are the 10 problems you will face, and how I would handle each. The first hard part is the machines. A coding benchmark gets by with a container and a shell. A computer use benchmark needs a whole desktop for every attempt, with its own apps and websites, and the agent takes hundreds of steps per task. The second is that one run may not be enough. Agents are not deterministic, so a single run can come out higher or lower just by luck. That is why it is good to run the benchmark a few times and average the results. Three runs of OSWorld 2.1 are 324 attempts and roughly 400 machine-hours. All that machine time is a plain reason so few published scores get re-run. My last article found it on OSWorld 2.0: {% embed x.com %} Every September flagship number there is self-reported, and the maintainers' own board has not run those models yet. Most teams run once: in Yutori's tracker of OSWorld-Verified results, 94 of the 111 entries that list a run count are a single run. Some do repeat: Anthropic averaged five runs for Claude Fable 5.1 on OSWorld 2.0, and Yutori reported three runs for its own Navigator n2. The 400 is my estimate from Simular's numbers. Simular published every run of its Sai agent on OSWorld 2.1. It took 14 runs and reruns to get one clean result on every task, and that clean pass took about 139 machine-hours, with a median task of 46 minutes and the longest at 8.9 hours. Three passes like that come to roughly 400. OSWorld-Verified, with 369 shorter tasks, costs about the same: Simular reports 4 to 5 hours per full run on 30 parallel environments, close to one 2.1 pass in machine time. !Chart titled One clean pass of OSWorld 2.1, task by task: every one of the 108 tasks as a dot on a time axis from 15 minutes to 8 hours, 69 under an hour and 39 over, median 46 minutes, mean 77 minutes, longest 8.9 hours. Each of the ten can change the score, and they come in order: on the way from one desktop to a daily run, each fix exposes the next one. The problems come from other people's runs I did not run a campaign. Every problem below was reported by people who did, in issues and pull requests on OSWorld, SWE-bench and Terminal-Bench. Most are known, and many are already fixed. The "how I would handle it" parts are my design, and some of it comes from agent runtimes I built before. The ten problems | Problem | What it does to the number | What to check | |---|---|---| | 1. Capacity | The slowest tasks hold the run | Valid results per hour | | 2. Preparation | Setup time hides inside task time | Each readiness step | | 3. Rate limits | Throttled tasks end early | Calls per attempt against quota | | 4. Isolation | Attempts change each other's results | Two attempts at once | | 5. Runtime failures | Crashes look like agent failures | Timeouts and real health checks | | 6. Retries | Actions run twice; best-of reads as mean | When the retry happened | | 7. Grading | Infrastructure errors become zeros | Ran, graded, evidence kept | | 8. Reproducibility | Same model, different number | The run manifest | | 9. Observability | Stuck looks like slow | One ID per attempt | | 10. Cost | Cheap runs that prove nothing | Cost per valid result | What happens in one attempt An attempt is one task run once, and it has four parts. The benchmark gives the task and the checks. The desktop gives the apps and the starting state. The agent decides what to do. The harness runs the loop between them: screenshot in, action out, until the agent stops. Then a grader checks what is left on the machine. !Diagram titled One attempt: the agent harness sits between the prepared sandbox and the model endpoint, sending observations and requests and returning decisions and actions, about 318 tool calls per task, and records a step trace of every step in the sandbox; when the episode ends, the sandbox final state is handed to the grader, which produces the score and evidence along with the step trace. In OSWorld 2.1 the desktop is a QEMU virtual machine inside a Docker container, with 4 GB of RAM and 4 virtual CPUs by default. The agent sees screenshots, and in the OSWorld 2.0 paper Claude Opus 4.7 averaged 318 tool calls per task. Once one task works, you know the model can use the desktop. What you need to know next is how long all 324 attempts take and how many of their results you can trust. 1. Capacity: rent the desktops or run them yourself On one desktop, the 324 attempts run one after another, and 400 machine-hours means 400 hours of waiting. Parallel desktops are the first answer, and they work on the wait, not the work: the total stays 400 machine-hours. On 40 desktops that is 10 hours if everything packs perfectly, and it never does, because the run ends when the slowest task ends. Simular reports 4 to 5 hours per OSWorld run on 30 parallel environments, sometimes 6, while the run waits for its last tasks. !Chart titled Parallelism changes elapsed time, not total work: on a log scale, wall time for one update falls from 416 hours on 1 desktop to 27.0 hours on 16, 11.6 hours on 40 and 3.9 hours on 108, while a dashed line holds total work at 416 machine-hours at every width; 40 desktops need 160 GB of RAM and 160 virtual CPUs at the default size. Parallel desktops need real hardware. Forty of them at the default size is 160 GB of guest RAM and 160 virtual CPUs before anything else runs on the host. Tianbao Xie, who leads OSWorld, described what AWS changed for them: {% embed x.com %} That hardware can move the score as well as the speed. Anthropic found that with strict per-task CPU and memory limits, up to 6% of Terminal-Bench tasks failed on pod errors unrelated to the model, and loosening the limits moved scores by up to 6 points. Their summary: {% embed x.com %} So capacity is a question of managing containers, one per desktop, and there are two ways to get them. One is to rent. The OSWorld team moved to AWS, and sandbox providers such as Cua now run the official OSWorld image on their own cloud. Their limits then become yours: one OSWorld user hit their AWS account's vCPU limit at five desktops. The other is to run them on your own servers, which makes you the orchestrator. You plan per container: 4 GB of RAM and 4 virtual CPUs each, plus headroom, because tight limits are what cost those 6 points. On one server, the OSWorld team ran 8 or 16 desktops at a time. Either way, I would size one container first, then add more in steps and watch completed, validly graded attempts per hour. If 32 desktops are already waiting on the model API, 64 mostly doubles the bill. 2. Preparation: boot desktops before the attempts need them Once you have the machines, the next wait is getting each one ready for its task. Ready is four steps: the image is cached on the host, the VM has booted, control works (the screenshot server answers and the apps open), and the task's files and websites are in place. Each step can be slow on its own. !Diagram titled Ready has four steps: 324 attempts wait in a queue for a warm pool of 10 desktops that are already cached, booted and control-verified; only the task setup happens per attempt, and a desktop goes back to the pool only after a verified reset. The OSWorld team saw LibreOffice take a minute to open on first launch, because AWS loads disks restored from snapshots lazily. They cut the image from 50 GB to 25 GB, and first launch went to 5 seconds. Coding benchmarks have the same problem with bigger images. Epoch found SWE-bench's task images totaled 684 GiB and rebuilt them down to 67: {% embed x.com %} The usual answer is a warm pool: a few desktops booted and checked before they are needed. I built one for agent containers in another system, and in the case I measured a deploy went from about 60 seconds to under a second. For a benchmark, warm also has to mean clean, so a desktop goes back to the pool only after a verified reset. Ten clean desktops keep up with attempts as they arrive. Keeping all 324 warm in advance is a separate spending decision. So I would gate each attempt on every readiness step, pin the image by digest, and cache it near the workers. 3. Rate limits: every service the run touches has a limit With the desktops ready, the agents start calling the model. At 318 tool calls per task, 324 attempts come to about 100,000 tool calls driven by the model, most of them carrying a screenshot. Providers cap that traffic. Princeton's HAL team ran 21,730 agent rollouts across nine benchmarks and found that parallel evaluation runs into rate limits at every provider. When they ran, Anthropic capped API spending at $5,000 a month by default, even on its highest usage tier, so larger runs needed special approval. The tiers have changed since, but the standard ones still have a monthly cap, and reaching it pauses the API with the same 429 error a rate limit returns. The model API is one of several services a run depends on, and each one has its own limit. Nebius hit rate limits on PyPI, Docker Hub and the Ubuntu package archive while building SWE-bench task images, before any model was involved. Benchmarks that touch the live web meet bot detection, which HUD described in a walkthrough on browser evals: {% embed x.com %} A provider's content policy can stop a task outright. When Cua tried to run its benchmark on Muse, a KiCad task was flagged as a violation: {% embed x.com %} In OSWorld, the grader and the simulated user can call models as well as the agent, and all of them draw on the same quota unless you split it. So I would do most of the work before the run starts. First, confirm with the provider the limits on the tier you will actually run on, requests and tokens per minute and the monthly spend cap, and ask for more before the campaign, not in the middle of it. Pull every image and package once into your own mirror, so the public registries see one download instead of one per desktop. Give live-web tasks proxies you control, and pin one IP to any task where a site checks for humans; the OSWorld team did both. And run each task once on the provider you plan to use, so a content policy flag shows up before the campaign instead of as a zero inside it. During the run, I would put one admission controller in front of the shared model budget. It counts the grader and the simulated user against that budget, starts a desktop only when there is capacity for its traffic, and records a rate-limited attempt as an infrastructure failure to rerun, never as a zero. 4. Isolation: give each attempt its own world, then test it With attempts running side by side, each one needs a clean world, and a fresh VM only cleans the desktop. A task can post a comment, create an account or upload a file, and those writes live on a server. The next attempt inherits them, and an attempt running at the same time can see them. !Diagram titled One identity per attempt: one attempt ID runs like a line through the desktop, the website services, the grader, the browser, the harness and the evidence. WebArena's tooling handles this by running tasks in a fixed order, and its maintainers note that dropping the order for more parallelism can cost 1 to 2 percent. OSWorld 2.0 took the other route: it ships 31 self-hosted websites and scopes each attempt's state by cookie, so attempts do not contaminate each other. A clean world also means one without the answer in it. SWE-bench images once kept the repository's future commits, and some agents found the fix with git log. The maintainers rebuilt the images and measured the effect as small, about 5 in 10,000 trajectories. So I would give each attempt one identity across desktop, browser and website, and test it directly: run two attempts at once, change each world differently, and check that neither sees the other. 5. Runtime failures: the desktop dies before the agent does An attempt that runs for an hour or more outlasts some of the parts under it. One of them crashes, expires or hangs, and the attempt fails for a reason that is not the agent's. A crash can come from the image itself. OSWorld's in-VM server leaked a display connection on every screenshot, so long tasks crashed after about 220 steps. The fix went into the code in May, but the published image is older than the fix. In the logs it looks like a network error at a random step, until someone counts the steps. Clocks end attempts the same way. OSWorld 2.1's AWS provider can schedule a VM's termination after 300 minutes by default, while H Company's Holo4 notes list a per-task budget of 500 steps and 6 hours, an hour more than that VM lives. Tokens, sandboxes and leases all have lifetimes, and each has to be longer than the longest task. A hang is harder to see, because nothing fails. An open pull request on OSWorld found that 10 of 14 controller calls into the VM had no timeout, so a desktop that stops answering leaves the call waiting forever. I built a watchdog for the same shape of problem in an agent runtime: the process looks alive, but its event loop has frozen and nothing answers. So I would put a timeout on every call, check health by asking the desktop to do something real, and set every lifetime above the task budget. 6. Retries: mean of three and best of three are different scores Once failures happen, the instinct is to retry. Before the agent's first action that is safe, because the desktop has not changed yet. After it, the desktop has state, and a retry can repeat something that already happened. An open OSWorld pull request shows how that goes unnoticed. The desktop stops any command that runs past 120 seconds and reports an error, and the controller retried every error, including commands the desktop had already partly run. In that runner's logs, 357 of 23,244 actions ran twice. When the repeated command was a ctrl key press, the key stayed down for the rest of the task, and the trajectory looked normal. Harbor's desktop runtime has the rule in a code comment: replaying a click is not idempotent. Ashton Chew's input daemon for computer use on Modal makes the same call: once a press may have reached the X server, it releases what it pressed, returns the error, and does not retry. {% embed x.com %} Retries at different layers multiply. A Harbor issue describes SDK and harness retries stacking until one hung call could take about 90 minutes. Rerunning a whole task brings back the policy question from my last article, which run counts. Simular's published runs give 0.7674 if you count the first clean run per task and 0.7938 if you count the best one, 2.6 points apart from the same campaign. So I would retry freely before the first action, start and log a new attempt after it, give every layer one shared deadline, and say in the report whether the score is a mean of three or a best of three. 7. Grading: check that it ran, was graded and kept its evidence When the run ends, you have a folder of result files, each with a score. A zero in that folder can mean the agent failed. It can also mean the grader crashed, the output went missing, or the evidence never uploaded, and the file looks the same in every case. Simular hit this while making its OSWorld 2.1 reference runs. A usage limit on the model key stopped tasks after one step, and they were logged as success with a score of 0. The average over the folder came out at 0.40 and, in their words, looked like a measurement. The step count was what gave it away. !Diagram titled An infrastructure error is not a zero: result files pass through acceptance checks (did it run, was it graded, is the evidence there); all pass goes to an accepted score, including a real zero, any failure goes to an invalid attempt that keeps its cause, such as a usage limit, a grader crash or missing evidence. Graders can write the same false zero themselves. The OSWorld maintainers have an open fix for 14 of the 108 tasks, whose graders turn a failed call to the judge model into a zero. The grader also has to be out of the agent's reach. Some graders read the answer by running commands inside the VM the agent controlled. An engineer at H Company suggested loading the grading code in memory, so it never touches a disk the agent can read. Teams that publish numbers decide before the run how infrastructure failures count. An engineer at poolside described their rule: {% embed x.com %} So I would record three things per attempt: did it run, was it graded, and is the evidence there. I would run the reference solutions before each campaign, because a grader that fails the right answer shows up there first. One team logged 240 SWE-bench rollouts at 0% and blamed their agent until the gold patches failed too. And I would report coverage, the share of attempts that were validly graded, next to the score. 8. Reproducibility: pin everything the number depends on Once a run is reliable, you run it again after the next harness change. If the number moves, you need to know whether your change moved it. The harness alone can do it. Anthropic re-ran older models after fixing a bug in its zoom tool and raising the token limit per turn, and wrote that it had been underreporting OSWorld across its model families. Claude Opus 4.7 went from 78.0% to 82.8% with no change to the model. The rest of the setup moves too. Between OSWorld 2.0 and 2.1, 174 pull requests to the task files were merged. Simular ran the same agent on two desktop images and got 0.7766 on one and 0.7414 on the other. HAL notes that OpenRouter can serve different quantizations of a model under the same endpoint name, so the same name does not guarantee the same model. So every run gets a manifest, with a line for each thing that can move the number: - task and grader revisions - image digest and website builds - the model and the provider route that actually served it - the judge model - harness commit - observation and action settings, screen size - step and time budgets - resources per desktop - retry and repeat policy: mean or best of three A manifest does not make a model deterministic. It tells you what stayed the same when the number moved. 9. Observability: you need to tell slow from stuck Hours into a run, you want to know which attempts are working, which are slow and which are stuck. The first thing you reach for is a live view of one desktop. Terminal-Bench had one early: {% embed x.com %} A live view shows one screen. It cannot tell you whether twenty other attempts are stuck in preparation or waiting on the model. A Harbor user spent about two days on a stall like that. Trials sat silent until a 30-minute timeout. The cause was two crashed trials whose 4 GB sandboxes kept running and filled the account's 10 GB memory limit, so every new sandbox was refused and retried without a word. You need the same visibility after the run, to explain a score. Simular found that OSWorld 2.1's rubric graders return only the total, so a partial score like 0.6969 cannot be explained or re-graded later. So I would carry one attempt ID from allocation through model calls, desktop actions, grading and cleanup, record how long each phase took, and keep the evidence of failed attempts instead of deleting it. 10. Cost: count cost per valid result Once the run works every day, it has a bill, and the model is only one line of it. The others are desktops, idle warm capacity, retries, failed preparation, storage, and anything left running after a crash. Simular published its model spend for the OSWorld 2.1 reference campaign: $2,417.63 over 27,662 gateway calls, about $18 per clean attempt. Desktop hours are not in that number. Leftovers add up quickly. OSWorld's Docker provider once left a 32 GB volume behind for every task, enough to fill a disk within an hour of a full run, until it was fixed. At scale, cost decides the method. HAL spent about $40,000 on its rollouts and still had to rely on single runs for most evaluations. Some models they did not run at all: {% embed x.com %} So I would track total cost divided by validly graded attempts, shown next to coverage. A cheap run that failed early has few valid attempts, so its cost per result goes up. A run that skipped the hard tasks shows up as lower coverage. A daily run has to hold all ten at once Every fix in this article is ordinary on its own. The trouble is that each one exposes the next problem, so a daily run has to hold all ten at once. Here is what that takes: eight stations every attempt passes through, and one attempt ID from allocation to cleanup. !Matrix titled What a daily run needs: eight stations, queue, admission, ready desktop, attempt, watchdog, grading, evidence and report, against the ten problems, with a dot where each station answers a problem, and one attempt ID from allocation to cleanup. You can find each station somewhere today. Parallel desktops, warm pools, retries and dashboards exist in OSWorld's own runner, Harbor, HUD, Cua and the sandbox providers. One setup that holds all of them, on the benchmark's exact package, every day, is still rare. That setup is the hardest benchmark. What to take away A benchmark score is the output of an operation. Before you trust one, yours or a lab's, ask how it was run. Save the table and the manifest. They work for SWE-bench and Terminal-Bench too. I build harnesses that let agents use computers. If you want the full notes with every source, reply and I will send them. --- Originally published on arthurkatcher.com.
给 agent 的"真机考场"：369 个真实任务跑在 Ubuntu/Windows 虚拟机，NeurIPS 2024。 纯截图或 a11y tree 观察，动作走 𝐩𝐲𝐚𝐮𝐭𝐨𝐠𝐮𝐢；评估脚本只验 VM 最终状态。 人类 72.36%，最强模型 12.24%——短板是 𝐆𝐔𝐈 𝐠𝐫𝐨𝐮𝐧𝐝𝐢𝐧𝐠。必跑基准。 xlang-ai/OSWorld t.co