I benchmarked my own agent skill, and today's Claude barely needed it
I wanted to write an agent skill that makes coding agents better at data analysis. The kind of mistakes I had in
mind are the ones that produce a confident, wrong number: a "Total" row summed together with the countries, a join
that duplicates orders, averaged p99s, timed-out requests that vanish from the latency percentiles, an A/B test that
was supposed to split 50/50 and came back 50.7/49.3.
Every rule in the skill comes from a paper or official documentation. But before claiming it helps, I wanted to
measure it.
The benchmark
I built 30 realistic cases, each with one planted trap and a correct answer computed by code:
- 19 data-analysis traps (subtotal rows, join fan-out, average of averages, NA parsed as null when it's Namibia,
partial weeks, sample ratio mismatch, segment fishing, Simpson's paradox, …)
- 8 performance and reliability traps (closed-loop load tests, a saturated load generator, retries counted as
requests, tail-sampled traces, regression to the mean after a "perf sprint", …)
- 3 tasks where the agent measures its own work (fix a flaky test and confirm it, report a speedup, measure
logging overhead)
Each case ran with and without the skill on Claude Haiku 5.5 and Sonnet 5.5, graded by an LLM judge (Sonnet,
3 votes per grader).
The result
| | With skill | Without skill |
|---|---|---|
| Haiku 5.5 | 61/61 | 58/61 |
| Sonnet 5.5 | 61/61 | 60/61 |
Today's Claude models already avoid nearly all of these traps, including the harder performance ones. The skill
fixed every baseline miss (-1 timeout sentinels, which both models missed, plus a closed-loop load test hiding a
stall and an overhead number reported without uncertainty on Haiku) and introduced no new failures. But with one
run per arm, that's "catches occasional slips", not a large effect. That's what the README says.
What I learned
1. Skills help most where the model's knowledge is weakest. The same week I wrote a skill for upgrading to
TypeScript 7, which shipped after the models' training cutoff. There the gains were real: graders passed went from
10/18 to 17/18 on Sonnet and 7/18 to 15/18 on Haiku, on small samples with offline agents. Mainstream analysis
practice is already in the model. Post-cutoff releases and exact operational details are not.
2. Check your test cases before you trust a baseline failure. Across both projects, several of the
"failures" I first saw were flaws in my own cases: an effect I hadn't intended in the data generator, a grader
that was too strict, a premise that contradicted itself. Once, Haiku correctly pointed out that a case's premise
was wrong.
3. A skill that doesn't activate doesn't help. In the benchmark the skill activated by itself in 22/30 Haiku
runs. Its description listed data types and trap names. The prompts it missed asked for an action: "should we
roll back?", "confirm it's fixed", "measure the overhead". I rewrote the description to say when to load it ("at
the start of any task whose answer will include a number … before reading files, running benchmarks or editing
code"). In a short trigger test on Haiku (first 3 turns, 2 rounds), activation on 10 queries the description
was never tuned on went from 7/20 to 20/20, with no activations on 10 unrelated queries.
Try it
The benchmark runs against any agent, with or without the skill. If you run it on another model or agent, I'd
like to see the results: boranfurkan/trustworthy-data-analysis
TypeScript 7 upgrade: the two silent default changes and the missing compiler API
TypeScript 7.0 shipped in July: the Go-native compiler, about 10x faster type-checking, still installed as
typescript and still run as tsc. Most of its breaking changes were announced as deprecations in 6.0, so the
upgrade looks easy. Three things catch people anyway.
1. types now defaults to []
Before 6.0, TypeScript loaded every @types/* package in node_modules. Now it loads none unless you list them.
The symptom:
error TS2580: Cannot find name 'process'.
error TS2582: Cannot find name 'describe'.
even though @types/node and @types/jest are installed. The fix is to list them:
"compilerOptions": {
"types": ["node", "jest"]
}
If you'd rather keep the old behaviour, "types": ["*"] brings it back. Listing them is better: test globals
like describe belong in the test tsconfig, not the one that builds your app.
2. rootDir now defaults to the tsconfig's directory
It used to be inferred as the common directory of your input files, usually src/. Now it's the directory that
contains tsconfig.json. If your tsconfig sits at the project root, src/server.ts is emitted to
dist/src/server.js, and node dist/server.js or your "main" field breaks.
"compilerOptions": {
"rootDir": "./src"
}
If a test config extends this one and includes test/, it needs its own rootDir (for example "."), or the test
files fall outside rootDir.
3. 7.0 ships without a compiler API
This is the big one. The 7.0 release notes say it plainly: 7.0 does not ship with an API. A new, different API is
planned for 7.1. Anything that loads typescript as a library can't run on 7.0: typescript-eslint (type-aware
rules), vue-tsc and other Volar-based tools, ts-jest, ts-node, ts-patch transformers, and your own scripts that call
ts.createProgram. As of early November typescript-eslint still doesn't support TS 7.
You don't have to choose. The TypeScript team publishes @typescript/typescript6, which re-exports the 6.0 API
and installs a tsc6 binary. Alias it as typescript so library consumers keep getting 6.0, and install 7.0 under
another name for the fast tsc:
{
"devDependencies": {
"typescript": "npm:@typescript/typescript6", // what typescript-eslint, vue-tsc, ts-jest import (bin: tsc6)
"@typescript/native": "npm:typescript@^7.0.2" // the 7.0 compiler (bin: tsc)
},
"scripts": {
"typecheck": "tsc --noEmit", // 7.0
"typecheck:vue": "vue-tsc --noEmit" // still 6.0 via the alias
}
}
Be explicit about which command runs on which version. In a Vue project, for example, the slow check is usually
vue-tsc, which stays on 6.0, so the 10x speedup may not show up where you expected it.
The hard errors (quick list)
Deprecated in 6.0, errors in 7.0, and "ignoreDeprecations": "6.0" won't save you:
target: es5, downlevelIteration, moduleResolution: node/node10/classic, baseUrl,
esModuleInterop: false, allowSyntheticDefaultImports: false, alwaysStrict: false, outFile,
module: amd/umd/system/none, the module keyword for namespaces, and import asserts (use with).
The safe order
1. Upgrade to 6.0 first. Turn on "stableTypeOrdering": true, remove ignoreDeprecations, fix everything. Code
that compiles cleanly there should compile the same on 7.0. (The flag can slow 6.0 by about 25%; it's a
diagnostic step, not a permanent setting.)
2. Set rootDir and types explicitly.
3. Move to 7.0, keeping API-dependent tools on 6.0 with the alias.
4. On small CI runners, --checkers <n> (default 4) limits the parallel type-checkers.
Why I wrote this down
TypeScript 7 shipped after the training cutoff of today's coding agents. In my tests, Claude knew the general
direction but hedged or guessed on these details, especially the @typescript/typescript6 setup. So I packaged
the facts from the official release notes as an agent skill. In small offline evals it raised graders passed from
10/18 to 17/18 on Sonnet and 7/18 to 15/18 on Haiku. Two runs per arm, and agents with web access may find
the facts themselves, so read those numbers as a hint, not a result:
boranfurkan/typescript-7-migration
Sources: Announcing TypeScript 7.0,
Announcing TypeScript 6.0,
typescript-eslint #12518.
Mentions
A benchmark of data-analysis traps for coding agents
I benchmarked my own agent skill, and today's Claude barely needed it I wanted to write an agent skill that makes coding agents better at data analysis. The kind of mistakes I had in mind are the ones that produce a confident, wrong number: a "Total" row summed together with the countries, a join that duplicates orders, averaged p99s, timed-out requests that vanish from the latency percentiles, an A/B test that was supposed to split 50/50 and came back 50.7/49.3. Every rule in the skill comes from a paper or official documentation. But before claiming it helps, I wanted to measure it. The benchmark I built 30 realistic cases, each with one planted trap and a correct answer computed by code: - 19 data-analysis traps (subtotal rows, join fan-out, average of averages, NA parsed as null when it's Namibia, partial weeks, sample ratio mismatch, segment fishing, Simpson's paradox, …) - 8 performance and reliability traps (closed-loop load tests, a saturated load generator, retries counted as requests, tail-sampled traces, regression to the mean after a "perf sprint", …) - 3 tasks where the agent measures its own work (fix a flaky test and confirm it, report a speedup, measure logging overhead) Each case ran with and without the skill on Claude Haiku 5.5 and Sonnet 5.5, graded by an LLM judge (Sonnet, 3 votes per grader). The result | | With skill | Without skill | |---|---|---| | Haiku 5.5 | 61/61 | 58/61 | | Sonnet 5.5 | 61/61 | 60/61 | Today's Claude models already avoid nearly all of these traps, including the harder performance ones. The skill fixed every baseline miss (-1 timeout sentinels, which both models missed, plus a closed-loop load test hiding a stall and an overhead number reported without uncertainty on Haiku) and introduced no new failures. But with one run per arm, that's "catches occasional slips", not a large effect. That's what the README says. What I learned 1. Skills help most where the model's knowledge is weakest. The same week I wrote a skill for upgrading to TypeScript 7, which shipped after the models' training cutoff. There the gains were real: graders passed went from 10/18 to 17/18 on Sonnet and 7/18 to 15/18 on Haiku, on small samples with offline agents. Mainstream analysis practice is already in the model. Post-cutoff releases and exact operational details are not. 2. Check your test cases before you trust a baseline failure. Across both projects, several of the "failures" I first saw were flaws in my own cases: an effect I hadn't intended in the data generator, a grader that was too strict, a premise that contradicted itself. Once, Haiku correctly pointed out that a case's premise was wrong. 3. A skill that doesn't activate doesn't help. In the benchmark the skill activated by itself in 22/30 Haiku runs. Its description listed data types and trap names. The prompts it missed asked for an action: "should we roll back?", "confirm it's fixed", "measure the overhead". I rewrote the description to say when to load it ("at the start of any task whose answer will include a number … before reading files, running benchmarks or editing code"). In a short trigger test on Haiku (first 3 turns, 2 rounds), activation on 10 queries the description was never tuned on went from 7/20 to 20/20, with no activations on 10 unrelated queries. Try it The benchmark runs against any agent, with or without the skill. If you run it on another model or agent, I'd like to see the results: boranfurkan/trustworthy-data-analysis
TypeScript 7 upgrade: the two silent default changes and the missing compiler API TypeScript 7.0 shipped in July: the Go-native compiler, about 10x faster type-checking, still installed as typescript and still run as tsc. Most of its breaking changes were announced as deprecations in 6.0, so the upgrade looks easy. Three things catch people anyway. 1. types now defaults to [] Before 6.0, TypeScript loaded every @types/* package in node_modules. Now it loads none unless you list them. The symptom: error TS2580: Cannot find name 'process'. error TS2582: Cannot find name 'describe'. even though @types/node and @types/jest are installed. The fix is to list them: "compilerOptions": { "types": ["node", "jest"] } If you'd rather keep the old behaviour, "types": ["*"] brings it back. Listing them is better: test globals like describe belong in the test tsconfig, not the one that builds your app. 2. rootDir now defaults to the tsconfig's directory It used to be inferred as the common directory of your input files, usually src/. Now it's the directory that contains tsconfig.json. If your tsconfig sits at the project root, src/server.ts is emitted to dist/src/server.js, and node dist/server.js or your "main" field breaks. "compilerOptions": { "rootDir": "./src" } If a test config extends this one and includes test/, it needs its own rootDir (for example "."), or the test files fall outside rootDir. 3. 7.0 ships without a compiler API This is the big one. The 7.0 release notes say it plainly: 7.0 does not ship with an API. A new, different API is planned for 7.1. Anything that loads typescript as a library can't run on 7.0: typescript-eslint (type-aware rules), vue-tsc and other Volar-based tools, ts-jest, ts-node, ts-patch transformers, and your own scripts that call ts.createProgram. As of early November typescript-eslint still doesn't support TS 7. You don't have to choose. The TypeScript team publishes @typescript/typescript6, which re-exports the 6.0 API and installs a tsc6 binary. Alias it as typescript so library consumers keep getting 6.0, and install 7.0 under another name for the fast tsc: { "devDependencies": { "typescript": "npm:@typescript/typescript6", // what typescript-eslint, vue-tsc, ts-jest import (bin: tsc6) "@typescript/native": "npm:typescript@^7.0.2" // the 7.0 compiler (bin: tsc) }, "scripts": { "typecheck": "tsc --noEmit", // 7.0 "typecheck:vue": "vue-tsc --noEmit" // still 6.0 via the alias } } Be explicit about which command runs on which version. In a Vue project, for example, the slow check is usually vue-tsc, which stays on 6.0, so the 10x speedup may not show up where you expected it. The hard errors (quick list) Deprecated in 6.0, errors in 7.0, and "ignoreDeprecations": "6.0" won't save you: target: es5, downlevelIteration, moduleResolution: node/node10/classic, baseUrl, esModuleInterop: false, allowSyntheticDefaultImports: false, alwaysStrict: false, outFile, module: amd/umd/system/none, the module keyword for namespaces, and import asserts (use with). The safe order 1. Upgrade to 6.0 first. Turn on "stableTypeOrdering": true, remove ignoreDeprecations, fix everything. Code that compiles cleanly there should compile the same on 7.0. (The flag can slow 6.0 by about 25%; it's a diagnostic step, not a permanent setting.) 2. Set rootDir and types explicitly. 3. Move to 7.0, keeping API-dependent tools on 6.0 with the alias. 4. On small CI runners, --checkers <n> (default 4) limits the parallel type-checkers. Why I wrote this down TypeScript 7 shipped after the training cutoff of today's coding agents. In my tests, Claude knew the general direction but hedged or guessed on these details, especially the @typescript/typescript6 setup. So I packaged the facts from the official release notes as an agent skill. In small offline evals it raised graders passed from 10/18 to 17/18 on Sonnet and 7/18 to 15/18 on Haiku. Two runs per arm, and agents with web access may find the facts themselves, so read those numbers as a hint, not a result: boranfurkan/typescript-7-migration Sources: Announcing TypeScript 7.0, Announcing TypeScript 6.0, typescript-eslint #12518.