nolanlawson

The agent called the framework's MCP server in 0 of 5 runs Nolan Lawson built the same Pokédex app 12 times, once per JavaScript framework, 5 iterations each, one agent per run, and published the whole thing with costs and Lighthouse scores. The cost table got the attention. The table I keep scrolling back to is the one that lists the AI tooling each framework's docs recommend for coding agents, next to whether the agent actually used it. Eight of the twelve frameworks had something to install. Across those eight, invocation ranged from 0 of 5 runs to 5 of 5. Two of them landed at zero on the headline mechanism their docs point you at. What the harness actually does The setup matters for reading the rest, and it's documented well enough to copy. Each agent gets the same SPEC.md, the same conformance suite, the same tooling, and the same one-line prompt: implement the app in this framework, use the latest stable version, work autonomously until npm test passes. One agent at a time in a fresh Podman container, 4 CPUs and 8 GB, Opus 5.5 at high effort, Claude Code pinned to 2.1.310. When the agent stops, only its app/ directory gets copied into a clean container to run the tests. Everything else it touched (tests, fixtures, node_modules, whatever it left running) stays behind, so it can't edit its way to a pass. Measurement happens in a third container behind Caddy, Lighthouse mobile only, 6 URLs, 5 runs each, median kept. The benchmark repo is forkable and the raw results are published separately. Costs land between $2.67 and $4.97 per run as medians of 5. Hono+htmx came in cheapest, Angular most expensive, and the spread between adjacent ranks is inside the noise. More on that at the end. Installed tooling and invoked tooling are two different numbers This is the part I'd steal. From the post's own table: | Framework | Tooling installed | Used by the agents | | --- | --- | --- | | Hono+htmx | Hono skills | skill tool in 5/5 runs | | React Router | React Router skill | skill tool 5/5, skill files 5/5 | | SvelteKit | Svelte Claude Code plugin | MCP in 5/5 runs | | Angular | Angular MCP + agent skills | MCP 3/5, skill tool 4/5, skill files 2/5 | | Astro | Astro MCP server | MCP in 2/5 runs | | TanStack Start | none official | skill files in 2/5 runs | | Next.js | next-devtools MCP + skill | skill files 1/5, AGENTS.md 2/5 | | Nuxt | Nuxt MCP | not used | | Qwik, SolidStart, Gracile+Lit, Vanilla | none | n/a | Nuxt's hosted MCP server was available in every run and invoked in none of them. Next.js got a stdio MCP server pinned through npx plus a skill, and the usage column only ever mentions the skill files and the AGENTS.md file. For SvelteKit the agents hit the MCP in all five runs. The harness installs this stuff the way the framework's own docs tell you to, and you can read the exact install path for each framework in setup-tooling.sh. Three shapes show up there. Skills copied into .claude/skills by the skills CLI (Hono, React Router, Angular). Hosted HTTP MCP servers registered in the MCP config (Nuxt, Astro). And a Claude Code plugin installed from a marketplace that starts its own MCP server through npx (SvelteKit). The three frameworks where the tooling got used in every run are the two skills installs and the plugin install. The two hosted HTTP servers are the bottom of the column. With 5 runs per framework, one model, one effort level, I'm not going to pretend that's a law about MCP transports. But if you maintain a framework's agent integration, "we shipped a server, the docs mention it" is not the same as "agents call it," and this is what the gap looks like when somebody finally counts. Lawson's own read is that the current crop of MCPs and skills may be doing more harm than good. Counting invocations is the cheap part of an eval The only reason that table exists is that the harness runs claude -p with --output-format stream-json and keeps the agent's repo. Every tool call is in the transcript. If you throw the transcript away, "with MCP" against "without MCP" becomes a comparison of two environments you can't check afterwards, and the result is unfalsifiable in the boring way: you can't tell a tool that helped from a tool that was never called. I ran into the same shape testing rules files. A reviewer's config file can be present, correct, and simply not loaded on the run you measured, which is why I put a canary word in the rules and checked whether it came back instead of reading the file and calling it configured. Installed, loaded, and invoked are three separate states and only the last one can change a result. So when you build an arm in an agent eval, log tool names per run and report the invocation rate next to the score. If the tool fired in 2 of 5 runs, label the arm as partially applied, because the other 3 runs are a control you didn't ask for and can't separate. Same question applies when you're the buyer rather than the benchmarker: comparing code review agents like Kodus, CodeRabbit or Greptile, the useful thing to ask is which checks ran on which pull requests, and whether the tool's report separates "no issues found" from "the review didn't run." What the harness records instead of pinning Frameworks and their AI tooling are deliberately left unpinned, since a real user installs whatever is current that week. The harness compensates by recording provenance instead. npx-launched MCP servers get resolved to an exact version at setup time and pinned to it. Hosted MCP servers can't be pinned, so mcp-probe.sh initializes a session, fetches the tool list, and stores a SHA-256 of it. Skill and plugin repos get their commit SHA. Alongside those, the run records the container image IDs, the Ubuntu base digest, Node, npm, Playwright, Claude Code, the model and effort level, and the app's own installed dependency versions. That's the right shape for a benchmark that cannot be frozen. The model, the frameworks, the docs and the search results all move, so reproducibility has to mean "here is the dated snapshot and everything that produced it" rather than "re-run this and get the same number." If your eval claims a number is reproducible while silently depending on whatever latest resolved to, you're describing the weather on the day you measured. The variance is printed next to the medians Cost per run for React Router: $3.81, range $2.82 to $3.97. Hono+htmx: $2.67, range $2.23 to $3.19. The cheapest React Router run beat the most expensive Hono+htmx run. That's a single-iteration comparison that would have flipped the other way on a different Tuesday, which is why the README says one iteration is an anecdote and why the results repo ships the per-run data instead of a leaderboard. The README also lists what the benchmark can't see. The app is content-heavy, which favors server-rendered MPAs on first-load metrics. Only page loads are measured, so client-side navigation and interactions where SPAs win back ground aren't. The harness's text helpers read innerText, which skips shadow-root content. And the data has to be fetched at request time, so a framework can't win by prerendering everything. Those caveats are the reason I trust the numbers that do survive them. Hono+htmx produced the same app in roughly 1,370 lines against Angular's 2,009, and Lawson's point about code size is the one worth carrying into agent work: in a codebase an agent keeps editing, line count is a recurring token cost, not a style preference. Astro shipped 0 KiB of JavaScript and an 810 ms LCP, Next.js 142 KiB and 2,090 ms, and the ranking in between those two is a shrug. The most useful thing in the whole exercise isn't which framework won. It's that a table of "did the agent call the thing we installed" exists at all, next to the cost numbers, in a post that also tells you how many iterations it took to make each one mean something. The HN thread spent most of its energy arguing that the app type flatters Astro, which is a fair fight to have and a fight the author already conceded in the limitations. I'd rather see the next round of these run the same tooling with and without the MCP servers, and print the invocation counts. Fork it, spend your own tokens.

Memlabの誤検知・検知漏れのメカニズムと対策 Memlab の誤検知・検知漏れのメカニズムと対策 はじめに Meta が公開しているメモリリーク検出ツール「Memlab」は、JavaScript ヒープスナップショットの解析が可能なツールです。しかし、実際にプロジェクトに導入すると「明らかなリークが出ない（検知漏れ）」「意図したキャッシュがリークとして報告される（誤検知）」が発生することがあります。 これらは Memlab 独自の抽出アルゴリズムが仕様通りに機能している結果です。本記事では、実践的な対策や CI での運用について考察します。 Memlab の概要と主要コマンド Memlab は、Puppeteer によるブラウザの自動操作と、V8 エンジンのヒープスナップショット解析を組み合わせた E2E のメモリリーク検出ツールです。 用途に応じて以下の CLI コマンドが用意されており、シナリオ実行からオフライン解析まで幅広く対応しています。 memlab run シナリオを実行し、スナップショットの取得からリーク検出までを一貫して自動で行う基本コマンド memlab runは、内部で memlab snapshot → memlab find-leaks と同等の処理を実行している memlab snapshot リーク解析は行わず、指定したシナリオや URL に対してヒープスナップショットの採取・保存のみを実行する memlab find-leaks 既にディスク上に保存されているヒープスナップショットファイル（SBP, STP, SFP）を読み込み、オフラインでリーク検出を行う memlab diff-leaks 2 つのヒープスナップショット間の差分を計算し、どのオブジェクトが増加・減少しているかを直接比較して抽出する memlab analyze <plugin> 取得したヒープスナップショットに対し、巨大な配列や不要なコレクションを検出する等の Memlab 内蔵プラグインを実行する memlab trace --node-id <id> ヒープ上の特定のノード ID を指定し、それがなぜメモリに残っているのかを示すリテーナートレース（保持経路）を出力する memlab heap 単一のヒープスナップショットを対話的（インタラクティブ）に探索します。 memlab view-heap 単一のヒープスナップショットを対話的に表示・確認します。 memlab measure ヒープサイズの変動（ノイズ）を把握するため、同じシナリオを複数回反復実行して測定します。 memlab warmup テスト前にターゲットのウェブアプリ（サーバー）をウォームアップし、キャッシュ等の初期化を済ませます。 memlab list 登録されているすべてのテストシナリオを一覧表示する memlab anonymize 取得したヒープスナップショットからユーザーデータやテキストを置換・削除し、第三者へ安全に共有できる状態（匿名化）にします。 memlab reset Memlab が一時的に生成したワークスペース内のスナップショットファイルやキャッシュデータを削除し、状態をリセットする コマンドラインから memlab run --scenario <シナリオファイル名> を実行する際、シナリオファイルは主に以下の 3 つの関数で構成されます url: テストを開始する初期画面の URL を指定する action: クリックなど、リークを発生させる可能性のある操作を定義する back: モーダルを閉じる、または元の画面に戻るための操作を定義する 基本的なシナリオファイルの実装サンプルは以下の通りです。 module.exports = { url: () => "example.com", action: async (page) => { await page.click("#open-modal"); }, back: async (page) => { await page.click('[aria-label="Close"]'); }, }; 誤検知と検知漏れのメカニズム Memlab の「誤検知」や「検知漏れ」を正しく理解するためには、デフォルトの挙動として内部で実行される以下の 2 つを知ることが重要です。 時間軸の差分比較による絞り込み Memlab は (STP \ SBP) ∩ SFP という計算式で差分を抽出する SBP（初期状態）に存在せず、STP（アクション後）に新規作成され、SFP（戻った後）にも残存するオブジェクトのみを対象とする SBP の時点で裏側のキャッシュ等に微量でも存在していた場合、今回のシナリオ起因の新規リークではないと正しく判定され除外される ヒューリスティクスによるノイズの切り捨て 全オブジェクトを抽出後、Detached DOM などリークの可能性が高い特定の種類のオブジェクトのみを厳格なフィルターで残す その他のプレーンな JavaScript オブジェクトや配列などは、ノイズとして意図的に報告から切り捨てられる クラスタリングによる代表経路の抽出 検出された膨大なリークオブジェクトを、類似する保持経路ごとにクラスタとしてグループ化する デフォルトでは各クラスタから代表的な 1 つの経路のみが出力されるため、細かいノイズクラスタが大量に生成されると、リークがログに埋もれて見逃されるリスクがある CLI オプションと挙動の変更 CLI オプションを付与することで、クラスタリング（類似したリーク経路のグループ化）や抽出対象の挙動を変更できます。 --save-trace-as-unclassified-cluster 全てのトレースを 1 つのクラスタに集約するのではなく、検出された各トレースをそれぞれ個別の未分類クラスタとして出力する挙動となる --ml-linkage-max-dist <0~1> クラスタリングにおける結合の最大距離を設定するものである 距離の閾値を 0 に設定した場合、2 つのトレース間の距離が完全に 0（内容が完全に同一）でない限り、それらが同じクラスタに結合されることはない --trace-all-objects 前述した「ヒューリスティクスによるノイズの切り捨て」を無効化する 差分計算 (STP \ SBP) ∩ SFP に合致したすべてのオブジェクトを対象としてトレースを出力する leakFilter を活用した V8 内部ノードの除外 CLI オプションで検出候補を増やすと、本当に見つけたいアプリケーション起因のリークかどうかの確認に要します。 シナリオファイル内にアプリケーション固有の leakFilter コールバックを定義することで、特定のオブジェクトを意図的に除外することができます。 例えば、V8 エンジンが自動生成する内部ノードの徹底的な除外です。 hidden や object shape (syntheticノード) など、V8 エンジンが裏側で自動生成する仮想的な C++バインディングノード これらは、アプリ側のオブジェクトがリークした結果として道連れでメモリに残存しているだけの「症状」に過ぎず、開発者が JS コードから直接破棄できるものではありません。 そのため、leakFilter を用いて明示的に return false とし、探索スコープから除外することができます。 具体的なシナリオファイルの実装例は以下の通りです module.exports = { url: () => "example.com", action: async (page) => { /* アクションの実行 */ }, back: async (page) => { /* アクションのクリーンアップ・戻る処理 */ }, leakFilter: (node, _snapshot, _leakedNodeIds) => { if (node.type === "hidden" || node.type === "synthetic") { return false; // 経路探索とリーク報告の対象から完全に除外 } // 任意: アプリ側で本当に深刻なリーク（例: 保持サイズ1MB以上）のみを追跡する return node.retainedSize > 1024 * 1024; }, }; 継続的なメモリリーク監視と運用の検討 Memlab は詳細なヒープ解析が可能ですが、アプリ規模によっては長い時間を要することもあるため、すべてのコミットに対する毎回の CI 実行に適さない場合があります。 下記は一例であり、アプリケーションの非機能要件に応じて適切な運用を検討する必要があります。 Memlab は Nightly ビルドで運用 通常の PR 同期 CI から切り離し、影響範囲の大きい改修後や夜間バッチでの非同期実行に限定する 毎回の CI には Playwright と CDP を活用 PR 単位の日常的なテストでは、Playwright から CDP（Chrome DevTools Protocol）の Memory.getHeapStats や Runtime.queryObjects 、 Memory.getDOMCounters を叩いて監視する ソークテストの導入 ページをリロードせずに長時間使用する SPA では、特定の UI 操作を数百回反復実行する「ソークテスト」を検討する Nolan Lawson 氏が開発した SPA 向け検証ツール fuite でもソークテストの概念を採用している まとめ Memlab を利用した調査は、ツールのアルゴリズム特性を把握し、CLI オプションの仕様を正しく理解した上で、leakFilter を用いて計算量を絞り込むことが必要です。 変更ごとのメモリリーク監視には、Memlab は時間がかかり過剰となる場合があるため、アプリケーションの非機能要件に応じて適切な運用を検討する必要があります。また、日本語での運用情報も少なく、調査コストがかかると感じました。 一方で、Memlab はメモリリークが発生しているとわかっている場合の原因究明では有効なツールと思います。 参考資料 Command Line Interface | memlab - Meta Open Source: facebook.github.io How memlab Works | memlab - Meta Open Source: facebook.github.io fuite GitHub Repository: nolanlawson/fuite memlab Issue #123: memlab#123

and here is my analysis so far: agent-framework-benchmark#2 (I'm updating it as I make changes)

and here is my analysis so far: agent-framework-benchmark#2 (I'm updating it as I make changes)

Le billet de Nolan Lawson, avec le protocole, les coûts estimés et les mesures côté navigateur : nolanlawson.com Le dépôt du benchmark, pour consulter les consignes et les résultats ou refaire le test : nolanlawson/agent-framework-benchmark

AI Coding Agents Do Not Make Framework Choice Irrelevant. They Make Its Costs Measurable A familiar argument is making the rounds in web development: if an AI agent writes most of the code, developer experience matters less, so teams should choose the framework that works best for agents. It sounds plausible. It also skips the hard question: what does “works best” mean, and how would you know? Nolan Lawson tried to answer that with a small benchmark. He asked Claude Opus 5.5 to build the same Pokémon catalog app in 12 JavaScript stacks, tested each result against a shared conformance suite, and measured cost, time, code size, and selected Lighthouse metrics. He ran five iterations per stack, except Angular, which had three measurable results for some metrics. The benchmark write-up and its open-source harness make the experiment inspectable. The headline is not that one framework won. The useful finding is that “agent-friendly” is not a single score. In this test, Hono with htmx was among the least expensive and shortest-code options, while Astro sent no JavaScript on the tested page. Angular cost more and generated more code, but this is one content-oriented application, one model, one benchmark specification, and a small number of runs. The evidence is enough to question vague claims. It is not enough to declare a universal winner. That distinction matters because framework decisions are long-lived. If the agent saves a few minutes today but creates more code to inspect and maintain tomorrow, the initial speedup may be a poor trade. The better response to AI-assisted development is not to pick a framework based on intuition about what models prefer. It is to extend the measurement window from “how fast did the agent finish?” to “what does this choice cost across the life of the application?” What the experiment actually measured A shared task, not a framework tutorial. Lawson specified a small but non-trivial Pokémon app using local data from PokéAPI. Agents had to implement the same user-visible features and pass the same conformance suite. They were allowed to use official framework MCP tools or skills where available, and the harness recorded whether the agent actually used them. The benchmark repository says each run used a fresh Podman container and that the generated app was tested separately from the agent environment. The repository README describes the controls and reproduction setup. More than token count. Lawson reported API-equivalent cost, wall-clock time, tokens, lines of code, and Lighthouse measurements for performance and accessibility. That is a useful choice: an agent can spend more tokens to produce code that is smaller, more accessible, or faster for users. Optimizing only for the first completion time would miss those trade-offs. The median reported cost ranged from $2.67 for Hono plus htmx to $4.97 for Angular. These are not the experimenter's cash charges. Lawson says he used a Claude Max plan and reported the API prices quoted by Claude Code. The cost range is therefore a way to compare the reported usage, not a billing receipt or a forecast of what a team will pay. The time and token figures also varied. Astro's median run took 8.2 minutes, while Angular's took 17.2 minutes. Median token use ranged from 3.85 million for Nuxt to 9.76 million for Angular. But those figures came from five runs per framework, with Angular's reduced sample in some measurements. The author's own write-up notes stochastic results and overlapping ranges. A median is a helpful summary, not proof that the same ordering will hold on your task. The counterintuitive result is the important one Raw JavaScript was not automatically the cheapest route for the agent. The benchmark's “Vanilla” condition used JavaScript or TypeScript with Vite and no UI framework. It was not simply “write the smallest possible plain JavaScript app.” The agent made architectural choices inside the allowed setup, and the result used a median of 5.00 million tokens and took 10.1 minutes. Several framework conditions used fewer tokens or completed sooner. That does not prove frameworks are better for agents. It demonstrates that removing a framework does not automatically remove complexity. A framework can supply conventions, routing, data loading, and a structure an agent can follow. Conversely, it can introduce conventions the model does not know well. Which effect dominates depends on the framework, task, tool setup, and model version. The web payload and the agent's build cost are different budgets. Astro's tested page transferred a median of 0 KiB of JavaScript, while Next.js transferred 142 KiB. Hono plus htmx used 18 KiB, and Angular 93 KiB. On that metric, the result separates framework bundles more clearly than the agent's cost rankings do. But a zero-JavaScript page is not a complete performance verdict: the application still has server work, network requests, content, and user interactions. The benchmark also reports different results for blocking time and largest contentful paint, with caveats around variance and the Lighthouse setup. If a team collapses those measurements into one “best” framework score, it will hide the very trade-offs that make the exercise useful. Agent usage cost is an engineering input. Browser payload and responsiveness are user-facing outputs. Maintainability is a future cost. They should not be treated as interchangeable just because they came from the same run. Code size is a proxy, not maintainability itself. Hono plus htmx generated a median of 1,370 non-blank source lines; Angular generated 2,009. That is a measurable difference, and smaller implementations may mean fewer tokens for later agent edits. Lawson explicitly connects code volume with the cost of future changes. Yet line count cannot tell you whether code is understandable, well-factored, secure, or easy for a human team to change. A short, opaque implementation is not automatically maintainable. Why “better for agents” is a weak architecture rule Framework advocates can point to model familiarity, official agent skills, MCP servers, and established examples. Those can matter. But the benchmark found that agents did not consistently use the available tooling. For example, the report says some frameworks' tools were used in only a subset of runs, and Nuxt's MCP was not used in the observed attempts. Installing an integration is not the same as proving it changed the result. There is also a training-data trap. Lawson suggests that SvelteKit's higher measured cost may partly reflect its recent release: the agent repeatedly read its migration guide, possibly because the latest version was not represented well in the model's training data. That explanation is plausible, but it is not isolated as a causal factor. A follow-up would need to vary model version, tool access, and framework documentation exposure independently to determine whether unfamiliarity caused the extra work. The result is a moving target. Framework versions change, models change, tools change, and the same agent can make different choices across runs. A one-time benchmark can help choose what to investigate. It cannot permanently rank a technology stack. That is especially true when official AI tooling is unevenly available or the agent ignores it. How to use this kind of benchmark without overfitting Start with your application shape. A content catalog is not a collaborative editor, admin dashboard, offline-first mobile web app, or real-time interface. Lawson says the sample is closer to a content site than a highly interactive product. If your application is dominated by complex client-side state, this result is a poor proxy. Pick a representative slice of your own product before treating any framework ranking as relevant. Define acceptance criteria before the run. Specify user-visible behavior, accessibility expectations, performance thresholds, supported browsers, and operational constraints. Keep the same tests across candidates. If one framework gets a different prompt or extra hints, the result measures prompt quality as much as framework choice. Measure failed attempts, not just successful output. In the benchmark, Angular failed to create a measurable app in two of five runs. Lawson retained the three usable iterations for some metrics and explained the difference. That transparency is important. In your own evaluation, record failures, retries, timeouts, test results, and cleanup effort. Do not quietly discard the runs that make a preferred tool look worse. Separate repeatability from generalization. Run enough iterations to see whether differences exceed run-to-run noise. Then repeat with a second task and, if relevant, a second model. The original author notes variance in both agent output and Lighthouse. Re-running one task reduces some uncertainty; it does not make that task representative of every product. Track the whole cost surface. At minimum, keep separate measures for: - Agent spend or token use per accepted change. - Human review and correction time. - Test pass rate and severity of defects. - Browser performance and accessibility. - Code size and complexity signals. - The effort required to add a second feature after the first implementation. That last measure is easy to skip and potentially more revealing than initial generation time. A framework that lets an agent produce a first screen quickly may not help when the next request changes data flow, permissions, or navigation. Add a follow-up task that tests whether the generated structure supports change without a rewrite. A decision checklist for framework evaluation Before you switch stacks because a model appears to prefer one, ask: 1. Is the benchmark task representative of our actual product? 2. Did each candidate receive the same specification, model, resource limits, and acceptance tests? 3. Were official agent tools installed, and did the agent use them? 4. Are the reported differences larger than the run-to-run spread? 5. Did we count failed runs and human repair work? 6. Are user outcomes measured independently from agent speed and cost? 7. Does the generated code remain workable after a second, realistic change? 8. Would we make the same choice if the current model or tool integration disappeared next month? The last question is a useful stress test. A framework should not be selected solely because one model has recently seen more examples of it. Model familiarity can improve today's result, but application needs, ecosystem support, and maintainability still matter after the next model update. The takeaway: benchmark claims, not vibes This experiment is valuable because it exposes its scope, shares its harness, reports multiple measures, and discusses limitations. Its results are not a framework league table. They are a reminder that the cost of a stack now includes both the work an agent spends producing code and the work people spend checking, operating, and changing that code. For a team choosing a framework, the practical move is modest: use public benchmarks to form hypotheses, then test those hypotheses on a small, representative feature with your own requirements. Keep the agent's cost, user-facing quality, reliability, and maintenance effort separate. If the decision changes when you add a second task or another model, you have learned something important about how fragile the original ranking was. I write about developer tools, infrastructure, and AI. Subscribe if you want evidence-based engineering analysis without treating a benchmark as a universal answer. Have you measured how your framework changes an AI agent's cost or the quality of the code it produces? What would you include in a fair test? Sources - Nolan Lawson: The AI cost of JavaScript frameworks - Open-source benchmark repository and reproduction notes

Best JS Framework for AI Coding: SvelteKit Burns 71% More Tokens Than Next.js If you asked me last month which JavaScript framework an AI coding agent would build fastest and cheapest, I'd have said SvelteKit. Small API, little boilerplate. In Nolan Lawson's new benchmark, Claude Opus 5.5 burned 8.53 million tokens building a small Pokédex app in SvelteKit. Next.js built the same app for 4.99 million. Nuxt did it for 3.85 million. The post is The AI cost of JavaScript frameworks, and it's the first real data I've seen on the "best JavaScript framework for AI coding" question that people keep settling on vibes. My take after reading it and the Hacker News thread: your framework choice now comes with a token bill, and that bill depends less on how elegant the framework is than on how well the model already knows its current API. What Lawson actually measured He gave Claude Opus 5.5 the same spec, the same tests and the same short prompt across 12 frameworks, changing only the framework name. The prompt template from the benchmark repo is: Implement the app described in SPEC.md using {framework}. Use the latest stable version. Work autonomously until `npm test` passes. The app is a Pokémon browser backed by a local copy of PokéAPI, with search and sorting. Each run happened in a fresh Podman container. Data had to be fetched at request time, so no static generation. Tailwind and component libraries were banned. Frameworks that ship an official MCP server or agent skills had them installed. Lawson took the median of 5 runs per framework, then measured the output with mobile Lighthouse (4x CPU throttling, Slow 4G). He's upfront about the limits: one small, content-heavy app, which favors server-rendered MPAs on first-load metrics, and interactions like search aren't measured, which is where SPAs could win back ground. It's still more real data than every "agents just work better with X" tweet I've read this year. React vs Svelte vs Vue vs vanilla JS: tokens, cost and Lighthouse results Medians from the post, sorted by tokens: | Framework | Tokens (M) | Cost (USD) | Time (min) | JS shipped (KiB) | LCP (ms) | Lines of code | |---|---|---|---|---|---|---| | Nuxt (Vue) | 3.85 | $2.82 | 9.0 | 84 | 1727 | 1415 | | Astro | 4.13 | $2.76 | 8.2 | 0 | 810 | 1461 | | Hono+htmx | 4.26 | $2.67 | 8.6 | 18 | 978 | 1370 | | Next.js | 4.99 | $3.14 | 10.9 | 142 | 2090 | 1539 | | Vanilla (Vite) | 5.00 | $2.98 | 10.1 | 7 | 1667 | 1492 | | Qwik | 5.64 | $3.27 | 10.7 | 40 | 979 | 1744 | | TanStack Start | 7.07 | $3.54 | 10.3 | 118 | 2028 | 1697 | | SolidStart | 7.13 | $3.79 | 11.2 | 40 | 1204 | 1630 | | React Router | 7.58 | $3.81 | 12.5 | 115 | 1729 | 1564 | | SvelteKit | 8.53 | $4.28 | 10.8 | 36 | 1130 | 1611 | | Gracile+Lit | 9.04 | $4.65 | 12.1 | 8 | 956 | 1717 | | Angular | 9.76 | $4.97 | 17.2 | 93 | 1954 | 2009 | Cost is at Claude Code API prices, not what he actually paid on a Max plan. Accessibility scored 100 everywhere, and Lighthouse performance landed between 97 and 100 for all of them, so every stack ended up with a working, decent app. Where they differ is the bill and how much JS reaches the browser. Angular used about 2.5x the tokens of Nuxt and took nearly twice as long. Two of its five runs didn't produce a measurable app because the agent built an SSR app that rejected non-localhost requests, even though the spec said it would sit behind a reverse proxy. Why SvelteKit cost so many tokens The agent read SvelteKit's v3 migration guide in every single run. The prompt asked for the latest stable version, which was @sveltejs/kit 3.1.0, and Lawson thinks that release is likely too new to be in the model's training data. So each session started by catching up on APIs the model didn't know. So yes, training-data bias, but not the version the "LLMs just default to React" crowd describes. React itself gets no special treatment. The model does well on API surface it has seen a lot of and that hasn't changed since, and Next.js and Nuxt are huge, mature and all over the training corpus. React Router and TanStack Start are both React too, and they still used 42-52% more tokens than Next.js, JSX and all. One HN commenter, ef2k, summed it up: "A significant part of the cost is that newer framework APIs are not known to the current agents." So the framework you love for shipping a clean new reactivity model is, right now, the one your agent has to re-learn on your dime. smrtinsert wrote that "sveltekit 2 was super quick to use for me," which fits the theory: same framework, but an API the model had already seen. Vanilla JS didn't save tokens either I went in expecting vanilla to be the cheapest option, on the theory that fewer abstractions means fewer APIs to hallucinate. It landed mid-pack at 5.00M tokens, basically tied with Next.js, and it was the only stack with layout shift (CLS 0.111). Its LCP of 1667 ms trailed Astro, Gracile+Lit, Hono+htmx, Qwik, SvelteKit and SolidStart. Lawson's read is that "having _some_ framework can be better than nothing (or better than raw Vite, anyway)." My guess at why: with raw Vite the agent has to invent routing, data loading and rendering conventions from scratch on every run, and that costs tokens too. oatmealsnap on HN made the long-term version of the argument: in a vanilla codebase, "your agents (or human devs) are going to write element.addEventListener and document.querySelectorAll countless times." The winners were the boring ones: server-first, stable, with a small API surface. Astro shipped 0 KiB of JS, had the best LCP (810 ms) and the shortest median build time (8.2 min). Hono+htmx had the lowest median cost at $2.67 and produced the least code, 1370 lines against Angular's 2009. Framework MCP servers and skills didn't lower the token bill Where a framework had official AI tooling, it was installed, and agents used it inconsistently: - SvelteKit's MCP/plugin, the React Router skill and the Hono skills: used in 5/5 runs - Angular MCP: 3/5 - Astro MCP: 2/5 - Next.js: skill files read in 1/5 runs, AGENTS.md in 2/5 - Nuxt MCP: never used Put that list next to the token table. Nuxt never touched its MCP server and was the cheapest on tokens. SvelteKit used its tooling every time and was the third most expensive. Lawson says he suspects the current crop of framework MCPs and skills "may be doing more harm than good." I'd go further: if your framework needs an MCP server so the agent can write it correctly, that's the framework tax showing up on your invoice. Why your agent keeps picking Next.js h0rv noted on HN that among their vibe-coder friends who let the agent decide, "the default stack is always Next.js, Vercel, and Supabase." In another subthread, buildfocus wrote: "I would be very surprised if the tool that agents use most effectively is also the one they recommend." Lawson's data partly vindicates the agent here, since Next.js was cheap to generate. But it also shipped the most JavaScript of any framework (142 KiB), had the worst median LCP (2090 ms) and posted 71 ms of total blocking time. Lawson pins the wide LCP spread across Next.js runs on lazy-loading: one app avoided it and the other four didn't. So Next.js was cheap to write and expensive to serve. You pay the token bill once, and your users pay for the 142 KiB on every page load. How I'd pick a JavaScript framework for agent-written code now Here's what I'm changing in my own setup: 1. Check whether the framework's current major version predates your model's training cutoff. If the agent has to read a migration guide to start, you'll pay for that in every fresh session. Pin the previous major if the new one buys you nothing for this project. 2. Default to server-rendered for content-heavy work. Nuxt, Astro and Hono+htmx were the three cheapest to build in this benchmark, and Astro (0 KiB) and Hono+htmx (18 KiB) shipped almost no JS. 3. Don't go vanilla to save tokens. You won't. Go vanilla because you want zero dependencies, and write the conventions down in your repo so the agent stops reinventing them. 4. Don't assume framework MCP servers help. Compare a few runs with and without them before adopting one team-wide. 5. Run your own version. The benchmark repo takes a framework name and a spec. Swap in a spec that looks like your app. The README recommends "3 or more iterations before drawing conclusions," and I'd take that seriously. n=5 on one Pokédex app is enough to make me suspicious of my defaults, and not much more. Lawson ends with the question I keep coming back to: "Are you really picking your tech stack based on evidence? Or are you just going with what you already know?" With agents in the loop, "what the model already knows" is part of that answer now, like it or not. Has your coding agent ever talked you into React or Next.js when you didn't need it? And which framework does it actually write best for you? If you've got token counts from your own projects that contradict this table, post them. I want to see them. Sources - Nolan Lawson, The AI cost of JavaScript frameworks (Oct 28, 2026) - nolanlawson/agent-framework-benchmark and results - Hacker News discussion

Nolan Lawson, The AI Cost of JavaScript Frameworks: nolanlawson.com Nolan Lawson, Agent Framework Benchmark: nolanlawson/agent-framework-benchmark Hacker News, The AI Cost of JavaScript Frameworks: news.ycombinator.com

source: nolanlawson.com repo: nolanlawson/agent-framework-benchmark fun one: nuxt had its mcp installed, the agents used it in 0 of 5 runs. still the fewest tokens (3.85M)