nolanlawson

Le billet de Nolan Lawson, avec le protocole, les coûts estimés et les mesures côté navigateur : nolanlawson.com Le dépôt du benchmark, pour consulter les consignes et les résultats ou refaire le test : nolanlawson/agent-framework-benchmark

AI Coding Agents Do Not Make Framework Choice Irrelevant. They Make Its Costs Measurable A familiar argument is making the rounds in web development: if an AI agent writes most of the code, developer experience matters less, so teams should choose the framework that works best for agents. It sounds plausible. It also skips the hard question: what does “works best” mean, and how would you know? Nolan Lawson tried to answer that with a small benchmark. He asked Claude Opus 5.5 to build the same Pokémon catalog app in 12 JavaScript stacks, tested each result against a shared conformance suite, and measured cost, time, code size, and selected Lighthouse metrics. He ran five iterations per stack, except Angular, which had three measurable results for some metrics. The benchmark write-up and its open-source harness make the experiment inspectable. The headline is not that one framework won. The useful finding is that “agent-friendly” is not a single score. In this test, Hono with htmx was among the least expensive and shortest-code options, while Astro sent no JavaScript on the tested page. Angular cost more and generated more code, but this is one content-oriented application, one model, one benchmark specification, and a small number of runs. The evidence is enough to question vague claims. It is not enough to declare a universal winner. That distinction matters because framework decisions are long-lived. If the agent saves a few minutes today but creates more code to inspect and maintain tomorrow, the initial speedup may be a poor trade. The better response to AI-assisted development is not to pick a framework based on intuition about what models prefer. It is to extend the measurement window from “how fast did the agent finish?” to “what does this choice cost across the life of the application?” What the experiment actually measured A shared task, not a framework tutorial. Lawson specified a small but non-trivial Pokémon app using local data from PokéAPI. Agents had to implement the same user-visible features and pass the same conformance suite. They were allowed to use official framework MCP tools or skills where available, and the harness recorded whether the agent actually used them. The benchmark repository says each run used a fresh Podman container and that the generated app was tested separately from the agent environment. The repository README describes the controls and reproduction setup. More than token count. Lawson reported API-equivalent cost, wall-clock time, tokens, lines of code, and Lighthouse measurements for performance and accessibility. That is a useful choice: an agent can spend more tokens to produce code that is smaller, more accessible, or faster for users. Optimizing only for the first completion time would miss those trade-offs. The median reported cost ranged from $2.67 for Hono plus htmx to $4.97 for Angular. These are not the experimenter's cash charges. Lawson says he used a Claude Max plan and reported the API prices quoted by Claude Code. The cost range is therefore a way to compare the reported usage, not a billing receipt or a forecast of what a team will pay. The time and token figures also varied. Astro's median run took 8.2 minutes, while Angular's took 17.2 minutes. Median token use ranged from 3.85 million for Nuxt to 9.76 million for Angular. But those figures came from five runs per framework, with Angular's reduced sample in some measurements. The author's own write-up notes stochastic results and overlapping ranges. A median is a helpful summary, not proof that the same ordering will hold on your task. The counterintuitive result is the important one Raw JavaScript was not automatically the cheapest route for the agent. The benchmark's “Vanilla” condition used JavaScript or TypeScript with Vite and no UI framework. It was not simply “write the smallest possible plain JavaScript app.” The agent made architectural choices inside the allowed setup, and the result used a median of 5.00 million tokens and took 10.1 minutes. Several framework conditions used fewer tokens or completed sooner. That does not prove frameworks are better for agents. It demonstrates that removing a framework does not automatically remove complexity. A framework can supply conventions, routing, data loading, and a structure an agent can follow. Conversely, it can introduce conventions the model does not know well. Which effect dominates depends on the framework, task, tool setup, and model version. The web payload and the agent's build cost are different budgets. Astro's tested page transferred a median of 0 KiB of JavaScript, while Next.js transferred 142 KiB. Hono plus htmx used 18 KiB, and Angular 93 KiB. On that metric, the result separates framework bundles more clearly than the agent's cost rankings do. But a zero-JavaScript page is not a complete performance verdict: the application still has server work, network requests, content, and user interactions. The benchmark also reports different results for blocking time and largest contentful paint, with caveats around variance and the Lighthouse setup. If a team collapses those measurements into one “best” framework score, it will hide the very trade-offs that make the exercise useful. Agent usage cost is an engineering input. Browser payload and responsiveness are user-facing outputs. Maintainability is a future cost. They should not be treated as interchangeable just because they came from the same run. Code size is a proxy, not maintainability itself. Hono plus htmx generated a median of 1,370 non-blank source lines; Angular generated 2,009. That is a measurable difference, and smaller implementations may mean fewer tokens for later agent edits. Lawson explicitly connects code volume with the cost of future changes. Yet line count cannot tell you whether code is understandable, well-factored, secure, or easy for a human team to change. A short, opaque implementation is not automatically maintainable. Why “better for agents” is a weak architecture rule Framework advocates can point to model familiarity, official agent skills, MCP servers, and established examples. Those can matter. But the benchmark found that agents did not consistently use the available tooling. For example, the report says some frameworks' tools were used in only a subset of runs, and Nuxt's MCP was not used in the observed attempts. Installing an integration is not the same as proving it changed the result. There is also a training-data trap. Lawson suggests that SvelteKit's higher measured cost may partly reflect its recent release: the agent repeatedly read its migration guide, possibly because the latest version was not represented well in the model's training data. That explanation is plausible, but it is not isolated as a causal factor. A follow-up would need to vary model version, tool access, and framework documentation exposure independently to determine whether unfamiliarity caused the extra work. The result is a moving target. Framework versions change, models change, tools change, and the same agent can make different choices across runs. A one-time benchmark can help choose what to investigate. It cannot permanently rank a technology stack. That is especially true when official AI tooling is unevenly available or the agent ignores it. How to use this kind of benchmark without overfitting Start with your application shape. A content catalog is not a collaborative editor, admin dashboard, offline-first mobile web app, or real-time interface. Lawson says the sample is closer to a content site than a highly interactive product. If your application is dominated by complex client-side state, this result is a poor proxy. Pick a representative slice of your own product before treating any framework ranking as relevant. Define acceptance criteria before the run. Specify user-visible behavior, accessibility expectations, performance thresholds, supported browsers, and operational constraints. Keep the same tests across candidates. If one framework gets a different prompt or extra hints, the result measures prompt quality as much as framework choice. Measure failed attempts, not just successful output. In the benchmark, Angular failed to create a measurable app in two of five runs. Lawson retained the three usable iterations for some metrics and explained the difference. That transparency is important. In your own evaluation, record failures, retries, timeouts, test results, and cleanup effort. Do not quietly discard the runs that make a preferred tool look worse. Separate repeatability from generalization. Run enough iterations to see whether differences exceed run-to-run noise. Then repeat with a second task and, if relevant, a second model. The original author notes variance in both agent output and Lighthouse. Re-running one task reduces some uncertainty; it does not make that task representative of every product. Track the whole cost surface. At minimum, keep separate measures for: - Agent spend or token use per accepted change. - Human review and correction time. - Test pass rate and severity of defects. - Browser performance and accessibility. - Code size and complexity signals. - The effort required to add a second feature after the first implementation. That last measure is easy to skip and potentially more revealing than initial generation time. A framework that lets an agent produce a first screen quickly may not help when the next request changes data flow, permissions, or navigation. Add a follow-up task that tests whether the generated structure supports change without a rewrite. A decision checklist for framework evaluation Before you switch stacks because a model appears to prefer one, ask: 1. Is the benchmark task representative of our actual product? 2. Did each candidate receive the same specification, model, resource limits, and acceptance tests? 3. Were official agent tools installed, and did the agent use them? 4. Are the reported differences larger than the run-to-run spread? 5. Did we count failed runs and human repair work? 6. Are user outcomes measured independently from agent speed and cost? 7. Does the generated code remain workable after a second, realistic change? 8. Would we make the same choice if the current model or tool integration disappeared next month? The last question is a useful stress test. A framework should not be selected solely because one model has recently seen more examples of it. Model familiarity can improve today's result, but application needs, ecosystem support, and maintainability still matter after the next model update. The takeaway: benchmark claims, not vibes This experiment is valuable because it exposes its scope, shares its harness, reports multiple measures, and discusses limitations. Its results are not a framework league table. They are a reminder that the cost of a stack now includes both the work an agent spends producing code and the work people spend checking, operating, and changing that code. For a team choosing a framework, the practical move is modest: use public benchmarks to form hypotheses, then test those hypotheses on a small, representative feature with your own requirements. Keep the agent's cost, user-facing quality, reliability, and maintenance effort separate. If the decision changes when you add a second task or another model, you have learned something important about how fragile the original ranking was. I write about developer tools, infrastructure, and AI. Subscribe if you want evidence-based engineering analysis without treating a benchmark as a universal answer. Have you measured how your framework changes an AI agent's cost or the quality of the code it produces? What would you include in a fair test? Sources - Nolan Lawson: The AI cost of JavaScript frameworks - Open-source benchmark repository and reproduction notes

Best JS Framework for AI Coding: SvelteKit Burns 71% More Tokens Than Next.js If you asked me last month which JavaScript framework an AI coding agent would build fastest and cheapest, I'd have said SvelteKit. Small API, little boilerplate. In Nolan Lawson's new benchmark, Claude Opus 5.5 burned 8.53 million tokens building a small Pokédex app in SvelteKit. Next.js built the same app for 4.99 million. Nuxt did it for 3.85 million. The post is The AI cost of JavaScript frameworks, and it's the first real data I've seen on the "best JavaScript framework for AI coding" question that people keep settling on vibes. My take after reading it and the Hacker News thread: your framework choice now comes with a token bill, and that bill depends less on how elegant the framework is than on how well the model already knows its current API. What Lawson actually measured He gave Claude Opus 5.5 the same spec, the same tests and the same short prompt across 12 frameworks, changing only the framework name. The prompt template from the benchmark repo is: Implement the app described in SPEC.md using {framework}. Use the latest stable version. Work autonomously until `npm test` passes. The app is a Pokémon browser backed by a local copy of PokéAPI, with search and sorting. Each run happened in a fresh Podman container. Data had to be fetched at request time, so no static generation. Tailwind and component libraries were banned. Frameworks that ship an official MCP server or agent skills had them installed. Lawson took the median of 5 runs per framework, then measured the output with mobile Lighthouse (4x CPU throttling, Slow 4G). He's upfront about the limits: one small, content-heavy app, which favors server-rendered MPAs on first-load metrics, and interactions like search aren't measured, which is where SPAs could win back ground. It's still more real data than every "agents just work better with X" tweet I've read this year. React vs Svelte vs Vue vs vanilla JS: tokens, cost and Lighthouse results Medians from the post, sorted by tokens: | Framework | Tokens (M) | Cost (USD) | Time (min) | JS shipped (KiB) | LCP (ms) | Lines of code | |---|---|---|---|---|---|---| | Nuxt (Vue) | 3.85 | $2.82 | 9.0 | 84 | 1727 | 1415 | | Astro | 4.13 | $2.76 | 8.2 | 0 | 810 | 1461 | | Hono+htmx | 4.26 | $2.67 | 8.6 | 18 | 978 | 1370 | | Next.js | 4.99 | $3.14 | 10.9 | 142 | 2090 | 1539 | | Vanilla (Vite) | 5.00 | $2.98 | 10.1 | 7 | 1667 | 1492 | | Qwik | 5.64 | $3.27 | 10.7 | 40 | 979 | 1744 | | TanStack Start | 7.07 | $3.54 | 10.3 | 118 | 2028 | 1697 | | SolidStart | 7.13 | $3.79 | 11.2 | 40 | 1204 | 1630 | | React Router | 7.58 | $3.81 | 12.5 | 115 | 1729 | 1564 | | SvelteKit | 8.53 | $4.28 | 10.8 | 36 | 1130 | 1611 | | Gracile+Lit | 9.04 | $4.65 | 12.1 | 8 | 956 | 1717 | | Angular | 9.76 | $4.97 | 17.2 | 93 | 1954 | 2009 | Cost is at Claude Code API prices, not what he actually paid on a Max plan. Accessibility scored 100 everywhere, and Lighthouse performance landed between 97 and 100 for all of them, so every stack ended up with a working, decent app. Where they differ is the bill and how much JS reaches the browser. Angular used about 2.5x the tokens of Nuxt and took nearly twice as long. Two of its five runs didn't produce a measurable app because the agent built an SSR app that rejected non-localhost requests, even though the spec said it would sit behind a reverse proxy. Why SvelteKit cost so many tokens The agent read SvelteKit's v3 migration guide in every single run. The prompt asked for the latest stable version, which was @sveltejs/kit 3.1.0, and Lawson thinks that release is likely too new to be in the model's training data. So each session started by catching up on APIs the model didn't know. So yes, training-data bias, but not the version the "LLMs just default to React" crowd describes. React itself gets no special treatment. The model does well on API surface it has seen a lot of and that hasn't changed since, and Next.js and Nuxt are huge, mature and all over the training corpus. React Router and TanStack Start are both React too, and they still used 42-52% more tokens than Next.js, JSX and all. One HN commenter, ef2k, summed it up: "A significant part of the cost is that newer framework APIs are not known to the current agents." So the framework you love for shipping a clean new reactivity model is, right now, the one your agent has to re-learn on your dime. smrtinsert wrote that "sveltekit 2 was super quick to use for me," which fits the theory: same framework, but an API the model had already seen. Vanilla JS didn't save tokens either I went in expecting vanilla to be the cheapest option, on the theory that fewer abstractions means fewer APIs to hallucinate. It landed mid-pack at 5.00M tokens, basically tied with Next.js, and it was the only stack with layout shift (CLS 0.111). Its LCP of 1667 ms trailed Astro, Gracile+Lit, Hono+htmx, Qwik, SvelteKit and SolidStart. Lawson's read is that "having _some_ framework can be better than nothing (or better than raw Vite, anyway)." My guess at why: with raw Vite the agent has to invent routing, data loading and rendering conventions from scratch on every run, and that costs tokens too. oatmealsnap on HN made the long-term version of the argument: in a vanilla codebase, "your agents (or human devs) are going to write element.addEventListener and document.querySelectorAll countless times." The winners were the boring ones: server-first, stable, with a small API surface. Astro shipped 0 KiB of JS, had the best LCP (810 ms) and the shortest median build time (8.2 min). Hono+htmx had the lowest median cost at $2.67 and produced the least code, 1370 lines against Angular's 2009. Framework MCP servers and skills didn't lower the token bill Where a framework had official AI tooling, it was installed, and agents used it inconsistently: - SvelteKit's MCP/plugin, the React Router skill and the Hono skills: used in 5/5 runs - Angular MCP: 3/5 - Astro MCP: 2/5 - Next.js: skill files read in 1/5 runs, AGENTS.md in 2/5 - Nuxt MCP: never used Put that list next to the token table. Nuxt never touched its MCP server and was the cheapest on tokens. SvelteKit used its tooling every time and was the third most expensive. Lawson says he suspects the current crop of framework MCPs and skills "may be doing more harm than good." I'd go further: if your framework needs an MCP server so the agent can write it correctly, that's the framework tax showing up on your invoice. Why your agent keeps picking Next.js h0rv noted on HN that among their vibe-coder friends who let the agent decide, "the default stack is always Next.js, Vercel, and Supabase." In another subthread, buildfocus wrote: "I would be very surprised if the tool that agents use most effectively is also the one they recommend." Lawson's data partly vindicates the agent here, since Next.js was cheap to generate. But it also shipped the most JavaScript of any framework (142 KiB), had the worst median LCP (2090 ms) and posted 71 ms of total blocking time. Lawson pins the wide LCP spread across Next.js runs on lazy-loading: one app avoided it and the other four didn't. So Next.js was cheap to write and expensive to serve. You pay the token bill once, and your users pay for the 142 KiB on every page load. How I'd pick a JavaScript framework for agent-written code now Here's what I'm changing in my own setup: 1. Check whether the framework's current major version predates your model's training cutoff. If the agent has to read a migration guide to start, you'll pay for that in every fresh session. Pin the previous major if the new one buys you nothing for this project. 2. Default to server-rendered for content-heavy work. Nuxt, Astro and Hono+htmx were the three cheapest to build in this benchmark, and Astro (0 KiB) and Hono+htmx (18 KiB) shipped almost no JS. 3. Don't go vanilla to save tokens. You won't. Go vanilla because you want zero dependencies, and write the conventions down in your repo so the agent stops reinventing them. 4. Don't assume framework MCP servers help. Compare a few runs with and without them before adopting one team-wide. 5. Run your own version. The benchmark repo takes a framework name and a spec. Swap in a spec that looks like your app. The README recommends "3 or more iterations before drawing conclusions," and I'd take that seriously. n=5 on one Pokédex app is enough to make me suspicious of my defaults, and not much more. Lawson ends with the question I keep coming back to: "Are you really picking your tech stack based on evidence? Or are you just going with what you already know?" With agents in the loop, "what the model already knows" is part of that answer now, like it or not. Has your coding agent ever talked you into React or Next.js when you didn't need it? And which framework does it actually write best for you? If you've got token counts from your own projects that contradict this table, post them. I want to see them. Sources - Nolan Lawson, The AI cost of JavaScript frameworks (Oct 28, 2026) - nolanlawson/agent-framework-benchmark and results - Hacker News discussion

Nolan Lawson, The AI Cost of JavaScript Frameworks: nolanlawson.com Nolan Lawson, Agent Framework Benchmark: nolanlawson/agent-framework-benchmark Hacker News, The AI Cost of JavaScript Frameworks: news.ycombinator.com

source: nolanlawson.com repo: nolanlawson/agent-framework-benchmark fun one: nuxt had its mcp installed, the agents used it in 0 of 5 runs. still the fewest tokens (3.85M)