simonw

LLM 0.37 fixes a subtle agent bug: Response.reply() now preserves tool approval callbacks across reply rounds. Previously, a rejected tool could still run. Practical lesson for AI builders: test denied actions on every execution path. Apache-2.0 code. github.com/simonw/llm/releases/tag/0.37

simonw/llm 今早更新，修了一个隐私坑：明明关了日志，聊天模式却仍在存回答。 新版会遵守关日志设置。但这只管本地留档；接云端模型，请求仍会发出去，处理客户资料时别混为一谈。 simonw/llm

链接 👇 GitHub：simonw/raccoon-heist 在线玩：simonw.github.io 作者博文：simonwillison.net

How I caught a crash by actually running someone's fix Most code review on open source PRs is reading. Someone looks at the diff, it makes sense, they leave a thumbs up. I've started doing something a bit different: I actually install the PR and run it. On Windows, which turns out to be the part that matters. Today I tried it on three open PRs in simonw/llm, the command-line tool for talking to language models. Here's what happened. PR 1: a clean exit that crashed on start This PR made llm chat exit cleanly when you hit end-of-input, instead of printing "Aborted!". The diff was small and looked fine. But it called click.termui._readline_prompt, a private function inside the click library. That function only exists in click 8.4 and newer, and llm doesn't pin a click version. So I set up a throwaway venv and ran it three ways: - main: "Aborted!" (the original bug) - the PR with click 8.5: clean exit, works - the PR with click 8.3.1: AttributeError the moment chat starts So the fix works on the newest click and breaks chat completely on anything older. Reading the diff, you'd never see that. I posted the results and suggested either pinning click>=8.4 or catching EOFError around a plain input(). PR 2: a Windows fix that still fails on Windows This one was about llm install / llm uninstall, running pip in a separate process so llm can update itself. On Windows I ran the self-reinstall case on main and on the PR. - main: pip refuses to modify itself - the PR: WinError 5 (access denied) on a .pyd file The running llm process has that compiled extension loaded, and Windows locks loaded files, so the child pip can't replace it. Different error, same result: it still can't reinstall itself while running. Again, nothing in the diff tells you that. You have to run it on Windows. PR 3: a cleanup that was just fine The third PR removed some old Windows-only readline setup lines. The author said they couldn't reproduce the original problem, so nobody was sure it was safe. I ran it on main and on the PR, checked that chat opens and arrow keys still work, and both behaved the same. So I left a short "tested on Windows, looks good as a cleanup". Not every test finds a bug, and a confirmed "it's fine" is useful too. What I took from it - Running a PR takes maybe 15 minutes with a throwaway venv. That's often more than the author could do, because they might not have your OS or your versions. - Private functions and unpinned dependencies are where "works on my machine" hides. Try the oldest version the project allows, not just the newest. - Windows is underrepresented among testers. If you're on Windows, you're already useful. - Post exactly what you ran and what happened. Let the maintainer decide what to do with it. --- Disclosure: I used AI (Claude) to help pick PRs, write the little test scripts, and draft this post. I ran every test myself on my own PC, and the results above are what I saw.

great report on webframe by @simonw ... gives me thoughts on next iteration. any feedback on priorities? github.com/simonw/research/tree/main/webframe-eva…

Night Scout: Gemma gathers stuff of interest so I wake up to a powerful digest and next steps This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady As with many of you, I browse Hacker News, then Reddit, or Bluesky, then maybe hit up a couple of YouTube videos. Occasionally, a few blogs. An hour later I've read a lot of headlines and a lot of summaries, and I've done nothing. There's a crap ton of exciting AI development, that leaves my head buzzing. There's just too much to wrap my head around and I'm pulled in a ton of directions. There's a bunch of ideas I want to chase and tools I want to try. I realize we need an AI curator of sources of interest. I don't want another summarizer. I want the night's few stories that matter waiting with the coffee, and for the things I'm building, a next step. What I Built Night Scout is an overnight scout. Before bed I do nothing. Overnight it reads the feeds I'd otherwise doom-scroll (YouTube channels, blogs, Hacker News, Reddit, Bluesky, Mastodon and GitHub releases), and a local Gemma model reads the most promising items. The Digest needs nothing but topics. I type the topics I follow in my own words ("Local LLMs", "AI coding agents", "Security"). Gemma files each item it reads under one of them with a strict score, and only stories at 80 % or more make the Digest, 25 at most. Each has a plain summary, three to five key points and, for a video, the player right in the card. Weak titles (a list of names, a wall of hashtags, another language) get a short Gemma headline, with the original underneath. Everything else from the night sits one click below, ranked by similarity, so nothing is hidden. For the night of Oct 22 the Digest opened with "Sixteen stories for your topics. Gemma read 40 of the 90 new items against your 4 topics." Nine were Local LLMs, four AI coding agents, three Security, none Model Context Protocol; the other 74 items were under Everything else. (The Digest is newer than that overnight run, so I filed that night's 40 judged items against the four topics afterwards, with the app's own code on the same Gemma.) Add your projects and it tells you what to do. Point Night Scout at project folders and a second page, Actions, appears. Each pick is labelled for the project it fits: an Issue to check, a Change to track, an Improvement to adopt, or simply Relevant. It shows which profile entry it matched and comes with a typed next step: - Try it: an install and smoke-test plan I can run in a throwaway sandbox with one click, after I read the exact commands and confirm. - Research more: Gemma reads the full article or video transcript and writes a short brief for that specific project. - Note for project: a markdown note dropped into the project's repo, only when I click. - Watch later, or 👍 / 👎 to tune what it lets through next time. I don't write a keyword list. Night Scout reads each project's README and dependency files, and Gemma turns them into a small, editable profile: what the project is for, its stack, its key dependencies, and the kinds of news that would change what I do next ("new releases of feedparser", "security advisories for pydantic"). Whatever I write there is exactly what Gemma judges against. Under the picks, Gemma suggests two small new project ideas inspired by the night's reading. Here's a real night, Oct 21, with a demo profile of three public projects (Night Scout itself, Simon Willison's llm CLI and the MCP TypeScript SDK): 12 feeds, 140 new items, 40 pre-filtered for Gemma, all 40 judged in about 14 minutes on a 16 GB M4 Mac with 0 failures, and 7 picks in the morning: - Issue: a compromised LiteLLM release on PyPI that installs a credential stealer. Next step: check whether LiteLLM is a transitive dependency and pin a known safe version. - Change: llm-tools-datasette 0.1.1, which fixes breakage from an httpx2 migration. - Improvement: llm-echo 0.5, a debug model for testing tool calls. - Relevant: a new Ollama release, an MCP memory server matched to the TypeScript SDK, and two more. (I added the labels a day later, by running the new judge over that night's seven picks.) The next night, Oct 22, was quieter: 90 new items, all 40 candidates judged in 11.6 minutes, 2 picks (an Ollama release as a Change, the OpenAI CLI as Relevant) and two project ideas. That's the point. Some mornings there's little worth doing, and Night Scout says so instead of filling a feed. Scout key terms. A feed only shows what's new. When I follow a source I can add a few terms, and Night Scout looks for them once across the whole source, at any age. Every night after that, new posts that mention one go to the front of Gemma's queue. I tried it on Oct 22 with the Higgsfield AI YouTube channel and the terms "kling, veo". YouTube's own channel search returned 9 videos for "kling" and 10 for "veo", but its search is loose, so Night Scout keeps only real whole-word mentions in the title or description: 7 and 4. Ten videos were queued for the first night; the search took between 11 and 43 seconds across my runs, depending on YouTube. Whole-channel search uses the YouTube Data API if I add a key, or yt-dlp without one. That's the one place Night Scout reads public web pages instead of a feed, and only when I ask for it. Each source also has a look back (1 to 14 days, default 3) for its first night. Ask. An Ask button opens a chat with Gemma on any page. It looks things up before it answers: the Guide, my setup and past runs, every night's items, and web tools (read a page, YouTube transcripts, Hacker News, and web search if I turn it on). Each lookup shows in the drawer as it happens. I can paste a screenshot, or attach text files and PDFs. If I ask it to follow a channel or add a topic, it shows a button under its answer, and nothing changes until I click. A source is checked first, so a card only appears for one that resolves. When Gemma guesses a GitHub owner that doesn't exist and only one repo has that name, the button offers that repo instead, marked "corrected from" the guess. Listen. With an optional add-on (pip install "night-scout[voice]"), a Listen button reads the Digest aloud: each headline, where it's from, and Gemma's summary, topic by topic. The voice is Kokoro, an open-weight speech model that runs on my Mac, with 28 English voices to pick from (Heart by default). After the nightly run it records the Digest, so it plays at once in the morning. The add-on is optional because it's big: about 1.1 GB for PyTorch plus a 330 MB voice model. Local by default, cloud optional per job. Out of the box everything runs on Ollama on my machine. Settings → AI Providers shows each job (Judge, Chat, Fallback, Embeddings) and which model does it. I can give a job to Anthropic, Google Gemini, OpenAI, OpenRouter, DeepSeek, Groq, Mistral or xAI instead, and the app marks that job "Leaves this machine". Embeddings always stay local. Keys go into the OS keychain, never into config files. Demo {% embed youtube.com %} Two minutes, built around the questions I kept asking myself, each answered by the app's real screens: - 0:00 How many channels, blogs and feeds do I follow, and how many did I actually get to? - 0:29 The Digest: what mattered last night, with summaries, key points and the video right in its card. - 0:40 Scout key terms: it searches a whole channel for my terms, keeps only the videos that really mention them, and reads the transcripts so I don't have to. - 0:52 Upgrade your projects: each action names the project it improves, and why. - 1:02 Try it safely: I read the commands, confirm, and a plugin installs and passes its smoke test inside a sandbox that can't see my home folder. - 1:14 Build something new: project ideas from what I read and what I already build; one click starts one, README and all. - 1:24 Ask: I paste a screenshot and Gemma reads it; ask it to follow a channel and it hands me a button. - 1:34 Listen: Kokoro, running locally, reads the Digest aloud. - 1:44 Local by default: every job on Gemma on this Mac, and where a cloud model could take one. The narration is an AI voice (Hume's "Male English Actor") over AI-generated music. The opening and closing stills are AI-generated, and the source chips in the opening are illustrations; everything else is screen capture of the app. !The Night Scout Digest Code {% embed saixiong/nightscout %} MIT licensed. Clone it and double-click Night Scout.command (macOS) or Night Scout.cmd (Windows): the first launch installs it and opens a walkthrough in the browser (model, Just the Digest or Digest + My projects, where 📁 Choose folder… picks project folders in Finder or File Explorer, then topics, sources and schedule). From a terminal it's pip install -e . and night-scout setup. After that the scout goes out every night on its own (launchd on macOS, systemd or cron on Linux; on Windows, night-scout schedule prints the one schtasks line to add), and ▶ Run now on the Digest or Actions page sends it out straight away. The README has the two-minute quick start. How I Built It The open pieces do all the thinking: - Gemma 4 12B (gemma4:12b via Ollama) judges every item, files it for the Digest, proposes actions and answers in Ask. A 4B variant is the configured fallback. - EmbeddingGemma 2 (embeddinggemma-2:270m) does the cheap pre-filter and finds near-duplicate stories across sources. - Python, FastAPI, feedparser, httpx and pydantic hold it together. The UI is plain vanilla JS on a local JSON API: no frontend build, a strict Content Security Policy, and it works at phone width too. - Models are swappable, local first. Ollama is the default; LM Studio or any OpenAI-compatible server works too. Anthropic has its own client; the other cloud providers go through OpenAI-compatible endpoints. One night is seven steps: fetch → dedupe → pre-filter → Gemma judges → Gemma proposes → digest → ideas 1. Fetch from official feeds and public APIs only, politely spaced, with a descriptive User-Agent. Each source fails on its own with a status line ("Reddit r/x: HTTP 429, skipped"), and the run goes on. 2. Dedupe by canonical URL (tracking parameters stripped, YouTube URL forms folded) and by embedding similarity. A story on both HN and Reddit becomes one item with a little extra "buzz". 3. Pre-filter: each interest becomes a retrieval query for EmbeddingGemma, so Gemma reads the 40 most likely items instead of 140. Key-term matches go first. 4. Judge: Gemma returns structured JSON for every item: its topic and score for the Digest, a summary and key points, and with projects a relevance score per project, a label, the matched profile entries and a next step. Matched entries are checked against the profile word for word, so the model can't invent an interest I don't have. - Ollama constrains the output to a JSON schema, and pydantic validates it. - An invalid answer gets one retry with the exact validation error. If it fails again, it's logged and skipped, never fatal. 5. Propose typed actions for items above the bar. 6. Digest as markdown plus the web UI. 7. Ideas: one more Gemma call turns the night's reading into two side-project ideas. Three things I learned the honest way: - The pre-filter is the product. On the first real run, 13 of Gemma's 40 slots went to items it then scored 0.0: PHP CVEs and game news. Meanwhile two releases that mattered to my demo projects were filtered out. Raw embedding cosines all sat between 0.35 and 0.45, so popularity decided everything. The fix: - use EmbeddingGemma's task prompts (task: search result | query: …); - turn every profile interest into its own query; - normalize similarity across the night's batch. The next run had 7 on-topic picks instead of 2, and the missing releases came back. - Schemas are a conversation, not a cage. Once, Gemma wrote a Python smoke test as separate shell steps (from rank_bm25 import … as a command). Now the Try-it schema rejects source code as a step, and that rejection becomes the feedback on the one retry. The model fixes it itself. - Read your tool's build. Homebrew's ollama formula currently ships without the grammar library, so every structured request returned 501 structured output is unavailable. The official build (the ollama-app cask) has it. Night Scout now detects the 501 and falls back to schema-in-prompt plus validation, so it works on both. The part I spent the most time on is safety. A tool that reads untrusted text from the internet and asks a model to suggest shell commands needs to be boring about it: - Nothing runs automatically. The nightly run only proposes, and Ask only offers buttons. - A denylist runs at proposal time and again before running. It covers sudo, curl | sh and download-then-run, global installs, credential paths, local services and backgrounded processes. A refused plan shows its reasons instead of a Run button. - Confirm page. You see the exact commands, where they'll run, the time limit and a supply-chain warning, then you tick a box. - The sandbox is a throwaway container if Docker is around. Otherwise it's a throwaway virtualenv with a scrubbed environment, fenced by the OS: sandbox-exec on macOS blocks my home folder, other drives, local services (including Ollama and the Night Scout UI), the clipboard and launching apps; bubblewrap does the job on Linux. I checked it live: pip install llm-echo plus an import check passed in 10.5 seconds, while reading ~, reaching 127.0.0.1:11434 or reading a keychain item from inside it all fail. - Feed text is treated as data. Every prompt says so, including the chat's tool results. Research more and Ask's page reader only fetch public addresses, connecting to the exact IP they checked. The only remote images are YouTube thumbnails (and the youtube-nocookie player, when I press play), allowed by name in a strict Content Security Policy. The UI needs a per-launch key, so other processes on the machine can't click buttons for me. Testing. Independent review passes audited the code against the spec after each milestone. The first found 3 high and 8 medium issues, including a Linux sandbox with no isolation; the second found 9 more. All are fixed with regression tests. The final build has 556 unit tests and seven Playwright end-to-end kits that drive the real app in a browser: setup, Digest, the guided tour, Digest plus chat, the assistant on real Gemma, sources and providers, 114 of 114 steps passing. Each kit runs sandboxed, with a throwaway home, a throwaway keychain entry and a faked scheduler. The kits earned their keep. They caught chat history dropping tool calls, so Gemma claimed buttons it never made; the drawer hiding proposal cards after a failed tool call; and YouTube feed outages blocking channel adds. All fixed before the final run. Why Does Open Innovation Matter? For this project, a closed API would have been a worse product, not just a cheaper one. - My project context never leaves the laptop. The profile is a map of what I'm building, including private repos. By default Night Scout reads public feeds in and sends nothing out: the model is on localhost. Cloud models are there if I want one for a job, and the app says plainly which jobs that sends off the machine. - $0 per night, forever. Forty judgements plus proposals and embeddings every night is a real token bill on a paid API. Here it's under 15 minutes of a laptop GPU while I sleep. A tool like this only works if it runs every single night without me thinking about cost. - It's mine to tune. I swapped embedding models, changed prompts and schemas, picked a smaller fallback for slower machines, and none of that needed anyone's permission or a pricing page. - Open weights plus open source means you can check the safety story yourself. Every line that decides what gets proposed, refused or run is in the repo. And it was simply better at one thing: the judge reads untrusted internet text all night. Running it locally means that text can't trick the model into sending my profile anywhere, because nothing in the loop can send it anywhere. The Lazy Payoff - Before: about 40 minutes a morning across seven sites, and most days nothing came of it. - After: the night's reading is done while I sleep. On Oct 22 that was 90 new items, 40 read by Gemma, sixteen stories in the Digest and two picks for my projects. Some mornings it just says "Nothing cleared the bar", and that's the point. Prior art, for credit and contrast: AI News Brief, Auto-News and news-agent all aggregate, summarize and rank, and they're good at it. Night Scout's bet is different: it files the night under your own topics, it knows your projects, and every pick ends in something you can do. Prize Categories - Best Use of Gemma: Gemma 4 12B files the Digest, judges and plans the Actions with schema-constrained output, and answers in Ask with tools and screenshots; EmbeddingGemma 2 is the pre-filter and dedupe. By default all of it runs locally through Ollama.

I made an interactive tool for visualising a codebase and how it has grown over time. I found it useful, so I wanted to share it Hi reddit, I wanted to share a tool I made (with Claude code of course) that I’ve found very helpful, in the hope that other people might benefit from it too. It’s called Repo Map. It turns a Git repository into a single interactive page showing: - a foldable file tree with line and character counts; - a treemap showing where the project’s size and complexity live; - language breakdowns; - a timeline you can scrub through to see how every file and folder evolved; - comparisons with earlier commits; - automatic hiding of generated, vendored, lock, and unusually large data files. I find it particularly good for monitoring for code bloat generated by coding agents working in my various repos. The video shows Repo Map pointed at Simon Willison’s open-source LLM project, a command-line tool and Python library for working with large language models. I built Repo Map iteratively with Claude Code. Claude handled much of the implementation while I directed the functionality, tested it against real repositories, identified problems, and refined the behavior and interface. I’d love feedback on: 1. Is this something you would actually use? 2. Is it worth spending more time polishing? 3. How would you prefer to access it? I currently use it as a Claude Code skill, so I can simply ask Claude to “map this repo.” However, the generator itself is standalone and could also become a Claude code plugin/mod or standalone tool like: - a VS Code extension; - a standalone CLI; - a GitHub Action or integration; - or something else I haven’t considered. It is fully open source and free to use. Repository: roderik-maker/repo-map I haven’t really shared any of my (coding) projects publicly before. I tend to be quite critical of my own work, but this latest tool has been useful enough that I think it could genuinely help other people too. Please let me know if this seems useful—or what would need to change before you’d want to use it yourself! Disclaimers: This is a vibe-coded early version, but it has worked well so far and I use it regularly. 😄

A walking meme? this anthropic lead of engineering dev @simonw just gave claude a body and calls it a "first kind of meme" ycombinator.community his github is : github.com/simonw t.co

98% of Battle Drones Was Written by Claude: I Audited 334 Commits With 140 Lines of Python On 20 October Simon Willison published Battle Drones, a browser RTS in the Command & Conquer mould. He built it over 12 major versions and 321 commits, mostly with Claude Code, and he is candid about it: "I've reviewed almost none of the code." The repo is public, which makes it the best open dataset I have seen for a question every team adopting coding agents should ask: how much of our codebase did the agent write, and what happened to that code afterwards? So I wrote a small, stdlib-only Python script that answers it from git log, cloned the repo and ran it. Below are the script, the real output and what I think it means for teams. Why measure this at all My team is small and agents now write a real share of our commits. "Is the agent productive?" is the wrong question. The useful ones are operational: - Review load. If most changed lines come from an agent, reviewers are the bottleneck. The cost of a merge is reading, not writing. - Churn. Code added and deleted within two weeks is work you paid for twice, and agents can produce a lot of it. - Fix-ups. If fix commits keep landing on files an agent just touched, that is where you need tests, a spec or a narrower prompt. - Hotspots. The files agents touch most are where a guardrail (a lint rule, a CODEOWNERS entry, a contract test) pays back fastest. None of this needs a vendor dashboard. Agents already leave fingerprints in git. How it detects agent commits Claude Code adds a Co-Authored-By: Claude ... <noreply@anthropic.com> trailer, and Claude Code on the web also adds a Claude-Session: line with a session URL. Copilot's coding agent commits as copilot-swe-agent[bot] or a +Copilot@users.noreply.github.com address. Cursor, Codex, Devin and Aider have their own patterns. The script matches one regex per agent against author, committer and the full message. Everything else counts as human. Merge commits and CI bots are skipped. Then, in one pass over the full patch history (git log -p with zero context lines), oldest commit first, it computes: - Share: commits and changed lines (added plus deleted) per class, ignoring generated paths such as dist/, lockfiles and minified bundles. - Churn: an added line counts as churned if a later commit deletes an identical line in the same file within N days (default 14). This is content matching, not true blame, so treat it as an estimate. - Fix-ups: commits whose subject says fix, bug, revert, regression and so on, split by whether they touch a file that agent or human work changed inside the window. - Hotspots and timeline: top files by agent-changed lines, plus a per-day table. The script #!/usr/bin/env python3 """agentaudit.py: how much of a git repo did coding agents write? Usage: python3 agentaudit.py [REPO] [--days 14] [--top 8] (stdlib only) """ import argparse, collections, datetime as dt, re, subprocess # One regex per agent, matched against "author / committer / full message". AGENTS = { "Claude": r"noreply@anthropic\.com|co-authored-by:[^\n]*claude|claude-session:" r"|claude\.ai/code/session|generated with \[claude code\]", "Codex": r"co-authored-by:[^\n]*codex|chatgpt-codex-connector|codex@openai\.com", "Copilot": r"copilot-swe-agent|\+copilot@users\.noreply|co-authored-by:[^\n]*copilot", "Cursor": r"cursoragent@cursor\.com|co-authored-by:[^\n]*cursor", "Devin": r"devin-ai-integration", "Aider": r"\(aider\)|co-authored-by:[^\n]*aider", } AGENTS = {k: re.compile(v, re.I | re.M) for k, v in AGENTS.items()} BOT = re.compile(r"\[bot\]|actions@users\.noreply|github-actions", re.I) # CI, not agents FIX = re.compile(r"^(revert|fixup!|hotfix)|\b(fix(es|ed)?|bug|regression|broke|broken|oops)\b", re.I) TRAILER = re.compile(r"^co-authored-by:\s*([^<\n]+?)\s*<", re.I | re.M) # Generated files inflate line counts; skip them (override with --skip). SKIP = r"(^|/)(dist|build|vendor|node_modules|docs/assets)/|\.lock$|-lock\.json$|\.min\.|\.map$" FMT = "%x00%H%x1f%an <%ae>%x1f%cn <%ce>%x1f%at%x1f%P%x1f%B%x1e" def who(author, committer, msg): text = f"{author}\n{committer}\n{msg}" return next((name for name, rx in AGENTS.items() if rx.search(text)), None) def audit(repo, days, skip_rx): skip_rx = re.compile(skip_rx) out = subprocess.run(["git", "-C", repo, "log", "--reverse", "-p", "-M", "--unified=0", "--no-color", "--format=" + FMT], capture_output=True, text=True, errors="replace", check=True).stdout window = days * 86400 s = collections.defaultdict(collections.Counter) # stat name -> {"agent"/"human": n} per_agent, models = collections.Counter(), collections.Counter() files, day = collections.defaultdict(collections.Counter), collections.defaultdict(collections.Counter) added = collections.defaultdict(lambda: collections.defaultdict(collections.deque)) last_touch = {} # path -> (ts, cls) for chunk in out.split("\x00")[1:]: head, patch = chunk.split("\x1e", 1) sha, author, committer, ts, parents, msg = head.split("\x1f") ts = int(ts) if len(parents.split()) > 1: s["merges"]["all"] += 1 continue agent = who(author, committer, msg) if not agent and BOT.search(author): s["merges"]["bots"] += 1 continue cls = "agent" if agent else "human" if agent: per_agent[agent] += 1 models.update(t for t in TRAILER.findall(msg) if AGENTS[agent].search(f"co-authored-by: {t}")) s["commits"][cls] += 1 if not agent and re.search(r"\(#\d+\)$", msg.split("\n", 1)[0].strip()): s["squash"]["human"] += 1 # squash-merged PR: author unknown d = dt.datetime.fromtimestamp(ts, dt.timezone.utc).strftime("%Y-%m-%d") day[d][cls] += 1 path, in_hunk, skip, touched = None, False, False, set() for line in patch.split("\n"): if line.startswith("diff --git "): m = re.match(r"diff --git a/(.*) b/(.*)", line) old, path = (m.group(1), m.group(2)) if m else (None, line) if old and old != path and old in added: # follow renames added[path] = added.pop(old) skip = bool(skip_rx.search(path)) if not skip: touched.add(path) in_hunk = False elif line.startswith("@@"): in_hunk = not skip elif in_hunk and line[:1] in ("+", "-"): content = line[1:].strip() s["lines"][cls] += 1 files[path][cls] += 1 day[d][cls + "_lines"] += 1 if line[0] == "+": s["added"][cls] += 1 if len(content) > 2: # skip "}", "", ");" added[path][content].append((ts, cls)) elif added[path][content]: t0, c0 = added[path][content].popleft() # oldest matching add if ts - t0 <= window: s["churn"][c0] += 1 if FIX.search(msg.split("\n", 1)[0]): s["fixes"][cls] += 1 hit = {last_touch[p][1] for p in touched if p in last_touch and ts - last_touch[p][0] <= window} s["fix_after"].update(hit) for p in touched: last_touch[p] = (ts, cls) return s, per_agent, models, files, day def pct(a, b): return f"{100 * a / b:5.1f}%" if b else " n/a" def report(repo, days, top, skip_rx): s, per_agent, models, files, day = audit(repo, days, skip_rx) c, l = s["commits"], s["lines"] nc, nl = sum(c.values()), sum(l.values()) print(f"repo: {repo} window: {days} days merges skipped: {s['merges']['all']} bot commits skipped: {s['merges']['bots']}") print(f"commits agent {c['agent']:>6} {pct(c['agent'], nc)} human {c['human']:>6} {pct(c['human'], nc)}") print(f" human commits that look like squash-merged PRs (true author unknown): " f"{s['squash']['human']}") print(f"lines agent {l['agent']:>6} {pct(l['agent'], nl)} human {l['human']:>6} {pct(l['human'], nl)}") print("agents " + ", ".join(f"{k} {v}" for k, v in per_agent.most_common())) print("models " + ", ".join(f"{k} {v}" for k, v in models.most_common(5))) for cls in ("agent", "human"): print(f"churn {cls}: {s['churn'][cls]} of {s['added'][cls]} added lines deleted within " f"{days}d ({pct(s['churn'][cls], s['added'][cls]).strip()})") print(f"fixes fix/revert commits: {sum(s['fixes'].values())} (by agent {s['fixes']['agent']}, " f"by human {s['fixes']['human']})") for cls in ("agent", "human"): print(f" touching files last changed by {cls} work <= {days}d earlier: " f"{s['fix_after'][cls]} ({s['fix_after'][cls] / max(c[cls], 1) * 100:.1f} per 100 {cls} commits)") print(f"\ntop {top} files by agent-changed lines:") for p, n in sorted(files.items(), key=lambda kv: -kv[1]["agent"])[:top]: print(f" {n['agent']:>6} {pct(n['agent'], n['agent'] + n['human'])} agent {p}") print("\nday commits A/H lines A/H agent share of lines") for d in sorted(day): n = day[d] share = n["agent_lines"] / max(n["agent_lines"] + n["human_lines"], 1) print(f"{d} {n['agent']:>4}/{n['human']:<4} {n['agent_lines']:>7}/{n['human_lines']:<6} " f"{'#' * round(share * 20):<20}") if __name__ == "__main__": ap = argparse.ArgumentParser(description=__doc__) ap.add_argument("repo", nargs="?", default=".") ap.add_argument("--days", type=int, default=14) ap.add_argument("--top", type=int, default=8) ap.add_argument("--skip", default=SKIP, help="regex of paths to ignore") a = ap.parse_args() report(a.repo, a.days, a.top, a.skip) Usage is just this, and it took about 4 seconds on Battle Drones: python3 agentaudit.py path/to/repo --days 14 --top 8 Battle Drones: the real output I cloned at commit 39d6f58 (20 Oct). The clone has 334 commits, 48 of them merges, so slightly more than the 321 counted in Simon's post. Dates are UTC. repo: . window: 14 days merges skipped: 48 bot commits skipped: 0 commits agent 267 93.4% human 19 6.6% human commits that look like squash-merged PRs (true author unknown): 0 lines agent 170299 98.1% human 3258 1.9% agents Claude 267 models Claude Opus 5.5 229, Claude Fable 5 28, Claude 1 churn agent: 12824 of 153755 added lines deleted within 14d (8.3%) churn human: 362 of 2875 added lines deleted within 14d (12.6%) fixes fix/revert commits: 39 (by agent 39, by human 0) touching files last changed by agent work <= 14d earlier: 38 (14.2 per 100 agent commits) touching files last changed by human work <= 14d earlier: 0 (0.0 per 100 human commits) top 8 files by agent-changed lines: 6362 100.0% agent src/ui/style.css 4640 100.0% agent DESIGN.md 3874 100.0% agent src/game/GameScene.ts 2747 100.0% agent src/ui/hud.ts 2680 99.8% agent SOUND-DUEL.md 2444 100.0% agent src/game/three/terrain.ts 2365 100.0% agent src/sim/ai/commander.ts 2339 100.0% agent public/assets/sprites/game.json day commits A/H lines A/H agent share of lines 2026-07-17 28/1 6504/2 #################### 2026-09-28 3/0 13953/0 #################### 2026-10-07 23/0 33545/0 #################### 2026-10-08 5/1 9589/581 ################### 2026-10-09 21/2 10002/473 ################### 2026-10-10 40/5 11759/683 ################### 2026-10-11 18/0 10034/0 #################### 2026-10-12 65/0 16785/0 #################### 2026-10-13 10/0 3075/0 #################### 2026-10-14 6/7 3934/1210 ############### 2026-10-15 9/0 2832/0 #################### 2026-10-16 3/0 2677/0 #################### 2026-10-19 15/0 33969/0 #################### 2026-10-20 20/3 11597/309 ################### 2026-10-21 1/0 44/0 #################### What I read from it: 98.1% of changed lines are agent-written. The history backs up "mostly by Claude Code": 267 of 286 non-merge commits carry a Claude signal. 229 name Claude Opus 5.5 in the trailer and 28 name Claude Fable 5, which matches his account of starting v1 on an earlier model in July. Generated files distort everything. My first run had no skip list. Agent share of lines came out at 95.4%, and five of the top eight "agent files" were build output under docs/assets/ plus package-lock.json. If you are measuring review load, exclude what nobody reviews. Squash merges nearly hid commits. Simon is the author on 28 non-merge commits. Nine of them are squash-merged agent PRs, with subjects like "Shift+click adds or removes a unit from the selection (#6)". Eight of those nine have no Co-Authored-By trailer at all and are caught only because the message keeps a Claude Code session link. Remove that one signal and the human commit count goes from 19 to 27. Churn is moderate. 8.3% of agent-added lines were deleted within 14 days. The human figure is higher at 12.6%, but on only 2,875 lines, so I would not read anything into the comparison. For a game rebalanced across 12 versions with overnight playtest runs, I expected more. Every fix commit came from the agent. There are 39 fix-style commits, all agent-authored, and 38 of them touch files that agent work changed in the previous two weeks. Most subjects read like "Fixes from the final review" or "Sound duel: fix the final review's three major findings". The fix loop here is agents reviewing and repairing agent code. That works for a game. It is the first number I would want visible before letting the same workflow near a billing service. Hotspots are where you would expect. src/ui/style.css, GameScene.ts, hud.ts and the AI's commander.ts lead. Two of the top five are design docs (DESIGN.md, SOUND-DUEL.md): the agent writes and maintains its own specs, which is a habit worth copying. A second repo: where attribution breaks For contrast I ran it on simonw/tools, his collection of single-file HTML tools, at commit befbcec. Top of the output: repo: . window: 14 days merges skipped: 6 bot commits skipped: 483 commits agent 226 24.5% human 698 75.5% human commits that look like squash-merged PRs (true author unknown): 48 lines agent 72709 9.1% human 727389 90.9% agents Claude 214, Copilot 12 models Claude 155, copilot-swe-agent[bot] 9, Claude Fable 5.1 6, Claude Opus 5.5 4, claude[bot] 3 churn agent: 1420 of 68797 added lines deleted within 14d (2.1%) churn human: 4639 of 716876 added lines deleted within 14d (0.6%) fixes fix/revert commits: 74 (by agent 40, by human 34) touching files last changed by agent work <= 14d earlier: 18 (8.0 per 100 agent commits) touching files last changed by human work <= 14d earlier: 42 (6.0 per 100 human commits) top 6 files by agent-changed lines: 5244 100.0% agent grok-mermaid/src/mermaid.rs 2929 100.0% agent chirp-pong.html 2343 100.0% agent video-compressor.html 2080 100.0% agent bluesky-bot-check.html 1614 100.0% agent latex.html 1613 100.0% agent sqlite-query-explainer.html Here only 24.5% of commits and 9.1% of lines are attributed to agents (Claude 214, Copilot 12). Agent code churns more than human code (2.1% vs 0.6%) and its files attract slightly more follow-up fix commits (8.0 vs 6.0 per 100 commits). But the "human" bucket is not what it looks like. Of the 698 commits the script calls human, 112 link to a shared Claude conversation and 48 to a ChatGPT one, and 48 look like squash-merged PRs. Code written in a chat window and pasted in by hand looks exactly like human code to git. Git records who committed, not who wrote. Caveats to take seriously - Trailers are voluntary. They can be disabled in settings, stripped by hand or lost in a rebase. Treat the agent share as a floor. - Squash merges erase authorship unless the squash message keeps the trailer or a session link. If you want this metric, make that a merge rule. - Chat-and-paste is invisible. The script only sees agents that commit. - Content-matched churn misses lines that were edited slightly and counts moved lines as churn. - "Fix" in a subject line is a crude signal. It is fine for trends inside one repo and poor for comparing teams. Most of these are solved by policy, not code: require the trailer, keep it through squash merges, and run something like this weekly in CI so the trend is visible before it becomes a surprise. Where this fits At Axionry we build LLM apps, agents, RAG pipelines and MCP tooling, and every build ships with observability and cost-per-request dashboards. A git audit like this is the development-side twin of those dashboards: production metrics tell you what an agent costs to run, this tells you what agent-written code costs to review and keep. If you are scoping an AI feature and want a rough number first, try our AI product cost estimator. The script is above, 140 lines, no dependencies. Point it at your own repo and see what your real agent share is. Sources: Simon Willison, "Battle Drones", simonw/battledrones on GitHub, simonw/tools on GitHub, AI Weekly coverage.

Not clear how you submit new listings to this site. I'd like to get my little RTS simonw/battledrones on there - I released it yesterday (Apache 2.0 license) and it's already attracted a fork that translates it to French and redoes the graphics to be 1900s Belle Époque steampunk: simonwillison.net