backnotprop

Before I Could Evaluate My Coding-Agent Harness, I Had to Isolate It I started writing an article about the relationship between coding-agent models and the harnesses around them. I wanted to explain something I had learned from comparing a few configurations, but the results never gave me a clear enough answer to finish it. Instead, I built yuurei, a CLI for running coding agents with isolated, versionable configurations, without swapping out my everyday global setup. This is the story of the article I did not finish, the comparison behind it, and why I decided that isolation and evaluation should be separate tools. Adapted from my Japanese article, originally published on September 14, 2026. Was my harness holding the model back? Here, by harness, I mean the workflow instructions I had accumulated in skills and files such as CLAUDE.md and AGENTS.md: understand the code first, review the design, stop when something looks risky, ask another agent to verify the result. I wanted to try the newly released GPT-6 Astra, so I casually asked Codex, which I rarely used at the time, to refactor one of my open-source projects. The output was better than I expected. Could this be better than what I was getting from Claude Code, where I had put much more work into the harness? I ran three configurations, using the same repository, starting commit, and prompt: | Run | Runtime and model | Harness | | --- | --- | --- | | PR1 | Codex CLI / GPT-6 Astra | Only the general-purpose pstack skill | | PR2 | Claude Code / Sonnet 5 | My custom harness | | PR3 | Claude Code / Sonnet 5 | Only pstack | pstack is a general-purpose collection of skills for coding agents, covering tasks such as investigation, design, and verification. “Only pstack” means I used those skills without my custom harness. My hypothesis was that the harness might be setting a ceiling on the agent's work. Perhaps all those constraints were good at giving it reasons to stop, but less good at helping it look for a safe alternative after stopping. That would have been something I could fix. I had written the harness. The comparison did not support that hypothesis. In PR3, removing the custom harness narrowed the agent's exploration. It did not produce dedicated tests or reach a design decision. The result was disappointing because the explanation I had hoped to find was in something I could change myself. Instead, differences between the models and execution environments remained possible explanations. I ran each condition only once: n = 1 per condition. This was not enough to establish a general effect or separate model effects from runtime effects. What I could say was narrower: in this comparison, taking away my custom harness did not produce the improvement I expected. I still wanted to keep comparing An inconclusive result did not make the question less interesting. Advice that helped an earlier model can become unnecessary as models improve. My harness had grown through repeated additions, and I could not say how much each part still contributed. I wanted to keep comparing configurations so I could make my own development output more consistent. To do that, I needed the comparisons to be practical to repeat. Even these three runs had been painful to set up. A Git worktree only solves half the problem For the repository, git worktree gave me separate working trees at the same starting point. The harness was harder. Skills, instruction files, settings, and hooks were spread across several places. Each runtime had its own configuration locations and loading rules. With Claude Code, I could point CLAUDE_CONFIG_DIR at another directory. But I could not simply assume that this isolated everything. I had to check the Setting sources section of /status each time to see what had actually been loaded. I used procedures and scripts to switch out the custom parts of my Claude Code harness. Restoring my normal environment afterward still took time because I had not put everything back correctly. I also started the Codex run assuming pstack would be absent because I rarely used Codex. Only afterward did I realize it had been active there too. That was one comparison with three runs. Removing individual harness components would quickly give me ten or twenty variants. Every variant would add another round of backing up, swapping, restarting, and restoring configuration. I could see myself getting stuck maintaining the comparison environment before I ever reached the thing I wanted to measure. Making the harness a versioned input That is why I built yuurei. For each run, it creates a temporary execution cell and runs the coding agent with the selected profile, keeping the runtime's configuration separate from my normal global setup. The basic model has three parts: | Concept | Meaning | | --- | --- | | Profile | A runtime and its native configuration files: skills, instructions, settings, and hooks | | Task | The instructions sent to the runtime | | Run | An execution using a profile and task, with its own trace and logs | Profiles live under .yuurei/, so I can keep them in version control. For me, that is the most useful part: a harness becomes a named artifact I can preserve instead of a collection of files I continuously overwrite. Here is a minimal example using the current CLI. It requires Node.js 22 or later, an installed Claude Code runtime, and an explicit authentication environment variable supported by the adapter; the README explains the authentication options. npm install -g yuurei mkdir yuurei-demo cd yuurei-demo yuurei init yuurei doctor yuurei profile list yuurei run hello The default scaffold selects Claude Code and creates a small greeting task. It is a way to check the setup, rather than a refactoring experiment. Authentication must be ready before the run; an existing interactive login alone is not necessarily sufficient. After a run, I can inspect its record using the run ID printed by the CLI: yuurei trace show <run-id> Each run has a trace.json recording information such as the runtime, execution outcome, timeout status, profile and task digests, and isolation verification result. Logs and the trace are retained under .yuurei/runs/<run-id>/. The digests let me check which configuration was used. They do not establish that the resulting code is correct. That distinction mattered in the original comparison. The PR that looked most autonomous also incorrectly reported that its own checks had passed. An agent's report needs verification. So does the infrastructure I use to measure it. There is also a specific limit to the word isolation here. Yuurei separates configuration and environment variables; its default isolation level uses a temporary HOME. It is not an OS sandbox, and code executed by the agent still has the access available to my user account. Likewise, reproducible configuration does not imply identical agent behavior across runs. Why I separated isolation from evaluation My first plan was to build one tool that both isolated runs and evaluated them. I abandoned that plan because I had not worked out the evaluation design. The differences that stood out in the original comparison concerned behavior, rather than simply the quality of the final artifact: - Did the agent detect a risky change? - Did it detect the risk and deliberately leave the change alone? That is different from never noticing it. - After deciding against that change, did it look for another improvement? - Did it hand the decision back to a human? A PR diff cannot answer those questions on its own. I did not yet know how to capture them as structured observations. If I put evaluation into the same tool, I would have to settle those questions before I could release the isolation mechanism. An unresolved evaluation design would block a piece of infrastructure whose responsibility was already clear. I wanted the dependency to run the other way: an evaluator could use yuurei's execution records, while yuurei would not need to know which evaluator consumed them. So I kept yuurei responsible for preparing isolated executions and emitting machine-readable records. A separate tool would decide what to evaluate and how to judge it. The trace records execution facts. It does not turn those facts into a judgment that an agent made a good decision, nor does it automatically capture all the behavioral evidence a future evaluator might need. This separation also has a practical benefit: evaluation criteria will change. What I consider useful behavior may change as models improve. I do not want every revision to those criteria to require rebuilding the isolation layer. I could make the execution setup repeatable while leaving the evaluation questions open. What the original version did—and did not do At the time of the Japanese article, the v0.3 scope was single-runtime execution for Claude Code and Codex. Each runtime × model × harness × task combination was treated independently. That scope did not include: - Porting a harness across runtimes. - Automatically scoring output quality. - A cost or ROI dashboard. - OS-level sandboxing. - Team authentication. There was also no concept for grouping repeated executions of the same profile and task. Agent behavior varies from run to run, so repetition would be necessary before drawing stronger conclusions. Implementation update, November 5, 2026: I checked this adaptation against v1.4.0. Yuurei now also supports OpenCode. Later additions include experimental repository seeding (--seed-repo) and opt-in pre-run observation (--observe). Pre-run observation records information about the prepared environment; it does not resolve the behavioral evaluation questions above. These additions do not change the original experiment or its results. The current public contract describes the implementation's guarantees and limits. The question I still wanted to test One possibility remained open: could an accumulated harness prevent me from getting improvements that a newer model would otherwise provide? The original comparison could not answer that. PR2 and PR3 both used Sonnet 5. I had changed the harness while keeping the model fixed. Instructions written to compensate for an older model's weaknesses might behave differently when the model itself changes. Testing that requires the reverse comparison: preserve the harness and change the model. It also requires the older harness to still exist when I want to run the comparison. If I keep editing it in place, it may already be gone by the time the next model arrives. That was what yuurei was preparing me to do. I had not yet measured the effect I wanted to measure. I had built a way to preserve and rerun the configurations I would need to investigate it.

@Furanshico Sim! uma Skill maravilhosa que mostra como tá as coisas dessa forma. Faz parte desse pacote aqui backnotprop/bro Funciona no codex tambem

Plannotator: Review AI Agent Plans in Your Browser Plannotator is an open source tool that opens AI coding agent plans and code diffs in your browser so you can annotate them and send feedback back to the agent. See the project at backnotprop/plannotator

Plannotator: Annotate and review coding agent plans and code diffs visually (9.7k stars) What it does All text below is quoted directly from the project README. >Annotate and review coding agent plans and code diffs visually, share with your team, send feedback to agents with one click. Plannotator: - Plannotator is a local, browser-based review surface for AI coding agents: Claude Code, Codex, Copilot CLI, Gemini CLI, OpenCode, Kiro, Droid, Amp, and Pi. - It plugs directly into your agent through its hooks and commands. - Annotate plans, specs, messages, html, then send the feedback to your agent. - Review local changes or remote PRs. Comment on diffs, suggest code. Your comments go back to the agent. - Works with Git, GitButler, Jujutsu (jj), Perforce (p4), GitHub, and GitLab. - Plannotator does not collect usage telemetry or analytics. >curl -fsSL plannotator.ai | bash Repository backnotprop/plannotator Repository stats |Metric|Value| |:-|:-| |Stars|9,719| |Forks|728| |Last release|v0.28.35 (2026-10-28)| |Languages|TypeScript 94.7%, CSS 2.3%, Shell 0.8%| |Contributors|180| |AI contributor accounts (Claude / Codex / Cursor / Copilot)|@claude (421 commits), @Copilot (4 commits)| |AI co-authored commits|Claude: 423, Codex: 1, Copilot: 4| Main contributor |Field|Value| |:-|:-| |Name|Michael Ramos| |GitHub|@backnotprop| |Commits to this repo|2,073 (top contributor)| |Company / Location|CA| |Followers|1.6k| ────────────────────────────── 📌 How many great GitHub repos have you discovered and completely forgotten about? Don't let another useful tool disappear into your Reddit feed. Get the community's best repos, organized by category, plus key takeaways from real developer discussions, delivered to your inbox twice a week. 📬 Never lose track of great repos again. Subscribe free.

show me big pics and little words (skill) github.com/backnotprop/bro/blob/main/skills/big-p… t.co

try this skill backnotprop/pstack setup Fable as judgement and Opus 5.5 as worker thanks me later

This was the case 10mo ago. And Ive been a memory-system bear for a long time. Novel fronteir for memory is probably dynamic filesytem. backnotprop/rg_history (but now agents dont need a skill to search their own history) t.co

Show HN: State Machines for Software Factories backnotprop/machine

herdr-annotate: Annotate terminal text, review documents and agent replies, and send the feedback straight back to the agent (0.7k stars) What it does All text below is quoted directly from the project README. >Annotate inside Herdr: comment on any terminal text, review whole Markdown documents and your agent's replies, and send the feedback straight back to the agent. Annotate terminal text: - Ctrl+B A: comment on the selected text · Ctrl+S saves - Ctrl+B Shift+A: copy all annotations as Markdown - Ctrl+B Ctrl+A: copy all annotations as Markdown, then archive them - Ctrl+B Ctrl+V: paste all annotations into the focused agent's prompt without sending, then archive them - Ctrl+B Ctrl+S: send all annotations as the focused agent's next message, then archive them - Full: annotate terminal text, review documents and agent replies, send feedback to the agent. - Lite: the simple version: select text, prefix+a, comment in a popover. >herdr plugin install plannotator/herdr-annotate Repository plannotator/herdr-annotate Repository stats |Metric|Value| |:-|:-| |Stars|684| |Forks|24| |Last release|None| |Languages|Rust 51.8%, Python 23.2%, PowerShell 17.8%| |Contributors|8| |AI contributor accounts (Claude / Codex / Cursor / Copilot)|claude: 2| |AI co-authored commits|Claude: 2| Main contributor |Field|Value| |:-|:-| |Name|Michael Ramos| |GitHub|@backnotprop| |Commits to this repo|46 (top contributor)| |Bio|github is the fun stuff. day to day is complex critical systems, mostly involving AI.| |Company / Location|CA, prev DC.| |X / Twitter|@backnotprop| |Followers|1,500| ────────────────────────────── 📌 How many great GitHub repos have you discovered and completely forgotten about? Don't let another useful tool disappear into your Reddit feed. Get the community's best repos, organized by category, plus key takeaways from real developer discussions, delivered to your inbox twice a week. 📬 Never lose track of great repos again. Subscribe free.

pstack：Cursor 工程師開源的 AI 開發工作流，怎麼用＋arena、swarm、no-comments 三招 #shorts 別再讓 AI 寫一大坨 slop 了。Lauren Tan（poteto）是 Cursor 工程師（Cursor 已被 SpaceX 併購，她現在在 SpaceX），她說自己一個月送出上千個 PR。她把自己不寫 slop 的整套 AI 開發工作流開源了，叫 pstack，是一整套 Agent Skills。 ▍怎麼裝 ・Cursor：/add-plugin pstack ・Claude Code（社群移植版）： /plugin marketplace add michael-denyer/pstack-claude /plugin install pstack@pstack-claude ・Codex（社群移植版）： codex plugin marketplace add michael-denyer/pstack-claude codex plugin add pstack@pstack-claude ・不綁工具（社群鏡像，純 skills）：npx skills add backnotprop/pstack ▍怎麼用 1. 先跑 /setup-pstack，選 reasoning budget 和要用的模型 2. 每次開工打 /poteto-mode ＋ 你的任務，它會從 23 個 playbook 裡自己挑對的流程（修 bug、抓效能、開 PR、整晚自動跑……） 不知道用哪個技能就問 /poteto-help。 ▍為什麼好用 ① arena：同一個題目讓多個模型各做一版（預設 Opus 5.5 和 Grok），再找一個模型當評審，挑最好的當底，把另一版的優點接上去。 ② swarm：一次派一群 agent 分頭驗證，每個都附證據，最後整理成一份報告。 ③ no-comments：把註解砍掉。註解一跟程式碼對不上，AI 有時只看註解就跟著錯。unslop 再把文字裡的 AI 腔刪掉。 ・24 條原則拆成 24 個小 skill，poteto-mode 開工先讀索引，要用哪條再翻哪條，不用整包塞進系統提示。 ・它很在意證據：像修 bug，會先重現失敗，修完在同一個地方再跑一次，前後結果都貼給你看。 ▍來源 ・pstack 官方（cursor/plugins）：github.com/cursor/plugins/tree/main/pstack ・Claude Code／Codex 移植：michael-denyer/pstack-claude ・純 skills 鏡像：backnotprop/pstack 畫面來源： ・pstack 官方 logo：github.com/cursor/plugins/tree/main/pstack #pstack #ClaudeCode #Cursor #AI懶人報