josdejong

Harness Engineering, Explained by Building One: An AI Agent Test Bench That Runs in Your Browser !HarnessBench: load a model, give it a task, watch every step, and get a verdict --- An agent that lies, politely I asked a small language model to do something simple: create three files called a.txt, b.txt and c.txt, then list them. It created one file. Then it announced: "File 'c.txt' has been created successfully." It never made the other two and it never listed anything. By its own account, the task was done. That isn't a bug in one model. It's normal behaviour for AI agents, especially small ones. They skip steps, do things out of order, call tools that don't exist, repeat themselves, and finish with a confident summary either way. If you grade an agent by its last message, you're letting it grade its own homework. The fix is not a better prompt. It's a different kind of engineering. --- What is harness engineering? Harness engineering is the practice of building the controlled environment an AI agent runs inside. The harness decides what the agent can see, what it is allowed to do, records everything it does, and checks the result against reality instead of the agent's own claims. The word comes from testing. In electronics and software, a test harness is the rig you clamp a component into so you can feed it inputs, watch its outputs, and measure whether it behaves. Think of a car on a crash-test sled. The car is the interesting part, but the sled, the sensors and the high-speed cameras are what turn a crash into data. For AI agents, the model is the car. The harness is everything else. Why it became its own discipline Working with language models has gone through roughly three layers of concern: - Prompt engineering: what do I say to the model? Wording, examples, instructions. - Context engineering: what information does the model see? Retrieved documents, memory, conversation history, tool results. - Harness engineering: what world does the model act in, and how do I know what it did? Tools, permissions, sandboxes, loops, logs and judges. The first two make a single answer better. Once a model starts acting (running code, editing files, calling APIs, spending money) a good answer isn't enough. You need to contain the actions, observe them, and verify the outcome. That is the harness's job. The five parts of a harness Almost every agent harness, from a research benchmark to a production coding assistant, comes down to the same five pieces: 1. The sandbox (environment). An isolated place where the agent's actions land. In production it might be a Docker container, a virtual machine or a staging database. The key property is that it is fresh and disposable, so every run starts from the same known state. 2. The tool registry (permissions). The explicit list of actions the agent may take, each with a defined input. Anything not on the list does not run. This is the agent's permission boundary. 3. The agent loop (control). The cycle of model decides → harness executes → result is fed back, with limits on steps, tokens and time so a confused agent can't run forever. 4. The recorder (observability). A step-by-step log of what the agent thought, what it called, with which parameters, and what came back. It works like a flight data recorder: when something goes wrong, you can replay exactly how. 5. The judge (evaluation). Code that inspects the final state of the sandbox and the action log and decides pass or fail. It never relies on the agent's description of what it did. Get these five right and an unpredictable model becomes something you can measure, compare and trust a little more. Leave any one out and you're back to taking the agent's word for it. Why you need one even for small experiments You don't need a harness only for big production systems. The moment you want to answer questions like "Is model A better than model B at this task?" or "Did my prompt change actually help?", you need the same five parts: identical starting conditions, the same allowed actions, a record of what happened, and a judge that applies the same rules every time. Without them, you are comparing anecdotes. The catch is that harnesses usually need infrastructure: servers, containers, API keys, somewhere to keep logs. That's a lot of setup just to learn the idea. So I built the smallest complete harness I could, and put it in a web page. --- HarnessBench: all five parts in one browser tab HarnessBench is a single web page. It downloads an open-source language model once, runs it on your own GPU through WebLLM and WebGPU, and wraps it in a complete harness. Nothing is sent to a server, and after the first visit the model loads from the browser's cache in seconds. Here is how each part of harness engineering maps onto the app: | Harness part | In HarnessBench | |---|---| | Sandbox | A fresh in-memory "file system" and variable store, reset before every run | | Tool registry | Six actions: create_file, read_file, list_files, delete_file, set_variable, get_variable, plus final_answer | | Agent loop | One JSON decision per step (thought, action, parameters), a step limit, and a cap on reply length | | Recorder | A live token stream, a step-by-step trace, and the full action log in the sandbox state | | Judge | Plain JavaScript that checks which actions were called and what is actually in the sandbox | !Model loaded and ready to run A run that passes Pick a task (or click one of the "Try" chips) and press Run Evaluation. Here the agent was asked: "Create a file called note.txt with the content 'hello world', then read it back." !A passing run: create_file, then read_file, then final_answer, judged PASSED Everything the harness knows is on screen: - The verdict at the top, with the judge's reason. Note what the reason says: "note.txt" exists: "hello world". That's a fact read from the sandbox, not a quote from the model. - The live token stream, showing the model's raw output as it's generated. - The agent trace, one card per step: what the model was thinking, which action it chose, and what happened. - The sandbox state, showing the actual files, variables and action log after the run. A longer task shows why the recorder matters. Asked to create three files and list them, Llama 3.2 3B created a.txt, b.txt and c.txt in order, listed them, and finished. It also listed them a second time for no reason. The run still passes, but the trace shows you the wasted step. A single score would have hidden it. !Multi-step trace: three create_file calls, list_files, then final_answer A run that fails, and why that's the useful part Here is that same three-file task given to a smaller model, SmolLM2 1.7B. This is the "polite liar" from the start of the article. It created only c.txt, then declared the job done. !The judge catches an incomplete run: list_files was never called and two files are missing The judge doesn't care what the model said. It sees that list_files was never called and that a.txt and b.txt don't exist, so the verdict is FAILED. This is the whole point of a harness: the agent's confidence has no effect on the score. The permission boundary in action The tool registry isn't a suggestion. Ask the agent to "Ring the bell using the ring_bell action, then create bell.txt" and it will dutifully try to call ring_bell. That action isn't registered, so the harness stops the run at once and nothing else executes. !Unknown action blocked: ring_bell is not a registered action In a real system, the same rule is what keeps an agent from calling a "delete everything" endpoint just because it believes one exists. The model can ask for anything; only registered actions ever run. --- How the judge decides The judge is ordinary code, not another AI model. That's deliberate: a judge should be predictable and give the same verdict for the same behaviour every time. It runs two checks: 1. Did the agent do every step? Words in the task map to the actions they require: "create" needs create_file, "read" needs read_file, "delete" needs delete_file, "list" needs list_files, and so on. If a required action never appears in the log, the run fails, whatever the agent claimed. 2. Is the world in the right state? Every file named in the task must exist at the end, or must be gone if the task asked for a delete. Variable tasks must leave a stored variable. Because every run starts from an empty sandbox and the judge is plain code, two runs with the same behaviour always get the same verdict. That reproducibility is what makes it fair to compare models side by side. Dealing with messy model output A harness also has to cope with the model breaking the rules of the conversation, not just the rules of the task. Small models are bad at clean structured output. While testing, a frequent habit was writing two decisions back to back, {...} {...}, which isn't valid JSON. Others wrap the answer in a chatty sentence or a code fence. HarnessBench handles this the way a production harness should: - It takes the first complete JSON object in the reply (one action per step) and uses jsonrepair to fix small mistakes such as a trailing comma or a missing closing brace. - If a reply still has no action, the harness tells the model its format was wrong and lets it try again. That retry uses up one of its steps. - Each reply is capped at 384 tokens, so a runaway generation can't stall a step for minutes. - Old steps are trimmed from the conversation so long runs stay inside a small model's memory window. None of this makes the model smarter. It makes the measurement more reliable, so that a failed run means the agent failed and not that the plumbing did. A harness has to test itself, too If the judge has a bug, every verdict is wrong. So HarnessBench includes a Harness Verification panel: 29 deterministic tests that run without any model. They cover every action, sandbox isolation, unknown-action blocking, judge pass and fail cases (including "only one of three files was created"), JSON repair and context trimming. !All 29 harness self-tests passing --- What I saw on a real laptop Every screenshot in this article comes from real models running on WebGPU in Chrome, on a laptop with an integrated Intel Iris Xe GPU: | Model | Example tasks passed | What it did | |---|---|---| | Llama 3.2 3B Instruct (q4f32) | 4 of 4 | Followed steps in order; sometimes repeated a harmless list_files | | SmolLM2 1.7B Instruct (q4f16) | 1 of 4 | Often stopped after the first step, or kept retrying a failing action | On this hardware the 3B model produces about 3 tokens per second. One step takes roughly 8 to 20 seconds, and a full task takes one to two minutes. The first visit downloads about 1.8 GB; after that it loads from cache in seconds. The interesting result isn't the scores. It's that the harness made the difference between the two models visible and explainable. You can see exactly where the smaller model gave up. Why this is useful - Learn harness engineering by touching it. All five parts are on one screen and in a few hundred lines of plain JavaScript, so you can see how they connect. - Compare models honestly. Swap the model, run the same tasks, and get verdicts based on what happened in the sandbox. - See how agents fail, not just that they fail. Early stopping, wrong order, invented tools and loops all show up in the trace. - Private by default. Prompts and results never leave your machine. - Zero setup. Share a link. Anyone with Chrome or Edge and a WebGPU-capable GPU can watch an agent run and get graded. Limitations To be clear about what this is: - The tasks are toy tasks. Six in-memory tools are enough to show the ideas, not to benchmark real-world agents. - The judge is keyword-based. It works for the included kinds of tasks. A task worded very differently might not map to the right required actions. - Results vary. Small models behave differently across hardware and library versions. Treat the table above as one observation, not a leaderboard. - WebGPU is required. It works in recent Chrome and Edge with hardware acceleration enabled. Bigger models need more GPU memory than many laptops have. If a model starts producing garbled text, clearing the site's data (which resets the cached model) usually fixes it. The takeaway Prompting makes an agent say better things. A harness tells you what it actually did. Before you trust any agent with real work, it's worth asking five questions: Where do its actions land? What is it allowed to do? Who stops it when it loops? Where is the record? And who checks the result, other than the agent itself? Open the live demo, load Llama 3.2 3B, click a "Try" chip and press Run Evaluation. Then try to break it: ask for a tool that doesn't exist, or give a small model a five-step task, and watch the harness catch it. --- Credits HarnessBench builds on open-source work: - WebLLM and MLC LLM by the MLC team, for in-browser LLM inference on WebGPU and the prebuilt quantized models. - Llama 3.2 by Meta and SmolLM2 by Hugging Face, plus the other model families offered in the app (Phi-3.5, Gemma 2). - jsonrepair by Jos de Jong, for recovering malformed JSON. - The WebGPU standard and its browser implementations. Disclaimer HarnessBench is an educational demo, not a production evaluation tool. Its verdicts come from simple rule-based checks on toy tasks and should not be used to make claims about any model's real-world capability or safety. Model weights are downloaded from their publishers and are covered by their own licenses. This is a personal project and is not affiliated with or endorsed by any of the projects or companies named above.

🔢 The JSON Numbers JavaScript Quietly Corrupts --- title: 🔢 The JSON Numbers JavaScript Quietly Corrupts published: true description: JSON.parse changes your data and never says so. Three distinct failure modes, only one of which is the famous one about large integers. tags: javascript, json, webdev, programming canonical_url: prettify.tools --- Overview This post will help identify the ways JSON.parse silently changes your data, and what you can do about it. We will cover why it happens, the three distinct failure modes it produces, how to check whether your own pipeline is affected, and the approach I use in my own tooling to prevent it. Each failure mode includes an example you can run in your browser console right now. > 💡 Every example in this post takes less than ten seconds to verify. Open your console and follow along. 📚 Table of Contents - Why it happens - The three ways it bites - Large integers round - Overflow becomes null - Formatting is destroyed even when the value survives - What most tools do about it - What we do in Prettify - How to check whether this affects you - Drawbacks - The short version First, try it yourself Paste this into a browser console: JSON.parse('{"id": 9007199254740993}').id You get 9007199254740992. Off by one. No error, no warning, no truncation notice. The value you asked for is gone and something that looks just like it is standing where it was. This is not a bug in JSON.parse. It is doing what the spec says. But it means that any JavaScript program that reads JSON, touches it, and writes it back is a program that can silently change your data, and most of them do not know it. Why it happens JSON has one numeric type and no size limit. The grammar is happy to describe a four-hundred-digit integer. JavaScript also has one numeric type: the IEEE-754 double. It has 53 bits of integer precision. Past Number.MAX_SAFE_INTEGER, which is 9007199254740991, consecutive integers stop being representable, and the parser rounds to whatever double is nearest. So JSON.parse maps an unbounded numeric grammar onto a bounded numeric type. There is no way to do that without losing something, and the standard's choice is to lose it silently. The three ways it bites Value loss is the famous one. It is not the only one, and it is not the most common. 1. Large integers round JSON.parse('{"id": 1071004833782874112}').id // 1071004833782874100 The last two digits are invented. That is a Discord-style snowflake ID, and the same applies to Twitter/X IDs, most database bigints, Stripe-style numeric identifiers, and anything else that packs a timestamp into a 64-bit integer. > 🚧 The insidious part is that IDs are exactly the values where a near-miss is worse than a crash. A rounded price is obviously wrong. A rounded ID is a valid-looking ID that points at nothing, or worse, at something else. 2. Overflow becomes null JSON.parse('{"v": 1e400}').v // Infinity JSON.stringify(Infinity) // "null" Read a number too large for a double and you get Infinity. Write it back out and JSON.stringify turns Infinity into null, because JSON has no infinity literal. So a round trip converts a number into a null. Downstream, that is not a precision problem. It is a type change, and whatever consumes the field next is going to be surprised. 3. Formatting is destroyed even when the value survives JSON.stringify(JSON.parse('{"v": 4.0}')) // {"v":4} 4.0 and 4 are the same number. They are not the same document. If you are round-tripping a config file, a fixture, an API contract, or anything a human wrote and will read again, 4.0 becoming 4 is a diff. It shows up in code review. It shows up in a git blame. Version-controlled JSON that a formatter "cleaned" is a classic source of noise commits. The same goes for 1.00, -0 becoming 0, and 5e+3 becoming 5000. And all three are silent Nothing throws. Nothing logs. The document parses, the program runs, and the tests pass because the fixtures were generated by the same broken round trip. You find out when a customer reports that a record will not load, and the ID in your logs is one digit off from the ID in theirs. What most tools do about it Nothing, mostly. Every JSON formatter, viewer, and prettifier that works by calling JSON.parse and then JSON.stringify has all of these behaviors. That is the standard implementation. Paste a Discord message dump into a random online JSON formatter and the IDs come back wrong. The workarounds people reach for: - Parse to BigInt. Handles large integers, does nothing for 4.0 or 1e400, and gives you a type that will not JSON.stringify without a custom replacer. - Parse numbers as strings. Safe, and it breaks every consumer that expected a number. - Ask the API for string IDs. Correct, and most public APIs already do this for exactly this reason. Also not available to you when you are just looking at somebody else's payload. What we do in Prettify The rule is: preserve the token, not the value. When parsing, each numeric token gets one question asked about it. Would writing this back out produce the exact bytes I read? If yes, it stays an ordinary JavaScript number. If no, it becomes a wrapper that holds the original digit string. The wrapper type and the two helpers below come from lossless-json. The policy around them is ours, and it is the part worth copying. function parseNumber(value: string): number | LosslessNumber { if (isSafeNumber(value) && String(parseFloat(value)) === value) { return parseFloat(value) } return parseLosslessNumber(value) } Two conditions, because there are two independent failure modes, and this is the part that most implementations miss. isSafeNumber catches value loss. It does not catch formatting loss: it says 4.0 is perfectly safe, because numerically it is. 4.0 and 4 are the same double. That is why the second check exists. String(parseFloat("4.0")) is "4", which is not "4.0", so the token gets preserved even though nothing about its value was ever at risk. > 💡 Checking only isSafeNumber is the natural implementation, and it silently reformats your document. Wrap only what needs wrapping The other half of the design is that most numbers are not wrapped. 1, 42, 0.1, -17.5 all stay ordinary numbers, because they round-trip exactly. Only the tokens that would change get the wrapper. This matters for a reason that is more practical than philosophical: a wrapper is typeof === 'object'. Every typeof x === 'number' check in the codebase is false for it, and every "is this a container?" check is true for it unless you exclude it. Wrapping everything would mean auditing every numeric code path in the application. Wrapping only unsafe tokens keeps the blast radius to the small set of documents that actually contain them. Viewing is not computing Preserving a number exactly and doing arithmetic on it are incompatible goals in a language with one float type. So the two are separated by policy: - View, select, copy, round-trip: exact. The digits you pasted are the digits you get back. - Compute: coerced, and you are told. Arithmetic, aggregation, and comparison flatten to a float, because they have to. When that happens, the app says so. That last clause is the whole point. The original sin of JSON.parse is not that it loses precision. It is that it loses precision without mentioning it. An answer you know is approximate is a usable answer. An answer you believe is exact and is not is a landmine. How to check whether this affects you Fastest test, in any JavaScript runtime: const original = yourJsonString const roundTripped = JSON.stringify(JSON.parse(original)) // If these differ, your pipeline is changing documents. That catches all three cases at once, formatting included. If it comes back different, the questions worth asking are: does anything downstream compare these documents, diff them, sign them, hash them, or use a numeric field as a key? If yes, you have a real problem rather than a cosmetic one. Drawbacks Being honest about the limits of this approach: - It only covers JSON and JSONL. The policy above applies to Prettify's JSON pipeline. YAML and CSV go through their own parsers and get ordinary JavaScript numbers, with all three behaviors described in this post. - Computing still coerces. Preserving a number exactly and doing math on it are incompatible in JavaScript. Arithmetic flattens to a float. The best a tool can do is tell you when it happened. - It does not help with data that was already corrupted upstream. If the ID was rounded before it reached you, no parser can recover the digits that were lost. The short version - JSON.parse silently rounds integers past 9007199254740991 - 1e400 round-trips to null, changing the type - 4.0 round-trips to 4, changing the document - None of it warns you - The fix is to preserve the original token when re-serializing it would differ, and to check formatting as well as value --- Prettify preserves these numbers byte for byte, in JSON and JSONL, in the browser: paste a payload with a snowflake ID into the JSON viewer and the digits come back unchanged. The jq engine applies the same policy, and flags any query that had to coerce. --- Happy Coding 🚀 --- Blog Post: prettify.tools --- If you liked this content, follow me on Twitter @alexi_be3 💙