Six hosted sandboxes say "isolated." None tells you which wall your agent ran behind
A Show HN went up yesterday for plimsoll, a Go daemon you run yourself that executes code an agent wrote. The sandbox part is ordinary. The interesting part is that every result carries the isolation tier the run actually got, as a value, and a request can demand a minimum tier before anything executes.
That second half is the rare bit. plimsoll's own docs say they read the current documentation of six hosted sandbox services in September 2026 (E2B, Modal, Daytona, Vercel Sandbox, Cloudflare and Northflank) and found that none of them returns the wall a run actually got, and none refuses a run that would land below a minimum the caller stated.
I checked two of those myself instead of taking the claim on faith.
Cloudflare's sandbox docs are explicit about the wall: "Each instance is a microVM with its own kernel and network, so no other workload shares it." That's a real statement about the boundary. It's also a statement about the product. Nothing on a run's result repeats it back per call, and there's no per-request floor to set.
Modal comes at it from the other side. Its Sandbox.create reference takes experimental_options, and passing {"vm_runtime": True} asks for a full virtual machine instead of the default gVisor layer. Nothing in what comes back tells you which of the two served a particular call. You asked. You don't get told which one you got.
Four tiers, as a field on the result
plimsoll's tiers are process, container, kernel, vm, weakest first. The tier is a value on every result rather than a sentence in a datasheet. A request sets MinimumIsolation, and if the daemon can't meet it you get ErrInsufficientIsolation and nothing ran.
That last clause is the part I care about more than the tier names. "Nothing ran" and "may have run" are different facts, and a plain error code can't tell them apart. InvalidArgument is usually a request that failed validation, but a worker can also refuse after its container started. A capacity refusal can come from the daemon's limiter or from the provider. So refusals raised before dispatch are marked with a reason (request, permission, protocol, unsupported, isolation, environment, capacity), and a refusal without the mark reads as "may have executed." That's the conservative default: a code path that forgot to mark its error can never be mistaken for a safe retry.
There's a detail under that which is easy to miss. The mark only counts if the answer carries your request ID back. Otherwise a confused proxy serving one request's answer for another would make a call that ran read as "nothing ran," and the retry runs it twice. The client sends a fresh 32-hex-digit ID with each request and the daemon echoes it.
The record is the part you can hand to someone else
Every answered run carries SHA-256 digests of the request as sent and the result as returned, plus the provider, the tier, the image identity and the timings. The daemon holds no signing key, on purpose, because it executes hostile code and one engine escape would otherwise be a key away from forging every record. A harness outside the daemon recomputes the digests from its own copy and signs the ones that match. For a deterministic workload you can replay the stored request and compare digests.
Their run report for the cart-pole example shows what that buys you. An agent-written controller, run twice, same fingerprint 63cfd676924a3896127d60c97333677f3b3c804dd764dba001121f6c234a937f. The agent's first draft, same structure with two gains of the wrong sign, different fingerprint, pole down at 5.96 s. The same run on five different providers, four of them different backends, hashed to that same value.
The page also says plainly what it doesn't show. The controller can read the plant and the runner, since they sit on the same read-only root. A fingerprint is identity, not a score, and recompiling the plant moves every fingerprint on the page.
The credential story is where this gets practical
Most of what I read about agent sandboxes stops at containment. The capability grants doc goes further into the thing that actually bites teams: the agent's code needs to call your API, and the usual fix is to hand the sandbox a token.
A grant here is permission for one run to call listed routes of one API. Without a grant, the run has no network at all. Docker runs keep --network none and send calls over a per-run Unix socket. The code calls a small client plimsoll places in the sandbox, and the broker outside makes the real call, checks the caller and the exact route, then attaches a credential minted per run. The token never enters the sandbox, and code that skips the client and calls out by hand gets nowhere because the broker, not the client, enforces the rules.
The route matching is where they've clearly been bitten. A path is accepted only if it's already decoded, in simplest form, and byte-identical to an approved route. Queries, .. traversal and percent-encoding tricks are refused before anything is sent. So are ; parameters inside a path segment, because Tomcat and Spring read /orgs/..;/repos/x as /repos/x. A * matches one segment of unreserved characters only. On an API following Google's custom-method convention, POST /v1/items/a:setIamPolicy is a different operation than POST /v1/items/a, and on OData POST /odata/$batch carries a batch of other requests, so a grant of POST /v1/items/* doesn't reach either.
There's a smaller piece I liked more than I expected. Three things describe the same set of routes (the allow list the broker enforces, the JavaScript client the agent calls, and the tool description the model reads), and they drift apart whenever a human keeps them in sync. plimsoll-specgen builds all three from one OpenAPI document, fails closed on ambiguous input, and warns instead of silently dropping what it can't express. It also emits every operation in the document as allowed, because a spec describes what an API has, not what an agent should reach. Pasting that unedited grants the whole API. The doc says so out loud.
Why I'm writing about this one
The docs say the tier is configuration plus startup smoke tests, and that it is not attestation, meaning cryptographic proof from the hardware of what software is actually running. They say the project is pre-1.0, has no outside security audit, and the bus factor is one. gVisor is Linux only, so a Mac gets the container tier. The wasm provider runs inside the daemon's own process and the docs say not to point it at hostile code. Those are the sentences that make the rest of it checkable, and they're the reason I'd rather read this than another launch post.
For my beat it's the same shape as every accuracy claim I've torn apart. "Isolated" is a claim like "96% accurate." The version you can falsify has a field on the result, a floor on the request, and a record somebody outside the box can recompute. This is also where the patch-versus-execution gap lands. If review reads the diff and never the run, then the execution record is the thing you have to trust, and most of the options today hand you a sentence instead.
Practical version, since I'm not asking anyone to take my word for it either: next time you're choosing where an agent's code runs, ask for one result object from a real run. If it doesn't state the boundary that run got, you're still reading marketing copy.
1 reaction · 2 comments
LLM Capability Gaps, OpenAI Revenue Shocks, and AI Decompilation
Today's AI landscape is marked by a stark contrast between theoretical capability and practical understanding, alongside significant corporate turbulence.
The Widening Gap in LLM Capability Understanding
Understanding the Divergence
Andrej Karpathy has highlighted a growing gap in how we understand LLM capabilities. As models evolve, the distance between what we think they can do and their actual, verified performance is widening, leading to fragmented benchmarks and inconsistent expectations across the industry.
Source: Twitter/Karpathy
OpenAI's Financial Turbulence
The Revenue Story
Reports indicate that OpenAI fired its own auditor shortly before a narrative shift occurred where its revenue projections shrank by approximately $20 billion. This raises critical questions about the sustainability of the current AI hype cycle and the actual monetization of LLMs at scale.
Source: The Inference
AI Agents in Game Decompilation
500B Tokens of Insight
In a fascinating application of agentic AI, researchers have used AI agents to decompile a first-person shooter game. By processing over 500 billion tokens, the agents were able to reconstruct logic and structure from binary data, showcasing the power of LLMs in reverse engineering.
Source: momo5502.com
Recursive Self-Improvement via Collective Intelligence
Sakana AI's Approach
Sakana AI is pushing the boundaries of recursive self-improvement. By utilizing collective intelligence, they are exploring how models can iteratively improve their own architectures and weights without requiring massive new datasets, potentially leading to a "self-evolving" AI loop.
Source: Sakana AI
Plimsoll: Secure Sandboxing for AI Code
Safe Execution Environments
The introduction of Plimsoll provides a self-hosted sandbox specifically designed for the code that AI agents write. This solves a critical safety gap, allowing developers to execute agent-generated code without risking their host system's integrity.
Source: GitHub/Plimsoll
FAQ
Why is the LLM capability gap widening?
The gap widens because model behavior is becoming more non-linear, and standard benchmarks are being "leaked" into training sets, making it harder to measure true generalization.
What does the OpenAI revenue shrink mean for the industry?
It suggests that the "AI bubble" may be facing a reality check regarding how much enterprises are actually paying for LLM integration versus the valuations assigned to the providers.
Can AI agents really decompile games?
Yes, by leveraging large context windows and pattern recognition over binary patterns, they can infer C++-like structures from machine code, though human verification is still required.
How does Sakana AI's recursive improvement work?
It involves creating a population of models that "compete" or "collaborate" to find better weights, effectively evolving the model through a simulated genetic process.
Is Plimsoll available for personal use?
Yes, it is a self-hosted project available on GitHub, designed for developers who want a local "safe zone" for AI agents.
Mentions
Plimsoll is a new self-hosted sandbox for code written by AI agents, with local and cloud isolation options from WebAssembly to microVMs. It can deny network access by default and issue scoped grants when a run needs to reach a service. plimsollmark/plimsoll
Six hosted sandboxes say "isolated." None tells you which wall your agent ran behind A Show HN went up yesterday for plimsoll, a Go daemon you run yourself that executes code an agent wrote. The sandbox part is ordinary. The interesting part is that every result carries the isolation tier the run actually got, as a value, and a request can demand a minimum tier before anything executes. That second half is the rare bit. plimsoll's own docs say they read the current documentation of six hosted sandbox services in September 2026 (E2B, Modal, Daytona, Vercel Sandbox, Cloudflare and Northflank) and found that none of them returns the wall a run actually got, and none refuses a run that would land below a minimum the caller stated. I checked two of those myself instead of taking the claim on faith. Cloudflare's sandbox docs are explicit about the wall: "Each instance is a microVM with its own kernel and network, so no other workload shares it." That's a real statement about the boundary. It's also a statement about the product. Nothing on a run's result repeats it back per call, and there's no per-request floor to set. Modal comes at it from the other side. Its Sandbox.create reference takes experimental_options, and passing {"vm_runtime": True} asks for a full virtual machine instead of the default gVisor layer. Nothing in what comes back tells you which of the two served a particular call. You asked. You don't get told which one you got. Four tiers, as a field on the result plimsoll's tiers are process, container, kernel, vm, weakest first. The tier is a value on every result rather than a sentence in a datasheet. A request sets MinimumIsolation, and if the daemon can't meet it you get ErrInsufficientIsolation and nothing ran. That last clause is the part I care about more than the tier names. "Nothing ran" and "may have run" are different facts, and a plain error code can't tell them apart. InvalidArgument is usually a request that failed validation, but a worker can also refuse after its container started. A capacity refusal can come from the daemon's limiter or from the provider. So refusals raised before dispatch are marked with a reason (request, permission, protocol, unsupported, isolation, environment, capacity), and a refusal without the mark reads as "may have executed." That's the conservative default: a code path that forgot to mark its error can never be mistaken for a safe retry. There's a detail under that which is easy to miss. The mark only counts if the answer carries your request ID back. Otherwise a confused proxy serving one request's answer for another would make a call that ran read as "nothing ran," and the retry runs it twice. The client sends a fresh 32-hex-digit ID with each request and the daemon echoes it. The record is the part you can hand to someone else Every answered run carries SHA-256 digests of the request as sent and the result as returned, plus the provider, the tier, the image identity and the timings. The daemon holds no signing key, on purpose, because it executes hostile code and one engine escape would otherwise be a key away from forging every record. A harness outside the daemon recomputes the digests from its own copy and signs the ones that match. For a deterministic workload you can replay the stored request and compare digests. Their run report for the cart-pole example shows what that buys you. An agent-written controller, run twice, same fingerprint 63cfd676924a3896127d60c97333677f3b3c804dd764dba001121f6c234a937f. The agent's first draft, same structure with two gains of the wrong sign, different fingerprint, pole down at 5.96 s. The same run on five different providers, four of them different backends, hashed to that same value. The page also says plainly what it doesn't show. The controller can read the plant and the runner, since they sit on the same read-only root. A fingerprint is identity, not a score, and recompiling the plant moves every fingerprint on the page. The credential story is where this gets practical Most of what I read about agent sandboxes stops at containment. The capability grants doc goes further into the thing that actually bites teams: the agent's code needs to call your API, and the usual fix is to hand the sandbox a token. A grant here is permission for one run to call listed routes of one API. Without a grant, the run has no network at all. Docker runs keep --network none and send calls over a per-run Unix socket. The code calls a small client plimsoll places in the sandbox, and the broker outside makes the real call, checks the caller and the exact route, then attaches a credential minted per run. The token never enters the sandbox, and code that skips the client and calls out by hand gets nowhere because the broker, not the client, enforces the rules. The route matching is where they've clearly been bitten. A path is accepted only if it's already decoded, in simplest form, and byte-identical to an approved route. Queries, .. traversal and percent-encoding tricks are refused before anything is sent. So are ; parameters inside a path segment, because Tomcat and Spring read /orgs/..;/repos/x as /repos/x. A * matches one segment of unreserved characters only. On an API following Google's custom-method convention, POST /v1/items/a:setIamPolicy is a different operation than POST /v1/items/a, and on OData POST /odata/$batch carries a batch of other requests, so a grant of POST /v1/items/* doesn't reach either. There's a smaller piece I liked more than I expected. Three things describe the same set of routes (the allow list the broker enforces, the JavaScript client the agent calls, and the tool description the model reads), and they drift apart whenever a human keeps them in sync. plimsoll-specgen builds all three from one OpenAPI document, fails closed on ambiguous input, and warns instead of silently dropping what it can't express. It also emits every operation in the document as allowed, because a spec describes what an API has, not what an agent should reach. Pasting that unedited grants the whole API. The doc says so out loud. Why I'm writing about this one The docs say the tier is configuration plus startup smoke tests, and that it is not attestation, meaning cryptographic proof from the hardware of what software is actually running. They say the project is pre-1.0, has no outside security audit, and the bus factor is one. gVisor is Linux only, so a Mac gets the container tier. The wasm provider runs inside the daemon's own process and the docs say not to point it at hostile code. Those are the sentences that make the rest of it checkable, and they're the reason I'd rather read this than another launch post. For my beat it's the same shape as every accuracy claim I've torn apart. "Isolated" is a claim like "96% accurate." The version you can falsify has a field on the result, a floor on the request, and a record somebody outside the box can recompute. This is also where the patch-versus-execution gap lands. If review reads the diff and never the run, then the execution record is the thing you have to trust, and most of the options today hand you a sentence instead. Practical version, since I'm not asking anyone to take my word for it either: next time you're choosing where an agent's code runs, ask for one result object from a real run. If it doesn't state the boundary that run got, you're still reading marketing copy.
Short-lived tokens only limit how long a credential works, but better than nothing. i.e. They don't stop an unwanted action while it's still valid. A practical pattern is an outside broker: the guest never sees the real credential and can't pick the destination, only a path from an allowlist. Shameless plug for my project, I maintain one of these brokers: Plimsoll's broker sits outside the sandbox, injects the upstream credential itself, matches method + exact path (no query strings, no traversal, no encoding tricks), and enforces (some limitations that I am want to improve) a total call budget plus per-route caps. So a task can be granted "GET /items/* plus at most one job submission" even when the underlying key is coarser. Worth a look if you're thinking about narrower credentials for services whose keys are still coarse: plimsollmark/plimsoll (docs/capability-grants.md, and the broker walkthrough at plimsollmark.github.io).
エージェントに書かせたコードをそのまま実行するのは一番危ない操作。Plimsollはサンドボックスの強度をprocess/container/kernel/vmの4段階で結果に必ず付け、要求より弱ければ実行しない。境界を資料でなく実行結果で証明する発想が良い。 plimsollmark/plimsoll
LLM Capability Gaps, OpenAI Revenue Shocks, and AI Decompilation Today's AI landscape is marked by a stark contrast between theoretical capability and practical understanding, alongside significant corporate turbulence. The Widening Gap in LLM Capability Understanding Understanding the Divergence Andrej Karpathy has highlighted a growing gap in how we understand LLM capabilities. As models evolve, the distance between what we think they can do and their actual, verified performance is widening, leading to fragmented benchmarks and inconsistent expectations across the industry. Source: Twitter/Karpathy OpenAI's Financial Turbulence The Revenue Story Reports indicate that OpenAI fired its own auditor shortly before a narrative shift occurred where its revenue projections shrank by approximately $20 billion. This raises critical questions about the sustainability of the current AI hype cycle and the actual monetization of LLMs at scale. Source: The Inference AI Agents in Game Decompilation 500B Tokens of Insight In a fascinating application of agentic AI, researchers have used AI agents to decompile a first-person shooter game. By processing over 500 billion tokens, the agents were able to reconstruct logic and structure from binary data, showcasing the power of LLMs in reverse engineering. Source: momo5502.com Recursive Self-Improvement via Collective Intelligence Sakana AI's Approach Sakana AI is pushing the boundaries of recursive self-improvement. By utilizing collective intelligence, they are exploring how models can iteratively improve their own architectures and weights without requiring massive new datasets, potentially leading to a "self-evolving" AI loop. Source: Sakana AI Plimsoll: Secure Sandboxing for AI Code Safe Execution Environments The introduction of Plimsoll provides a self-hosted sandbox specifically designed for the code that AI agents write. This solves a critical safety gap, allowing developers to execute agent-generated code without risking their host system's integrity. Source: GitHub/Plimsoll FAQ Why is the LLM capability gap widening? The gap widens because model behavior is becoming more non-linear, and standard benchmarks are being "leaked" into training sets, making it harder to measure true generalization. What does the OpenAI revenue shrink mean for the industry? It suggests that the "AI bubble" may be facing a reality check regarding how much enterprises are actually paying for LLM integration versus the valuations assigned to the providers. Can AI agents really decompile games? Yes, by leveraging large context windows and pattern recognition over binary patterns, they can infer C++-like structures from machine code, though human verification is still required. How does Sakana AI's recursive improvement work? It involves creating a population of models that "compete" or "collaborate" to find better weights, effectively evolving the model through a simulated genetic process. Is Plimsoll available for personal use? Yes, it is a self-hosted project available on GitHub, designed for developers who want a local "safe zone" for AI agents.
Show HN: Plimsoll, a self-hosted sandbox for code your AI agent writes
Show HN: Plimsoll, a self-hosted sandbox for code your AI agent writes
Show HN: Plimsoll, a self-hosted sandbox for code your AI agent writes plimsollmark/plimsoll
Show HN: Plimsoll, a self-hosted sandbox for code your AI agent writes comments · posted on 2026.10.11 at 19:47:59 (c=1, p=3)
Show HN: Plimsoll, a self-hosted sandbox for code your AI agent writes Discussion