harness-sdk

The harness is your attack surface You install a coding agent and point it at a repo. Which files can it read, and what stops it? For some setups the honest answer is "everything your user can." The harness gives you control points, but the boundary and the policy are yours. A note on method. I read the current documentation of two open-source agent projects, OpenHands and the Strands Agents Python SDK, plus a harness-engineering survey. Every factual statement below carries a marker like `` that maps to one of those three sources. Where I give an opinion or a reading of a README, I say so. I do not call any project "safer" than another, because I found no comparison made on equal terms. 1. The default question: where does the agent run? Start with the most uncomfortable line in the OpenHands README. It documents that running the agent without a sandbox gives it full access to the host filesystem. What matters is where the code runs in that mode. Without a sandbox, the OpenHands agent-server runs directly on the host machine rather than in a container. Fine for a throwaway laptop. A very different choice on a developer machine that holds SSH keys, cloud credentials and ten other repositories. I'd settle the run mode before the first task, not after the first surprise. A cheap habit: in your own wrapper, print the run mode at the start of every task and stop if it is not the one you expect. 2. Docker helps, but only as far as you mount OpenHands can run each new conversation in its own Docker container, with its own Agent Server and tools. That narrows what one conversation can break. !Two columns: a narrow mount exposes one project for one task; a broad PROJECTS_PATH exposes every repository under it. The catch is the mount. In the Docker sandbox option, the agent can access any project under PROJECTS_PATH, a host directory the user chooses. The container is only as narrow as the directory you hand it. Here is the difference, as an illustration of the idea rather than a copy of any official setup: # Narrow: one project only. The narrow path is the whole point: one task, one repo. export PROJECTS_PATH="$HOME/work/payments-service" # Broad: every project you have ever cloned (commented out so this block is safe to paste) # export PROJECTS_PATH="$HOME/work" With the second value, a task about one service can reach every other repository under that directory, because the documented behaviour is access to any project under the path. 3. The control plane sits outside the sandbox Isolating where the agent executes does not harden the service around it. Per-conversation Docker isolation in OpenHands does not sandbox the whole Canvas or automation service. Treat that service as its own system to harden. For internet-facing installs, OpenHands points readers to its self-hosting guide for security hardening, and the project is MIT licensed. I did not open that guide, so read it yourself before exposing an install. 4. What Strands gives you: control points, not a policy The Strands Agents SDK takes a different shape. Its README lists a set of control points. I group them by the job they do: !A three-step chain: set limits such as turn limits and budgets, add hooks to log each step, then use steering to recover. - Observe. Hooks let you intercept any step to log it, validate it or redirect it. - Limit. Guardrails catch mistakes before they run. Lifecycle controls include turn limits, token budgets, cancellation and stop reasons. - Recover. Steering handlers let agents correct themselves instead of failing silently. My opinion on order: set the limits first, then add hooks so you can see what the limits catch. A single create_harness() call in Python, or createHarness() in TypeScript, returns an optimized agent, in the README's words. The SDK is Apache-2.0 licensed. These are mechanisms. My reading of the README excerpt is that it states no default policy, so what to block, what budget to set and what to log would fall to whoever deploys it; check the repository before relying on that. The README excerpt I used shows no hook example, so for working code, go to the repository. 5. What the survey says is still open The harness-engineering survey gives an outside view. It says it covers 110+ papers and 23+ deployed systems, and identifies nine open challenges. Security and sandboxing is challenge number one. The survey attributes 15–35% frontier-model container escape rates to a benchmark it calls SandboxEscapeBench, and I saw that figure only second-hand, so treat it as the survey's claim. The survey's advice is blunt: treat sandboxing as a hard requirement and apply defense-in-depth. It also lists a harness-specific threat model, covering tool hijacking, context poisoning and agent impersonation, as future work. For anyone planning multi-agent setups, one more line stands out. The survey says Byzantine fault tolerance remains unsolved for adversarial multi-agent settings. The term comes from the problem of getting generals who communicate only by messenger to agree on a plan when some may be traitors, per Lamport, Shostak and Pease. 6. What I would ask for before a pilot This section is my opinion, not a sourced finding. If I were setting a baseline for coding agents in a company, I would ask for these before the first pilot: !A six-item checklist: sandbox on by default, narrow mounts, hardened control plane, explicit budgets, logging hooks, human approval for high-stakes steps. 1. Sandbox on by default, with the run mode checked in tooling, so nobody discovers the host-level mode by accident. 2. Narrow mounts: one project per task, so one mistake stays small. 3. The control plane treated as its own system to harden, separate from the execution sandbox, since the sandbox does not cover it. 4. Budgets and turn limits set explicitly, so a runaway task stops. 5. Hooks that log every step, so you can see what happened. 6. Human approval for high-stakes steps, so a person owns the risky calls. The last items line up with the survey's governance layer, which covers security constraints, human handoff points, HITL policy, audit trails and compliance enforcement. Separately, in its section on multi-agent fault tolerance, the survey recommends gating high-stakes coordination on human approval. Each item is cheap to check before a pilot and expensive to retrofit. Ask each team to show, not describe, the run mode, the mount, the budget and the log. If any of the four is "we will look into it", the pilot is not ready. None of this needs a particular product; it needs someone to own the answers, hence the question below. Draw the boundary before the agent runs. Which boundary do you enforce today for coding agents, and who owns it on your team?

Can we actually build the thing now? Part 1 ended with "Now we can actually build it." So in Part 2 we build it on both frameworks, and give each one the same request to compare the courses they make. The first iteration built by yours truly Claude Opus 5.5 had 235 tests, and every single one of them passed. But when I started poking (ok, so this has been my huge limitation with Claude. It doesn't surface gaps until I start poking), I realized not one of those tests had ever talked to a real model 🙃 When I pointed it at a real model, it failed on the very first call. And when it finally did produce a Terraform course, the quiz answers sat right under the questions and every exercise came with the finished files. So I deleted the entire folder and started from scratch. The rebuild came out much better, and it's the one we'll walk through in this post. Rant Aside. Because accuracy is super matters to me, whenever I'm learning something new, I search for "the most up to date resource", and more than once I've bought a course only to realise a few lessons in that it's already out of date. Case in point: I started Part 1 a couple of weeks ago, and Strands has already shipped a whole agent harness. We'll start with what using it looks like, then build it up side by side the way we did in Part 1. Everything here ran between 27 and 29 September 2026, on the same versions as Part 1 (strands-agents 1.55.1 and google-adk 2.9.0), and the code is in the companion repo. Both frameworks are actively developed, and strands-agents 1.61.1 and google-adk 2.12.0 were out by the time this went up, so some APIs may have changed since. What should a learning system actually do? When I started over, I wrote down what I actually want from this thing: 1. I say what I want to learn, and it asks a couple of questions to get the course right. 2. It covers what people actually do with it at work. For example, pinning a shared module's version. 3. It's as current as the docs on the day it's built. 4. I learn by doing, and there are tests that verify my work. 5. I have to try before I see the answer. 6. It doesn't cost a fortune to build, and it tells me what it spent. What does using it look like? You'll need uv. Then install it once, straight from the repo: uv tool install "git+https://github.com/coolchigi/strands-vs-adk#subdirectory=part2-learning-system" and start a course from anywhere: learn "I want to learn TypeScript so I can build a small web app" The first time, it asks for a Gemini API key and saves it, so you never set it up again. That key is for the ADK build, which runs on Gemini and is the default. To build with Strands, which runs Claude on Amazon Bedrock, add --engine strands. You'll need AWS credentials and access to Claude in Bedrock, and the Strands Python quickstart walks through setting both up: learn --engine strands "I want to learn TypeScript so I can build a small web app" If you already have AWS credentials and no Gemini key, it picks Strands on its own and doesn't ask for a key. Every course has a spending cap, $5 on Gemini and $10 on Claude by default, and we'll get to how it works in the bill section. Then it asks what it needs to know, and you answer right there: A few questions first, so I build the right course: What is your background with programming, particularly JavaScript? > I know JavaScript well but have never used TypeScript What kind of web app are you looking to build, and do you have a specific framework in mind (e.g., React, Next.js, Node.js)? > a small web app with a backend API Got it. Building your TypeScript course in ~/courses/typescript This usually takes 30 to 90 minutes. You can stop it any time, and running the same command again picks up where it left off. Looking for the official TypeScript documentation ✓ Found it: TypeScript, version 7.0 ✓ Worked out 15 things you'll need to be able to do ✓ Checked 126 facts, word for word, against 17 official pages ✓ Planned 5 units, 12 lessons Writing lesson 1 of 12: Compiler CLI and Strict Type Validation ✓ passed its checks ... Your course is ready: ~/courses/typescript 12 lessons, and it cost about $2.94 to build. That's a trimmed output of the real run. Once the build finishes, you get a folder with a README.md for each lesson. Each one has the explanation, where every fact links to the page it came from, then a worked example, then an exercise with files for you to change. You work through it with learn: cd ~/courses/typescript learn next # where you are, and what's next learn check # run the exercise's tests learn hint # one hint at a time learn solution # the answer, when you want it learn quiz # answer first, then see the answer learn review # questions you missed come back after 1, 3, 7 and 21 days Lesson 1's exercise gives you a formatter.ts with two TODO comments, which mark the code you have to write. One of them is to give a parameter a type. Here's what learn check says if you fix the code but skip the type: $ learn check Checking units/u1-strict-compiler-configuration-and-compilation/u1-l1-compiler-cli-and-strict-type-validation/exercise ... formatter.ts(1,28): error TS7006: Parameter 'input' implicitly has an 'any' type. Not yet. Read the messages above, change your files, and check again. Stuck? learn hint Add the type and it passes, then points you to learn quiz. That error comes from the TypeScript compiler itself, because learn check runs each subject's real tools. So what you need installed depends on what you're learning. A Terraform course needs Terraform, a Python course needs Python 3.9 or newer, and a JavaScript or TypeScript course needs Node. Anything else has no tests, so you compare your work with the solution. How is it built? Behind those learn commands is the system we sketched in Part 1, with one new agent at the front: learner ─▶ intake ─▶ researcher ─▶ curriculum builder ─▶ teacher ─▶ your course ▲ ▲ ▲ └──── the feedback agent reviews each stage, sends it back with reasons - The intake asks what it needs to know, like where you're starting from and what you want to build. - The researcher reads the official docs and makes a list of what the course has to teach (i.e. its objectives). It only keeps facts from the live pages. - The curriculum builder turns that list into units and lessons, and picks an exercise or a quiz to check each one. - The teacher writes each lesson, including the tests that check your exercise. - The feedback agent checks each stage's work and sends it back if something's wrong. Code checks what it can, like running every exercise, and a model reviews the rest. Most of the system, like fetching pages and running tests, is plain Python that both sides share. The frameworks handle the parts this post compares: reaching the docs, getting typed answers back, running the feedback loop and keeping the bill down. Both frameworks plug in through one method: class Brain(Protocol): def ask(self, name: str, instruction: str, prompt: str, schema: type[T], research: bool = False) -> T: """One agent call, returning `schema`. With research=True the agent can search and read.""" Each framework implements ask its own way and runs the stages in its own graph. The Strands side uses Claude Sonnet 4.6 on Amazon Bedrock, which is Strands' default, and the ADK side uses Gemini 3.8 Flash. So when the two sides behave differently, it can be the framework or the model, and we'll say which one it was each time. How does the researcher reach the docs? The researcher needs tools to find the right pages and read them. Some of those tools come from MCP servers. MCP (Model Context Protocol) is an open source standard for connecting AI apps to outside tools and data, so the same server works with both Strands and ADK. Here's how they both do it: | Strands | ADK | |---|---| | No search in the package | google_search, Google Search built into Gemini only | | MCP servers go straight into tools | MCP servers come in through McpToolset | | web_fetch reads a page live | load_web_page, which I couldn't use | Both sides also get the same two extra tools. One is AWS's documentation server, which searches and reads AWS's docs with no key. The other is list_docs_pages, a small tool we wrote that lists every page a docs site has, from the site's own sitemap. Strands Strands ships tools for fetching pages and none for searching, so I used an MCP server. Here we create an MCPClient and pass it straight into tools, and Strands opens and closes the connection for us. This is what connecting to the AWS docs server looks like: from strands import Agent from strands.tools.mcp import MCPClient aws_docs = MCPClient(url="https://knowledge-mcp.global.api.aws") researcher = Agent(system_prompt="Answer from the AWS documentation, in one sentence.", tools=[aws_docs], callback_handler=None) print(researcher("How big can a single S3 object be?")) A single Amazon S3 object can be a maximum of **5 terabytes (TB)** in size, though the largest object that can be uploaded in a single PUT request is **5 GB** ... Keep that 5 TB in mind, because we'll ask the ADK side the same question. For general web search I used Exa's hosted MCP server because it didn't need an API key. Its free tier ran out partway through a single course build, so on Strands, web search stays off unless you add an Exa or Tavily key. For now, you don't need one, because list_docs_pages finds the pages on the docs site itself. To read a page, web_fetch has two modes. The default, agentic, hands the page to a second agent and returns that agent's summary, and markdown returns the page itself. We use markdown mode because, for every fact, the researcher copies the sentence on the page that backs it up, and our code checks that sentence is really there. from strands.vended_tools.web_fetch import make_web_fetch fetch = make_web_fetch(mode="markdown", max_content_chars=50_000) It needs the web-fetch extra, so pip install "strands-agents[web-fetch]". ADK ADK ships search as google_search, and it only works on Gemini. On any other model, like Claude, the agent still builds, but the first call fails with ValueError: Google search tool is not supported for model .... MCP servers come in through McpToolset. This is the same AWS docs server, connected next to google_search: import asyncio from google.adk.agents import Agent from google.adk.runners import InMemoryRunner from google.adk.tools import google_search from google.adk.tools.mcp_tool import McpToolset, StreamableHTTPConnectionParams from google.genai import types aws_docs = McpToolset(connection_params=StreamableHTTPConnectionParams(url="https://knowledge-mcp.global.api.aws")) researcher = Agent( name="researcher", model="gemini-3.8-flash", instruction="Answer from the AWS documentation, in one sentence.", tools=[google_search, aws_docs], generate_content_config=types.GenerateContentConfig( tool_config=types.ToolConfig(include_server_side_tool_invocations=True) ), ) async def main() -> None: runner = InMemoryRunner(agent=researcher, app_name="docs") session = await runner.session_service.create_session(app_name="docs", user_id="me") message = types.Content(role="user", parts=[types.Part(text="How big can a single S3 object be?")]) async for event in runner.run_async(user_id="me", session_id=session.id, new_message=message): if event.is_final_response() and event.content and event.content.parts: print("".join(part.text or "" for part in event.content.parts)) await aws_docs.close() asyncio.run(main()) According to AWS documentation, a single Amazon S3 object can be up to 50 TB in size ... The Strands agent said 5 TB and the ADK agent said 50 TB, and AWS's own page says "each object can be up to 50 TB in size". Both agents had the same AWS docs server. Claude searched once, got a result that mentioned both 5 TB and 50 TB, and answered 5 TB. Gemini kept going, with more searches, and in both runs I traced it read the S3 FAQ page itself before answering. That's why nothing a model says goes into a course unless it comes with a quote from the live page. !The same S3 question asked on both sides, against the same AWS docs server. Claude searches once and answers 5 TB. Gemini searches, reads the S3 FAQ page, searches again, and answers 50 TB If you use McpToolset yourself, run all your agent calls on one event loop. The toolset keeps its MCP session on the loop it first ran on. My first version called asyncio.run for every agent call, and asyncio.run closes its loop when it's done, so every call after the first found its session on a closed loop. ADK has its own page loader, load_web_page, but it kept losing the pages we needed. Terraform's docs redirect a lot, and it won't follow a redirect, so it came back with Failed to fetch url: ... instead of the page. When a page did load, it threw away every line of 3 words or fewer, which took out 84 of the 181 lines in one tutorial's code examples, $ terraform init and required_providers { included. So on the ADK side, the agents find the pages, and our own code fetches them in full. The ADK example also had a generate_content_config line we haven't explained yet. It's there because of the very first real call. What happened on the first real call? The researcher's first job is to find the official docs and the current version, and hand them back as a typed plan through output_schema, which is how ADK asks a model for a typed answer. The first real call came back with this: 400 INVALID_ARGUMENT: Please enable tool_config.include_server_side_tool_invocations to use Built-in tools with Function calling. Hmm, function calling. That's weird because I hadn't given it a function. ADK had. When an agent has both output_schema and tools, ADK quietly adds a function tool called set_model_response, and the model hands back its typed answer by calling it. So my agent was sending Google Search, a built in tool, together with a function tool, and the Gemini API wants you to say that's on purpose. I tried GoogleSearchTool(bypass_multi_tools_limit=True) first and got the same 400, because ADK adds set_model_response after that setting is checked. The fix was what the error asked for, turning on include_server_side_tool_invocations: import asyncio from google.adk.agents import Agent from google.adk.runners import InMemoryRunner from google.adk.tools import google_search from google.genai import types from pydantic import BaseModel class Plan(BaseModel): official_domains: list[str] current_version: str | None = None planner = Agent( name="plan", model="gemini-3.8-flash", instruction="Find the official docs domain and the current release of the subject. Use google_search.", tools=[google_search], output_schema=Plan, output_key="plan", generate_content_config=types.GenerateContentConfig( tool_config=types.ToolConfig(include_server_side_tool_invocations=True) ), ) async def main() -> None: runner = InMemoryRunner(agent=planner, app_name="plan") session = await runner.session_service.create_session(app_name="plan", user_id="me") message = types.Content(role="user", parts=[types.Part(text="Terraform")]) async for _ in runner.run_async(user_id="me", session_id=session.id, new_message=message): pass done = await runner.session_service.get_session(app_name="plan", user_id="me", session_id=session.id) print(done.state["plan"]) if __name__ == "__main__": asyncio.run(main()) {'official_domains': ['developer.hashicorp.com', 'terraform.io'], 'current_version': '1.16.4'} Without the generate_content_config line, the 400 comes back. None of my 235 tests could have caught this, because they ran against a fake model, and a fake model accepts any request. Which one found the current version? The plan step asks for the subject's current version. On 27 September Terraform's was 1.16.4, per HashiCorp's own release API. Here's every plan step I ran while the Strands side searched through Exa: | Answer | Strands (Claude + Exa) | ADK (Gemini + Google Search) | |---|---|---| | 1.16.4 | 1 | 8 | | 1.16.3 | 5 | 0 | | 1.16.2 | 3 | 0 | | "1.9.x (1.9 series, latest stable as of mid-2024)" | 1 | 0 | Twice Claude quoted the Terraform install page as saying "Version: 1.16.3" when the live page said 1.16.4. Exa's copy of the page was a few days behind, and Claude was quoting the copy. ADK's search is Google's index, and Strands' search is whatever server you plug in, so checking how fresh it is falls to you. So now the version gets checked like every other fact. The plan has to name an official page that states it, with the exact words, and our code fetches that page and checks the words are there. Once Strands had web_fetch to read the live page, it got 1.16.4 too. There's one bug left here. Every course's README says which version it was written against. If the feedback agent sends research back for more facts, a version our code couldn't find on the live page, like Claude's 1.16.3, can still end up in that README. How do we get typed answers back? Every agent in the system hands back a typed object, like a plan, a list of facts or a lesson. Both frameworks take a Pydantic model and give you something shaped like it, and they differ in where you pick it up. Here's how they both do it: | Strands | ADK | |---|---| | structured_output_model= on the call or the agent | output_schema= on the agent | | The object comes back on result.structured_output | A dict lands in session state under output_key | | Works next to any tools | Adds a function tool, so built in tools need include_server_side_tool_invocations | Strands We pass the model on the call, and get the object back on the result: from pydantic import BaseModel from strands import Agent class Question(BaseModel): stem: str options: list[str] answer: int # index into options explanation: str agent = Agent(system_prompt="You write quiz questions about Terraform.", callback_handler=None) result = agent("One question on what terraform init does.", structured_output_model=Question) question = result.structured_output print(type(question).__name__) print(question.stem) print(question.options[question.answer]) Question What is the primary purpose of the `terraform init` command? It initializes a working directory by downloading providers, modules, and setting up the backend result.structured_output is a real Question. Strands gives the model a tool shaped like our schema and makes it call it, and if the model never does, you get StructuredOutputException. So the Strands side catches that and asks again, with the reason. ADK We put the schema on the agent, and the answer lands in session state: import asyncio from google.adk.agents import Agent from google.adk.runners import InMemoryRunner from google.genai import types from pydantic import BaseModel class Question(BaseModel): stem: str options: list[str] answer: int # index into options explanation: str agent = Agent( name="quiz", model="gemini-3.8-flash", instruction="You write quiz questions about Terraform.", output_schema=Question, output_key="question", ) async def main() -> None: runner = InMemoryRunner(agent=agent, app_name="quiz") session = await runner.session_service.create_session(app_name="quiz", user_id="me") message = types.Content(role="user", parts=[types.Part(text="One question on what terraform init does.")]) async for _ in runner.run_async(user_id="me", session_id=session.id, new_message=message): pass done = await runner.session_service.get_session(app_name="quiz", user_id="me", session_id=session.id) raw = done.state["question"] print(type(raw).__name__) question = Question.model_validate(raw) print(question.stem) print(question.options[question.answer]) asyncio.run(main()) dict Which of the following actions is performed by running the 'terraform init' command? Initializes the backend configuration, downloads provider plugins, and installs referenced modules It comes back as a plain dict, so turning it into a Question is our job. And if the agent ends its turn without writing an answer, state stays empty. That's what stopped the first TypeScript build, with nothing was written to session state under 'answer', so the ADK side also asks again with the reason. Where does the feedback loop live? As we saw in Part 1, both frameworks can loop inside a graph. Where they differ is what a node can be, and what stops the loop. Here's how they both do it: | Strands | ADK | |---|---| | GraphBuilder, conditional edges | Workflow, routed edges | | set_max_node_executions() caps the loop | Your router stops the loop | | A node has to be an agent or a MultiAgentBase | A node can be any function | | Hitting the cap ends the graph as FAILED | Nothing to hit | The first difference matters here, because our stages aren't agents. The research stage, for example, is Python code that calls several agents and checks every quote in between. Strands In Strands, a graph node has to be an agent or a MultiAgentBase, and you only find that out when the graph runs. add_node accepts a plain function, so does build(), and then the graph reaches that node and stops with ValueError: Node 'stage' of type '<class 'function'>' is not supported. So each stage gets wrapped in a small MultiAgentBase subclass. Here's the loop with stand-in stages that just print, so it runs without a model or any keys: import asyncio from strands.multiagent import GraphBuilder from strands.multiagent.base import MultiAgentBase, MultiAgentResult, Status board = {"reviews": 0, "verdict": None} class Stage(MultiAgentBase): def __init__(self, name, work): super().__init__() self.name = name self.work = work async def invoke_async(self, task, invocation_state=None, **kwargs): # the graph runs nodes on its event loop, and real stages block, so off it they go await asyncio.to_thread(self.work) # results stays empty: Strands walks into it to build the next node's input, # so it has to be real results or nothing return MultiAgentResult(status=Status.COMPLETED, results={}) def teach(): print("teaching") def review(): board["reviews"] += 1 board["verdict"] = "revise" if board["reviews"] < 3 else "approved" print(f"review {board['reviews']}: {board['verdict']}") builder = GraphBuilder() builder.add_node(Stage("teaching", teach), "teaching") builder.add_node(Stage("feedback", review), "feedback") builder.add_edge("teaching", "feedback") builder.add_edge("feedback", "teaching", condition=lambda state: board["verdict"] == "revise") builder.set_entry_point("teaching") builder.set_max_node_executions(10) builder.reset_on_revisit(True) graph = builder.build() if __name__ == "__main__": result = graph("build the course") print(result.status) teaching review 1: revise teaching review 2: revise teaching review 3: approved Status.COMPLETED In Stage, results={} has to stay empty. If you put the stage's output there, Strands tries to read it as the next node's input and crashes with AttributeError: 'str' object has no attribute 'results'. So the stages share their work through board, a plain dict, and the graph only decides what runs next. If the reviewer never approves, the same file stops after 10 node executions and ends Status.FAILED. It doesn't raise, so check result.status. Without set_max_node_executions, build() only logs a warning, "Graph without execution limits may run indefinitely if cycles exist". ADK In ADK, a node can be a plain function, no subclass needed: import asyncio from google.adk import Event, Workflow from google.adk.runners import InMemoryRunner from google.genai import types board = {"reviews": 0, "verdict": None} MAX_REVIEWS = 4 async def teaching() -> None: # nodes run on the workflow's event loop, and real stages block, so off it they go await asyncio.to_thread(print, "teaching") def feedback() -> None: board["reviews"] += 1 board["verdict"] = "revise" if board["reviews"] < 3 else "approved" print(f"review {board['reviews']}: {board['verdict']}") def route() -> Event: if board["verdict"] == "approved" or board["reviews"] >= MAX_REVIEWS: return Event(route="stop") return Event(route="revise") def stop() -> None: return None root_agent = Workflow(name="revision_loop", edges=[ ("START", teaching, feedback, route), (route, {"revise": teaching, "stop": stop}), ]) async def main() -> None: runner = InMemoryRunner(agent=root_agent, app_name="loop") session = await runner.session_service.create_session(app_name="loop", user_id="me") message = types.Content(role="user", parts=[types.Part(text="build the course")]) async for _ in runner.run_async(user_id="me", session_id=session.id, new_message=message): pass if __name__ == "__main__": asyncio.run(main()) It prints the same thing, without the status line. In ADK, route is the only thing that ends this loop. Without the board["reviews"] >= MAX_REVIEWS check, a reviewer that never approves keeps it looping, because nothing in ADK limits how many times a node runs. What the real system does with it The learning system's graphs are those two shapes, with each review as a node of its own: !A run through the learning system. The research review sends research round again with its reasons, then forward. The curriculum review finds the problem started in research and sends the run back upstream, and research keeps what it verified before the run goes forward to done After each stage, a review decides what happens next. The run moves on, the stage runs again with the reviewer's reasons, or the run goes back to research if that's where the problem started. A stage gets 3 tries, and after that the run moves on and the course lists what's still wrong. Going back to research used to start it again from nothing. On the Strands side, a reviewer found a few gaps, the rerun threw away 306 facts that had already been checked, and that one trip cost $11.81. Now research that comes back keeps everything it already checked and only looks for what's missing. How do we keep the bill down? A course takes hundreds of model calls, from picking pages to writing and reviewing every lesson, and more when a review sends something back. The Strands side runs on Claude Sonnet 4.6, which costs 4 times what Gemini 3.8 Flash does per token. Part 1 covered the limits each framework has built in. Strands takes limits= on a call, and ADK takes RunConfig(max_llm_calls=...) on a run. But those only count within one call or one run, and a course is hundreds of them, so I wanted one number for the whole course, in dollars. I added a spending cap to every course, and a meter that counts what every agent has spent. Before each model call, the cap checks the meter and refuses the call if we've exceeded the cap. Both frameworks give us a hook in exactly the right place, and they attach differently. Here's how they both do it: | Strands | ADK | |---|---| | add_hook on an agent you already have | before_model_callback on the agent | | Usage is on agent.event_loop_metrics | Usage is on each llm_response | | Raise, and it comes out as is, or wrapped once a tool has run | Raise, and it comes back as an error event first | Strands Strands already counts each agent's tokens in agent.event_loop_metrics, so the hook just reads them: from strands import Agent from strands.hooks import BeforeModelCallEvent PRICE = (3.00, 15.00) # USD per million tokens, in and out, for Claude Sonnet 4.6 CAP = 0.001 class BudgetExceeded(RuntimeError): pass def spent(agent: Agent) -> float: usage = agent.event_loop_metrics.accumulated_usage return (usage["inputTokens"] * PRICE[0] + usage["outputTokens"] * PRICE[1]) / 1_000_000 def enforce_cap(event: BeforeModelCallEvent) -> None: if spent(event.agent) >= CAP: raise BudgetExceeded(f"spent ${spent(event.agent):.4f}, the cap is ${CAP}") agent = Agent(system_prompt="Answer in one short paragraph.", callback_handler=None) agent.add_hook(enforce_cap) print(agent("What does terraform init do?")) print(f"spent ${spent(agent):.4f}") try: print(agent("And terraform plan?")) except BudgetExceeded as e: print("stopped:", e) `terraform init` initializes a Terraform working directory by downloading ... spent $0.0018 stopped: spent $0.0018, the cap is $0.001 BeforeModelCallEvent also has a cancel field, which looks like the obvious tool. Cancelling hands the loop a fake assistant message, "model call denied by hook", so a plain call returns that as if the model had said it, and a structured call fails blaming a model that never got the call. Raising says what actually happened. There's one more catch, and it only shows up in a real run. On the first model call of an invocation, your exception comes out exactly as you raised it, which is all the example above ever does. On any call after a tool has run, Strands wraps it in its own EventLoopException. My research agents use tools, so the first time the cap fired for real, except BudgetExceeded didn't match and the run died with a traceback. The fix is to look through the exception's causes. ADK Here we pass ADK the callbacks when we build the agent, and count the tokens as each response comes back: import asyncio from google.adk.agents import Agent from google.adk.runners import InMemoryRunner from google.genai import types PRICE = (0.75, 3.75) # USD per million tokens, in and out, for Gemini 3.8 Flash CAP = 0.0001 tokens = {"in": 0, "out": 0} class BudgetExceeded(RuntimeError): pass def spent() -> float: return (tokens["in"] * PRICE[0] + tokens["out"] * PRICE[1]) / 1_000_000 def enforce_cap(callback_context, llm_request): if spent() >= CAP: raise BudgetExceeded(f"spent ${spent():.6f}, the cap is ${CAP}") def count(callback_context, llm_response): usage = llm_response.usage_metadata if usage: tokens["in"] += usage.prompt_token_count or 0 # thinking tokens are billed as output and reported separately tokens["out"] += (usage.candidates_token_count or 0) + (usage.thoughts_token_count or 0) agent = Agent( name="tutor", model="gemini-3.8-flash", instruction="Answer in one short paragraph.", before_model_callback=enforce_cap, after_model_callback=count, ) async def ask(runner, session_id, text): message = types.Content(role="user", parts=[types.Part(text=text)]) async for event in runner.run_async(user_id="me", session_id=session_id, new_message=message): if event.error_code: print("error event:", event.error_code) elif event.is_final_response(): print(event.content.parts[0].text) async def main(): runner = InMemoryRunner(agent=agent, app_name="tutor") session = await runner.session_service.create_session(app_name="tutor", user_id="me") await ask(runner, session.id, "What does terraform init do?") print(f"spent ${spent():.6f}") try: await ask(runner, session.id, "And terraform plan?") except BudgetExceeded as e: print("stopped:", e) asyncio.run(main()) The `terraform init` command initializes a working directory containing Terraform ... spent $0.001424 error event: BudgetExceeded stopped: spent $0.001424, the cap is $0.0001 Two things are different here. Gemini 3.8 Flash thinks, and thinking is billed as output but reported separately as thoughts_token_count, so a meter that only adds candidates_token_count undercounts every call. And the exception comes back twice, first as an error event with error_code set to its class name, then raised again when the stream ends. So every loop over ADK's events checks for an error event and stops straight away. The bigger difference is that ADK can put the cap on every agent at once. In both frameworks we can add enforce_cap to an agent after it's built, with add_hook on Strands, or by setting the agent's before_model_callback on ADK. But ADK also has plugins, which we register once for the whole app. A plugin's before_model_callback runs before every model call, for every agent. Strands has nothing like that, so each Strands agent needs its own hook. What did it cost? Here's what each course cost, from the meters, at list prices: | Course | Built with | Cost | |---|---|---| | Terraform | ADK, Gemini 3.8 Flash | $3.35 for the lessons (research and planning ran before the meter saved anything) | | Terraform | Strands, Claude Sonnet 4.6 | $18.36: research $7.06, planning $0.87, 23 lessons $10.43 | | TypeScript | ADK, Gemini 3.8 Flash, through learn | $2.94, everything included | Across the 2 days of building, with the lost runs, the restarts and the bugs, Claude on my Bedrock account came to about $59, going by AWS's Cost Explorer. Two bugs let it get that high. The Strands meter remembered each agent's usage by id(agent), and Python hands a new object the id of one that's been thrown away, so fresh agents inherited dead ones' totals and the meter undercounted. One run reported -834,706 input tokens for a single role. And the cap lived in the process, so every restart got a fresh budget. Both are fixed now. Each agent's count lives in its own hooks, and spend is saved to disk the moment it's counted, so a restarted course carries on from what it already spent. A cap that just stops leaves you with half a course, though. So now, halfway to the cap, the course switches to a cheaper model to finish, and tells you: $5.00 spent of your $10.00 cap, so the rest of this course is written by a cheaper model (global.anthropic.claude-haiku-4-5-20251001-v1:0). On Strands that's Claude Haiku 4.5, at $1 in and $5 out per million tokens against Sonnet 4.6's $3 and $15. On ADK it's Gemini 3.5 Flash-Lite, at $0.30 and $2.50. Before trusting either, I asked each one to find Terraform's current version through the system's own Brain, with its tools and typed answers. Both found 1.16.4 on the live install page. Swapping the model mid-course is easy on both sides, because each call builds a fresh agent anyway. Strands takes a different BedrockModel, and ADK takes a different model name. The cap, the switch point and the cheaper model are all settings in one file. Bedrock's US-only inference profile costs 10% more than the global one, which I only found by asking AWS's price list. And Gemini 3.8 Flash's price doubles on 1 January 2027, to $1.50 in and $7.50 out. Newer, cheaper Claude models came out while I was writing this. Sonnet 5.5 came to Bedrock on 28 September at $2 in and $10 out, and Haiku 5.5 on 7 October, from $0.10 in and $0.50 out. Is the course any good? Here's what each side produced from the same request, "I'd like to deeply understand Terraform and build projects along the way." | | Strands (Claude) | ADK (Gemini) | |---|---|---| | Facts checked against a live page | 546, from 39 sources | 162, from 19 sources | | Lessons | 23 in 5 units | 12 in 4 units | | Exercises that pass every check | 12 of 23 | 12 of 12 | | Lessons that ship with a known problem | 23 | 1 | | Cost | $18.36 | $3.35, plus research and planning | The TypeScript course from earlier was built through learn. It has 12 lessons, all 12 exercises pass every check, and nothing is left on its known problems list, for $2.94. I planned to judge the courses by their review scores. Then I did the lessons myself, and found more than the reviewers had. The exercises The first finished ADK course had 5 exercises out of 12 that failed their own checks, and 4 of them hit the same error, Unknown condition value ... This means you have executed a run block with command = plan. That one was my fault. I'd told the teacher to write the tests with command = plan, and at plan time things like IDs don't exist yet. The tests run against a mock provider (i.e. a fake AWS), so command = apply doesn't create anything real or need an AWS account. It also makes up values like IDs, so the tests have something to check. After that fix, I rebuilt the ADK course and worked through lesson 1 myself. I left one of its 3 TODOs alone, filled in the other two, and learn check still said "Passed". The system's rule was that an untouched exercise (i.e. every TODO left alone) has to fail its tests before the course ships. If the tests check just one TODO, that rule passes, so the TODO I left alone was never tested. So now the system also checks each TODO on its own. It fills in every TODO with the answer except one, and the tests have to fail. It does that for each TODO in turn, so you can't skip any of them and still pass. 3 of the 12 lessons in that course had a TODO you could skip. Even with every TODO checked on its own, a wrong answer can still pass. In unit 3, lesson 3 of the same course, a TODO asks you to output the ID of the EC2 instance the lesson builds. If you output its ARN instead, learn check still says "Passed". The mock provider makes up both values as random 8-character strings, so the test can't tell them apart. TypeScript had a sneakier version. Node runs TypeScript by stripping the types out without checking them, so an exercise about types could pass with no types at all. So TypeScript exercises also go through the real compiler in strict mode, which is the TS7006 you saw earlier. The quizzes Here's where the right answer sat: | Right answer is option | 1 | 2 | 3 | 4 | |---|---|---|---|---| | ADK (Gemini) | 30 | 5 | 1 | 0 | | Strands (Claude) | 14 | 40 | 15 | 0 | Gemini put the right answer first 83% of the time, and Claude put it second 58% of the time, so a learner who always picked 1 would pass most of ADK's quizzes. The course writer shuffles the options now, and after shuffling, the ADK course's 36 right answers are spread 10, 7, 9 and 10 across the 4 positions. The reviewers Gemini's reviewer signed off on all 12 of ADK's final lessons. Claude's reviewer, with the same prompt, didn't sign off on any of the Strands lessons it reviewed. So I swapped them. Claude reviewed ADK's 12 lessons and found 29 problems, at least one in every lesson, and most were real. One was a worked example that says terraform fmt scans the .terraform folder (that's where providers are cached), and another was a lesson about terraform state mv whose exercise never has you run it. Gemini reviewed Strands' 23 and passed 13 of them. So Claude's course looks worse on paper because its reviewer is harder to please, and Gemini's looks clean partly because its reviewer isn't. What the courses missed Remember item 2 on my list, a course that covers what people actually do at work? Where I work, we publish Terraform modules, and every team that uses one pins a version, with ?ref= on a Git source or version on a registry module, and bumps it when there's a new release. We do that all the time. Neither course taught it as something you do. The ADK course mentions it in one lesson's worked example and quiz, and the Strands course mentions it once in passing. Both researchers built their plans from how the docs are organised, and the docs don't put "publish a module and pin it downstream" on the front page. That's a gap in the research, so that's where the fix went. The researcher now writes a list of what someone who uses the subject at work does routinely, and names the objective that teaches each item. A plain check flags any item with no objective, and the research reviewer reads each pairing and says when an objective only teaches part of an item. Then I ran the same Terraform research twice more. The check did its job, and in both runs the reviewer caught items that were only half taught, and the research added objectives for them. Module pinning still didn't make it in, because neither run put it on the list in the first place. Of the 6 research runs that wrote this list, 2 put module pinning on it. The check can only hold the objectives to the list, and the list is still the model's idea of what people do at work. What I think fixes it is asking. The intake already asks where you're starting from. If it also asked how your team uses the subject, I'd have told it we pin shared modules, and the research would have had to cover that. Both courses say their exercises run against LocalStack, an AWS emulator (i.e. it pretends to be AWS on your own machine), but neither shows you how to set LocalStack up. In the ADK course, that just means the LocalStack steps fail. The Strands course could cost you money, though, because its last 6 lessons set Terraform up to talk to real AWS. Two of those lessons tell you to run terraform apply, so if you have AWS credentials on your machine, you'd create a real VPC in your account. Terraform would also try to launch EC2 instances there, and those cost money once they're running. learn check is safe in every lesson, because its tests run against the mock provider and can't create anything in AWS. The Strands course's README warns about this at the top now. What I think fixes it is running AWS exercises against an emulator by default, and letting you switch to your own AWS account if you want the real thing. So which one? Here's the whole comparison in one place: | | Strands | ADK | |---|---|---| | Web search | Bring an MCP server | google_search, Gemini only | | Reading a page | web_fetch, markdown or agentic | load_web_page, which I couldn't use | | Typed answers | structured_output_model= on the call or the agent, the object on the result | output_schema on the agent, a dict in session state, plus a flag next to built in tools | | The feedback loop | GraphBuilder, capped by set_max_node_executions | Workflow, stopped by your router | | A stage that isn't an agent | Subclass MultiAgentBase | Any function | | A spending cap | add_hook on each agent | A callback on each agent, or one plugin for every agent | | Errors from your hooks | Wrapped in EventLoopException once a tool has run | An error event first, raised at the end | Both frameworks ran the same stages through the same kind of graph, with the same typed answers. How good the courses came out was down to the models and to my checks. Claude found 3 times as many facts and wrote twice as many lessons, at 4 times Gemini's price per token, and its reviewer was much harder to please. Gemini was cheaper, and its reviewer passed its own side's lessons. Most of the problems I found doing the lessons came from a check I hadn't written yet, or one I'd written wrong. So which one would I pick? For myself, Strands. But ADK did some things better, and if those matter more for what you're building, it's the better pick. Strands was easier to start with. Once AWS credentials and access to the model in Bedrock are set up, an agent is one Python file and three lines of code, and as it runs it prints each tool it calls. It also had set_max_node_executions, which stops the graph once it has run a set number of nodes. ADK has nothing like it, and it was my backstop in case the feedback loop ever went around forever. ADK is lighter to set up, because a Gemini API key is enough to run it on your machine. adk web gives you a chat in the browser with a trace of every call next to it. Any plain function can be a stage in the workflow, where Strands needs it wrapped in a class. And its google_search tool found the current Terraform version 8 times out of 8, though it only works with Gemini. And whichever framework you pick, get a second model to review the first. Both Terraform courses and the TypeScript course are in the repo, and each lesson lists its known problems at the top. If you work through one and find something they missed, open an issue and tell me.

今まで Strands で Nova2 の thinking 使えなかったんだ harness-sdk#4853

Inference on AWS: Precision, Memory and Six Backends Behind One Strands Agent --- title: "Inference on AWS: Precision, Memory and Six Backends Behind One Strands Agent" published: true description: "The companion to the talk: what bits per weight, memory bandwidth and the serving engine do to Gemma 4 on Bedrock, SageMaker, EC2 g6 and g5g, Inferentia2 and Trainium, with the cost of each, measured on one day." tags: aws, ai, machinelearning, gemma cover_image: raw.githubusercontent.com --- This article is the written companion to the talk "Inference on AWS". One Strands agent sends the same question to six AWS backends: Amazon Bedrock, a SageMaker endpoint, vLLM on EC2 g6 and g5g, and a hand-ported Gemma 4 on Inferentia2 and Trainium. The talk adds two ideas that explain most of the results: the precision a model is stored and computed in, and the memory bandwidth of the chip that serves it. xbill9/aws-inference-strands The slides: docs.google.com --- What Is the Problem? AWS sells a model as an API, as a managed endpoint, as a GPU by the hour and as two families of its own chips. The same small model, Gemma 4 E2B, runs at 36 tokens a second on one of them and 132 on another, and the hourly prices do not explain the gap. Three properties do: how many bits each weight takes, how fast the chip can read them, and how well the serving engine uses the chip. --- Which Backends? !One Strands agent sends the same question to four kinds of backend: a managed API, a managed endpoint, your own GPU and your own AWS chip | Backend | What serves the model | Model | |---|---|---| | Amazon Bedrock | managed API | us.amazon.nova-micro-v1:0 | | SageMaker ml.g6.xlarge | AWS vLLM 0.31.0 container | xbill9/gemma-4-E2B-it-qat-q4_0-w4a16-ct-text-emb4 | | EC2 g6.xlarge (NVIDIA L4) | vLLM 0.30 | the same repack | | EC2 g5g.2xlarge (Graviton2 + T4G) | vLLM 0.27.2rc0, patched for sm_75 | google/gemma-4-E2B-it, fp16 | | EC2 inf2.xlarge (Inferentia2) | hand-ported Gemma 4, xbill9/gemma4-optb:slim | Gemma 4 E2B, compiled for Neuron | | EC2 trn1.2xlarge (Trainium) | the same image | the same build | --- Step 1 — Precision: What You Trade Every format below stores the same Gemma 4 E2B. Fewer bits per weight means less memory, less disk and less data to move for every token. It buys speed only where the chip multiplies that format natively, and it costs accuracy unless the weights were trained for it. !Three panels for Gemma 4 E2B: weights in memory per format, suite accuracy per format, and which formats the L4, T4G and Neuron multiply natively | Format | E2B weights in memory | Suite accuracy, 3,880 records | |---|---:|---:| | bf16 | 9.42 GiB | 0.683 | | fp8, from Google's QAT weights | 7.07 GiB | 0.670 | | int8, from Google's QAT weights | 7.07 GiB | 0.686 | | int8, rounded from bf16 | — | 0.670 | | int4, QAT | 6.32 GiB | 0.678 | | int4 with int4 embedding tables | 2.85 GiB | 0.681 | Two results carry the slide. The int4 builds made from Google's quantization-aware-trained (QAT) checkpoint stay within half a point of bf16, while an int8 build rounded from bf16 loses 1.3 points. And E2B's embedding tables are most of its weight: packing them to int4 takes the model from 6.32 GiB to 2.85 GiB. Native support decides whether the smaller format is also faster: | Chip | bf16 | fp8 | int8 | int4 | |---|---|---|---|---| | NVIDIA L4 (g6, SageMaker) | yes | yes | yes | widened to 16-bit | | NVIDIA T4G (g5g) | emulated | no | yes | widened to 16-bit | | NeuronCore-v2 (inf2, trn1) | yes | in hardware; compiler support pending | yes | widened to bf16, 27x per call | Memory figures come from the E2B data-type sweep on one AMD MI300X (2026-10-07) and accuracy from the same checkpoint files on one TPU v5e (2026-09-29), both in gemma4-dev. --- Step 2 — Memory and Bandwidth Across AWS Generating a token reads every weight once, so memory bandwidth sets the ceiling on tokens per second for one request. The engine decides how close a backend gets to that ceiling. | Backend | Accelerator | Memory | Bandwidth | Host RAM | tok/s, one request | |---|---|---|---|---|---:| | g6.xlarge | NVIDIA L4 | 24 GB | 300 GB/s | 16 GiB | 131.9 | | SageMaker ml.g6.xlarge | NVIDIA L4 | 24 GB | 300 GB/s | 16 GiB | 96.5 | | g5g.2xlarge | NVIDIA T4G | 16 GB | 320 GB/s, 277 measured | 16 GiB | 34.9 | | inf2.xlarge | Inferentia2 | 2 x 16 GiB | ~817 GB/s | 16 GiB | 36.5 | | trn1.2xlarge | Trainium | 2 x 16 GiB | ~613 GB/s | 32 GiB | 36.0 | | Bedrock | Nova Micro | not disclosed | not disclosed | — | 130.4 | The Neuron chips have 2.0 to 2.7 times the L4's bandwidth and decode at 0.28 times its speed. Their 32 GB is two 16 GiB pools, one per NeuronCore, so a model either splits across both cores or fits in 16 GiB. AWS publishes Neuron bandwidth per instance; the per-chip figures here divide the largest instance's 9.8 TB/s by its chip count. --- 🔎 Tip: Quota Is Permission to Ask Quota does not reserve a machine. On the rehearsal day the first zone tried in us-east-2 had no capacity for g6 or trn1, and on an earlier run inf2 found no capacity in us-east-2a with 80 vCPUs of quota free. stage/up.sh tries every zone and then the next region, so a launch script that does the same saves a demo. --- Step 3 — Bedrock and SageMaker: the Managed Pair Bedrock needs an IAM permission and an inference profile id, and bills per token: $0.035 per million input tokens and $0.14 per million output tokens for Nova Micro. from strands.models import BedrockModel model = BedrockModel(model_id="us.amazon.nova-micro-v1:0", region_name="us-east-1") SageMaker runs a container you choose on an instance you choose: !You create a model, an endpoint config and an endpoint; SageMaker runs the vLLM container on an ml.g6.xlarge with an L4, health-checks it and logs to CloudWatch; the agent calls InvokeEndpoint and the weights come from Hugging Face at start-up Three API calls build it, create-model, create-endpoint-config and create-endpoint, and vLLM flags pass as SM_VLLM_* environment variables. On the rehearsal day the endpoint reached InService about 9 minutes after deploy on the AWS vLLM 0.31.0 container. It serves the same weights on the same GPU as the g6, and each request carries about 0.24 s of extra time through the managed path: 87 tokens took 0.902 s against 0.660 s at the g6's rate. --- Step 4 — vLLM on an L4 The g6.xlarge runs vLLM 0.30 with two flags that turn on tool calls: --enable-auto-tool-choice --tool-call-parser gemma4 | NVIDIA L4 | | |---|---| | Family | Ada Lovelace, compute capability 8.9 | | Memory | 24 GB | | Bandwidth | 300 GB/s | | BF16 / FP16 tensor | 121 TFLOPS dense | | FP8 tensor | 242.5 TFLOPS dense | | Power | 72 W, single-slot PCIe | It was ready 16 minutes after launch and decoded 131.9 tokens a second for one request, the fastest of the five instances. --- Step 5 — Arm and CUDA, Then and Now The g5g pairs a Graviton2 Arm host with an NVIDIA T4G. AWS calls G5g "the first Arm-based instances in a major cloud to feature GPU acceleration." Serving Gemma 4 on it needs vLLM built from source for sm_75 with a shared-memory patch, because the arm64 image ships without Turing kernels. The same pairing reaches the desk this fall. NVIDIA announced RTX Spark on May 31, 2026: | | EC2 g5g | NVIDIA RTX Spark | |---|---|---| | CPU | AWS Graviton2, Arm | 20-core NVIDIA Grace, Arm | | GPU | NVIDIA T4G, Turing | Blackwell RTX, 6,144 CUDA cores | | CPU to GPU | PCIe | NVLink-C2C | | Memory | 16 GB GPU + 16 GB host | up to 128 GB unified | The aarch64 and CUDA build work that the g5g required is the platform those machines run. --- Step 6 — Why Neuron Is Hard to Serve On The Inferentia2 and Trainium rows run the same hand-ported image, measured on inf2.xlarge: - No vLLM release serves Gemma 4 on Neuron, so the engine is a hand port and ignores tool definitions. - The graph compiles ahead of time for fixed shapes, 512 tokens in total and 128 for the prompt. A new shape costs about 3.9 s to compile. - Memory is 2 x 16 GiB, one per NeuronCore. A single gather above roughly 4 to 5 GB returns zeros with no error. - int4 weights get a dense bf16 copy before each matrix multiply: 27 times slower per call than plain bf16. That is the 0.28x in Step 2: the chips have the bandwidth, and the software stack for this model has not caught up. --- Step 7 — One Strands Agent, Every Backend The agent asks how many of eleven ids are 10 or more and has a count_ids tool that computes it. Every backend that supports tool calls, three runs each on the rehearsal day: bedrock run 1/3 ok quoted=8 filters=['id >= 10'] 1.19s g6 run 2/3 ok quoted=8 filters=['id >= 10'] 0.4s g5g run 1/3 ok quoted=8 filters=['id >= 10'] 2.35s sagemaker run 1/3 ok quoted=8 filters=['id >= 10'] 0.63s All twelve runs answered 8 and sent the filter id >= 10. Grading the filter as well as the number matters: a wrong filter returns an exact number with a clean source. A second agent on Bedrock calls all six backends as tools and quotes a scoreboard sorted in code: | vLLM on EC2 g6 (L4) | Paris | 0.151 | 13.2 | | SageMaker endpoint (vLLM, L4) | Paris | 0.426 | 4.7 | | Amazon Bedrock | Paris | 0.861 | 2.3 | --- Step 8 — What Each Backend Costs cost.py pulls On-Demand prices from the AWS Pricing API and does the arithmetic: | Backend | $/hr | $/month 24x7 | $ per 1M output tokens, 1 request | $ per 1M output tokens, 16 parallel | |---|---:|---:|---:|---:| | EC2 g6.xlarge | 0.8048 | 588 | 1.69 | 0.150 | | SageMaker ml.g6.xlarge | 1.1267 | 822 | 3.24 | 0.291 | | EC2 g5g.2xlarge | 0.556 | 406 | 4.43 | not measured | | EC2 inf2.xlarge | 0.7582 | 553 | 5.77 | not measured | | EC2 trn1.2xlarge | 1.3438 | 981 | 10.37 | not measured | | Bedrock, Nova Micro | — | 0 when idle | 0.14 | 0.14 | A g6 running all month costs more than Bedrock Nova Micro until it averages 710 output tokens a second around the clock, at five input tokens per output token. --- 🔎 Tip: Debug With the Small Model Every path in the talk was brought up with E2B first. Flags, port forwards and tool parsing fail the same way at any size, and E2B loads on Neuron in about 2 minutes against close to 4 for the 26B. --- Compare and Contrast | Backend | Precision served | Bandwidth | tok/s | $ per 1M output tokens, 1 request | Tool calls | |---|---|---|---:|---:|---| | 🥇 EC2 g6, vLLM | int4 weights, int4 tables | 300 GB/s | 131.9 | 1.69 | yes | | 🥈 Bedrock | managed | not disclosed | 130.4 | 0.14 | yes | | 🥉 SageMaker, vLLM | int4 weights, int4 tables | 300 GB/s | 96.5 | 3.24 | yes | | EC2 g5g, vLLM | fp16 | 320 GB/s | 34.9 | 4.43 | yes | | EC2 inf2, hand-ported | compiled for Neuron | ~817 GB/s | 36.5 | 5.77 | no | | EC2 trn1, hand-ported | compiled for Neuron | ~613 GB/s | 36.0 | 10.37 | no | --- So, Which One? Start on Bedrock when one of its models does the job: nothing to run and the lowest cost per token here. Move to vLLM on a g6 for your own weights, quantized from a QAT checkpoint so the smaller format keeps its accuracy; it was the fastest and cheapest instance. SageMaker runs the same container as a managed endpoint for about twice the cost per token. The g5g is the Arm and CUDA path, and Inferentia2 and Trainium pay off once the serving stack supports the model. --- Summary The goal of this article was to explain the talk's results through precision, memory bandwidth and the serving engine. The key to the solution was one Strands agent driving six backends on the same day, with every count, ranking and price computed in code. The results were: - 🟢 int4 from QAT weights stays within half a point of bf16, and int4 embedding tables shrink E2B from 6.32 GiB to 2.85 GiB. - 🟢 vLLM on a g6 decoded fastest, 131.9 tok/s for one request at $1.69 per million output tokens, $0.150 at 16 in parallel. - 🟢 Every tool-calling backend picked the right filter, 12 of 12 runs. - ⚠️ SageMaker adds about 0.24 s per request on the same GPU and costs about twice the g6 per token. - ⚠️ Bandwidth sets a ceiling the engine has to reach: Neuron has 2.0 to 2.7 times the L4's bandwidth and decodes at 0.28 times its speed. - ❌ No vLLM release serves Gemma 4 on Neuron, so those backends run a hand port with fixed shapes and no tool calls. Scope: one rehearsal on 2026-10-12 in us-east-2, with the g5g and Bedrock in us-east-1, three timed requests per backend, client on a laptop through SSM port forwards. Memory per format comes from one MI300X and accuracy from one TPU v5e, both serving the same Gemma 4 E2B checkpoint files, and the 16-parallel prices from earlier compare.py runs of the E2B QAT build on a g6 and an ml.g6.xlarge. Bedrock serves Nova Micro, a different model from the Gemma 4 builds on the other rows. On-demand list prices from the AWS Pricing API on 2026-10-12. The strategy for serving Gemma 4 on AWS from Bedrock to Trainium behind one Strands agent was validated with an incremental step by step approach. --- References - aws-inference-strands | GitHub - Inference on AWS slides | Google Slides - gemma4-dev | GitHub - Strands Agents | GitHub - NVIDIA L4 | NVIDIA - NVIDIA and Microsoft Reinvent Windows PCs for the Age of Personal AI | NVIDIA Newsroom - Amazon EC2 G5g instances | AWS - Amazon EC2 Inf2 instances | AWS - Amazon EC2 Trn1 instances | AWS - AWS Neuron data types | AWS - Amazon SageMaker AI real-time inference | AWS - Amazon Bedrock pricing | AWS

Five ways Claude prompt caching quietly breaks on Amazon Bedrock --- title: Five ways Claude prompt caching quietly breaks on Amazon Bedrock published: true description: "Requests still return 200. You just pay full price again. Five causes from this quarter's bug reports, and a limit I measured: 21 blocks hit, 22 miss." tags: aws, ai, llm, claude canonical_url: harisfarooq.substack.com --- Originally published on my Substack. Prompt caching on Amazon Bedrock can stop working without a single error. I run a platform of Claude agents on Bedrock, and I went through this quarter’s bug reports about Bedrock caching in the big agent frameworks. Five failures kept coming back. For each one: what happens, how to tell if it is happening to you, and what to change. In July, a LiteLLM upgrade changed where the system prompt went in Bedrock requests. Cache hits on Claude Code traffic fell from about 90% to 25-45%, and daily spend went up 2-3x for six days before anyone traced it. Every request returned 200. AWS documents the caching rules, but the one that bites agents hardest, how far back Bedrock looks for an earlier cache entry, is given only as “approximately 20 content blocks”. So I measured it. If this is new to you Datadog found that in March 2026, 69% of all input tokens in its customers’ traces were system prompts: instructions, policies and tool guidance sent again on every call. Prompt caching is what keeps that cheap. Bedrock can reuse the start of a request, meaning your instructions, tool list and earlier conversation, and bill it at about a tenth of the normal input price. You mark where the reusable part ends with a cache point. The next request reuses it only if everything before that point is exactly the same, and only if Bedrock can find the earlier entry. When either fails, the request still works. You just pay full price again, plus a 25% premium to write the cache a second time. The five are ordered roughly the way an agent meets them: first where the cache point sits, then what a busy turn does to it, then the prompt itself, the library that builds it, and the stream that rebuilds it. 1. Move the cache point forward in agent loops In an agent loop, most new tokens are tool calls and tool results, not user messages. If the cache point only goes on the last user message, it stays on the first prompt while the tool history grows behind it. Every call then pays full price for that history. This is what happened in Cline 4.1.0. Its Bedrock code put the cache point on the last message with the role “user”, and in its message format tool results have their own role. So the point never moved past the opening prompt. Users saw one cache write at the start of each session and none after (cline/cline#13738; 4.0.12 was fine). The test: in a long agent session, cache reads should grow call by call. If they stop growing while input tokens keep climbing, the point isn’t moving. The fix is to put the cache point on the last block of the newest message, whatever its role. 2. Add a second cache point after wide tool fan-outs When Bedrock looks for an earlier cache entry, it only searches back a fixed number of content blocks from the new cache point. I measured the edge: with 21 blocks added since the old cache point the next request still hit, and with 22 it missed, every time. Every tool call and every tool result is a block. So one turn with ten or more parallel tool calls can push the old entry out of reach. [](substackcdn.com) pydantic-ai saw the same edge in its own tests. After 10 parallel calls (21 new blocks) the next request read 6,827 tokens from cache. After 11 calls (23 blocks) it read 0 and wrote 7,534 again (pydantic/pydantic-ai#9404). A Strands user saw an 18-call turn followed by 0 tokens read and 134,082 written (strands-agents/harness-sdk#3348). The same requests on Anthropic’s own API didn’t miss, because there a run of tool calls counts as one position. Bedrock counts each one. The test: look at the calls right after your widest turns. A read of zero there, with normal reads elsewhere, is this. The fix: keep the previous cache point when a turn added more than about 18 blocks. Bedrock allows four cache points per request, so there is usually room. 3. Keep anything that changes per request after the cache point Caching is a prefix match. A date, a request ID, a user’s name or a per-item instruction placed before the cache point makes every request unique, so every call writes the cache and none read it. An AWS sample app showed it cleanly. Over 28 calls reviewing the same document, CloudWatch recorded 196,680 tokens written to cache and 0 read, because the cache point was placed after each item’s own instructions. Moving it to right after the document turned the second item into a 4,624-token read (aws-samples/review-and-assessment-powered-by-intelligent-documentation#327). Subtler versions exist too. In one Mastra setup, storing messages in a Postgres jsonb column changed the order of attribute keys on the way back, which changed the prompt bytes and broke the cache (mastra-ai/mastra#25929). The test: run the same request twice and compare the bytes up to the cache point. They have to be identical. The fix is to put dynamic text after the last cache point, and to serialize anything structured in a stable order. 4. Check your library still sends the cache point Most teams don’t write cache points by hand. A framework or gateway adds them, often only for model IDs it recognises. Upgrade the library, switch to a new model ID or an application inference profile ARN, and the cache point can quietly disappear. That’s the LiteLLM incident above. It isn’t rare: MLflow’s AI Gateway turns Bedrock chat messages into plain strings and never sends a cache point at all (mlflow/mlflow#26477), and LangChain’s openwiki found its caching middleware skipped application inference profile ARNs (langchain-ai/openwiki#853). The test: if cache reads and cache writes are both zero, no cache point is reaching Bedrock. Check this after every library upgrade and every model change. 5. Watch how streaming rebuilds the message When you stream a response and save it back into the conversation, your library rebuilds the assistant message from the chunks. If it rebuilds it differently than one piece of text, the history changes shape. In LangChain.js, streaming with extended thinking on Bedrock dropped the block index on text chunks, so one answer was saved as hundreds of separate text blocks. That pushed the next cache point far past the lookback limit (langchain-ai/langchainjs#11357). The test: count the content blocks in a saved assistant message. One reply should be one text block, not hundreds. Checking your own account Two CloudWatch metrics in the AWS/Bedrock namespace tell you most of it: CacheReadInputTokenCount and CacheWriteInputTokenCount, split by model ID. In steady traffic, reads should be many times writes. Writes with reads near zero means caching is on but missing, which costs more than not caching at all. Both near zero means no cache point is being sent. If the hit rate dips at busy times while the prompts stay identical, look at cross-region inference. AWS notes that at times of high demand its routing “may lead to increased cache writes”, because a request can land in a Region that doesn’t hold the cached copy. For the why, turn on Bedrock model invocation logging. The logs carry the cache counts for every call and who made it. What I would still like Two things. Cache diagnostics on Bedrock: Anthropic’s Console now explains why a request missed the cache, and Bedrock doesn’t. And an exact lookback number in the AWS docs instead of “approximately 20”, since one tool call more or less decides whether an agent pays 0.1x or 1.25x for its history. Sources and disclosure The lookback numbers are my own measurement: two requests per size on Amazon Bedrock, Claude Sonnet 4.6, us-west-2, October 2026, run twice each with fresh prompts so an old cache entry could not fake a hit (scripts/live_lookback_boundary.py in the CacheCanary repo). Prices and multipliers are AWS’s published on-demand rates. The other numbers come from LiteLLM’s incident write-up (docs.litellm.ai); Datadog, “Monitor prompt caching to optimize your token usage” (September 2026); Anthropic, “Reducing cost and improving performance with Claude Platform” (September 2026); the AWS Bedrock prompt caching docs; and the public GitHub issues linked in each section. I build the agent platform at Ruby AI, which runs Claude on Bedrock. I also made a small free tool for the checks above, CacheCanary (Haarris/cachecanary). It reads invocation logs and shows hit rate and what the misses cost, and it compares two saved requests to tell you why the second one missed. Since this post I've also written tested guides for LiteLLM, Strands Agents and LangChain that show exactly what each one sends to Bedrock and where the cache is lost. I write these when something breaks in production and the docs did not warn me. The next one goes out first on my Substack.

strandsのdefault model opus-5になってそうなので、Accessできないアカウントだとエラーになりそう harness-sdk#4661