Contextual Retrieval in 80 Lines: Anthropic's Chunk-Prefix Trick, Measured on Your Docs
- Book: RAG Pocket Guide
- Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go
- My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
- Me: xgabriel.com | GitHub
---
Here is a chunk from a quarterly filing: "The company's revenue grew by 3% over the previous quarter." Which company? Which quarter? Grew from what? Inside the filing, the sentence made sense. Cut out, embedded and indexed on its own, it carries none of that, and a query about ACME's Q2 2023 revenue has almost nothing to match against.
That example comes from Anthropic's Contextual Retrieval write-up, published September 19, 2024. The fix they proposed is small. Before you embed a chunk or index it for BM25, ask a model to write a short context for it using the whole document, and glue that context to the front of the chunk. Their chunk becomes: "This chunk is from an SEC filing on ACME corp's performance in Q2 2023; the previous quarter's revenue was $314 million. The company's revenue grew by 3% over the previous quarter."
On their test sets, contextual embeddings plus contextual BM25 cut the top-20 retrieval failure rate by 49%, from 5.7% to 2.9%. Adding a reranker on top took the reduction to 67%, down to 1.9%. Those are their numbers on their data. This post gets you yours: about 80 lines of Python that index your docs both ways, contextualize every chunk with prompt caching so each document is paid for once, and print recall@5 before and after on 30 questions you label yourself.
What Anthropic measured, and what it can't tell you
The metric in the write-up is 1 minus recall@20: the share of relevant chunks that did not show up in the top 20 results. The test data covered codebases, fiction, ArXiv papers and science papers. The headline figures used Gemini Text 004 embeddings and Claude 3 Haiku to write the contexts, which usually came out at 50 to 100 tokens (same source).
The companion cookbook notebook reports at k=5, closer to what this post measures. Its printed run shows Pass@5 going from 80.92% to 88.12% with contextual embeddings alone. The same notebook's summary table and prose don't fully agree with its printed output, which is one more reason to treat any published number as a hint about direction.
None of this tells you whether your docs have the problem. Chunks that lean on their surroundings ("the endpoint above", "this setting", a table whose header lives two chunks back) have the most to gain. A folder of self-contained FAQ entries may barely move. You find out by measuring.
Anthropic's 2024 write-up also suggests that if your whole knowledge base is under 200,000 tokens, you put all of it in the prompt and skip retrieval.
!A single paper card torn from a larger document page, with a burnt-orange strip clipped along its top edge and a thin thread tying it back to the page it came from
The whole trick: the chunk keeps a short note about where it came from.
The script
Install the four dependencies and set ANTHROPIC_API_KEY and OPENAI_API_KEY:
pip install anthropic openai rank-bm25 numpy
The script uses the Anthropic Python SDK for the contexts, the OpenAI Python SDK for embeddings, rank-bm25 for keyword search and NumPy for the vector math. Save it as ctx_eval.py:
import json, re, sys
import numpy as np
from anthropic import Anthropic
from openai import OpenAI
from rank_bm25 import BM25Okapi
claude, oai = Anthropic(), OpenAI()
usage = {"written": 0, "read": 0, "uncached": 0}
PROMPT = (
"Here is the chunk we want to situate within the whole "
"document\n<chunk>\n{chunk}\n</chunk>\n"
"Please give a short succinct context to situate this "
"chunk within the overall document for the purposes of "
"improving search retrieval of the chunk. Answer only "
"with the succinct context and nothing else."
)
def chunk(text, size=500):
words = text.split()
return [" ".join(words[i:i + size])
for i in range(0, len(words), size)]
def situate(doc, piece):
resp = claude.messages.create(
model="claude-haiku-5-5",
max_tokens=300,
thinking={"type": "disabled"},
messages=[{"role": "user", "content": [
{"type": "text",
"text": f"<document>\n{doc}\n</document>",
"cache_control": {"type": "ephemeral"}},
{"type": "text", "text": PROMPT.format(chunk=piece)},
]}],
)
u = resp.usage
usage["written"] += u.cache_creation_input_tokens or 0
usage["read"] += u.cache_read_input_tokens or 0
usage["uncached"] += u.input_tokens
return resp.content[0].text.strip()
def embed(texts):
vecs = []
for i in range(0, len(texts), 100):
r = oai.embeddings.create(
model="text-embedding-3-small",
input=texts[i:i + 100])
vecs += [d.embedding for d in r.data]
return np.array(vecs)
def toks(s):
return re.findall(r"\w+", s.lower())
class Index:
def __init__(self, texts):
self.vecs = embed(texts)
self.bm25 = BM25Okapi([toks(t) for t in texts])
def search(self, q, k=5, pool=20):
dense = np.argsort(-(self.vecs @ embed([q])[0]))[:pool]
bm = self.bm25.get_scores(toks(q))
sparse = [i for i in np.argsort(-bm)[:pool] if bm[i] > 0]
fused = {}
for ranking in (dense, sparse):
for rank, i in enumerate(ranking, 1):
fused[i] = fused.get(i, 0) + 1 / (60 + rank)
return sorted(fused, key=fused.get, reverse=True)[:k]
def build(docs, contextual):
ids, texts = [], []
for name, doc in docs.items():
for n, piece in enumerate(chunk(doc)):
if contextual:
piece = situate(doc, piece) + "\n\n" + piece
ids.append(f"{name}#{n}")
texts.append(piece)
return ids, Index(texts)
def recall(ids, index, qa, k=5):
def hit(item):
gold = set(item["gold"])
top = {ids[i] for i in index.search(item["q"], k)}
return len(top & gold) / len(gold)
return sum(hit(x) for x in qa) / len(qa)
if __name__ == "__main__":
qa = json.load(open(sys.argv[1]))
docs = {p: open(p).read() for p in sys.argv[2:]}
for contextual in (False, True):
ids, index = build(docs, contextual)
r = recall(ids, index, qa)
print(f"contextual={contextual} recall@5={r:.1%}")
print("prompt tokens:", usage)
Run it as python ctx_eval.py qa.json docs/*.md. Here is what each part is doing.
chunk splits on fixed 500-word windows. Swap in your production chunker; both runs just have to use the same one, because the gold labels point at chunk positions.
situate sends Anthropic's contextualizer prompt, copied from the cookbook. The document goes in the first content block with cache_control, and the chunk goes in a second block that is not cached. That ordering matters: the cache covers the prefix up to the marked block, so the document has to come first. The model is Claude Haiku 5.5, whose thinking is on by default. The thinking docs say Haiku 5.5 accepts {"type": "disabled"} at high effort or below, and the models overview lists its default effort as medium. Writing one sentence of context does not need reasoning tokens billed as output.
Index.search runs both retrievers. The dense side is a plain dot product, which works as cosine similarity because OpenAI embeddings are normalized to length 1. The BM25 side lowercases and splits on word characters itself, since rank-bm25 does no preprocessing of its own. Chunks with a zero BM25 score are dropped so they don't sneak into the fusion. Anthropic's write-up says to merge the two result sets with "rank fusion techniques" without naming one, so this uses reciprocal rank fusion with the k=60 constant Elasticsearch defaults to.
build is where the experiment lives. The contextual flag changes the text that gets embedded and indexed, and nothing else. Chunk IDs stay the same in both runs, so one label file scores both.
recall averages, over all questions, the fraction of gold chunks that land in the top 5. For a question with one gold chunk, that is hit or miss.
Why the prompt cache does most of the work
Without caching, every chunk call re-sends the entire document. Ten chunks from one document means paying for that document ten times as input. With cache_control on the document block, the first call writes it to the cache and the next nine read it back.
The prompt caching docs price a 5-minute cache write at 1.25 times base input and a cache read at 0.1 times. On the pricing page as of October 2026, Claude Haiku 5.5 for prompts up to 100,000 tokens is $0.10 per million input tokens, $0.125 for 5-minute cache writes, $0.01 for cache hits and $0.50 for output. Prompts over 100,000 tokens move to a rate five times higher, so very long documents cost more per chunk.
Some arithmetic from that table, using the shape Anthropic used for its own estimate: an 8,000-token document, ten 800-token chunks, about 50 tokens of instructions and 100 tokens of generated context per chunk. With caching, that is roughly $0.001 for the one cache write, $0.0007 for nine cache reads, $0.0009 for the uncached chunk text and $0.0005 for output. About $0.003 per document, or around $0.38 per million document tokens. Without caching, the same job comes to roughly $1.17 per million. That is my arithmetic from list prices, not a measured bill. Anthropic's 2024 figure for the same shape was $1.02 per million document tokens, on Claude 3 Haiku pricing.
!A large document block sealed once into a cache box by a thick burnt-orange arrow, with many small chunk cards drawing from the box through thin dotted lines
One write, many cheap reads: the document is cached and each chunk call reuses it.
Three details from the caching docs decide whether you actually get those reads:
- Minimum length. Haiku 5.5 caches prompts of 512 tokens or more. Anything shorter runs uncached, with no error. A document under that size shows zero in both cache counters.
- Concurrency. The docs say a cache entry "only becomes available after the first response begins." The script runs chunks one after another for that reason. If you parallelize, send each document's first chunk alone, wait for it to start responding, then fan out the rest.
- Lifetime. The default cache lives 5 minutes and is refreshed for free on each hit, so sequential calls keep it warm.
The prompt tokens line at the end tells you how it went on your run. For comparison, the cookbook's own 737-chunk ingestion reported that 61.83% of all input tokens were read from cache, lower than the 70 to 80% its prose suggests. Read your counters before you trust anyone's estimate, including the one above.
Build the 30 questions before the first run
The number this script prints is only as honest as qa.json. To label anything, you first need to see the chunk IDs. A helper that reuses the same chunker:
import sys
from ctx_eval import chunk
for path in sys.argv[1:]:
for n, piece in enumerate(chunk(open(path).read())):
print(f"{path}#{n}\t{piece[:90]}")
Importing ctx_eval creates both API clients, so keep the environment variables set. Then write the label file:
[
{"q": "How long do refunds take on annual plans?",
"gold": ["docs/billing.md#3"]},
{"q": "Which header carries the idempotency key?",
"gold": ["docs/api.md#7", "docs/api.md#8"]}
]
A few rules keep the set from flattering either run:
- Write questions the way your users ask them. Search logs and support tickets are the best source if you have them.
- Don't copy wording from the chunk into the question. Shared vocabulary hands BM25 easy wins in both runs and hides the effect you are trying to see. The same goes for questions an LLM generates from a chunk, unless you rewrite them.
- Don't hand-pick questions you expect contextual retrieval to fix. Sample across your docs.
- Mark every chunk that answers the question as gold, not only the best one.
- Freeze the file before the first run. Editing labels after seeing results turns the eval into a mirror.
With 30 questions, each one is worth about 3.3 points of recall@5. A gap of one or two questions between runs is within noise. So look at the flips as well as the totals. Inside the __main__ block, after each build, this prints a hit or miss per question:
for item in qa:
top = {ids[i] for i in index.search(item["q"])}
print(bool(top & set(item["gold"])), item["q"][:60])
Questions that went from miss to hit tell you where the prefix helped. Questions that went from hit to miss deserve a closer read: when every chunk of one document gets a similar opening sentence, those chunks may start to look alike to the retriever.
Reading your number, and what to try next
If recall@5 barely moves, that is a real result. Your chunks may already be self-contained, or your baseline may already be high, and you just saved yourself an indexing step on every document update.
If it moves, Anthropic's write-up lists the next levers. They reranked by pulling the top 150 and narrowing to 20 with a Cohere reranker, which is where the 67% figure came from. They tried passing 5, 10 and 20 chunks and found 20 worked best for them, while telling readers to test it on their own use case. They also suggest a contextualizer prompt tailored to your domain, such as one that includes a glossary of terms defined elsewhere in the knowledge base. Change one lever at a time and rerun the same frozen 30 questions.
Keep the indexing cost in mind once this goes past an experiment. Contexts are generated at index time from the whole document, so when a document changes, any of its chunks' contexts can go stale. Re-contextualize the whole document, not only the edited chunk.
---
If this was useful
The script is the measuring stick; the decisions around it (how you chunk, when hybrid search beats dense alone, when a reranker earns its latency) are what the RAG Pocket Guide covers, under its subtitle of retrieval, chunking and reranking patterns for production. Run the 30 questions first, then pick the chapter that matches the misses.
3 reactions · 3 comments
Prompt Caching: Put the Stable Stuff First (and Stop Paying Twice for the Same Tokens)
Most LLM apps send the same few thousand tokens on every request: the tool definitions, the system prompt, a block of reference docs, a handful of examples. Only the last few hundred tokens, the user's actual message, change. Without caching, you pay full input price to reprocess that identical prefix every single time.
Prompt caching fixes that, but only if you build your prompt in the right order. This is the one lesson worth taking away today: stable stuff first, changing stuff last.
What actually gets cached
When a model reads your prompt, it computes intermediate key-value (KV) states for every token. Prompt caching saves those states for a reusable prefix, the unchanged tokens at the start of the prompt. If your next request starts with exactly the same prefix, the provider reuses the saved work and only processes what's new.
The key word is exactly. Both OpenAI and Anthropic match on the full rendered prefix. Change one token near the top and everything after it is a miss.
Think of it like a bookmark in a long book: the model can jump straight back to where it left off, but only if nobody rewrote the earlier pages.
Why it's worth caring about
The numbers from the providers' own docs:
- Anthropic: cache reads are billed at 0.1x the base input price on most Claude models. Writing to the default 5-minute cache costs 1.25x, and the 1-hour cache costs 2x. The 5-minute cache refreshes for free each time it's used.
- OpenAI: caching is on by default for supported models. On GPT-5.6 and later, cache writes cost 1.25x and reads 0.1x the uncached input rate, and the minimum cacheable prefix is 1,024 tokens. OpenAI's own worked example: one write plus nine full reads costs 2.15x the input price of the prefix, versus 10x without caching.
Caching also cuts time-to-first-token, because the model skips reprocessing the long prefix.
The ordering that makes it work
Build every request in this order:
1. Tools and the system prompt. These rarely change. On Anthropic, the cache hierarchy is literally tools, then system, then messages.
2. Shared reference material. Docs, a policy, few-shot examples, a long contract you're asking questions about.
3. Conversation history. Append new turns; don't rewrite old ones.
4. The new user message and anything dynamic. Timestamps, user names, per-request context.
The mistake almost everyone makes
Putting something dynamic at the top. The classic one is a system prompt that starts with Current date and time: 2026-10-12 09:04:31. That one line changes on every request, so the prefix never matches and you never get a cache hit.
Anthropic's docs describe a nastier version of the same trap: if you put cache_control on the block that contains the timestamp and the user message, you pay for a fresh cache write on every request and never get a read. Move the breakpoint to the last block that stays identical, and every later request reads from cache.
Other quiet cache-breakers from the docs:
- Changing tool names, descriptions, schemas, or their order.
- Languages that randomize JSON key order when serializing tool calls (Anthropic calls out Swift and Go).
- Toggling settings that get rendered into the prompt, such as tool_choice, images, or reasoning/thinking settings.
- Summarizing or compacting history, which rewrites the prefix.
What it looks like in code
Anthropic, with an explicit breakpoint at the end of the stable part:
response = client.messages.create(
model="<your-claude-model>",
max_tokens=1024,
system=[
{"type": "text", "text": "You are a support assistant for Acme..."},
{
"type": "text",
"text": "<full product docs and policy here>",
"cache_control": {"type": "ephemeral"}, # cache everything up to here
},
],
messages=[
{"role": "user", "content": f"(sent at {now}) {user_question}"}, # dynamic, after the breakpoint
],
)
print(response.usage) # look at cache_read_input_tokens / cache_creation_input_tokens
OpenAI needs no code change for basic caching on supported models. Keep the prefix stable and check usage.input_tokens_details.cached_tokens to confirm it's working.
Verify it, don't assume it
Caching fails silently. A prompt below the minimum length just isn't cached, with no error. So measure:
- Anthropic: cache_read_input_tokens and cache_creation_input_tokens. If both are 0, nothing was cached.
- OpenAI: usage.input_tokens_details.cached_tokens, plus the Prompt Caching dashboard.
Track a simple ratio, cached tokens divided by total input tokens, per day. If it drops after a deploy, someone put something dynamic in the prefix.
Quick checklist
- Tools and system prompt first, identical every call
- Docs and examples next
- User input, timestamps, and per-user data last
- Never reorder tools or let JSON keys shuffle
- Put the breakpoint on the last block that never changes
- Watch the cached-token fields after every prompt change
Before you spend a week shrinking your prompt to cut cost, check whether you're even getting cache hits. Reordering is usually the cheaper fix.
References
- OpenAI: Prompt caching guide
- Anthropic: Prompt caching docs
- Anthropic: Prompt caching cookbook
What's your cache-hit rate right now, and have you ever checked it?
1 comment
Mentions
A collection of notebooks/recipes showcasing some fun and effective ways of using Claude. anthropics/claude-cookbooks
Claude Cookbooks: notebooks and recipes showcasing effective ways of using Claude (54k stars) What it does All text below is quoted directly from the project README. >A collection of notebooks/recipes showcasing some fun and effective ways of using Claude. Table of recipes: - Classification: Explore techniques for text and data classification using Claude. - Retrieval Augmented Generation: Learn how to enhance Claude's responses with external knowledge. - Summarization: Discover techniques for effective text summarization with Claude. - Tool use: Learn how to integrate Claude with external tools and functions to extend its capabilities. - Sub-agents: Learn how to use Haiku as a sub-agent in combination with Opus. - Automated evaluations: Use Claude to automate the prompt evaluation process. - Prompt caching: Learn techniques for efficient prompt caching with Claude. - Cost optimization: Work through a cost optimization checklist on an agent, measuring pass rate and cost per task to find the Pareto-optimal configuration. >While the code examples are primarily written in Python, the concepts can be adapted to any programming language that supports interaction with the Claude API. Repository anthropics/claude-cookbooks Repository stats |Metric|Value| |:-|:-| |Stars|53,858| |Forks|6.5k| |Last release|None| |Languages|Jupyter Notebook 94.8%, Python 5.1%, Other 0.1%| |Contributors|87| |AI contributor accounts (Claude / Codex / Cursor / Copilot)|@claude (78 commits)| |AI co-authored commits|Claude: 71| Main contributor |Field|Value| |:-|:-| |Name|Alex Notov| |GitHub|@zealoushacker| |Commits to this repo|83 (top contributor)| |Company / Location|Zealoushacker, Inc. / Colorado| |Followers|414| ────────────────────────────── 📌 How many great GitHub repos have you discovered and completely forgotten about? Don't let another useful tool disappear into your Reddit feed. Get the community's best repos, organized by category, plus key takeaways from real developer discussions, delivered to your inbox twice a week. 📬 Never lose track of great repos again. Subscribe free.
Como Orquestrar Agentes de IA com Uma Skill DE GRAÇA #claudecode #codex #agentesdeia #ia #iagratis Orquestrar agentes de IA sem baixar nenhum app. Uma skill que roda no Claude Code, no Codex, no OpenCode e na IA que você já usa. O modelo caro planeja e os baratos fazem a leitura pesada, como na receita plan big, execute small da Anthropic. A skill lê as IAs instaladas no seu PC e a sua cota, e monta playbooks pra economizar token. Ela quebra a tarefa em partes, roda vários agentes ao mesmo tempo e passa pra outra IA quando uma trava ou o limite acaba. Nesse vídeo eu te mostro como instalar a alethe-orchestrate, configurar com o /alethe setup e orquestrar o backend de uma dashboard com SQLite e Prisma, incluindo o report com o custo de cada worker. A skill é open source e de graça. ⏱️ TIMESTAMPS 0:00 - Intro 1:18 - O problema dos subagents nativos 2:26 - Plan big, execute small (cookbook da Anthropic) 7:01 - Como a skill funciona 7:54 - Instalando a skill 9:58 - /alethe setup: suas IAs e sua cota 13:08 - /alethe editor: playbooks e perfis 17:10 - Demo: o backend da dashboard 20:30 - O Codex travou e o Sonnet assumiu 21:20 - O report: custo e tokens 22:59 - Criando um playbook ao vivo 24:38 - Resumo e encerramento 🔗 FERRAMENTAS DO VÍDEO → alethe-orchestrate (a skill): Kc1t/alethe-orchestrate-skill → Plan big, execute small (cookbook da Anthropic): github.com/anthropics/claude-cookbooks/blob/main/… → Alethe (o app de orquestração, open source): Kc1t/alethe-agents → Dashboard usada na demo: Kiranism/next-shadcn-dashboard-starter 💻 COMANDOS (na ordem do vídeo) # instalar a skill (precisa de Node) npx skills add Kc1t/alethe-orchestrate-skill # ou pelo plugin do Claude Code /plugin marketplace add Kc1t/alethe-orchestrate-skill /plugin install alethe-orchestrate@alethe # ou pelo plugin do Codex codex plugin marketplace add Kc1t/alethe-orchestrate-skill codex plugin add alethe-orchestrate@alethe # ler as IAs do seu PC, a cota e montar os perfis /alethe setup # ver os playbooks no navegador /alethe editor # orquestrar uma tarefa: é /alethe e o seu pedido /alethe coloque um banco de dados com SQLite e Prisma # acompanhar e ver o relatório com custo e tokens /alethe status /alethe report html 🔗 VÍDEOS RELACIONADOS → Pare de Usar 1 Agente de IA Por Vez: Orquestre Agentes de IA: youtu.be → Como Usar o 9Router: IA pra Programar Infinita e DE GRAÇA: youtu.be → Economize MILHÕES de Tokens com IA (37 Milhões): youtu.be ⚠️ FICA A DICA Orquestrar não deixa o token dos workers de graça. O lead gasta menos contexto, mas cada worker consome a cota da IA dele, e uma tarefa pequena sai mais cara delegada do que feita direto. No report, só o Claude informa o custo em dólar; o Codex mostra os tokens, mas não o valor. Se você usa assinatura, esse dólar é referência: o que sai de verdade é a sua cota. E no Windows o sandbox do Codex pode bloquear a escrita dos workers. Quando isso acontece a skill passa a tarefa pra outra IA, mas vale ajustar a permissão antes. 📱 ME SEGUE → Instagram: instagram.com → Portfolio: kc1t.com → GitHub: github.com/kc1t 💬 ENTRA NO DISCORD → discord.gg --- Se esse vídeo te ajudou, deixa o like e se inscreve! Comenta aí: qual combinação de IAs você colocaria no seu playbook? 🚀 #claudecode #codex #agentesdeia #ia #orquestracaodeagentes #multiagentes #subagents #claudeskills #agentskills #opencode #iagratis #vibecoding #aiagents #aicoding #anthropic #openai #tokens #programacao #inteligenciaartificial #claude [en-US] How to Orchestrate AI Agents with a FREE Skill #claudecode #codex #aiagents #ai #freeai [es-US] Cómo orquestar agentes de IA con una skill GRATIS #claudecode #codex #agentesdeia #ia #iagratis
O Claude Cookbooks reúne receitas e código prontos para construir com o Claude. → Receitas de classificação, RAG, resumo e ferramentas → Licença MIT → O porém: exige chave de API do Claude anthropics/claude-cookbooks t.co
I HAVE JUST FOUND SOMETHING INSANE IN ANTHROPICS GITHUB Anthropic's official Claude cookbook has a default vision test image, nine dogs github.com/anthropics/claude-cookbooks/blob/main/… every time a dev tests Claude vision they count the dogs but Anthropic never named them never been run wtf?? So genny $9 Dogs
Claude Cookbooks: Copy-Paste Recipes for Building AI Apps #Shorts Claude Cookbooks: Copy-Paste Recipes for Building AI Apps. Examples and notebooks for building applications with Claude. GitHub: anthropics/claude-cookbooks 💡 Why it's useful: Good implementation reference; many practical patterns. ⚠️ Keep in mind: Examples may need adaptation as APIs/models change. Which repo should we break down next? Tell us in the comments. Subscribe to @developerosindia for one AI GitHub repo explained every day. #Shorts #AICoding #Claude #GitHub #OpenSource
Contextual Retrieval in 80 Lines: Anthropic's Chunk-Prefix Trick, Measured on Your Docs - Book: RAG Pocket Guide - Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go - My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools - Me: xgabriel.com | GitHub --- Here is a chunk from a quarterly filing: "The company's revenue grew by 3% over the previous quarter." Which company? Which quarter? Grew from what? Inside the filing, the sentence made sense. Cut out, embedded and indexed on its own, it carries none of that, and a query about ACME's Q2 2023 revenue has almost nothing to match against. That example comes from Anthropic's Contextual Retrieval write-up, published September 19, 2024. The fix they proposed is small. Before you embed a chunk or index it for BM25, ask a model to write a short context for it using the whole document, and glue that context to the front of the chunk. Their chunk becomes: "This chunk is from an SEC filing on ACME corp's performance in Q2 2023; the previous quarter's revenue was $314 million. The company's revenue grew by 3% over the previous quarter." On their test sets, contextual embeddings plus contextual BM25 cut the top-20 retrieval failure rate by 49%, from 5.7% to 2.9%. Adding a reranker on top took the reduction to 67%, down to 1.9%. Those are their numbers on their data. This post gets you yours: about 80 lines of Python that index your docs both ways, contextualize every chunk with prompt caching so each document is paid for once, and print recall@5 before and after on 30 questions you label yourself. What Anthropic measured, and what it can't tell you The metric in the write-up is 1 minus recall@20: the share of relevant chunks that did not show up in the top 20 results. The test data covered codebases, fiction, ArXiv papers and science papers. The headline figures used Gemini Text 004 embeddings and Claude 3 Haiku to write the contexts, which usually came out at 50 to 100 tokens (same source). The companion cookbook notebook reports at k=5, closer to what this post measures. Its printed run shows Pass@5 going from 80.92% to 88.12% with contextual embeddings alone. The same notebook's summary table and prose don't fully agree with its printed output, which is one more reason to treat any published number as a hint about direction. None of this tells you whether your docs have the problem. Chunks that lean on their surroundings ("the endpoint above", "this setting", a table whose header lives two chunks back) have the most to gain. A folder of self-contained FAQ entries may barely move. You find out by measuring. Anthropic's 2024 write-up also suggests that if your whole knowledge base is under 200,000 tokens, you put all of it in the prompt and skip retrieval. !A single paper card torn from a larger document page, with a burnt-orange strip clipped along its top edge and a thin thread tying it back to the page it came from The whole trick: the chunk keeps a short note about where it came from. The script Install the four dependencies and set ANTHROPIC_API_KEY and OPENAI_API_KEY: pip install anthropic openai rank-bm25 numpy The script uses the Anthropic Python SDK for the contexts, the OpenAI Python SDK for embeddings, rank-bm25 for keyword search and NumPy for the vector math. Save it as ctx_eval.py: import json, re, sys import numpy as np from anthropic import Anthropic from openai import OpenAI from rank_bm25 import BM25Okapi claude, oai = Anthropic(), OpenAI() usage = {"written": 0, "read": 0, "uncached": 0} PROMPT = ( "Here is the chunk we want to situate within the whole " "document\n<chunk>\n{chunk}\n</chunk>\n" "Please give a short succinct context to situate this " "chunk within the overall document for the purposes of " "improving search retrieval of the chunk. Answer only " "with the succinct context and nothing else." ) def chunk(text, size=500): words = text.split() return [" ".join(words[i:i + size]) for i in range(0, len(words), size)] def situate(doc, piece): resp = claude.messages.create( model="claude-haiku-5-5", max_tokens=300, thinking={"type": "disabled"}, messages=[{"role": "user", "content": [ {"type": "text", "text": f"<document>\n{doc}\n</document>", "cache_control": {"type": "ephemeral"}}, {"type": "text", "text": PROMPT.format(chunk=piece)}, ]}], ) u = resp.usage usage["written"] += u.cache_creation_input_tokens or 0 usage["read"] += u.cache_read_input_tokens or 0 usage["uncached"] += u.input_tokens return resp.content[0].text.strip() def embed(texts): vecs = [] for i in range(0, len(texts), 100): r = oai.embeddings.create( model="text-embedding-3-small", input=texts[i:i + 100]) vecs += [d.embedding for d in r.data] return np.array(vecs) def toks(s): return re.findall(r"\w+", s.lower()) class Index: def __init__(self, texts): self.vecs = embed(texts) self.bm25 = BM25Okapi([toks(t) for t in texts]) def search(self, q, k=5, pool=20): dense = np.argsort(-(self.vecs @ embed([q])[0]))[:pool] bm = self.bm25.get_scores(toks(q)) sparse = [i for i in np.argsort(-bm)[:pool] if bm[i] > 0] fused = {} for ranking in (dense, sparse): for rank, i in enumerate(ranking, 1): fused[i] = fused.get(i, 0) + 1 / (60 + rank) return sorted(fused, key=fused.get, reverse=True)[:k] def build(docs, contextual): ids, texts = [], [] for name, doc in docs.items(): for n, piece in enumerate(chunk(doc)): if contextual: piece = situate(doc, piece) + "\n\n" + piece ids.append(f"{name}#{n}") texts.append(piece) return ids, Index(texts) def recall(ids, index, qa, k=5): def hit(item): gold = set(item["gold"]) top = {ids[i] for i in index.search(item["q"], k)} return len(top & gold) / len(gold) return sum(hit(x) for x in qa) / len(qa) if __name__ == "__main__": qa = json.load(open(sys.argv[1])) docs = {p: open(p).read() for p in sys.argv[2:]} for contextual in (False, True): ids, index = build(docs, contextual) r = recall(ids, index, qa) print(f"contextual={contextual} recall@5={r:.1%}") print("prompt tokens:", usage) Run it as python ctx_eval.py qa.json docs/*.md. Here is what each part is doing. chunk splits on fixed 500-word windows. Swap in your production chunker; both runs just have to use the same one, because the gold labels point at chunk positions. situate sends Anthropic's contextualizer prompt, copied from the cookbook. The document goes in the first content block with cache_control, and the chunk goes in a second block that is not cached. That ordering matters: the cache covers the prefix up to the marked block, so the document has to come first. The model is Claude Haiku 5.5, whose thinking is on by default. The thinking docs say Haiku 5.5 accepts {"type": "disabled"} at high effort or below, and the models overview lists its default effort as medium. Writing one sentence of context does not need reasoning tokens billed as output. Index.search runs both retrievers. The dense side is a plain dot product, which works as cosine similarity because OpenAI embeddings are normalized to length 1. The BM25 side lowercases and splits on word characters itself, since rank-bm25 does no preprocessing of its own. Chunks with a zero BM25 score are dropped so they don't sneak into the fusion. Anthropic's write-up says to merge the two result sets with "rank fusion techniques" without naming one, so this uses reciprocal rank fusion with the k=60 constant Elasticsearch defaults to. build is where the experiment lives. The contextual flag changes the text that gets embedded and indexed, and nothing else. Chunk IDs stay the same in both runs, so one label file scores both. recall averages, over all questions, the fraction of gold chunks that land in the top 5. For a question with one gold chunk, that is hit or miss. Why the prompt cache does most of the work Without caching, every chunk call re-sends the entire document. Ten chunks from one document means paying for that document ten times as input. With cache_control on the document block, the first call writes it to the cache and the next nine read it back. The prompt caching docs price a 5-minute cache write at 1.25 times base input and a cache read at 0.1 times. On the pricing page as of October 2026, Claude Haiku 5.5 for prompts up to 100,000 tokens is $0.10 per million input tokens, $0.125 for 5-minute cache writes, $0.01 for cache hits and $0.50 for output. Prompts over 100,000 tokens move to a rate five times higher, so very long documents cost more per chunk. Some arithmetic from that table, using the shape Anthropic used for its own estimate: an 8,000-token document, ten 800-token chunks, about 50 tokens of instructions and 100 tokens of generated context per chunk. With caching, that is roughly $0.001 for the one cache write, $0.0007 for nine cache reads, $0.0009 for the uncached chunk text and $0.0005 for output. About $0.003 per document, or around $0.38 per million document tokens. Without caching, the same job comes to roughly $1.17 per million. That is my arithmetic from list prices, not a measured bill. Anthropic's 2024 figure for the same shape was $1.02 per million document tokens, on Claude 3 Haiku pricing. !A large document block sealed once into a cache box by a thick burnt-orange arrow, with many small chunk cards drawing from the box through thin dotted lines One write, many cheap reads: the document is cached and each chunk call reuses it. Three details from the caching docs decide whether you actually get those reads: - Minimum length. Haiku 5.5 caches prompts of 512 tokens or more. Anything shorter runs uncached, with no error. A document under that size shows zero in both cache counters. - Concurrency. The docs say a cache entry "only becomes available after the first response begins." The script runs chunks one after another for that reason. If you parallelize, send each document's first chunk alone, wait for it to start responding, then fan out the rest. - Lifetime. The default cache lives 5 minutes and is refreshed for free on each hit, so sequential calls keep it warm. The prompt tokens line at the end tells you how it went on your run. For comparison, the cookbook's own 737-chunk ingestion reported that 61.83% of all input tokens were read from cache, lower than the 70 to 80% its prose suggests. Read your counters before you trust anyone's estimate, including the one above. Build the 30 questions before the first run The number this script prints is only as honest as qa.json. To label anything, you first need to see the chunk IDs. A helper that reuses the same chunker: import sys from ctx_eval import chunk for path in sys.argv[1:]: for n, piece in enumerate(chunk(open(path).read())): print(f"{path}#{n}\t{piece[:90]}") Importing ctx_eval creates both API clients, so keep the environment variables set. Then write the label file: [ {"q": "How long do refunds take on annual plans?", "gold": ["docs/billing.md#3"]}, {"q": "Which header carries the idempotency key?", "gold": ["docs/api.md#7", "docs/api.md#8"]} ] A few rules keep the set from flattering either run: - Write questions the way your users ask them. Search logs and support tickets are the best source if you have them. - Don't copy wording from the chunk into the question. Shared vocabulary hands BM25 easy wins in both runs and hides the effect you are trying to see. The same goes for questions an LLM generates from a chunk, unless you rewrite them. - Don't hand-pick questions you expect contextual retrieval to fix. Sample across your docs. - Mark every chunk that answers the question as gold, not only the best one. - Freeze the file before the first run. Editing labels after seeing results turns the eval into a mirror. With 30 questions, each one is worth about 3.3 points of recall@5. A gap of one or two questions between runs is within noise. So look at the flips as well as the totals. Inside the __main__ block, after each build, this prints a hit or miss per question: for item in qa: top = {ids[i] for i in index.search(item["q"])} print(bool(top & set(item["gold"])), item["q"][:60]) Questions that went from miss to hit tell you where the prefix helped. Questions that went from hit to miss deserve a closer read: when every chunk of one document gets a similar opening sentence, those chunks may start to look alike to the retriever. Reading your number, and what to try next If recall@5 barely moves, that is a real result. Your chunks may already be self-contained, or your baseline may already be high, and you just saved yourself an indexing step on every document update. If it moves, Anthropic's write-up lists the next levers. They reranked by pulling the top 150 and narrowing to 20 with a Cohere reranker, which is where the 67% figure came from. They tried passing 5, 10 and 20 chunks and found 20 worked best for them, while telling readers to test it on their own use case. They also suggest a contextualizer prompt tailored to your domain, such as one that includes a glossary of terms defined elsewhere in the knowledge base. Change one lever at a time and rerun the same frozen 30 questions. Keep the indexing cost in mind once this goes past an experiment. Contexts are generated at index time from the whole document, so when a document changes, any of its chunks' contexts can go stale. Re-contextualize the whole document, not only the edited chunk. --- If this was useful The script is the measuring stick; the decisions around it (how you chunk, when hybrid search beats dense alone, when a reranker earns its latency) are what the RAG Pocket Guide covers, under its subtitle of retrieval, chunking and reranking patterns for production. Run the 30 questions first, then pick the chapter that matches the misses. 
OTel is available for Claude Platform! 2 setup steps to trace SDKs calls _and_ API calls (1) Add OTel destination from the platform.claude.com (2) Setup your SDKs with OTLP (probably just an env var) Here's a cookbook: github.com/anthropics/claude-cookbooks/blob/main/… Also, it's baked into the `/claude-api` skill! so you can drive with Claude Code.
Prompt Caching: Put the Stable Stuff First (and Stop Paying Twice for the Same Tokens) Most LLM apps send the same few thousand tokens on every request: the tool definitions, the system prompt, a block of reference docs, a handful of examples. Only the last few hundred tokens, the user's actual message, change. Without caching, you pay full input price to reprocess that identical prefix every single time. Prompt caching fixes that, but only if you build your prompt in the right order. This is the one lesson worth taking away today: stable stuff first, changing stuff last. What actually gets cached When a model reads your prompt, it computes intermediate key-value (KV) states for every token. Prompt caching saves those states for a reusable prefix, the unchanged tokens at the start of the prompt. If your next request starts with exactly the same prefix, the provider reuses the saved work and only processes what's new. The key word is exactly. Both OpenAI and Anthropic match on the full rendered prefix. Change one token near the top and everything after it is a miss. Think of it like a bookmark in a long book: the model can jump straight back to where it left off, but only if nobody rewrote the earlier pages. Why it's worth caring about The numbers from the providers' own docs: - Anthropic: cache reads are billed at 0.1x the base input price on most Claude models. Writing to the default 5-minute cache costs 1.25x, and the 1-hour cache costs 2x. The 5-minute cache refreshes for free each time it's used. - OpenAI: caching is on by default for supported models. On GPT-5.6 and later, cache writes cost 1.25x and reads 0.1x the uncached input rate, and the minimum cacheable prefix is 1,024 tokens. OpenAI's own worked example: one write plus nine full reads costs 2.15x the input price of the prefix, versus 10x without caching. Caching also cuts time-to-first-token, because the model skips reprocessing the long prefix. The ordering that makes it work Build every request in this order: 1. Tools and the system prompt. These rarely change. On Anthropic, the cache hierarchy is literally tools, then system, then messages. 2. Shared reference material. Docs, a policy, few-shot examples, a long contract you're asking questions about. 3. Conversation history. Append new turns; don't rewrite old ones. 4. The new user message and anything dynamic. Timestamps, user names, per-request context. The mistake almost everyone makes Putting something dynamic at the top. The classic one is a system prompt that starts with Current date and time: 2026-10-12 09:04:31. That one line changes on every request, so the prefix never matches and you never get a cache hit. Anthropic's docs describe a nastier version of the same trap: if you put cache_control on the block that contains the timestamp and the user message, you pay for a fresh cache write on every request and never get a read. Move the breakpoint to the last block that stays identical, and every later request reads from cache. Other quiet cache-breakers from the docs: - Changing tool names, descriptions, schemas, or their order. - Languages that randomize JSON key order when serializing tool calls (Anthropic calls out Swift and Go). - Toggling settings that get rendered into the prompt, such as tool_choice, images, or reasoning/thinking settings. - Summarizing or compacting history, which rewrites the prefix. What it looks like in code Anthropic, with an explicit breakpoint at the end of the stable part: response = client.messages.create( model="<your-claude-model>", max_tokens=1024, system=[ {"type": "text", "text": "You are a support assistant for Acme..."}, { "type": "text", "text": "<full product docs and policy here>", "cache_control": {"type": "ephemeral"}, # cache everything up to here }, ], messages=[ {"role": "user", "content": f"(sent at {now}) {user_question}"}, # dynamic, after the breakpoint ], ) print(response.usage) # look at cache_read_input_tokens / cache_creation_input_tokens OpenAI needs no code change for basic caching on supported models. Keep the prefix stable and check usage.input_tokens_details.cached_tokens to confirm it's working. Verify it, don't assume it Caching fails silently. A prompt below the minimum length just isn't cached, with no error. So measure: - Anthropic: cache_read_input_tokens and cache_creation_input_tokens. If both are 0, nothing was cached. - OpenAI: usage.input_tokens_details.cached_tokens, plus the Prompt Caching dashboard. Track a simple ratio, cached tokens divided by total input tokens, per day. If it drops after a deploy, someone put something dynamic in the prefix. Quick checklist - Tools and system prompt first, identical every call - Docs and examples next - User input, timestamps, and per-user data last - Never reorder tools or let JSON keys shuffle - Put the breakpoint on the last block that never changes - Watch the cached-token fields after every prompt change Before you spend a week shrinking your prompt to cut cost, check whether you're even getting cache hits. Reordering is usually the cheaper fix. References - OpenAI: Prompt caching guide - Anthropic: Prompt caching docs - Anthropic: Prompt caching cookbook What's your cache-hit rate right now, and have you ever checked it?
Anthropic updated its GitHub for the new Claude update with two AI test instances named “Bob” and “Alice” both had tokens and isolated sessions inside Claude’s multi-tenant system to test session ownership and access control github.com/anthropics/claude-cookbooks/blob/main/… github.com/anthropics/claude-cookbooks/blob/main/… t.co