zilliztech

Why Pay for the Smartest Model Every Time? --- title: Why Pay for the Smartest Model Every Time? published: true description: 'Most questions don''t need your most expensive model. Routing, cascades, distillation and caching: the four ways to send each request to the cheapest model that can handle it, and how to know it''s still good enough.' tags: machinelearning, inference, efficiency, cost canonical_url: wrightfunctions.com ai_disclosure: some_ai cover_image: wrightfunctions.com --- Originally published on wrightfunctions.com, where the demos are live. A restaurant doesn't send every order to the head chef. Toast goes to the line cook. The head chef gets the dish that can go wrong. Plenty of AI apps do the opposite: every request, from "what's your refund policy?" to a thorny legal question, goes to the same big, expensive model. The price gap is large. As of September 2026, Anthropic's cheapest current model, Claude Haiku 4.5, costs $1 per million input tokens, and its priciest, Claude Fable 5.1, costs $10 (pricing page). That's ten times the price for questions the small one could often answer just as well. This post covers the four common ways to stop overpaying, and the one thing that keeps them honest. Routing: decide before you ask Model routing puts a quick decision in front of your models. A small, fast classifier reads each request and picks who answers: the cheap model, the expensive one, or a person. That classifier can be a decision model like Jev, which TypeSafe documents for exactly this. It works at scale. When OpenAI launched GPT-5 in August 2025, it described it as fast and deeper-thinking models behind "a real-time router that quickly decides which to use". Researchers at UC Berkeley and Anyscale built routers that learn from human preference data. Their RouteLLM project reported costs cut by more than 85% on one benchmark while keeping 95% of GPT-4's quality. AWS offers a managed router in Bedrock and claims it can cut costs by up to 30%. Cascades: try cheap, escalate when unsure A cascade skips the upfront guess. The small model answers every request and says how sure it is. Below a confidence threshold, the request goes up to the big model. The 2023 FrugalGPT paper chained models into a cascade, with a small scoring model judging each answer, and matched the best single model "with up to 98% cost reduction" on its tests. The catch is that escalated requests pay twice, and the whole thing rests on calibration: a small model that's confidently wrong never escalates. On the original post, you can drag the threshold. > ▶ Interactive demo: try it on the original post Set it too low and wrong answers slip through cheaply. Set it too high and you're paying for both models on most requests. The sweet spot is where quality stops rising, and you find it by measuring. Distillation: teach the small model your job If the small model is almost good enough, make it better at your specific task. Distillation trains a small model on a big model's answers. The name comes from Hinton and colleagues in 2015, who built on earlier work squeezing big models into small ones. A famous early result, DistilBERT, came out 40% smaller and 60% faster while keeping 97% of its teacher's language understanding. In January 2025, DeepSeek released six smaller models, from 1.5 to 70 billion weights, distilled from its R1 reasoning model. For your own app, it's the same move: log what the big model answers on your real traffic, then use fine-tuning to train a small model on those answers. Your cascade then escalates less often. Caching: the cheapest answer is one you already gave Plenty of requests repeat. A semantic cache stores answers and returns one when a new question means the same thing as an old one, matching them by embeddings rather than exact text. Open-source tools like GPTCache do this. Providers also offer prompt caching for the part of a prompt that never changes, like a long system prompt or a reference document. On Anthropic's API, a cache hit on most models costs a tenth of the normal input price. Caching stacks with everything above. Keeping it honest Every one of these trades a little quality risk for a lot of money, so the only safe way to run them is with measurement. Keep a test set of real requests with known good answers. Run it against the cheap path and the expensive path before any change to thresholds, routes or distilled models goes out. That's the eval gate from the first post in this series, applied to cost: a change ships only if quality holds. Then watch it in production, because routing failures are silent: a misrouted hard question gets a confident, wrong, cheap answer, and nothing crashes. Traffic also drifts: a new product launch fills your stream with questions the small model has never seen, and quality slides a percent at a time. Sample escalations and non-escalations, grade them, and track the numbers like any other SLO. So why pay for the smartest model every time? You shouldn't: most of your traffic is toast. Let a line cook handle it, send the hard dishes to the head chef, reuse what you've already cooked, and train the line cook on the chef's recipes. Along with quantization and decision models, it's one of the big levers on the AI bill. The one rule is that you can't cut the cost of quality you aren't measuring. References & further reading 1. Claude pricing (Anthropic · as of September 2026). Haiku 4.5 at $1 / $5 per million input / output tokens, Fable 5.1 at $10 / $50. 2. Intent Routing (TypeSafe AI · docs). A decision model classifies each request and sends it to fixed logic, a specialist LLM or a human. 3. Introducing GPT-5 (OpenAI · August 2025). A fast model, a deeper reasoning model, and "a real-time router that quickly decides which to use." 4. RouteLLM: An Open-Source Framework for Cost-Effective LLM Routing (LMSYS · July 2024). Routers trained on preference data cut costs by over 85% on MT Bench while keeping 95% of GPT-4's performance. Paper at arXiv 2406.18665. 5. Amazon Bedrock Intelligent Prompt Routing (AWS). Routes within one model family and claims up to 30% lower costs without losing accuracy. 6. FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance (Chen, Zaharia & Zou · 2023). Learned cascades, with a small DistilBERT scoring each answer, matched GPT-4 with up to 98% cost reduction, or beat it by 4% at the same cost. 7. Distilling the Knowledge in a Neural Network (Hinton, Vinyals & Dean · 2015). Names distillation and builds on Caruana's earlier work compressing an ensemble into a single model that's much easier to deploy. 8. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter (Sanh et al. · Hugging Face · 2019). 40% smaller, 60% faster, 97% of the language understanding. 9. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (DeepSeek-AI · January 2025). Six dense models from 1.5B to 70B, distilled from R1 on top of Qwen and Llama. 10. GPTCache (Zilliz · GitHub). An open-source semantic cache that matches new questions to past ones by embedding similarity. 11. Prompt caching (Anthropic · docs). Cache reads cost 10% of the base input price on most models, even less on a few. --- AI-assisted: I write these with help from AI tools.

Clientes preguntan “¿cuánto demora el envío a Valparaíso?” de 15 formas distintas. GPTCache las reconoce como la misma consulta y responde desde caché. Solo la primera paga tokens. zilliztech/GPTCache 🛒 ¿Tu negocio ya vende online? Te la armamos gratis y tú te llevas el 85% de cada venta nueva. smarteck.cl

Semantic Caching and Live Internet Search: Tools That Let AI Cache Answers to Save Money, and Pull Data From the Web When It Needs To Say you open the logs of your chat system and they are full of questions like these: - "How do I change my password?" - "Forgot my password, what do I do?" - "Where do I reset my password?" All three mean the same thing. But the system sent each one to the AI, had it write a fresh answer three times, and paid for the tokens three times. Multiply that by thousands or tens of thousands of users, and the API bill gets... sigh. At the same time, there is another kind of question the AI cannot answer from memory at all: "What's the gold price today?", "Is it going to rain today?", or "What promotions are on this month?" These answers change every day, and the data a model was trained on stops at a certain date, its knowledge cutoff. Anything new after that date, the model simply does not know. This article covers two techniques, one for each side. Semantic caching lets an AI remember answers to questions that keep coming back with the same meaning, so you stop paying for them again. Live internet search lets the AI pull fresh data from the web, or a search engine, when a question needs it. And we will look at how to decide which questions should be "remembered" and which should be "looked up again." The short version - Semantic caching keeps past answers. When a new question is close enough in meaning, it sends the old answer straight back, without calling the AI again and again. - If around 90% or more of your questions can be answered from the cache, you might save about 10x on tokens. The research I found reached roughly 60 to 70%, which works out to about 3x cheaper. - A cache has three main risks: a wrong answer because two questions looked like the same question, an outdated answer, and someone else's answer. - Live internet search lets an AI look up and answer current questions with sources. But each call costs more, because you also pay for the search API, and answers are slower because the system has to search and summarize first. Web pages it finds can also contain harmful instructions. - The two work together, as long as you have a step that sorts questions first: which ones can use the cache, which ones need a fresh search, and which ones must never share an answer with other users. Think of your AI as a call-center agent with a notebook A call-center agent who has been on the job for a while knows customers ask the same things over and over, so they write the answers down in a notebook. When someone calls and asks "I lost my card, what do I do?", they open the notebook and answer right away, without digging through the policy manual again. The idea here is pretty much the same. For example: | The customer asks | What a good agent does | In an AI system | |---|---|---| | "I lost my card. What do I do?" | Opens the notebook and answers | Semantic cache | | "What's the exchange rate today?" | Checks whether today's rate is already in the notebook. If not, looks at the live screen, then writes it down again | Live internet search + cache (per day) | | "What's my outstanding balance?" | Never answers from another customer's notes; opens this customer's account | Fetch from the real system with permissions; no cache shared with other users | | "I lost my credit card" vs. "I lost my debit card..." | Listens carefully first, because the steps may differ | Watch out for wrong matches (false hits) | What a cache is, and why it pays off with AI A cache is a place where you keep results you have already worked out, so next time you can grab them instead of doing the work again. Websites and apps have used this technique for a long time. With AI it pays off even more, because every time a model writes an answer, you pay per token and wait several seconds. Every word you send in and every word that comes back costs money, and output tokens cost more than input. If you want the background, I wrote an earlier article on tokens. Three kinds of cache in the AI world 1. Exact-match cache: the whole question is the key, so it only hits when every character matches. "How do I change my password" and "How do I change my password?" count as two different questions. 2. Prompt caching from model providers: providers such as Anthropic and OpenAI remember the beginning of a prompt you send repeatedly, such as a long system prompt or the same set of documents, and charge less for that part. On many models, the part read from cache costs about 0.1x the normal input price. The catch: it has to match character for character from the start, and the model still writes a new answer every time, so you pay full price for output. 3. Semantic cache: you keep "question and answer" pairs on your own application side and compare questions by meaning. If the meaning is close enough, you send the old answer back and skip the model call. | Type | Matches on | Saves | Limitation | |---|---|---|---| | Exact-match cache | Identical text | The whole AI call | Type it slightly differently and it misses | | Prompt caching | Identical prompt prefix | Input cost for the repeated part, and input processing time | You still pay for output and still wait for the model | | Semantic cache | Similar meaning | The whole AI call | May answer in the wrong context; needs care with stale data and data permissions | When you see "10x cheaper," part of it usually comes from that 0.1x price on prompt caching. But that discount only covers repeated input. It is not 10x off the whole bill. How semantic caching works Most setups have four steps. 1. Turn the question into numbers that represent its meaning (an embedding). This is the same method as vector search. I covered it in my earlier article on hybrid search. 2. Search the cache for the old question closest in meaning. 3. Compare it against a similarity threshold. If it is closer than the threshold, that is a cache hit, and the old answer goes straight back. 4. If it does not reach the threshold, it is a miss. The question goes to the AI as usual, and the new question and answer pair is saved for next time. The clearest win: on a hit, you pay nothing for the model and you do not wait for it, so users get their answer much faster. Turning the question into an embedding and searching the cache cost far less than having the AI answer. So how much can you actually save? To get a feel for it, let's do some simple math. If we ignore the cost of embeddings and the cache system itself, your AI cost comes down to the questions that miss. | Share of questions that hit | Still sent to the AI | Roughly cheaper by | |---|---|---| | 30% | 70% | 1.4x | | 50% | 50% | 2x | | 69% | 31% | 3.2x | | 90% | 10% | 10x | To get 10x, about 90% of requests need to hit. That means almost every question has to repeat the meaning of one you have seen before, like a bot that answers FAQs. The GPT Semantic Cache paper (Regmi and Pun, 2024) tested repetitive, customer-service-style questions. It got cache hit rates of about 61.6% to 68.8%, cut API calls by up to 68.8%, and over 97% of the answers served from cache were correct, by the authors' own measurement. Run that through the table above and it works out to roughly 3x cheaper. (More in the references at the end.) Even 3x is a very good deal. The risks of a semantic cache you need to know first 1. A wrong answer because two questions looked like the same question (false hit) "How do I cancel my credit card?" and "How do I cancel my debit card?" differ by one word. Their embeddings may be very close, but the answers may be completely different. The same goes for sentences that differ only slightly, by a number, a date, or even a different context. Redis's documentation says it plainly: semantic caching does not guarantee that a hit is always correct. The vCache paper (Schroeder et al., 2025) makes the same point from another angle: using one fixed similarity threshold for every question does not guarantee correctness either. To lower the risk, start with a strict threshold, then loosen it step by step, and spot-check the answers that hit the cache along the way. Test with real questions in the languages your users actually write in, because every embedding model understands each language a bit differently. 2. An outdated answer (stale answer) Some things have already changed, a promotion for example, but the cache keeps serving the old answer. The fix is to give each answer an expiry time (TTL) that matches how often the source data changes. Redis gives the example that real-time data like stock prices or weather should expire within 15 to 30 minutes, while FAQs that rarely change can be kept for days or weeks. And when a source document gets edited, clear the related cache entries right away. 3. Someone else's answer This one is the most serious. If a customer asks "What's my outstanding balance?" and the answer gets saved into a shared cache, the next customer who asks something similar could get the first customer's answer. That is bad. Redis gives a multi-tenant example too: Company A's policy could end up answering Company B if the cache is not separated. There is also research (Song et al., 2024) showing that by watching for responses that come back unusually fast, because they hit the cache, someone may be able to guess what other people asked. So a cache shared across users is a privacy issue as well, not only an accuracy issue. The fix is to separate the cache by user, organization, or permission level, and to not cache answers built from personal data at all unless you really need to. 4. Questions where users want a new answer Requests like "Help me come up with a caption" or "Give me name ideas for my shop" should get a fresh answer each time. Serving a repeat from the cache feels weird. GPTCache itself skips the cache more often when the temperature is set high. Same question, same context The same question asked at a different time or in a different context may need a different answer. So the cache key should be tied to its context, for example: - Model version - System prompt version - Language - The asker's permissions - Version of the documents used to answer (if you run a RAG system) If any one of these changes, the old answer may no longer be up to date. Live internet search: going out to find real data Even with a RAG system that can search your company's documents (I wrote an article explaining RAG, if you want to go back to it), some questions still need the latest data: news, prices, newly issued regulations, or software versions. That information is neither inside the model nor in your internal documents. A typical live internet search flow looks like this: 1. Decide whether this question needs a search 2. Turn the question into a search query, then call the search 3. Read the search results or the content of the relevant pages 4. Pick the passages that answer the question 5. Answer with sources There are mostly two ways to do it: 1. use the web search tool your model provider builds in, or 2. use a separate search API service and wire up the steps yourself. The limits of live search - Cost: search APIs charge per call, and results are long, so the tokens sent into the model are many times higher than for a normal question. - Speed: you wait for the search results, then load them into the model. - Source quality: results may come from unreliable sites, or the information may contradict itself. - Web pages may carry harmful instructions: the text on a web page is data we do not control. To lower the risk, you can restrict domains for topics that need accuracy, for example only searching official sites or a whitelist of trusted ones, and show sources users can check. Cap the number of searches per question. And do not give an AI that reads the web permission to do important things at the same time, because it may end up running or doing something unsafe. Sort first: cache it or search again? Put a router in front that sorts the questions first. | Question type | Example | Best approach | Answer lifetime (TTL) | |---|---|---|---| | Data that rarely changes | "What is RAG?", "How do I change my password?" | Semantic cache | Long, days or weeks, and cleared when the documents change | | Data that changes over time | "Gold price today", "Latest news on..." | Live search, then cache the search result briefly | Minutes, with the data's timestamp shown | | Personal or permission-bound data | "What's my outstanding balance?" | Fetch from the real system with permissions | No shared cache, or a per-user cache | | Design or creative work | "Help me come up with a caption" | Always let the AI answer fresh | No cache | Takeaway: cache what rarely changes, search what keeps updating, and be careful with caches shared between users Semantic caching really can lower costs and make AI answer faster. But caching has to be checked carefully and tightly. You need to weigh the benefit against the risk for each kind of question. If the AI gives a wrong answer without having any idea it is wrong, the savings may not be worth it compared with simply calling the LLM or the API again. Live internet search lets an AI look things up to answer current questions, but it costs both money and time. And the key point is that we cannot control anything about content that comes from the web, so we need a way to handle that too. Once you have a solid step that sorts questions into "cache" or "search again," your AI starts working like a great call-center agent: quick to answer, remembers what rarely changes, checks or looks up new information when it makes sense, answers accurately, does not leak data, stays safe, and saves money. If you can get all of that, that is the complete system everyone wants. References - GPTCache (Zilliz) on GitHub - Regmi and Pun: GPT Semantic Cache: Reducing LLM Costs and Latency via Semantic Embedding Caching - Schroeder et al.: vCache: Verified Semantic Prompt Caching - Song et al.: The Early Bird Catches the Leak: Unveiling Timing Side Channels in LLM Serving Systems - Redis: LangCache concepts - Redis: 10 techniques for semantic cache optimization - Anthropic: Prompt caching - OpenAI: Prompt caching - Anthropic: Web search tool - Microsoft: Bing Search API retirement ___ Explore the NEXT4I journey and read the original article at: go.next4i.com _Originally published at:_ next4i.com

How to Reduce the Cost of AI Systems with LLMs Without Sacrificing Quality --- title: "How to Reduce the Cost of AI Systems with LLMs Without Sacrificing Quality" published: true description: "Six evidence-based levers to reduce LLM costs while preserving quality, security, and governance." tags: ai, llm, architecture, governance canonical_url: doi.org --- Article DOI: 10.5281/zenodo.23732760 Technical report · Critical review and extension SGAEIA Research Series — Article 23 An evidence-based review of six levers, their limits, and a governed reference architecture Aridio Silva Independent Researcher, Brazil Creator of SGAEIA — Secure Governed Autonomous Edge Intelligence Architecture ORCID: 0009-0008-2411-6995 © Aridio Silva &#64;aridiosilva · October 2026 · Version 1.1 · Text and figures licensed CC BY 4.0 Abstract On 26 October 2026 R. Bhardwaj published an infographic claiming that a well-designed GenAI and agentic AI stack can cut monthly LLM spend by 70%, from US$10,000 to US$2,926, while keeping answer quality the same or better [[9](#en-ref-bhardwaj2026)]. This report breaks that proposal into its parts: a central thesis, six cost levers, a reference architecture, a cost decomposition and a before-and-after quality table. It then checks each part against primary sources: peer-reviewed papers, arXiv preprints and the current API documentation of the three largest model providers. The literature supports every lever, often with larger effects than the post assumes. Cascades and routers matched the strongest model's accuracy at a fraction of its cost in [[10](#en-ref-chen2023), [13](#en-ref-ding2024), [39](#en-ref-ong2024), [51](#en-ref-yue2023)]. Providers charge 50% less for batch work [[4](#en-ref-anthropicBatch), [18](#en-ref-googleBatch), [40](#en-ref-openaiBatch)] and as little as 0.1× the input price for cached prompt reads [[5](#en-ref-anthropicCache)]. Three weaknesses stand out. The 70% figure is illustrative, not measured. It leaves out the overhead the levers add (embeddings, routers, judges, evaluation). And the quality table reports differences smaller than the sampling error of a typical evaluation set [[35](#en-ref-miller2024)]. The report adds a formal cost model, break-even conditions for cascades and caches, a sensitivity analysis yielding an illustrative 61–68% net-saving scenario under stated assumptions, without presenting it as an empirical estimate, a KPI tree centred on cost per successful task [[14](#en-ref-erol2025)], a phased implementation roadmap, a glossary and an audited bibliography of 56 sources whose links were checked on 27 October 2026. Keywords: LLMOps; inference cost; model routing; LLM cascades; retrieval-augmented generation; prompt caching; semantic caching; quality gates; cost-of-pass; AI governance. Contents 1. [Introduction](#en-s1) 2. [Method](#en-s2) 3. [The central thesis](#en-s3) 4. [A cost model for LLM systems](#en-s4) 5. [The six levers](#en-s5) 6. [Reference architecture](#en-s6) 7. [Context optimisation engine](#en-s7) 8. [Caching: prompt, semantic and KV](#en-s8) 9. [Model routing and cascades](#en-s9) 10. [Quality gates and escalation](#en-s10) 11. [Cost decomposition and sensitivity](#en-s11) 12. [Evaluation and KPIs](#en-s12) 13. [Observability and governance](#en-s13) 14. [Relationship to SGAEIA](#en-s14) 15. [Implementation roadmap](#en-s15) 16. [Discussion and threats to validity](#en-s16) 17. [Conclusion](#en-s17) 18. [Glossary](#en-gloss) 19. [References (audited)](#en-refs) <a id="en-s1"></a> 1 Introduction 1.1 Context LLM inference is now a recurring operating cost, not a research expense. Each request is billed per token, and per-token prices across models differed by up to two orders of magnitude when FrugalGPT surveyed them [[10](#en-ref-chen2023)]. Agentic systems make this worse. One user request can trigger many model calls, tool calls and retries [[2](#en-ref-anthropic2024), [50](#en-ref-yao2022)]. Teams therefore face an engineering problem: deliver the quality the task needs at the lowest cost per successful outcome. 1.2 Object of analysis The object is a LinkedIn post and infographic by Rajeev Bhardwaj [[9](#en-ref-bhardwaj2026)]. Its text reads: cut LLM costs "by up to 70%, not by compromising intelligence", through smarter model routing, context, caching and agentic workflows. The infographic has six parts: - a headline result (US$10,000 to US$2,926 per month); - six "key levers"; - a reference architecture with six stages and a governance layer; - an "illustrative cost breakdown" waterfall; - a quality table comparing baseline and optimised metrics; - a takeaway: spend more intelligence only when the task actually needs it. 1.3 Contributions 1. A structured restatement of the proposal, re-drawn as 13 original figures in English and Portuguese. 2. For each component, the primary literature that supports, qualifies or contradicts it, with the concepts explained for a non-specialist technical reader. 3. A formal cost model (Eq. 1) that maps each lever to one factor of spend, plus break-even conditions for cascades (Eq. 4) and prompt caching (Eq. 3). 4. A check of the arithmetic in the post, and a sensitivity analysis of the 70% claim that includes the overhead the levers add. 5. A KPI tree, a phased roadmap with exit gates, a glossary and an audited bibliography. <a id="en-s2"></a> 2 Method The review followed four steps. 1. Source decomposition. Every claim in the infographic was transcribed and grouped by component: thesis, levers, architecture, decomposition, quality table, takeaway. The post text and publication date were read from the public LinkedIn page. 2. Literature selection. For each component we searched for (a) peer-reviewed papers or well-cited arXiv preprints that define or measure the technique, (b) the providers' own documentation for prices and limits, and (c) recognised standards for governance and reliability. Preference went to primary sources over blog summaries. 3. Audit. Every reference was resolved on 27 October 2026. arXiv records were checked through the arXiv API for title, authors and submission date. Every other URL was fetched and its HTTP status recorded. Numeric claims quoted in this report (for example "up to 98%" for FrugalGPT, or the 0.1× cache-read multiplier) were copied from the source abstract or documentation page, not from secondary summaries. 4. Synthesis and extension. The claims were rebuilt into a cost model and tested with simple analytical and sensitivity calculations. Where the post gives no data, figures are marked illustrative. Model-side efficiency techniques such as quantisation, speculative decoding and serving-system scheduling are surveyed elsewhere [[33](#en-ref-miao2023), [54](#en-ref-zhou2024)] and are outside the scope of this report, which treats the model as a priced API. Limitations of the method. This is a structured review, not a systematic review with a registered protocol. No experiment was run. Provider prices change often and are quoted as read on the audit date. <a id="en-s3"></a> 3 The central thesis 3.1 Statement The thesis is that cost and quality do not have to be traded against each other across the board. Most requests in a real workload are easy. Paying frontier-model prices for them buys capability that goes unused. A system that matches capability to difficulty can therefore sit on a better point of the cost–quality curve than one that always uses the largest model. !Figure 1. The central thesis on the cost–quality plane. The two filled points are the baseline and optimised values reported in the post; the hollow points and the frontier are illustrative. Figure 1. The central thesis on the cost–quality plane. The two filled points are the baseline and optimised values reported in the post; the hollow points and the frontier are illustrative.© 2026 Aridio Silva, Independent Researcher, ORCID 0009-0008-2411-6995. CC BY 4.0. 3.2 The Pareto view A configuration is Pareto-optimal when no other configuration is cheaper without also being worse. Kapoor et al. showed that agent benchmarks that report accuracy alone push the field toward "needlessly complex and costly" agents, and they argue that cost and accuracy should be optimised together [[25](#en-ref-kapoor2024)]. HELM had already argued for multi-metric evaluation that includes efficiency [[28](#en-ref-liang2022)]. In this view the "always frontier" baseline in the post is a dominated point (Figure 1). The routed mix reaches the same success band at 29% of the cost per successful task. 3.3 The right unit: cost per successful task The most important choice in the infographic is the last row of its quality table: cost per successful task (US$1.00 to US$0.29). Erol et al. formalise the same idea as cost-of-pass: the expected money spent to obtain one correct solution [[14](#en-ref-erol2025)]. With per-attempt cost c and success probability p: CPST = c / p This metric stops a cheap but unreliable model from looking good. A model that costs a tenth as much but succeeds half as often is only five times cheaper per success, and that is before the cost of handling its failures. Strengths of the thesis - Matches the empirical finding that routers and cascades keep quality at a fraction of the cost [[10](#en-ref-chen2023), [13](#en-ref-ding2024), [39](#en-ref-ong2024)]. - Gives a single, business-readable target metric. - Applies to agents as well as chat [[25](#en-ref-kapoor2024)]. Limits - It depends on the workload. If most requests are hard, there is little to route away. - "Same quality" must be shown statistically, not asserted (Section 12). - Quality on the long tail of rare, high-stakes queries can fall while the average looks fine. <a id="en-s4"></a> 4 A cost model for LLM systems The post lists levers but does not say which part of the bill each one acts on. We make that explicit with a monthly cost model: C = N · (1 − h) · k · [ *p*in(m) · (*T*u + δ·*T*c) + *p*out(m) · *T*out ] · (1 − β·b) + *C*ovh(1) Here N is requests per month, h the semantic-cache hit rate, k the mean number of LLM calls per request, m the model chosen with its input and output prices p, *T*u and *T*c the uncached and cached input tokens, δ the cached-read price multiplier, *T*out the output tokens, β the batch discount, b the share of traffic sent as batch, and *C*ovh the overhead the optimisation itself adds. !Figure 2. Each lever attacks a different factor of Eq. 1. The red card is the overhead that the levers themselves introduce and that the post leaves out. Figure 2. Each lever attacks a different factor of Eq. 1. The red card is the overhead that the levers themselves introduce and that the post leaves out.© 2026 Aridio Silva, Independent Researcher, ORCID 0009-0008-2411-6995. CC BY 4.0. Two properties of Eq. 1 explain the shape of the post's waterfall. First, the factors multiply, so savings compound: five reductions of 15%, 40%, 25%, 15% and 10% give 1 − 0.85·0.60·0.75·0.85·0.90 = 70.7%, not the 105% a naive sum would suggest. Second, levers on the same factor do not stack fully. Once routing has moved traffic to a cheaper model, trimming its tokens saves money at the cheaper price. The order in which savings are counted changes how much each step appears to contribute, but not the final product. Batch processing illustrates how to read the model. All three major providers discount batch work by 50% [[4](#en-ref-anthropicBatch), [18](#en-ref-googleBatch), [40](#en-ref-openaiBatch)], so β = 0.5. The post's 10% batch step therefore implies b = 20%: one request in five can wait up to 24 hours. <a id="en-s5"></a> 5 The six levers !Figure 3. The six levers, the factor of Eq. 1 each one targets, the strongest effect reported in the literature, and the main trade-off. Figure 3. The six levers, the factor of Eq. 1 each one targets, the strongest effect reported in the literature, and the main trade-off.© 2026 Aridio Silva, Independent Researcher, ORCID 0009-0008-2411-6995. CC BY 4.0. 5.1 Lever 1: intelligent model routing Concept. A router sends each request to the cheapest model expected to answer it well. A cascade is a sequential variant: try a cheap model first, check its answer, and escalate only if the check fails. Evidence. - FrugalGPT learned which combination of LLM APIs to query per request. It matched GPT-4's performance "with up to 98% cost reduction", or improved accuracy by 4% at the same cost [[10](#en-ref-chen2023)]. - Hybrid LLM routed between a small and a large model by predicted difficulty and made "up to 40% fewer calls to the large model, with no drop in response quality" [[13](#en-ref-ding2024)]. - RouteLLM trained routers on human preference data and cut cost "by over 2 times in certain cases". Its routers kept working when the strong and weak models were swapped at test time [[30](#en-ref-routellmRepo), [39](#en-ref-ong2024)]. - Answer-consistency cascades reached GPT-4-level reasoning accuracy at about 40% of its cost [[51](#en-ref-yue2023)]. - AutoMix used few-shot self-verification and a POMDP router to cut cost by over 50% at comparable performance [[1](#en-ref-aggarwal2023)]. Trade-offs. Routers need labelled or preference data and drift as models change. A wrong "easy" verdict on a hard request hurts quality most on rare, high-stakes queries. Cascades pay twice for every escalated request (Section 9). 5.2 Lever 2: context engineering Concept. Context engineering means deciding which tokens enter the model's context window and in what order. Anthropic describes it as finding "the smallest possible set of high-signal tokens" for the task [[3](#en-ref-anthropic2025)]. Evidence. - Models use information placed in the middle of a long context worse than information at the start or end ("lost in the middle") [[29](#en-ref-liu2023)]. - Performance degrades as input length grows, even on simple tasks [[11](#en-ref-chroma2025)]. - LLMLingua compressed prompts "up to 20x" with little loss [[23](#en-ref-jiang2023a), [34](#en-ref-llmlinguaRepo)]. LongLLMLingua extended this to long-context RAG [[24](#en-ref-jiang2023b)]. - Budget-aware prompting can shorten reasoning output as well [[22](#en-ref-han2024)]. Fewer tokens therefore save money and can also raise quality. Trade-offs. Cutting too hard loses the one passage that mattered (a recall loss). Compression adds a model call and latency. Aggressive budgets need evaluation per task type. 5.3 Lever 3: prompt caching Concept. When many requests share the same prompt prefix (system instructions, tool schemas, few-shot examples, a long document), the provider can reuse the already-processed attention state of that prefix instead of recomputing it. Gim et al. showed the systems basis for this: reusing attention states cut time-to-first-token by 8× on GPU and 60× on CPU without changing outputs [[17](#en-ref-gim2023)]. Current provider terms (audited 27 Oct 2026). - Anthropic: cache reads at 0.1× the base input price (lower on some newer models). Cache writes at 1.25× for a 5-minute TTL or 2× for a 1-hour TTL. These multipliers stack with the batch discount [[5](#en-ref-anthropicCache)]. - OpenAI: caching is on by default for supported models, with cached input "discounted up to 95%" and a minimum cacheable prefix of 1,024 tokens on recent models [[41](#en-ref-openaiCache)]. - Google: implicit caching on Gemini, with explicit caching available on some models [[19](#en-ref-googleCache)]. Trade-offs. The prefix must be byte-identical across requests, so templates must put stable content first and variable content last. Writes cost more than ordinary input. Gu et al. found that caches shared across users in seven API providers create a timing side channel that can leak information about other users' prompts [[20](#en-ref-gu2025)]. 5.4 Lever 4: semantic caching Concept. A semantic cache stores whole answers and returns one when a new query is close in meaning to an earlier one, measured by embedding similarity [[45](#en-ref-reimers2019)]. GPTCache is the reference open-source implementation [[7](#en-ref-bang2023), [55](#en-ref-gptcacheRepo)]. MeanCache reports that about 31% of queries in its setting were repeats, and warns that naive semantic caches have "unacceptable false hit-and-miss rates" [[16](#en-ref-gill2024)]. Trade-offs. A false hit returns a confident answer to a different question. Answers go stale when sources change. A cache shared across tenants can leak data. Section 8 gives the required controls. 5.5 Lever 5: agentic loop optimisation Concept. Agents interleave reasoning and tool use [[50](#en-ref-yao2022)]. Each step is a paid call, and redundant steps, retries and hand-offs multiply k in Eq. 1. Evidence. Kapoor et al. found that simple baselines matched state-of-the-art agents on accuracy at much lower cost [[25](#en-ref-kapoor2024)]. Anthropic's guidance is to start with the simplest pattern that works (a single call, then a fixed workflow) and add autonomy only when it pays for itself [[2](#en-ref-anthropic2024)]. Trade-offs. Hard caps on steps and retries can stop an agent just before it would have succeeded. Caps should be set per task type from traces. 5.6 Lever 6: batch processing Concept. Work that does not need an immediate answer is submitted asynchronously in bulk. Examples are enrichment, classification of backlogs, evaluation runs and report generation. Evidence. Anthropic, OpenAI and Google all price batch requests at 50% of the synchronous rate and return results within 24 hours, often much sooner [[4](#en-ref-anthropicBatch), [18](#en-ref-googleBatch), [40](#en-ref-openaiBatch)]. Trade-offs. It only applies to latency-tolerant work, and it adds queueing and result-handling code. <a id="en-s6"></a> 6 Reference architecture The post arranges the levers into a pipeline. We re-draw it with its control loops made explicit (Figure 4). !Figure 4. Reference architecture, re-drawn from the post. Three control loops are added explicitly: cache-hit bypass (amber), escalation (violet) and cache write-back after the gate (green). Figure 4. Reference architecture, re-drawn from the post. Three control loops are added explicitly: cache-hit bypass (amber), escalation (violet) and cache write-back after the gate (green).© 2026 Aridio Silva, Independent Researcher, ORCID 0009-0008-2411-6995. CC BY 4.0. 1. Request. Carries the query, tenant, identity and access rights, and task type. Task type is the cheapest routing signal and should be captured at the edge. 2. Semantic cache and policy check. Returns a stored answer only if it is authorised, fresh and semantically equivalent (Section 8). 3. Context optimisation engine. Hybrid retrieval, rerank, filter and token budget (Section 7). This is retrieval-augmented generation as introduced by Lewis et al. [[27](#en-ref-lewis2020)]. 4. Quality-aware router. Picks a small, medium or frontier model (Section 9). 5. Quality gate. Checks format, groundedness, completeness and safety, and escalates on failure (Section 10). 6. Final response. Returns the answer with citations and emits a trace with cost. A cross-cutting observability and governance layer records tokens, cost, quality and latency per stage (Section 13). Three design principles follow from the literature: - Cheap decisions first. Cache lookup and deterministic checks cost far less than a model call, so they run before it. - Close the loop. Only answers that passed the gate are written to the cache, which keeps errors from being replayed. - Prefer self-correcting retrieval. Self-RAG [[6](#en-ref-asai2023)] and Corrective RAG [[49](#en-ref-yan2024)] add retrieval-quality checks inside the generation loop. They are natural extensions of stages 3 and 5. <a id="en-s7"></a> 7 Context optimisation engine !Figure 5. The context engine narrows about 100 candidate passages to a budgeted prompt. Counts are illustrative. The layout puts the cacheable prefix first and the strongest evidence at the edges. Figure 5. The context engine narrows about 100 candidate passages to a budgeted prompt. Counts are illustrative. The layout puts the cacheable prefix first and the strongest evidence at the edges.© 2026 Aridio Silva, Independent Researcher, ORCID 0009-0008-2411-6995. CC BY 4.0. 7.1 Hybrid retrieval Lexical retrieval with BM25 scores documents by term frequency, inverse document frequency and length normalisation [[46](#en-ref-robertson2009)]. It handles exact identifiers, codes and rare terms well. Dense retrieval embeds queries and passages in a vector space [[45](#en-ref-reimers2019)] and finds paraphrases. Approximate nearest-neighbour indexes such as HNSW make it fast at scale [[31](#en-ref-malkov2016)]. Running both and merging the results covers the weaknesses of each. 7.2 Fusion with RRF Reciprocal rank fusion merges ranked lists without calibrating their scores [[12](#en-ref-cormack2009)]: RRF(d) = Σr ∈ R 1 / (k + rank*r*(d)), k = 60(2) 7.3 Reranking and filtering A cross-encoder reads the query and each candidate passage together and scores relevance much more accurately than the first-stage retriever, at higher cost per pair [[38](#en-ref-nogueira2019)]. LLMs can also act as rerankers [[48](#en-ref-sun2023)]. After reranking, filters remove near-duplicates, passages the user may not see (ACL), outdated versions and passages below a minimum score. 7.4 Token budgeting and layout The packer fills a fixed budget with the best passages. If the budget is exceeded, it compresses [[24](#en-ref-jiang2023b)] rather than truncating blindly. The layout follows two rules: - Stable content (system prompt, tool schemas, examples) goes first so that it can be cached [[5](#en-ref-anthropicCache)]. - The strongest evidence goes at the beginning and end of the dynamic part, where models use it best [[29](#en-ref-liu2023)]. Table 1. Context-engine techniques: benefit and cost | Technique | Main benefit | Main cost or risk | | --- | --- | --- | | BM25 + dense hybrid | Recall on both exact terms and paraphrases | Two indexes to maintain | | RRF fusion | No score calibration needed | Ignores score magnitude | | Cross-encoder rerank | Higher precision in the top results | Latency grows with candidate count | | Prompt compression | Up to 20× fewer tokens [[23](#en-ref-jiang2023a)] | Extra model call; may drop details | | Token budget | Predictable cost per request | Recall loss if set too low | <a id="en-s8"></a> 8 Caching: prompt, semantic and KV The post mentions "prompt caching" and "semantic caching" as separate levers, and they are often confused with each other and with the inference engine's KV cache. Figure 6 separates them. !Figure 6. Three caching layers. They reuse different things, match on different rules and carry different risks. Figure 6. Three caching layers. They reuse different things, match on different rules and carry different risks.© 2026 Aridio Silva, Independent Researcher, ORCID 0009-0008-2411-6995. CC BY 4.0. The third layer matters only to teams that host their own models. Inference engines keep the attention key-value (KV) tensors of each request in GPU memory. PagedAttention manages them in fixed-size blocks to cut fragmentation [[26](#en-ref-kwon2023)], and RadixAttention shares them across requests through a prefix tree [[53](#en-ref-zheng2024)]. Provider-side prompt caching is the priced, managed form of the same idea. 8.1 The economics of prompt caching Take a prefix of P tokens reused by n requests within the cache lifetime, at Anthropic's multipliers (write 1.25×, read 0.1×) [[5](#en-ref-anthropicCache)]. The relative cost of the prefix is: costcache / costno cache = [1.25 + 0.1·(n − 1)] / n(3) Caching pays from the second use (1.35 vs 2.0). At n = 10 the prefix costs 21.5% of the uncached price. The lever is low-risk and high-yield when prefixes are long and traffic is steady. It loses money only when a written prefix is never reused before it expires. 8.2 Semantic caching done safely !Figure 7. Semantic-cache decision flow. A hit requires similarity, freshness and a fresh authorisation check. The threshold chart is illustrative. Figure 7. Semantic-cache decision flow. A hit requires similarity, freshness and a fresh authorisation check. The threshold chart is illustrative.© 2026 Aridio Silva, Independent Researcher, ORCID 0009-0008-2411-6995. CC BY 4.0. The flow in Figure 7 adds four controls that the infographic only implies ("authorized, fresh, semantically equivalent"). 1. Scope key. Search only among entries with the same tenant, access rights, corpus version and prompt or model version. This prevents cross-tenant leakage, which OWASP lists under sensitive information disclosure and vector and embedding weaknesses [[43](#en-ref-owasp2025)]. 2. Calibrated threshold τ. Raising τ cuts false hits faster than it cuts hit rate, up to a point. τ must be tuned per domain on labelled pairs [[16](#en-ref-gill2024)]. 3. Freshness. Use a TTL plus invalidation when the source document version changes. 4. Write-back only after the gate. Store provenance with the answer so that it can be audited or purged. Benefits - A hit avoids the whole pipeline: the largest saving per event. - Lower latency on repeated questions. Risks - False hits are confident wrong answers. - Staleness and cross-tenant leakage. - Personalised or time-sensitive answers are poor candidates. <a id="en-s9"></a> 9 Model routing and cascades !Figure 8. Router and cascade, their expected costs, and the break-even escalation rate. Cost ratios are illustrative. Figure 8. Router and cascade, their expected costs, and the break-even escalation rate. Cost ratios are illustrative.© 2026 Aridio Silva, Independent Researcher, ORCID 0009-0008-2411-6995. CC BY 4.0. Let *c*s, *c*L, *c*v and *c*r be the costs of the small model, the large model, the verifier and the router. A router sends a share q of traffic to the large model. A cascade escalates a share r after verification. E[*C*router] = *c*r + (1 − q)·*c*s + q·*c*L E[*C*cascade] = *c*s + *c*v + r·*c*L(4) A cascade beats "always large" only while r < 1 − (*c*s + *c*v)/*c*L. With *c*s = 0.10 and *c*v = 0.02 (relative to *c*L), the break-even is r* = 0.88. At r = 30% the cascade costs 0.42, a 58% saving. That is close to the post's 40% routing step once the medium tier is included. Routing signals in the literature. - Predicted difficulty from a trained classifier [[13](#en-ref-ding2024)]. - Human-preference-trained routers [[39](#en-ref-ong2024)]. - Answer consistency across sampled reasoning paths [[51](#en-ref-yue2023)]. - Self-verification with a POMDP policy [[1](#en-ref-aggarwal2023)]. - Token-level uncertainty, which corrects the length bias of whole-sequence confidence [[21](#en-ref-gupta2024)]. Table 2. Router versus cascade | | Router (decide first) | Cascade (try, then escalate) | | --- | --- | --- | | Latency | One model call plus a small router | Escalated requests pay two calls in series | | Signal quality | Must predict difficulty before seeing an answer | Judges an actual answer, which is more reliable | | Data needs | Labelled or preference data to train | A verifier or consistency check | | Best for | Interactive, latency-sensitive traffic | Batch or accuracy-sensitive traffic | <a id="en-s10"></a> 10 Quality gates and escalation !Figure 9. Quality gate: four check tiers run from cheapest to most expensive, with four possible outcomes and a budget guard. Figure 9. Quality gate: four check tiers run from cheapest to most expensive, with four possible outcomes and a budget guard.© 2026 Aridio Silva, Independent Researcher, ORCID 0009-0008-2411-6995. CC BY 4.0. The gate is what makes aggressive routing safe. It should run checks in order of cost and stop at the first hard failure. 1. Deterministic checks. JSON schema, required fields, and whether every cited ID actually exists. These are cheap and catch the most frequent agent failures. 2. Classifiers. Safety, PII and prompt-injection signals (OWASP LLM01 and LLM02) [[43](#en-ref-owasp2025)]. 3. Grounding checks. Is each claim supported by the retrieved evidence? RAGAS defines reference-free metrics such as faithfulness and context relevance [[15](#en-ref-es2023), [44](#en-ref-ragasRepo)]. 4. Model-based judgement. LLM-as-a-judge reaches over 80% agreement with human preferences but shows position, verbosity and self-enhancement biases [[52](#en-ref-zheng2023)]. SelfCheckGPT detects hallucinations by checking consistency across samples [[32](#en-ref-manakul2023)]. Because these checks cost as much as a generation, run them on a sample or only on high-risk routes. Outcomes. Each answer ends in one of four states: - pass: respond, log and possibly cache; - repair: one retry with the validator's error message; - escalate: send to the next model tier, at most one hop; - abstain or hand off: refuse safely or pass to a human. A budget guard caps retries, escalations and spend per request, which addresses OWASP's "unbounded consumption" risk (LLM10) [[43](#en-ref-owasp2025)]. <a id="en-s11"></a> 11 Cost decomposition and sensitivity !Figure 10. The post's waterfall, re-drawn to scale. Step percentages and cumulative savings were computed by the author from the post's values. Figure 10. The post's waterfall, re-drawn to scale. Step percentages and cumulative savings were computed by the author from the post's values.© 2026 Aridio Silva, Independent Researcher, ORCID 0009-0008-2411-6995. CC BY 4.0. 11.1 Checking the arithmetic The post's values are internally consistent: 10,000 × 0.85 × 0.60 × 0.75 × 0.85 × 0.90 = 2,926. Routing is the largest single step (−40% of what remains; 34 of the 70.7 percentage points). The post labels the chart illustrative, so these are design targets, not measurements. 11.2 Sensitivity !Figure 11. One-at-a-time sensitivity of the final monthly cost. Ranges are the author's assumptions around the post's step factors. The five teal rows vary those factors up and down; the amber row is not a variation but the overhead the levers add, so it only increases cost. Figure 11. One-at-a-time sensitivity of the final monthly cost. Ranges are the author's assumptions around the post's step factors. The five teal rows vary those factors up and down; the amber row is not a variation but the overhead the levers add, so it only increases cost.© 2026 Aridio Silva, Independent Researcher, ORCID 0009-0008-2411-6995. CC BY 4.0. Varying each factor within a hypothetical author-selected range shows that, in this scenario, routing dominates the outcome (−64% to −78%). Batch and deduplication matter least. The post leaves out overhead: embeddings and the vector store for the semantic cache, the router, judge calls, and evaluation runs. If that overhead is assumed to add 3–9% of the original baseline, the scenario’s net saving falls to about 61–68%. This calculation demonstrates sensitivity rather than an expected range for real workloads; that range must be measured per workload. Table 3. Costs the levers add (not shown in the post) | Item | Introduced by | How to keep it small | | --- | --- | --- | | Query embeddings, vector search | L4, L2 | Small embedding model; cache embeddings | | Router inference | L1 | Lightweight classifier, not an LLM | | Verifier and judge calls | Gate | Deterministic checks first; sample model judges | | Cache writes (1.25×–2×) | L3 | Cache only prefixes that will be reused | | Evaluation runs and labelling | All | Run evaluations through the batch API (−50%) | | Engineering and operations time | All | Phase the work; quick wins first (Section 15) | <a id="en-s12"></a> 12 Evaluation and KPIs !Figure 12. KPI tree. Cost per successful task is the north-star metric; cost drivers, quality guards and operational metrics explain its movement. Figure 12. KPI tree. Cost per successful task is the north-star metric; cost drivers, quality guards and operational metrics explain its movement.© 2026 Aridio Silva, Independent Researcher, ORCID 0009-0008-2411-6995. CC BY 4.0. 12.1 Reading the quality table critically The post reports task success 92% → 92–93%, groundedness 90% → 90–91% and critical errors 0.5% → 0.4%. Miller recommends treating evaluation items as a sample from a larger population and reporting confidence intervals [[35](#en-ref-miller2024)]. With 500 items, the standard error of a 92% success rate is √(0.92 · 0.08 / 500) ≈ 1.2 points, so the 95% interval is roughly ±2.4 points. The reported changes are inside that noise. "Same quality" is plausible but not shown. A rigorous version would: - evaluate both systems on the same items (a paired design); - pre-register a non-inferiority margin, for example "success no more than 1 point lower"; - size the evaluation set so the interval is narrower than that margin. 12.2 KPI definitions Table 4. Recommended KPIs | KPI | Definition | Why it matters | | --- | --- | --- | | Cost per successful task | Total cost ÷ successful tasks [[14](#en-ref-erol2025)] | North-star; stops "cheap but wrong" from looking good | | Tokens per request (in/out) | Mean and P95, per route | Shows context bloat and verbose outputs | | Model-tier mix | Share of requests per tier | Main driver of p(m) | | Prompt-cache read share | Cached ÷ total input tokens | Verifies that Lever 3 works | | Semantic-cache hit and false-hit rates | Hits ÷ lookups; wrong hits ÷ audited hits | Saving and safety of Lever 4 | | Escalation rate r | Escalated ÷ gated requests | Cascade break-even (Eq. 4) | | Task success, groundedness, critical errors | On a fixed eval set, with 95% CIs [[35](#en-ref-miller2024)] | Quality guards | | P50/P95 latency and TTFT | End to end and per stage | User experience; latency SLOs [[8](#en-ref-beyer2016)] | | Calls per request k | LLM calls ÷ requests | Agent-loop efficiency (Lever 5) | <a id="en-s13"></a> 13 Observability and governance None of the levers can be tuned without per-request telemetry. The OpenTelemetry semantic conventions for generative AI define standard span attributes for model, token usage and operations [[42](#en-ref-otel)]. Using them makes cost per span portable across vendors. Governance has three anchors: - the NIST AI Risk Management Framework, with its Govern, Map, Measure and Manage functions [[36](#en-ref-nist2023)]; - the NIST Generative AI Profile [[37](#en-ref-nist2024)]; - the OWASP Top 10 for LLM Applications 2025, which names the main risks this architecture must control: prompt injection, sensitive information disclosure, vector and embedding weaknesses, and unbounded consumption [[43](#en-ref-owasp2025)]. Sculley et al.'s warning about hidden technical debt in ML systems applies directly [[47](#en-ref-sculley2015)]. Every cache, router and judge is another component whose configuration can drift. Each one needs an owner, a metric and an alert. <a id="en-s14"></a> 14 Relationship to SGAEIA: efficiency under governance and evidence Classification of this section: SGAEIA INTERPRETATION — a public conceptual connection, not normative architecture or implementation evidence. For SGAEIA — Secure Governed Autonomous Edge Intelligence Architecture — cost reduction is not a standalone objective. An optimisation is acceptable only when it preserves authority boundaries, policy enforcement, traceability, revocation, tenant separation, and enough evidence to explain why each request followed a particular route. Cost per successful task must therefore be paired with critical-error rates, gate false positives and false negatives, per-route latency, trace completeness, and residual risk; a cheaper answer that weakens those controls is not a net gain for a governed autonomous system [[56](#en-ref-sgaeia2026)]. The six levers reviewed here relate most directly to four SGAEIA responsibility planes. Routing and cascades belong to integration and substitution: changing the model must not silently expand available authority or invalidate security assumptions. Caches and the context engine cross identity and trust as well as authority and delegation, because a reused answer must respect the current tenant, current authorisation, provenance, and corpus version. Step, retry, and spend limits support governed execution, while per-request telemetry, route decisions, and gate outcomes support observability and evidence. This connection yields a candidate property that is not yet normative: cost optimisation in an AI system should not degrade authority boundaries, policy enforcement, revocation capability, or the completeness of evidence required for audit. The six-lever architecture in this report is external evidence and a possible engineering pattern; it does not become part of SGAEIA merely by appearing in this publication. Any adoption would require an existing-coverage audit, impact analysis, the applicable Research-to-Architecture gates, acceptance criteria, and adversarial verification before a decision by the project owner. This reading also changes the practical ordering of the levers. Instrumentation, duplicate-call removal, loop limits, and prompt caching offer gains with a smaller risk surface. Routing, model judges, and semantic caching require non-inferiority evaluation, leakage tests, false-hit analysis, model-substitution checks, and safe failure modes. The objective is not the lowest possible bill, but the lowest cost compatible with governed execution and continuous, evidence-backed assurance. <a id="en-s15"></a> 15 Implementation roadmap !Figure 13. Phased roadmap. Each phase ends at a gate whose exit criterion must hold on the evaluation set before the next phase starts. Weeks are indicative. Figure 13. Phased roadmap. Each phase ends at a gate whose exit criterion must hold on the evaluation set before the next phase starts. Weeks are indicative.© 2026 Aridio Silva, Independent Researcher, ORCID 0009-0008-2411-6995. CC BY 4.0. Table 5. Phases, actions and supporting sources | Phase | Actions | Exit criterion | Support | | --- | --- | --- | --- | | 0 · Baseline | Trace every call with tokens and cost; build an evaluation set with confidence intervals | Baseline cost per successful task known ± CI | [[14](#en-ref-erol2025), [35](#en-ref-miller2024), [42](#en-ref-otel)] | | 1 · Quick wins | Remove duplicate calls; cap loops and retries; restructure prompts for prefix caching | No quality regression; cache read share at target | [[2](#en-ref-anthropic2024), [5](#en-ref-anthropicCache), [25](#en-ref-kapoor2024), [41](#en-ref-openaiCache)] | | 2 · Context and batch | Hybrid retrieval, rerank, token budget, compression; move async jobs to batch | Tokens per request down; groundedness ≥ baseline | [[12](#en-ref-cormack2009), [24](#en-ref-jiang2023b), [29](#en-ref-liu2023), [38](#en-ref-nogueira2019), [40](#en-ref-openaiBatch)] | | 3 · Routing and gates | 2–3 tier router; gate tiers; bounded escalation; run in shadow mode before switching | Cost per successful task down; success non-inferior | [[13](#en-ref-ding2024), [15](#en-ref-es2023), [39](#en-ref-ong2024), [51](#en-ref-yue2023), [52](#en-ref-zheng2023)] | | 4 · Semantic cache | Scoped cache; τ calibration on labelled pairs; TTL and invalidation; leakage tests | False-hit rate below the agreed limit | [[7](#en-ref-bang2023), [16](#en-ref-gill2024), [43](#en-ref-owasp2025)] | The order reverses the post's visual order on purpose: the lowest-risk, highest-certainty levers come first. Semantic caching comes last because it is the only lever that can return a wrong answer with no model involved, so it needs the most mature governance. <a id="en-s16"></a> 16 Discussion and threats to validity - Workload dependence. Every number in the post depends on the request mix. A workload of hard, unique, multi-step tasks has little to route or cache. - Price volatility. Provider prices and cache multipliers change several times a year, and the gap between model tiers can shrink [[14](#en-ref-erol2025)]. The architecture stays valid; the size of the saving does not. - Latency claim. The post reports a lower P95 latency (2.8 s → 2.6 s). That is plausible if cache hits and smaller models outweigh the extra stages, but cascades add serial calls for escalated requests. It must be measured per route. - Evaluation validity. Agent benchmarks often lack proper hold-out sets [[25](#en-ref-kapoor2024)]. Routers tuned on the evaluation set will overstate gains. - Security of caches. Both cache types create cross-user information channels if shared [[20](#en-ref-gu2025), [43](#en-ref-owasp2025)]. - Scope of this report. This is a literature-grounded analysis, not an experiment. Its numeric extensions (Figures 8 and 11) are illustrative models. <a id="en-s17"></a> 17 Conclusion Bhardwaj's proposal is technically sound and matches the published evidence. Every one of its six levers has primary-source support, and several published results exceed the reductions it assumes. Its main contribution is a principle: spend intelligence where the task needs it, and measure the result as cost per successful task. Its main weakness is the headline number. It is illustrative, it omits the overhead the levers add, and its quality comparison lacks statistical support. Adding those pieces makes the modelled scenario yield a 61–68% net saving under the selected assumptions, but it does not establish an expected production range. Demonstrating a reduction near 70% requires workload telemetry, complete costs, and a paired non-inferiority test. The safest route is the one in Section 15: measure first, take the low-risk levers, and add routing and semantic caching only behind quality gates and a non-inferiority test. <a id="en-gloss"></a> Glossary Agentic loop - The cycle in which an agent reasons, calls a tool, reads the result and decides the next step. Each step is a paid model call. ANN (approximate nearest neighbour) - Search that finds vectors close to a query vector quickly, at the price of occasionally missing the exact nearest one. Batch API - An asynchronous endpoint that processes many requests within a time window (24 h) at a discount (50% at the major providers). BM25 - A lexical ranking function based on term frequency, inverse document frequency and document length. Cascade - A sequence of models from cheap to expensive. A request moves to the next model only when a check on the current answer fails. Confidence interval (CI) - A range that, under repeated sampling, contains the true value at the stated rate (for example 95%). Context engineering - Choosing which tokens enter the model's context window, and in what order, for a given step. Context window - The maximum number of tokens a model can read in one call, including instructions, evidence and history. Cost-of-pass / cost per successful task - Expected spend to obtain one correct result: cost per attempt divided by success probability. Cross-encoder - A model that reads query and passage together to score their relevance. More accurate and slower than comparing separate embeddings. Deferral rule - The rule a cascade uses to decide whether to keep an answer or pass the request to a stronger model. Dense retrieval / embedding - Representing text as vectors so that texts with similar meaning are close together, then searching by vector similarity. Escalation - Re-sending a request to a more capable model after a quality check fails. False hit - A semantic-cache hit that returns an answer to a different question. Frontier model - The most capable and usually most expensive model tier available at a given time. Groundedness / faithfulness - The degree to which every claim in an answer is supported by the evidence given to the model. Hallucination - Fluent output that is not supported by the input or by facts. HNSW - Hierarchical navigable small world graph: a widely used index for fast approximate nearest-neighbour search. Hybrid retrieval - Running lexical (BM25) and dense retrieval together and merging their results. KV cache - The attention key and value tensors an inference engine keeps in memory so that earlier tokens are not recomputed. LLM-as-a-judge - Using a language model to grade another model's output against a rubric. Lost in the middle - The finding that models use information in the middle of a long context worse than information at its start or end. Model routing - Choosing, per request, which model should answer it, before any model is called. Non-inferiority test - A test that shows a new system is not worse than the old one by more than a pre-set margin. Observability / trace / span - Recording each request as a trace made of spans (one per stage), with attributes such as tokens, cost and latency. Pareto frontier - The set of options where no option is better on one objective without being worse on another. Prompt (prefix) caching - Provider-side reuse of the processed state of a prompt prefix that repeats across requests, billed at a reduced rate. Prompt compression - Removing low-information tokens from a prompt with a smaller model while keeping its meaning. Prompt injection - Input crafted to override a model's instructions. OWASP's top risk for LLM applications. RAG (retrieval-augmented generation) - Retrieving relevant documents and adding them to the prompt so that the model answers from them. Reranking - A second, more accurate scoring pass over the top candidates from a first-stage retriever. RRF (reciprocal rank fusion) - Merging ranked lists by summing 1/(k + rank) for each document. Semantic cache - An application-level cache that returns a stored answer when a new query is similar enough in meaning to a past one. Shadow mode - Running a new component on live traffic without using its output, to compare it with the current one. SLO / error budget - A target for a service metric, and the amount of failure allowed before action is required. Token - The unit of text a model reads and writes, and the unit of billing. Roughly ¾ of an English word. Token budget - A fixed cap on the tokens a stage may send or generate. TTFT (time to first token) - Latency from sending a request to receiving the first output token. TTL (time to live) - How long a cache entry stays valid before it expires. <a id="en-refs"></a> References (audited) All links were resolved on 27 October 2026. arXiv records were checked against the arXiv API (title, authors, date). Other links were fetched and their HTTP status recorded. Where a publisher blocks automated access (HTTP 403), the DOI was confirmed to resolve to the publisher's record. Venues are given only where confirmed; otherwise the arXiv version is cited. <a id="en-ref-aggarwal2023"></a> 1. Aggarwal, P., Madaan, A., Anand, A., Potharaju, S. P., et al. (2023). AutoMix: Automatically mixing language models. arXiv preprint arXiv:2310.12963. arxiv.orgVerified: arXiv API metadata match, HTTP 200 · Accessed 2026-10-27 <a id="en-ref-anthropic2024"></a> 2. Anthropic (2024, December 19). Building effective AI agents. Anthropic Engineering. anthropic.comVerified: HTTP 200 at the current URL shown · Accessed 2026-10-27 <a id="en-ref-anthropic2025"></a> 3. Anthropic (2025, September 29). Effective context engineering for AI agents. Anthropic Engineering. anthropic.comVerified: HTTP 200 at the current URL shown · Accessed 202

05 / Claude Context 把整個codebase存進向量資料庫，用語意搜尋只把相關程式碼放進context 大型專案省掉每次載入整個資料夾的成本 需要準備向量資料庫和embedding的API金鑰 zilliztech/claude-context

DeepSearcher reasons over your private data DeepSearcher reasons over your private data ​ DeepSearcher combines large language models with vector databases to search and reason over private data. It supports models like OpenAI o3, DeepSeek, Claude, and Llama, and works with Milvus and Zilliz Cloud. Ideal for teams that need accurate answers from their own internal documents. ​ zilliztech/deep-searcher ​ #AI #Opensource #DeepSearcher #VectorDatabase ​ Follow for daily open source drops 👇 [ar] DeepSearcher يحلل بياناتك الخاصة ويستنبط منها النتائج يُحلل DeepSearcher بياناتك الخاصة يجمع DeepSearcher بين نماذج لغوية ضخمة وقواعد بيانات متجهة للبحث في البيانات الخاصة وتحليلها. يدعم نماذج مثل OpenAI o3 وDeepSeek وClaude وLlama، ويعمل مع Milvus وZilliz Cloud. مثالي للفرق التي تحتاج إلى إجابات دقيقة من وثائقها الداخلية. zilliztech/deep-searcher #الذكاء_الاصطناعي #المصادر_المفتوحة #DeepSearcher #قاعدة_بيانات_متجهة تابعنا للحصول على تحديثات يومية للمصادر المفتوحة 👇 [bn] DeepSearcher আপনার ব্যক্তিগত ডেটা নিয়ে যৌক্তিক বিশ্লেষণ করে DeepSearcher আপনার ব্যক্তিগত ডেটার উপর যুক্তি প্রয়োগ করে DeepSearcher ব্যক্তিগত ডেটা অনুসন্ধান ও তার উপর যুক্তি প্রয়োগের জন্য বৃহৎ ল্যাঙ্গুয়েজ মডেল এবং ভেক্টর ডেটাবেসকে একত্রিত করে। এটি OpenAI o3, DeepSeek, Claude, এবং Llama-এর মতো মডেল সমর্থন করে এবং Milvus ও Zilliz Cloud-এর সাথে কাজ করে। যেসব টিমের নিজস্ব অভ্যন্তরীণ নথি থেকে সঠিক উত্তরের প্রয়োজন, তাদের জন্য এটি আদর্শ। zilliztech/deep-searcher #AI #Opensource #DeepSearcher #VectorDatabase প্রতিদিন নতুন ওপেন সোর্স আপডেটের জন্য ফলো করুন 👇 [de-DE] DeepSearcher analysiert Ihre privaten Daten DeepSearcher analysiert Ihre privaten Daten DeepSearcher kombiniert große Sprachmodelle mit Vektordatenbanken, um private Daten zu durchsuchen und zu analysieren. Es unterstützt Modelle wie OpenAI o3, DeepSeek, Claude und Llama und ist mit Milvus und Zilliz Cloud kompatibel. Ideal für Teams, die präzise Antworten aus ihren internen Dokumenten benötigen. zilliztech/deep-searcher #KI #OpenSource #DeepSearcher #Vektordatenbank Folgen Sie uns für tägliche Open-Source-Veröffentlichungen 👇 [es-US] DeepSearcher analiza tus datos privados DeepSearcher analiza tus datos privados DeepSearcher combina modelos de lenguaje avanzados con bases de datos vectoriales para buscar y analizar datos privados. Es compatible con modelos como OpenAI o3, DeepSeek, Claude y Llama, y ​​funciona con Milvus y Zilliz Cloud. Ideal para equipos que necesitan respuestas precisas a partir de sus propios documentos internos. zilliztech/deep-searcher #IA #CódigoAbierto #DeepSearcher #BaseDeDatosVectoriales Síguenos para recibir actualizaciones diarias de código abierto 👇 [fr-FR] DeepSearcher analyse vos données privées DeepSearcher analyse vos données privées DeepSearcher combine de vastes modèles de langage avec des bases de données vectorielles pour rechercher et analyser des données privées. Il prend en charge des modèles tels que OpenAI o3, DeepSeek, Claude et Llama, et fonctionne avec Milvus et Zilliz Cloud. Idéal pour les équipes qui ont besoin de réponses précises à partir de leurs documents internes. zilliztech/deep-searcher #IA #OpenSource #DeepSearcher #BaseDeDonnéesVectorielle Suivez-nous pour découvrir chaque jour de nouveaux projets open source 👇 [hi] DeepSearcher आपके निजी डेटा का विश्लेषण करता है DeepSearcher आपके निजी डेटा पर तर्क करता है DeepSearcher निजी डेटा पर खोज और तर्क करने के लिए बड़े भाषा मॉडल और वेक्टर डेटाबेस को जोड़ता है। यह OpenAI o3, DeepSeek, Claude और Llama जैसे मॉडल को सपोर्ट करता है और Milvus और Zilliz Cloud के साथ काम करता है। उन टीमों के लिए आदर्श है जिन्हें अपने आंतरिक दस्तावेज़ों से सटीक उत्तर चाहिए। zilliztech/deep-searcher #AI #Opensource #DeepSearcher #VectorDatabase रोज़ाना ओपन सोर्स अपडेट के लिए फॉलो करें 👇 [id] DeepSearcher menalar data pribadi Anda DeepSearcher menganalisis data pribadi Anda DeepSearcher menggabungkan model bahasa besar dengan basis data vektor untuk mencari dan menganalisis data pribadi. Ia mendukung model seperti OpenAI o3, DeepSeek, Claude, dan Llama, serta bekerja dengan Milvus dan Zilliz Cloud. Ideal untuk tim yang membutuhkan jawaban akurat dari dokumen internal mereka sendiri. zilliztech/deep-searcher #AI #Opensource #DeepSearcher #VectorDatabase Ikuti untuk rilis open source harian 👇 [it] DeepSearcher analizza i tuoi dati privati DeepSearcher analizza i tuoi dati privati DeepSearcher combina modelli linguistici di grandi dimensioni con database vettoriali per ricercare e analizzare dati privati. Supporta modelli come OpenAI o3, DeepSeek, Claude e Llama e funziona con Milvus e Zilliz Cloud. Ideale per i team che necessitano di risposte accurate dai propri documenti interni. zilliztech/deep-searcher #AI #Opensource #DeepSearcher #VectorDatabase Seguici per aggiornamenti quotidiani sui progetti open source 👇 [iw] DeepSearcher מסיק מסקנות על הנתונים הפרטיים שלכם DeepSearcher מתחשב בנתונים הפרטיים שלך DeepSearcher משלב מודלים של שפה גדולה עם מסדי נתונים וקטוריים כדי לחפש ולהסביר נתונים פרטיים. הוא תומך במודלים כמו OpenAI o3, DeepSeek, Claude ו-Llama, ועובד עם Milvus ו-Zilliz Cloud. אידיאלי לצוותים הזקוקים לתשובות מדויקות מהמסמכים הפנימיים שלהם. zilliztech/deep-searcher #AI #OpenSource #DeepSearcher #VectorDatabase עקבו אחר פרסומים יומיים של קוד פתוח 👇 [ja] DeepSearcher：プライベートデータを活用した推論・検索ツール DeepSearcherは、プライベートデータから推論を行います。 DeepSearcherは、大規模な言語モデルとベクトルデータベースを組み合わせ、プライベートデータから検索と推論を行います。OpenAI o3、DeepSeek、Claude、Llamaなどのモデルをサポートし、MilvusおよびZilliz Cloudと連携します。社内文書から正確な回答を必要とするチームに最適です。 zilliztech/deep-searcher #AI #Opensource #DeepSearcher #VectorDatabase 毎日更新されるオープンソース情報はこちら👇 [ko] DeepSearcher가 당신의 개인 데이터를 분석하고 추론합니다 DeepSearcher는 개인 데이터에 대한 추론 기능을 제공합니다. DeepSearcher는 대규모 언어 모델과 벡터 데이터베이스를 결합하여 개인 데이터를 검색하고 추론합니다. OpenAI o3, DeepSeek, Claude, Llama 등의 모델을 지원하며 Milvus 및 Zilliz 클라우드와 연동됩니다. 내부 문서에서 정확한 답변을 찾아야 하는 팀에 이상적입니다. zilliztech/deep-searcher #AI #오픈소스 #DeepSearcher #벡터데이터베이스 매일 새로운 오픈소스를 받아보시려면 팔로우하세요 👇 [ml] DeepSearcher നിങ്ങളുടെ സ്വകാര്യ ഡാറ്റയെ വിശകലനം ചെയ്യുന്നു നിങ്ങളുടെ സ്വകാര്യ ഡാറ്റയെക്കുറിച്ചുള്ള DeepSearcher കാരണങ്ങൾ ​ സ്വകാര്യ ഡാറ്റയെക്കുറിച്ചുള്ള തിരയലിനും യുക്തിക്കും വേണ്ടി DeepSearcher വലിയ ഭാഷാ മോഡലുകളെ വെക്റ്റർ ഡാറ്റാബേസുകളുമായി സംയോജിപ്പിക്കുന്നു. ഇത് OpenAI o3, DeepSeek, Claude, Llama തുടങ്ങിയ മോഡലുകളെ പിന്തുണയ്ക്കുന്നു, കൂടാതെ Milvus, Zilliz Cloud എന്നിവയുമായി പ്രവർത്തിക്കുന്നു. സ്വന്തം ആന്തരിക പ്രമാണങ്ങളിൽ നിന്ന് കൃത്യമായ ഉത്തരങ്ങൾ ആവശ്യമുള്ള ടീമുകൾക്ക് അനുയോജ്യം. ​ zilliztech/deep-searcher ​ #AI #Opensource #DeepSearcher #VectorDatabase ​ ദിവസേനയുള്ള ഓപ്പൺ സോഴ്‌സ് ഡ്രോപ്പുകൾക്കായി പിന്തുടരുക 👇 [nl-NL] DeepSearcher redeneert op basis van jouw privégegevens DeepSearcher redeneert over uw privégegevens DeepSearcher combineert grote taalmodellen met vectordatabases om te zoeken en te redeneren over privégegevens. Het ondersteunt modellen zoals OpenAI o3, DeepSeek, Claude en Llama, en werkt met Milvus en Zilliz Cloud. Ideaal voor teams die nauwkeurige antwoorden nodig hebben op basis van hun eigen interne documenten. zilliztech/deep-searcher #AI #Opensource #DeepSearcher #VectorDatabase Volg voor dagelijkse open source-updates 👇 [pa] DeepSearcher ਤੁਹਾਡੇ ਨਿੱਜੀ ਡੇਟਾ ਦਾ ਵਿਸ਼ਲੇਸ਼ਣ ਕਰਦਾ ਹੈ DeepSearcher ਤੁਹਾਡੇ ਨਿੱਜੀ ਡੇਟਾ 'ਤੇ ਤਰਕ ਦਿੰਦਾ ਹੈ DeepSearcher ਨਿੱਜੀ ਡੇਟਾ 'ਤੇ ਖੋਜ ਅਤੇ ਤਰਕ ਕਰਨ ਲਈ ਵੈਕਟਰ ਡੇਟਾਬੇਸ ਦੇ ਨਾਲ ਵੱਡੇ ਭਾਸ਼ਾ ਮਾਡਲਾਂ ਨੂੰ ਜੋੜਦਾ ਹੈ। ਇਹ OpenAI o3, DeepSeek, Claude, ਅਤੇ Llama ਵਰਗੇ ਮਾਡਲਾਂ ਦਾ ਸਮਰਥਨ ਕਰਦਾ ਹੈ, ਅਤੇ Milvus ਅਤੇ Zilliz Cloud ਨਾਲ ਕੰਮ ਕਰਦਾ ਹੈ। ਉਹਨਾਂ ਟੀਮਾਂ ਲਈ ਆਦਰਸ਼ ਜਿਨ੍ਹਾਂ ਨੂੰ ਆਪਣੇ ਅੰਦਰੂਨੀ ਦਸਤਾਵੇਜ਼ਾਂ ਤੋਂ ਸਹੀ ਜਵਾਬਾਂ ਦੀ ਲੋੜ ਹੁੰਦੀ ਹੈ। zilliztech/deep-searcher ​ #AI #Opensource #DeepSearcher #VectorDatabase ​ ਰੋਜ਼ਾਨਾ ਓਪਨ ਸੋਰਸ ਡ੍ਰੌਪਸ ਲਈ ਫਾਲੋ ਕਰੋ 👇 [pl] DeepSearcher analizuje Twoje prywatne dane DeepSearcher analizuje Twoje prywatne dane DeepSearcher łączy duże modele językowe z wektorowymi bazami danych, aby wyszukiwać i analizować prywatne dane. Obsługuje modele takie jak OpenAI o3, DeepSeek, Claude i Llama oraz współpracuje z Milvus i Zilliz Cloud. Idealny dla zespołów, które potrzebują dokładnych odpowiedzi z własnych dokumentów wewnętrznych. zilliztech/deep-searcher #AI #Opensource #DeepSearcher #VectorDatabase Obserwuj, aby otrzymywać codzienne informacje o open source 👇 [pt-BR] DeepSearcher analisa seus dados privados com raciocínio DeepSearcher analisa seus dados privados ​ O DeepSearcher combina grandes modelos de linguagem com bancos de dados vetoriais para pesquisar e analisar dados privados. Ele suporta modelos como OpenAI o3, DeepSeek, Claude e Llama, e funciona com Milvus e Zilliz Cloud. Ideal para equipes que precisam de respostas precisas a partir de seus próprios documentos internos. ​ zilliztech/deep-searcher ​ #IA #CódigoAberto #DeepSearcher #BancoDeDadosVetoriais ​ Siga para receber dicas diárias de código aberto 👇 [ru] DeepSearcher анализирует ваши приватные данные DeepSearcher анализирует ваши личные данные DeepSearcher объединяет большие языковые модели с векторными базами данных для поиска и анализа личных данных. Он поддерживает такие модели, как OpenAI o3, DeepSeek, Claude и Llama, и работает с Milvus и Zilliz Cloud. Идеально подходит для команд, которым нужны точные ответы на основе собственных внутренних документов. zilliztech/deep-searcher #AI #Opensource #DeepSearcher #VectorDatabase Подписывайтесь на ежедневные обновления открытого исходного кода 👇 [ta] DeepSearcher உங்கள் தனிப்பட்ட தரவுகளை ஆய்வு செய்து பதிலளிக்கிறது டீப்சர்ச்சர் உங்கள் தனிப்பட்ட தரவுகளைப் பகுத்தறிகிறது டீப்சர்ச்சர், பெரிய மொழி மாதிரிகளை வெக்டர் தரவுத்தளங்களுடன் இணைத்து, தனிப்பட்ட தரவுகளைத் தேடிப் பகுத்தறிகிறது. இது OpenAI o3, DeepSeek, Claude, மற்றும் Llama போன்ற மாதிரிகளை ஆதரிக்கிறது, மேலும் Milvus மற்றும் Zilliz Cloud உடன் இணைந்து செயல்படுகிறது. தங்களின் சொந்த உள் ஆவணங்களிலிருந்து துல்லியமான பதில்கள் தேவைப்படும் குழுக்களுக்கு இது மிகவும் ஏற்றது. zilliztech/deep-searcher #AI #Opensource #DeepSearcher #VectorDatabase தினசரி ஓப்பன் சோர்ஸ் வெளியீடுகளுக்குப் பின்தொடரவும் 👇 [te] DeepSearcher మీ వ్యక్తిగత డేటాను విశ్లేషించి సమాధానాలను ఇస్తుంది డీప్‌సెర్చర్ మీ ప్రైవేట్ డేటాపై తర్కిస్తుంది డీప్‌సెర్చర్, ప్రైవేట్ డేటాను శోధించడానికి మరియు దానిపై తర్కించడానికి పెద్ద భాషా నమూనాలను వెక్టర్ డేటాబేస్‌లతో మిళితం చేస్తుంది. ఇది OpenAI o3, DeepSeek, Claude, మరియు Llama వంటి నమూనాలకు మద్దతు ఇస్తుంది మరియు Milvus మరియు Zilliz Cloudతో పనిచేస్తుంది. తమ సొంత అంతర్గత పత్రాల నుండి కచ్చితమైన సమాధానాలు అవసరమయ్యే బృందాలకు ఇది చాలా అనువైనది. zilliztech/deep-searcher #AI #Opensource #DeepSearcher #VectorDatabase రోజువారీ ఓపెన్ సోర్స్ అప్‌డేట్‌ల కోసం ఫాలో అవ్వండి 👇 [uk] DeepSearcher аналізує ваші конфіденційні дані DeepSearcher обґрунтовує ваші особисті дані DeepSearcher поєднує великі мовні моделі з векторними базами даних для пошуку та обґрунтування особистих даних. Він підтримує такі моделі, як OpenAI o3, DeepSeek, Claude та Llama, а також працює з Milvus та Zilliz Cloud. Ідеально підходить для команд, яким потрібні точні відповіді з власних внутрішніх документів. zilliztech/deep-searcher #AI #Opensource #DeepSearcher #VectorDatabase Підпишіться на щоденні публікації про відкритий вихідний код 👇

Zilliz CLI v1.5.0 adds access-control workflows—check the cloud and database boundary Access-control automation gets confusing when a cloud project role and a database role are treated as the same thing. For Milvus users operating on Zilliz Cloud, that distinction is worth checking before handing an admin workflow to a script or coding agent. We're sharing the CLI v1.5.0 access-control update with the community. It adds an ACL command tree for roles, members, groups, and policy grants, plus a generated skill that describes the same workflow. Policies can be supplied inline, loaded from a JSON file, or built interactively. Each path validates against an embedded action catalog before sending the request. That gives developers an earlier check on the policy they are asking the service to accept; it should not be read as a promise that a request will pass backend authorization. There are important scope and availability details in the release notes: - Project roles belong to one project. Non-interactive automation needs the relevant project context instead of relying on a terminal prompt. - Organization roles are read-only in this command set because the backend does not define create, modify, or delete actions for them. - The ACL commands remain hidden from help and shell completion until the corresponding /v2 routes are published on the API gateway. The generated zilliz:acl skill uses the same action catalog as local validation. It also distinguishes cloud organization/project access control from database RBAC. An agent choosing a command still needs to know which layer it is supposed to administer. Before adapting an existing workflow, identify the target resource, the role's scope, and the account that will execute it. Then check the release's availability notes against your environment. The release notes include supported platforms and installation options: github.com/zilliztech/zilliz-cli/releases/tag/zil…

Zilliz CLI v1.5.0 adds access-control workflows—check the cloud and database boundary Access-control automation gets confusing when a cloud project role and a database role are treated as the same thing. For Milvus users operating on Zilliz Cloud, that distinction is worth checking before handing an admin workflow to a script or coding agent. We're sharing the CLI v1.5.0 access-control update with the community. It adds an ACL command tree for roles, members, groups, and policy grants, plus a generated skill that describes the same workflow. Policies can be supplied inline, loaded from a JSON file, or built interactively. Each path validates against an embedded action catalog before sending the request. That gives developers an earlier check on the policy they are asking the service to accept; it should not be read as a promise that a request will pass backend authorization. There are important scope and availability details in the release notes: - Project roles belong to one project. Non-interactive automation needs the relevant project context instead of relying on a terminal prompt. - Organization roles are read-only in this command set because the backend does not define create, modify, or delete actions for them. - The ACL commands remain hidden from help and shell completion until the corresponding /v2 routes are published on the API gateway. The generated zilliz:acl skill uses the same action catalog as local validation. It also distinguishes cloud organization/project access control from database RBAC. An agent choosing a command still needs to know which layer it is supposed to administer. Before adapting an existing workflow, identify the target resource, the role's scope, and the account that will execute it. Then check the release's availability notes against your environment. The release notes include supported platforms and installation options: github.com/zilliztech/zilliz-cli/releases/tag/zil…

Zilliz CLI v1.5.0 adds access-control workflows—check the cloud and database boundary Access-control automation gets confusing when a cloud project role and a database role are treated as the same thing. For Milvus users operating on Zilliz Cloud, that distinction is worth checking before handing an admin workflow to a script or coding agent. We're sharing the CLI v1.5.0 access-control update with the community. It adds an ACL command tree for roles, members, groups, and policy grants, plus a generated skill that describes the same workflow. Policies can be supplied inline, loaded from a JSON file, or built interactively. Each path validates against an embedded action catalog before sending the request. That gives developers an earlier check on the policy they are asking the service to accept; it should not be read as a promise that a request will pass backend authorization. There are important scope and availability details in the release notes: - Project roles belong to one project. Non-interactive automation needs the relevant project context instead of relying on a terminal prompt. - Organization roles are read-only in this command set because the backend does not define create, modify, or delete actions for them. - The ACL commands remain hidden from help and shell completion until the corresponding /v2 routes are published on the API gateway. The generated zilliz:acl skill uses the same action catalog as local validation. It also distinguishes cloud organization/project access control from database RBAC. An agent choosing a command still needs to know which layer it is supposed to administer. Before adapting an existing workflow, identify the target resource, the role's scope, and the account that will execute it. Then check the release's availability notes against your environment. The release notes include supported platforms and installation options: github.com/zilliztech/zilliz-cli/releases/tag/zil…

Claude Context: Code search MCP for Claude Code. Make entire codebase the context for any coding agent (13k stars) What it does All text below is quoted directly from the project README. >Claude Context is an MCP plugin that adds semantic code search to Claude Code and other AI coding agents, giving them deep context from your entire codebase. Your Entire Codebase as Context: : Claude Context uses semantic search to find all relevant code from millions of lines. No multi-round discovery needed. It brings results straight into the Claude's context. Cost-Effective for Large Codebases: Instead of loading entire directories into Claude for every request, which can be very expensive, Claude Context efficiently stores your codebase in a vector database and only uses related code in context to keep your costs manageable. Available Tools 1. index_codebase Index a codebase directory for hybrid search (BM25 + dense vector). 2. search_code Search the indexed codebase using natural language queries with hybrid search (BM25 + dense vector). 3. clear_index Clear the search index for a specific codebase. 4. get_indexing_status Get the current indexing status of a codebase. Shows progress percentage for actively indexing codebases and completion status for indexed codebases. Implementation Details - Hybrid Code Search: Ask questions like find functions that handle user authentication and get relevant, context-rich code instantly using advanced hybrid search (BM25 + dense vector). - Context-Aware: Discover large codebase, understand how different parts of your codebase relate, even across millions of lines of code. - Incremental Indexing: Efficiently re-index only changed files using Merkle trees. - Intelligent Code Chunking: Analyze code in Abstract Syntax Trees (AST) for chunking. - Scalable: Integrates with Zilliz Cloud for scalable vector search, no matter how large your codebase is. - Customizable: Configure file extensions, ignore patterns, and embedding models. Repository zilliztech/claude-context