lm-sys

How to Reduce the Cost of AI Systems with LLMs Without Sacrificing Quality --- title: "How to Reduce the Cost of AI Systems with LLMs Without Sacrificing Quality" published: true description: "Six evidence-based levers to reduce LLM costs while preserving quality, security, and governance." tags: ai, llm, architecture, governance canonical_url: doi.org --- Article DOI: 10.5281/zenodo.23732760 Technical report · Critical review and extension SGAEIA Research Series — Article 23 An evidence-based review of six levers, their limits, and a governed reference architecture Aridio Silva Independent Researcher, Brazil Creator of SGAEIA — Secure Governed Autonomous Edge Intelligence Architecture ORCID: 0009-0008-2411-6995 © Aridio Silva &#64;aridiosilva · October 2026 · Version 1.1 · Text and figures licensed CC BY 4.0 Abstract On 26 October 2026 R. Bhardwaj published an infographic claiming that a well-designed GenAI and agentic AI stack can cut monthly LLM spend by 70%, from US$10,000 to US$2,926, while keeping answer quality the same or better [[9](#en-ref-bhardwaj2026)]. This report breaks that proposal into its parts: a central thesis, six cost levers, a reference architecture, a cost decomposition and a before-and-after quality table. It then checks each part against primary sources: peer-reviewed papers, arXiv preprints and the current API documentation of the three largest model providers. The literature supports every lever, often with larger effects than the post assumes. Cascades and routers matched the strongest model's accuracy at a fraction of its cost in [[10](#en-ref-chen2023), [13](#en-ref-ding2024), [39](#en-ref-ong2024), [51](#en-ref-yue2023)]. Providers charge 50% less for batch work [[4](#en-ref-anthropicBatch), [18](#en-ref-googleBatch), [40](#en-ref-openaiBatch)] and as little as 0.1× the input price for cached prompt reads [[5](#en-ref-anthropicCache)]. Three weaknesses stand out. The 70% figure is illustrative, not measured. It leaves out the overhead the levers add (embeddings, routers, judges, evaluation). And the quality table reports differences smaller than the sampling error of a typical evaluation set [[35](#en-ref-miller2024)]. The report adds a formal cost model, break-even conditions for cascades and caches, a sensitivity analysis yielding an illustrative 61–68% net-saving scenario under stated assumptions, without presenting it as an empirical estimate, a KPI tree centred on cost per successful task [[14](#en-ref-erol2025)], a phased implementation roadmap, a glossary and an audited bibliography of 56 sources whose links were checked on 27 October 2026. Keywords: LLMOps; inference cost; model routing; LLM cascades; retrieval-augmented generation; prompt caching; semantic caching; quality gates; cost-of-pass; AI governance. Contents 1. [Introduction](#en-s1) 2. [Method](#en-s2) 3. [The central thesis](#en-s3) 4. [A cost model for LLM systems](#en-s4) 5. [The six levers](#en-s5) 6. [Reference architecture](#en-s6) 7. [Context optimisation engine](#en-s7) 8. [Caching: prompt, semantic and KV](#en-s8) 9. [Model routing and cascades](#en-s9) 10. [Quality gates and escalation](#en-s10) 11. [Cost decomposition and sensitivity](#en-s11) 12. [Evaluation and KPIs](#en-s12) 13. [Observability and governance](#en-s13) 14. [Relationship to SGAEIA](#en-s14) 15. [Implementation roadmap](#en-s15) 16. [Discussion and threats to validity](#en-s16) 17. [Conclusion](#en-s17) 18. [Glossary](#en-gloss) 19. [References (audited)](#en-refs) <a id="en-s1"></a> 1 Introduction 1.1 Context LLM inference is now a recurring operating cost, not a research expense. Each request is billed per token, and per-token prices across models differed by up to two orders of magnitude when FrugalGPT surveyed them [[10](#en-ref-chen2023)]. Agentic systems make this worse. One user request can trigger many model calls, tool calls and retries [[2](#en-ref-anthropic2024), [50](#en-ref-yao2022)]. Teams therefore face an engineering problem: deliver the quality the task needs at the lowest cost per successful outcome. 1.2 Object of analysis The object is a LinkedIn post and infographic by Rajeev Bhardwaj [[9](#en-ref-bhardwaj2026)]. Its text reads: cut LLM costs "by up to 70%, not by compromising intelligence", through smarter model routing, context, caching and agentic workflows. The infographic has six parts: - a headline result (US$10,000 to US$2,926 per month); - six "key levers"; - a reference architecture with six stages and a governance layer; - an "illustrative cost breakdown" waterfall; - a quality table comparing baseline and optimised metrics; - a takeaway: spend more intelligence only when the task actually needs it. 1.3 Contributions 1. A structured restatement of the proposal, re-drawn as 13 original figures in English and Portuguese. 2. For each component, the primary literature that supports, qualifies or contradicts it, with the concepts explained for a non-specialist technical reader. 3. A formal cost model (Eq. 1) that maps each lever to one factor of spend, plus break-even conditions for cascades (Eq. 4) and prompt caching (Eq. 3). 4. A check of the arithmetic in the post, and a sensitivity analysis of the 70% claim that includes the overhead the levers add. 5. A KPI tree, a phased roadmap with exit gates, a glossary and an audited bibliography. <a id="en-s2"></a> 2 Method The review followed four steps. 1. Source decomposition. Every claim in the infographic was transcribed and grouped by component: thesis, levers, architecture, decomposition, quality table, takeaway. The post text and publication date were read from the public LinkedIn page. 2. Literature selection. For each component we searched for (a) peer-reviewed papers or well-cited arXiv preprints that define or measure the technique, (b) the providers' own documentation for prices and limits, and (c) recognised standards for governance and reliability. Preference went to primary sources over blog summaries. 3. Audit. Every reference was resolved on 27 October 2026. arXiv records were checked through the arXiv API for title, authors and submission date. Every other URL was fetched and its HTTP status recorded. Numeric claims quoted in this report (for example "up to 98%" for FrugalGPT, or the 0.1× cache-read multiplier) were copied from the source abstract or documentation page, not from secondary summaries. 4. Synthesis and extension. The claims were rebuilt into a cost model and tested with simple analytical and sensitivity calculations. Where the post gives no data, figures are marked illustrative. Model-side efficiency techniques such as quantisation, speculative decoding and serving-system scheduling are surveyed elsewhere [[33](#en-ref-miao2023), [54](#en-ref-zhou2024)] and are outside the scope of this report, which treats the model as a priced API. Limitations of the method. This is a structured review, not a systematic review with a registered protocol. No experiment was run. Provider prices change often and are quoted as read on the audit date. <a id="en-s3"></a> 3 The central thesis 3.1 Statement The thesis is that cost and quality do not have to be traded against each other across the board. Most requests in a real workload are easy. Paying frontier-model prices for them buys capability that goes unused. A system that matches capability to difficulty can therefore sit on a better point of the cost–quality curve than one that always uses the largest model. !Figure 1. The central thesis on the cost–quality plane. The two filled points are the baseline and optimised values reported in the post; the hollow points and the frontier are illustrative. Figure 1. The central thesis on the cost–quality plane. The two filled points are the baseline and optimised values reported in the post; the hollow points and the frontier are illustrative.© 2026 Aridio Silva, Independent Researcher, ORCID 0009-0008-2411-6995. CC BY 4.0. 3.2 The Pareto view A configuration is Pareto-optimal when no other configuration is cheaper without also being worse. Kapoor et al. showed that agent benchmarks that report accuracy alone push the field toward "needlessly complex and costly" agents, and they argue that cost and accuracy should be optimised together [[25](#en-ref-kapoor2024)]. HELM had already argued for multi-metric evaluation that includes efficiency [[28](#en-ref-liang2022)]. In this view the "always frontier" baseline in the post is a dominated point (Figure 1). The routed mix reaches the same success band at 29% of the cost per successful task. 3.3 The right unit: cost per successful task The most important choice in the infographic is the last row of its quality table: cost per successful task (US$1.00 to US$0.29). Erol et al. formalise the same idea as cost-of-pass: the expected money spent to obtain one correct solution [[14](#en-ref-erol2025)]. With per-attempt cost c and success probability p: CPST = c / p This metric stops a cheap but unreliable model from looking good. A model that costs a tenth as much but succeeds half as often is only five times cheaper per success, and that is before the cost of handling its failures. Strengths of the thesis - Matches the empirical finding that routers and cascades keep quality at a fraction of the cost [[10](#en-ref-chen2023), [13](#en-ref-ding2024), [39](#en-ref-ong2024)]. - Gives a single, business-readable target metric. - Applies to agents as well as chat [[25](#en-ref-kapoor2024)]. Limits - It depends on the workload. If most requests are hard, there is little to route away. - "Same quality" must be shown statistically, not asserted (Section 12). - Quality on the long tail of rare, high-stakes queries can fall while the average looks fine. <a id="en-s4"></a> 4 A cost model for LLM systems The post lists levers but does not say which part of the bill each one acts on. We make that explicit with a monthly cost model: C = N · (1 − h) · k · [ *p*in(m) · (*T*u + δ·*T*c) + *p*out(m) · *T*out ] · (1 − β·b) + *C*ovh(1) Here N is requests per month, h the semantic-cache hit rate, k the mean number of LLM calls per request, m the model chosen with its input and output prices p, *T*u and *T*c the uncached and cached input tokens, δ the cached-read price multiplier, *T*out the output tokens, β the batch discount, b the share of traffic sent as batch, and *C*ovh the overhead the optimisation itself adds. !Figure 2. Each lever attacks a different factor of Eq. 1. The red card is the overhead that the levers themselves introduce and that the post leaves out. Figure 2. Each lever attacks a different factor of Eq. 1. The red card is the overhead that the levers themselves introduce and that the post leaves out.© 2026 Aridio Silva, Independent Researcher, ORCID 0009-0008-2411-6995. CC BY 4.0. Two properties of Eq. 1 explain the shape of the post's waterfall. First, the factors multiply, so savings compound: five reductions of 15%, 40%, 25%, 15% and 10% give 1 − 0.85·0.60·0.75·0.85·0.90 = 70.7%, not the 105% a naive sum would suggest. Second, levers on the same factor do not stack fully. Once routing has moved traffic to a cheaper model, trimming its tokens saves money at the cheaper price. The order in which savings are counted changes how much each step appears to contribute, but not the final product. Batch processing illustrates how to read the model. All three major providers discount batch work by 50% [[4](#en-ref-anthropicBatch), [18](#en-ref-googleBatch), [40](#en-ref-openaiBatch)], so β = 0.5. The post's 10% batch step therefore implies b = 20%: one request in five can wait up to 24 hours. <a id="en-s5"></a> 5 The six levers !Figure 3. The six levers, the factor of Eq. 1 each one targets, the strongest effect reported in the literature, and the main trade-off. Figure 3. The six levers, the factor of Eq. 1 each one targets, the strongest effect reported in the literature, and the main trade-off.© 2026 Aridio Silva, Independent Researcher, ORCID 0009-0008-2411-6995. CC BY 4.0. 5.1 Lever 1: intelligent model routing Concept. A router sends each request to the cheapest model expected to answer it well. A cascade is a sequential variant: try a cheap model first, check its answer, and escalate only if the check fails. Evidence. - FrugalGPT learned which combination of LLM APIs to query per request. It matched GPT-4's performance "with up to 98% cost reduction", or improved accuracy by 4% at the same cost [[10](#en-ref-chen2023)]. - Hybrid LLM routed between a small and a large model by predicted difficulty and made "up to 40% fewer calls to the large model, with no drop in response quality" [[13](#en-ref-ding2024)]. - RouteLLM trained routers on human preference data and cut cost "by over 2 times in certain cases". Its routers kept working when the strong and weak models were swapped at test time [[30](#en-ref-routellmRepo), [39](#en-ref-ong2024)]. - Answer-consistency cascades reached GPT-4-level reasoning accuracy at about 40% of its cost [[51](#en-ref-yue2023)]. - AutoMix used few-shot self-verification and a POMDP router to cut cost by over 50% at comparable performance [[1](#en-ref-aggarwal2023)]. Trade-offs. Routers need labelled or preference data and drift as models change. A wrong "easy" verdict on a hard request hurts quality most on rare, high-stakes queries. Cascades pay twice for every escalated request (Section 9). 5.2 Lever 2: context engineering Concept. Context engineering means deciding which tokens enter the model's context window and in what order. Anthropic describes it as finding "the smallest possible set of high-signal tokens" for the task [[3](#en-ref-anthropic2025)]. Evidence. - Models use information placed in the middle of a long context worse than information at the start or end ("lost in the middle") [[29](#en-ref-liu2023)]. - Performance degrades as input length grows, even on simple tasks [[11](#en-ref-chroma2025)]. - LLMLingua compressed prompts "up to 20x" with little loss [[23](#en-ref-jiang2023a), [34](#en-ref-llmlinguaRepo)]. LongLLMLingua extended this to long-context RAG [[24](#en-ref-jiang2023b)]. - Budget-aware prompting can shorten reasoning output as well [[22](#en-ref-han2024)]. Fewer tokens therefore save money and can also raise quality. Trade-offs. Cutting too hard loses the one passage that mattered (a recall loss). Compression adds a model call and latency. Aggressive budgets need evaluation per task type. 5.3 Lever 3: prompt caching Concept. When many requests share the same prompt prefix (system instructions, tool schemas, few-shot examples, a long document), the provider can reuse the already-processed attention state of that prefix instead of recomputing it. Gim et al. showed the systems basis for this: reusing attention states cut time-to-first-token by 8× on GPU and 60× on CPU without changing outputs [[17](#en-ref-gim2023)]. Current provider terms (audited 27 Oct 2026). - Anthropic: cache reads at 0.1× the base input price (lower on some newer models). Cache writes at 1.25× for a 5-minute TTL or 2× for a 1-hour TTL. These multipliers stack with the batch discount [[5](#en-ref-anthropicCache)]. - OpenAI: caching is on by default for supported models, with cached input "discounted up to 95%" and a minimum cacheable prefix of 1,024 tokens on recent models [[41](#en-ref-openaiCache)]. - Google: implicit caching on Gemini, with explicit caching available on some models [[19](#en-ref-googleCache)]. Trade-offs. The prefix must be byte-identical across requests, so templates must put stable content first and variable content last. Writes cost more than ordinary input. Gu et al. found that caches shared across users in seven API providers create a timing side channel that can leak information about other users' prompts [[20](#en-ref-gu2025)]. 5.4 Lever 4: semantic caching Concept. A semantic cache stores whole answers and returns one when a new query is close in meaning to an earlier one, measured by embedding similarity [[45](#en-ref-reimers2019)]. GPTCache is the reference open-source implementation [[7](#en-ref-bang2023), [55](#en-ref-gptcacheRepo)]. MeanCache reports that about 31% of queries in its setting were repeats, and warns that naive semantic caches have "unacceptable false hit-and-miss rates" [[16](#en-ref-gill2024)]. Trade-offs. A false hit returns a confident answer to a different question. Answers go stale when sources change. A cache shared across tenants can leak data. Section 8 gives the required controls. 5.5 Lever 5: agentic loop optimisation Concept. Agents interleave reasoning and tool use [[50](#en-ref-yao2022)]. Each step is a paid call, and redundant steps, retries and hand-offs multiply k in Eq. 1. Evidence. Kapoor et al. found that simple baselines matched state-of-the-art agents on accuracy at much lower cost [[25](#en-ref-kapoor2024)]. Anthropic's guidance is to start with the simplest pattern that works (a single call, then a fixed workflow) and add autonomy only when it pays for itself [[2](#en-ref-anthropic2024)]. Trade-offs. Hard caps on steps and retries can stop an agent just before it would have succeeded. Caps should be set per task type from traces. 5.6 Lever 6: batch processing Concept. Work that does not need an immediate answer is submitted asynchronously in bulk. Examples are enrichment, classification of backlogs, evaluation runs and report generation. Evidence. Anthropic, OpenAI and Google all price batch requests at 50% of the synchronous rate and return results within 24 hours, often much sooner [[4](#en-ref-anthropicBatch), [18](#en-ref-googleBatch), [40](#en-ref-openaiBatch)]. Trade-offs. It only applies to latency-tolerant work, and it adds queueing and result-handling code. <a id="en-s6"></a> 6 Reference architecture The post arranges the levers into a pipeline. We re-draw it with its control loops made explicit (Figure 4). !Figure 4. Reference architecture, re-drawn from the post. Three control loops are added explicitly: cache-hit bypass (amber), escalation (violet) and cache write-back after the gate (green). Figure 4. Reference architecture, re-drawn from the post. Three control loops are added explicitly: cache-hit bypass (amber), escalation (violet) and cache write-back after the gate (green).© 2026 Aridio Silva, Independent Researcher, ORCID 0009-0008-2411-6995. CC BY 4.0. 1. Request. Carries the query, tenant, identity and access rights, and task type. Task type is the cheapest routing signal and should be captured at the edge. 2. Semantic cache and policy check. Returns a stored answer only if it is authorised, fresh and semantically equivalent (Section 8). 3. Context optimisation engine. Hybrid retrieval, rerank, filter and token budget (Section 7). This is retrieval-augmented generation as introduced by Lewis et al. [[27](#en-ref-lewis2020)]. 4. Quality-aware router. Picks a small, medium or frontier model (Section 9). 5. Quality gate. Checks format, groundedness, completeness and safety, and escalates on failure (Section 10). 6. Final response. Returns the answer with citations and emits a trace with cost. A cross-cutting observability and governance layer records tokens, cost, quality and latency per stage (Section 13). Three design principles follow from the literature: - Cheap decisions first. Cache lookup and deterministic checks cost far less than a model call, so they run before it. - Close the loop. Only answers that passed the gate are written to the cache, which keeps errors from being replayed. - Prefer self-correcting retrieval. Self-RAG [[6](#en-ref-asai2023)] and Corrective RAG [[49](#en-ref-yan2024)] add retrieval-quality checks inside the generation loop. They are natural extensions of stages 3 and 5. <a id="en-s7"></a> 7 Context optimisation engine !Figure 5. The context engine narrows about 100 candidate passages to a budgeted prompt. Counts are illustrative. The layout puts the cacheable prefix first and the strongest evidence at the edges. Figure 5. The context engine narrows about 100 candidate passages to a budgeted prompt. Counts are illustrative. The layout puts the cacheable prefix first and the strongest evidence at the edges.© 2026 Aridio Silva, Independent Researcher, ORCID 0009-0008-2411-6995. CC BY 4.0. 7.1 Hybrid retrieval Lexical retrieval with BM25 scores documents by term frequency, inverse document frequency and length normalisation [[46](#en-ref-robertson2009)]. It handles exact identifiers, codes and rare terms well. Dense retrieval embeds queries and passages in a vector space [[45](#en-ref-reimers2019)] and finds paraphrases. Approximate nearest-neighbour indexes such as HNSW make it fast at scale [[31](#en-ref-malkov2016)]. Running both and merging the results covers the weaknesses of each. 7.2 Fusion with RRF Reciprocal rank fusion merges ranked lists without calibrating their scores [[12](#en-ref-cormack2009)]: RRF(d) = Σr ∈ R 1 / (k + rank*r*(d)), k = 60(2) 7.3 Reranking and filtering A cross-encoder reads the query and each candidate passage together and scores relevance much more accurately than the first-stage retriever, at higher cost per pair [[38](#en-ref-nogueira2019)]. LLMs can also act as rerankers [[48](#en-ref-sun2023)]. After reranking, filters remove near-duplicates, passages the user may not see (ACL), outdated versions and passages below a minimum score. 7.4 Token budgeting and layout The packer fills a fixed budget with the best passages. If the budget is exceeded, it compresses [[24](#en-ref-jiang2023b)] rather than truncating blindly. The layout follows two rules: - Stable content (system prompt, tool schemas, examples) goes first so that it can be cached [[5](#en-ref-anthropicCache)]. - The strongest evidence goes at the beginning and end of the dynamic part, where models use it best [[29](#en-ref-liu2023)]. Table 1. Context-engine techniques: benefit and cost | Technique | Main benefit | Main cost or risk | | --- | --- | --- | | BM25 + dense hybrid | Recall on both exact terms and paraphrases | Two indexes to maintain | | RRF fusion | No score calibration needed | Ignores score magnitude | | Cross-encoder rerank | Higher precision in the top results | Latency grows with candidate count | | Prompt compression | Up to 20× fewer tokens [[23](#en-ref-jiang2023a)] | Extra model call; may drop details | | Token budget | Predictable cost per request | Recall loss if set too low | <a id="en-s8"></a> 8 Caching: prompt, semantic and KV The post mentions "prompt caching" and "semantic caching" as separate levers, and they are often confused with each other and with the inference engine's KV cache. Figure 6 separates them. !Figure 6. Three caching layers. They reuse different things, match on different rules and carry different risks. Figure 6. Three caching layers. They reuse different things, match on different rules and carry different risks.© 2026 Aridio Silva, Independent Researcher, ORCID 0009-0008-2411-6995. CC BY 4.0. The third layer matters only to teams that host their own models. Inference engines keep the attention key-value (KV) tensors of each request in GPU memory. PagedAttention manages them in fixed-size blocks to cut fragmentation [[26](#en-ref-kwon2023)], and RadixAttention shares them across requests through a prefix tree [[53](#en-ref-zheng2024)]. Provider-side prompt caching is the priced, managed form of the same idea. 8.1 The economics of prompt caching Take a prefix of P tokens reused by n requests within the cache lifetime, at Anthropic's multipliers (write 1.25×, read 0.1×) [[5](#en-ref-anthropicCache)]. The relative cost of the prefix is: costcache / costno cache = [1.25 + 0.1·(n − 1)] / n(3) Caching pays from the second use (1.35 vs 2.0). At n = 10 the prefix costs 21.5% of the uncached price. The lever is low-risk and high-yield when prefixes are long and traffic is steady. It loses money only when a written prefix is never reused before it expires. 8.2 Semantic caching done safely !Figure 7. Semantic-cache decision flow. A hit requires similarity, freshness and a fresh authorisation check. The threshold chart is illustrative. Figure 7. Semantic-cache decision flow. A hit requires similarity, freshness and a fresh authorisation check. The threshold chart is illustrative.© 2026 Aridio Silva, Independent Researcher, ORCID 0009-0008-2411-6995. CC BY 4.0. The flow in Figure 7 adds four controls that the infographic only implies ("authorized, fresh, semantically equivalent"). 1. Scope key. Search only among entries with the same tenant, access rights, corpus version and prompt or model version. This prevents cross-tenant leakage, which OWASP lists under sensitive information disclosure and vector and embedding weaknesses [[43](#en-ref-owasp2025)]. 2. Calibrated threshold τ. Raising τ cuts false hits faster than it cuts hit rate, up to a point. τ must be tuned per domain on labelled pairs [[16](#en-ref-gill2024)]. 3. Freshness. Use a TTL plus invalidation when the source document version changes. 4. Write-back only after the gate. Store provenance with the answer so that it can be audited or purged. Benefits - A hit avoids the whole pipeline: the largest saving per event. - Lower latency on repeated questions. Risks - False hits are confident wrong answers. - Staleness and cross-tenant leakage. - Personalised or time-sensitive answers are poor candidates. <a id="en-s9"></a> 9 Model routing and cascades !Figure 8. Router and cascade, their expected costs, and the break-even escalation rate. Cost ratios are illustrative. Figure 8. Router and cascade, their expected costs, and the break-even escalation rate. Cost ratios are illustrative.© 2026 Aridio Silva, Independent Researcher, ORCID 0009-0008-2411-6995. CC BY 4.0. Let *c*s, *c*L, *c*v and *c*r be the costs of the small model, the large model, the verifier and the router. A router sends a share q of traffic to the large model. A cascade escalates a share r after verification. E[*C*router] = *c*r + (1 − q)·*c*s + q·*c*L E[*C*cascade] = *c*s + *c*v + r·*c*L(4) A cascade beats "always large" only while r < 1 − (*c*s + *c*v)/*c*L. With *c*s = 0.10 and *c*v = 0.02 (relative to *c*L), the break-even is r* = 0.88. At r = 30% the cascade costs 0.42, a 58% saving. That is close to the post's 40% routing step once the medium tier is included. Routing signals in the literature. - Predicted difficulty from a trained classifier [[13](#en-ref-ding2024)]. - Human-preference-trained routers [[39](#en-ref-ong2024)]. - Answer consistency across sampled reasoning paths [[51](#en-ref-yue2023)]. - Self-verification with a POMDP policy [[1](#en-ref-aggarwal2023)]. - Token-level uncertainty, which corrects the length bias of whole-sequence confidence [[21](#en-ref-gupta2024)]. Table 2. Router versus cascade | | Router (decide first) | Cascade (try, then escalate) | | --- | --- | --- | | Latency | One model call plus a small router | Escalated requests pay two calls in series | | Signal quality | Must predict difficulty before seeing an answer | Judges an actual answer, which is more reliable | | Data needs | Labelled or preference data to train | A verifier or consistency check | | Best for | Interactive, latency-sensitive traffic | Batch or accuracy-sensitive traffic | <a id="en-s10"></a> 10 Quality gates and escalation !Figure 9. Quality gate: four check tiers run from cheapest to most expensive, with four possible outcomes and a budget guard. Figure 9. Quality gate: four check tiers run from cheapest to most expensive, with four possible outcomes and a budget guard.© 2026 Aridio Silva, Independent Researcher, ORCID 0009-0008-2411-6995. CC BY 4.0. The gate is what makes aggressive routing safe. It should run checks in order of cost and stop at the first hard failure. 1. Deterministic checks. JSON schema, required fields, and whether every cited ID actually exists. These are cheap and catch the most frequent agent failures. 2. Classifiers. Safety, PII and prompt-injection signals (OWASP LLM01 and LLM02) [[43](#en-ref-owasp2025)]. 3. Grounding checks. Is each claim supported by the retrieved evidence? RAGAS defines reference-free metrics such as faithfulness and context relevance [[15](#en-ref-es2023), [44](#en-ref-ragasRepo)]. 4. Model-based judgement. LLM-as-a-judge reaches over 80% agreement with human preferences but shows position, verbosity and self-enhancement biases [[52](#en-ref-zheng2023)]. SelfCheckGPT detects hallucinations by checking consistency across samples [[32](#en-ref-manakul2023)]. Because these checks cost as much as a generation, run them on a sample or only on high-risk routes. Outcomes. Each answer ends in one of four states: - pass: respond, log and possibly cache; - repair: one retry with the validator's error message; - escalate: send to the next model tier, at most one hop; - abstain or hand off: refuse safely or pass to a human. A budget guard caps retries, escalations and spend per request, which addresses OWASP's "unbounded consumption" risk (LLM10) [[43](#en-ref-owasp2025)]. <a id="en-s11"></a> 11 Cost decomposition and sensitivity !Figure 10. The post's waterfall, re-drawn to scale. Step percentages and cumulative savings were computed by the author from the post's values. Figure 10. The post's waterfall, re-drawn to scale. Step percentages and cumulative savings were computed by the author from the post's values.© 2026 Aridio Silva, Independent Researcher, ORCID 0009-0008-2411-6995. CC BY 4.0. 11.1 Checking the arithmetic The post's values are internally consistent: 10,000 × 0.85 × 0.60 × 0.75 × 0.85 × 0.90 = 2,926. Routing is the largest single step (−40% of what remains; 34 of the 70.7 percentage points). The post labels the chart illustrative, so these are design targets, not measurements. 11.2 Sensitivity !Figure 11. One-at-a-time sensitivity of the final monthly cost. Ranges are the author's assumptions around the post's step factors. The five teal rows vary those factors up and down; the amber row is not a variation but the overhead the levers add, so it only increases cost. Figure 11. One-at-a-time sensitivity of the final monthly cost. Ranges are the author's assumptions around the post's step factors. The five teal rows vary those factors up and down; the amber row is not a variation but the overhead the levers add, so it only increases cost.© 2026 Aridio Silva, Independent Researcher, ORCID 0009-0008-2411-6995. CC BY 4.0. Varying each factor within a hypothetical author-selected range shows that, in this scenario, routing dominates the outcome (−64% to −78%). Batch and deduplication matter least. The post leaves out overhead: embeddings and the vector store for the semantic cache, the router, judge calls, and evaluation runs. If that overhead is assumed to add 3–9% of the original baseline, the scenario’s net saving falls to about 61–68%. This calculation demonstrates sensitivity rather than an expected range for real workloads; that range must be measured per workload. Table 3. Costs the levers add (not shown in the post) | Item | Introduced by | How to keep it small | | --- | --- | --- | | Query embeddings, vector search | L4, L2 | Small embedding model; cache embeddings | | Router inference | L1 | Lightweight classifier, not an LLM | | Verifier and judge calls | Gate | Deterministic checks first; sample model judges | | Cache writes (1.25×–2×) | L3 | Cache only prefixes that will be reused | | Evaluation runs and labelling | All | Run evaluations through the batch API (−50%) | | Engineering and operations time | All | Phase the work; quick wins first (Section 15) | <a id="en-s12"></a> 12 Evaluation and KPIs !Figure 12. KPI tree. Cost per successful task is the north-star metric; cost drivers, quality guards and operational metrics explain its movement. Figure 12. KPI tree. Cost per successful task is the north-star metric; cost drivers, quality guards and operational metrics explain its movement.© 2026 Aridio Silva, Independent Researcher, ORCID 0009-0008-2411-6995. CC BY 4.0. 12.1 Reading the quality table critically The post reports task success 92% → 92–93%, groundedness 90% → 90–91% and critical errors 0.5% → 0.4%. Miller recommends treating evaluation items as a sample from a larger population and reporting confidence intervals [[35](#en-ref-miller2024)]. With 500 items, the standard error of a 92% success rate is √(0.92 · 0.08 / 500) ≈ 1.2 points, so the 95% interval is roughly ±2.4 points. The reported changes are inside that noise. "Same quality" is plausible but not shown. A rigorous version would: - evaluate both systems on the same items (a paired design); - pre-register a non-inferiority margin, for example "success no more than 1 point lower"; - size the evaluation set so the interval is narrower than that margin. 12.2 KPI definitions Table 4. Recommended KPIs | KPI | Definition | Why it matters | | --- | --- | --- | | Cost per successful task | Total cost ÷ successful tasks [[14](#en-ref-erol2025)] | North-star; stops "cheap but wrong" from looking good | | Tokens per request (in/out) | Mean and P95, per route | Shows context bloat and verbose outputs | | Model-tier mix | Share of requests per tier | Main driver of p(m) | | Prompt-cache read share | Cached ÷ total input tokens | Verifies that Lever 3 works | | Semantic-cache hit and false-hit rates | Hits ÷ lookups; wrong hits ÷ audited hits | Saving and safety of Lever 4 | | Escalation rate r | Escalated ÷ gated requests | Cascade break-even (Eq. 4) | | Task success, groundedness, critical errors | On a fixed eval set, with 95% CIs [[35](#en-ref-miller2024)] | Quality guards | | P50/P95 latency and TTFT | End to end and per stage | User experience; latency SLOs [[8](#en-ref-beyer2016)] | | Calls per request k | LLM calls ÷ requests | Agent-loop efficiency (Lever 5) | <a id="en-s13"></a> 13 Observability and governance None of the levers can be tuned without per-request telemetry. The OpenTelemetry semantic conventions for generative AI define standard span attributes for model, token usage and operations [[42](#en-ref-otel)]. Using them makes cost per span portable across vendors. Governance has three anchors: - the NIST AI Risk Management Framework, with its Govern, Map, Measure and Manage functions [[36](#en-ref-nist2023)]; - the NIST Generative AI Profile [[37](#en-ref-nist2024)]; - the OWASP Top 10 for LLM Applications 2025, which names the main risks this architecture must control: prompt injection, sensitive information disclosure, vector and embedding weaknesses, and unbounded consumption [[43](#en-ref-owasp2025)]. Sculley et al.'s warning about hidden technical debt in ML systems applies directly [[47](#en-ref-sculley2015)]. Every cache, router and judge is another component whose configuration can drift. Each one needs an owner, a metric and an alert. <a id="en-s14"></a> 14 Relationship to SGAEIA: efficiency under governance and evidence Classification of this section: SGAEIA INTERPRETATION — a public conceptual connection, not normative architecture or implementation evidence. For SGAEIA — Secure Governed Autonomous Edge Intelligence Architecture — cost reduction is not a standalone objective. An optimisation is acceptable only when it preserves authority boundaries, policy enforcement, traceability, revocation, tenant separation, and enough evidence to explain why each request followed a particular route. Cost per successful task must therefore be paired with critical-error rates, gate false positives and false negatives, per-route latency, trace completeness, and residual risk; a cheaper answer that weakens those controls is not a net gain for a governed autonomous system [[56](#en-ref-sgaeia2026)]. The six levers reviewed here relate most directly to four SGAEIA responsibility planes. Routing and cascades belong to integration and substitution: changing the model must not silently expand available authority or invalidate security assumptions. Caches and the context engine cross identity and trust as well as authority and delegation, because a reused answer must respect the current tenant, current authorisation, provenance, and corpus version. Step, retry, and spend limits support governed execution, while per-request telemetry, route decisions, and gate outcomes support observability and evidence. This connection yields a candidate property that is not yet normative: cost optimisation in an AI system should not degrade authority boundaries, policy enforcement, revocation capability, or the completeness of evidence required for audit. The six-lever architecture in this report is external evidence and a possible engineering pattern; it does not become part of SGAEIA merely by appearing in this publication. Any adoption would require an existing-coverage audit, impact analysis, the applicable Research-to-Architecture gates, acceptance criteria, and adversarial verification before a decision by the project owner. This reading also changes the practical ordering of the levers. Instrumentation, duplicate-call removal, loop limits, and prompt caching offer gains with a smaller risk surface. Routing, model judges, and semantic caching require non-inferiority evaluation, leakage tests, false-hit analysis, model-substitution checks, and safe failure modes. The objective is not the lowest possible bill, but the lowest cost compatible with governed execution and continuous, evidence-backed assurance. <a id="en-s15"></a> 15 Implementation roadmap !Figure 13. Phased roadmap. Each phase ends at a gate whose exit criterion must hold on the evaluation set before the next phase starts. Weeks are indicative. Figure 13. Phased roadmap. Each phase ends at a gate whose exit criterion must hold on the evaluation set before the next phase starts. Weeks are indicative.© 2026 Aridio Silva, Independent Researcher, ORCID 0009-0008-2411-6995. CC BY 4.0. Table 5. Phases, actions and supporting sources | Phase | Actions | Exit criterion | Support | | --- | --- | --- | --- | | 0 · Baseline | Trace every call with tokens and cost; build an evaluation set with confidence intervals | Baseline cost per successful task known ± CI | [[14](#en-ref-erol2025), [35](#en-ref-miller2024), [42](#en-ref-otel)] | | 1 · Quick wins | Remove duplicate calls; cap loops and retries; restructure prompts for prefix caching | No quality regression; cache read share at target | [[2](#en-ref-anthropic2024), [5](#en-ref-anthropicCache), [25](#en-ref-kapoor2024), [41](#en-ref-openaiCache)] | | 2 · Context and batch | Hybrid retrieval, rerank, token budget, compression; move async jobs to batch | Tokens per request down; groundedness ≥ baseline | [[12](#en-ref-cormack2009), [24](#en-ref-jiang2023b), [29](#en-ref-liu2023), [38](#en-ref-nogueira2019), [40](#en-ref-openaiBatch)] | | 3 · Routing and gates | 2–3 tier router; gate tiers; bounded escalation; run in shadow mode before switching | Cost per successful task down; success non-inferior | [[13](#en-ref-ding2024), [15](#en-ref-es2023), [39](#en-ref-ong2024), [51](#en-ref-yue2023), [52](#en-ref-zheng2023)] | | 4 · Semantic cache | Scoped cache; τ calibration on labelled pairs; TTL and invalidation; leakage tests | False-hit rate below the agreed limit | [[7](#en-ref-bang2023), [16](#en-ref-gill2024), [43](#en-ref-owasp2025)] | The order reverses the post's visual order on purpose: the lowest-risk, highest-certainty levers come first. Semantic caching comes last because it is the only lever that can return a wrong answer with no model involved, so it needs the most mature governance. <a id="en-s16"></a> 16 Discussion and threats to validity - Workload dependence. Every number in the post depends on the request mix. A workload of hard, unique, multi-step tasks has little to route or cache. - Price volatility. Provider prices and cache multipliers change several times a year, and the gap between model tiers can shrink [[14](#en-ref-erol2025)]. The architecture stays valid; the size of the saving does not. - Latency claim. The post reports a lower P95 latency (2.8 s → 2.6 s). That is plausible if cache hits and smaller models outweigh the extra stages, but cascades add serial calls for escalated requests. It must be measured per route. - Evaluation validity. Agent benchmarks often lack proper hold-out sets [[25](#en-ref-kapoor2024)]. Routers tuned on the evaluation set will overstate gains. - Security of caches. Both cache types create cross-user information channels if shared [[20](#en-ref-gu2025), [43](#en-ref-owasp2025)]. - Scope of this report. This is a literature-grounded analysis, not an experiment. Its numeric extensions (Figures 8 and 11) are illustrative models. <a id="en-s17"></a> 17 Conclusion Bhardwaj's proposal is technically sound and matches the published evidence. Every one of its six levers has primary-source support, and several published results exceed the reductions it assumes. Its main contribution is a principle: spend intelligence where the task needs it, and measure the result as cost per successful task. Its main weakness is the headline number. It is illustrative, it omits the overhead the levers add, and its quality comparison lacks statistical support. Adding those pieces makes the modelled scenario yield a 61–68% net saving under the selected assumptions, but it does not establish an expected production range. Demonstrating a reduction near 70% requires workload telemetry, complete costs, and a paired non-inferiority test. The safest route is the one in Section 15: measure first, take the low-risk levers, and add routing and semantic caching only behind quality gates and a non-inferiority test. <a id="en-gloss"></a> Glossary Agentic loop - The cycle in which an agent reasons, calls a tool, reads the result and decides the next step. Each step is a paid model call. ANN (approximate nearest neighbour) - Search that finds vectors close to a query vector quickly, at the price of occasionally missing the exact nearest one. Batch API - An asynchronous endpoint that processes many requests within a time window (24 h) at a discount (50% at the major providers). BM25 - A lexical ranking function based on term frequency, inverse document frequency and document length. Cascade - A sequence of models from cheap to expensive. A request moves to the next model only when a check on the current answer fails. Confidence interval (CI) - A range that, under repeated sampling, contains the true value at the stated rate (for example 95%). Context engineering - Choosing which tokens enter the model's context window, and in what order, for a given step. Context window - The maximum number of tokens a model can read in one call, including instructions, evidence and history. Cost-of-pass / cost per successful task - Expected spend to obtain one correct result: cost per attempt divided by success probability. Cross-encoder - A model that reads query and passage together to score their relevance. More accurate and slower than comparing separate embeddings. Deferral rule - The rule a cascade uses to decide whether to keep an answer or pass the request to a stronger model. Dense retrieval / embedding - Representing text as vectors so that texts with similar meaning are close together, then searching by vector similarity. Escalation - Re-sending a request to a more capable model after a quality check fails. False hit - A semantic-cache hit that returns an answer to a different question. Frontier model - The most capable and usually most expensive model tier available at a given time. Groundedness / faithfulness - The degree to which every claim in an answer is supported by the evidence given to the model. Hallucination - Fluent output that is not supported by the input or by facts. HNSW - Hierarchical navigable small world graph: a widely used index for fast approximate nearest-neighbour search. Hybrid retrieval - Running lexical (BM25) and dense retrieval together and merging their results. KV cache - The attention key and value tensors an inference engine keeps in memory so that earlier tokens are not recomputed. LLM-as-a-judge - Using a language model to grade another model's output against a rubric. Lost in the middle - The finding that models use information in the middle of a long context worse than information at its start or end. Model routing - Choosing, per request, which model should answer it, before any model is called. Non-inferiority test - A test that shows a new system is not worse than the old one by more than a pre-set margin. Observability / trace / span - Recording each request as a trace made of spans (one per stage), with attributes such as tokens, cost and latency. Pareto frontier - The set of options where no option is better on one objective without being worse on another. Prompt (prefix) caching - Provider-side reuse of the processed state of a prompt prefix that repeats across requests, billed at a reduced rate. Prompt compression - Removing low-information tokens from a prompt with a smaller model while keeping its meaning. Prompt injection - Input crafted to override a model's instructions. OWASP's top risk for LLM applications. RAG (retrieval-augmented generation) - Retrieving relevant documents and adding them to the prompt so that the model answers from them. Reranking - A second, more accurate scoring pass over the top candidates from a first-stage retriever. RRF (reciprocal rank fusion) - Merging ranked lists by summing 1/(k + rank) for each document. Semantic cache - An application-level cache that returns a stored answer when a new query is similar enough in meaning to a past one. Shadow mode - Running a new component on live traffic without using its output, to compare it with the current one. SLO / error budget - A target for a service metric, and the amount of failure allowed before action is required. Token - The unit of text a model reads and writes, and the unit of billing. Roughly ¾ of an English word. Token budget - A fixed cap on the tokens a stage may send or generate. TTFT (time to first token) - Latency from sending a request to receiving the first output token. TTL (time to live) - How long a cache entry stays valid before it expires. <a id="en-refs"></a> References (audited) All links were resolved on 27 October 2026. arXiv records were checked against the arXiv API (title, authors, date). Other links were fetched and their HTTP status recorded. Where a publisher blocks automated access (HTTP 403), the DOI was confirmed to resolve to the publisher's record. Venues are given only where confirmed; otherwise the arXiv version is cited. <a id="en-ref-aggarwal2023"></a> 1. Aggarwal, P., Madaan, A., Anand, A., Potharaju, S. P., et al. (2023). AutoMix: Automatically mixing language models. arXiv preprint arXiv:2310.12963. arxiv.orgVerified: arXiv API metadata match, HTTP 200 · Accessed 2026-10-27 <a id="en-ref-anthropic2024"></a> 2. Anthropic (2024, December 19). Building effective AI agents. Anthropic Engineering. anthropic.comVerified: HTTP 200 at the current URL shown · Accessed 2026-10-27 <a id="en-ref-anthropic2025"></a> 3. Anthropic (2025, September 29). Effective context engineering for AI agents. Anthropic Engineering. anthropic.comVerified: HTTP 200 at the current URL shown · Accessed 202

Theo's 60% stat is an outage chart: 43 hours of Codex incidents Originally published at tensorust.com. TL;DR: @theo posted that over 60% of T3 Code users who used Codex also used Claude Code, and sold it as proof that the harness is the product. I pulled both companies' own status pages for the same two weeks. Codex sat under an open incident for 42.7 hours of those 14 days (12.7% of the time). Claude Code: 34 minutes. Over 80 days, Claude Code had 42.2 hours of incidents and Codex had 56.0, but both were broken at the same moment for only 101 minutes. People are not using both because of taste. They use both because each one breaks, and switching deletes 96% of the downtime. Meanwhile my own "smart" LLM router picked the same model for 244 out of 244 real prompts. Model switching is a backup generator. Theo is selling it as a lifestyle. If you came here from Google asking "Claude Code vs Codex, which one?", the short answer from the data: neither, alone. Each had 40 to 56 hours of status-page incidents in 80 days. Together they were down at the same time for under two hours. Two $20 plans buy you roughly 99.9% coverage instead of about 97 to 98%. The rest of this article is how I measured that, what it says about the stat everyone quoted this weekend, and what my own router got embarrassingly wrong. !Animated timeline from Aug 1 to Oct 19 2026 with three lanes. Claude Code, Anthropic: 28 incidents, 42.2 hours. Codex, OpenAI: 12 incidents, 56.0 hours. The incident bars fill in from left to right, then a third lane appears: both broken at the same time, 2 episodes, 101 minutes. Final line: switch when one breaks and 96% of the downtime is gone. Claude Code and Codex incidents from both companies' status page APIs, Aug 1 to Oct 19 2026 10 AM ET, any severity. Static frame: timeline.png. The stat that went around this weekend At 12:17 AM Eastern on October 19, Theo Browne, the creator behind t3.gg, T3 Chat and T3 Code, posted: "For the last 2 weeks, over 60% of T3 Code users who used Codex also used Claude Code as well. The majority of Codex users are Claude Code users." It did 69.1K views by the time I captured it. !Screenshot of a post by @theo, Oct 19 2026 12:17 AM, 69.1K views: For the last 2 weeks, over 60% of T3 Code users who used Codex also used Claude Code as well. The majority of Codex users are Claude Code users. @theo on X, Oct 19 2026, captured 10:12 AM ET Oct 19 2026, real and unedited. The replies read it the way it was framed: the models are interchangeable, the harness that holds both of them is where the value lives, and T3 Code is that harness. A few hours earlier the same account asked "If you haven't moved to T3 Code yet, why?" That is the context. This is a product stat, posted by the product's owner, about why you need the product. Here is my take, stated plainly: the 60% is not a preference number. It is an outage number with a marketing caption. A product that sits on top of both labs makes money every time one of them falls over, and Theo knows exactly which way that stat cuts. He will never call T3 Code outage insurance, because "insurance" does not trend. "The majority of Codex users are Claude Code users" does. What I measured, and how Both labs publish machine-readable incident history. Anthropic runs a Statuspage at status.claude.com with a public /api/v2/incidents.json, where every incident lists the components it hit ("Claude Code" is one of them). OpenAI's page at status.openai.com exposes per-component impacts with start and end times, including "Codex API", "Codex Web", "CLI", "VS Code extension" and "Codex in ChatGPT Desktop". I pulled both on October 19 at about 10:10 AM Eastern: 50 Anthropic incidents (newest first, back to July 27) and 105 OpenAI incidents (back to July 21). Then a 60-line Python script did three things: 1. Kept every incident that touched Claude Code, and every OpenAI component impact on a Codex component. 2. Merged overlapping intervals per product, so two incidents at once count once. 3. Intersected the two timelines: the minutes when both products had an open incident at the same time. The window is August 1 to October 19, 10 AM Eastern: 79.6 days. A second run used exactly Theo's window, the 14 days before his post. Two honest limits, and then I will stop hedging. A status-page incident is not a total outage: "degraded performance" can mean 5% of requests fail, or a slow model. And labs tag components their own way: OpenAI's longest Codex entry, 25 hours on October 5 and 6, is titled "Elevated Work Mode errors" and is tagged on a Codex component. That is how OpenAI filed it, so that is how I counted it. If anything, status pages under-report: they are written by the company, about the company. 80 days of receipts: each one broke for 40+ hours From August 1 to October 19: - Claude Code: 28 incidents, 2,533 minutes (42.2 hours) under an open incident. That is 97.79% clean time. 15 of those incidents were rated major or critical. - Codex: 12 incidents, 3,358 minutes (56.0 hours). 97.07% clean time. - Both at the same time: 101 minutes, in 2 episodes. 85 minutes on September 3 and 16 minutes on October 5. Counting only major incidents and full or partial outages, the overlap is zero. So a person who switches the moment one breaks goes from about 97 to 98% to 99.91%. Switching removes 96.0% of a Claude Code user's incident time and 97.0% of a Codex user's. !Bar chart of hours under an open status page incident. Aug 1 to Oct 19, 80 days: Claude Code 42.2 hours, Codex 56.0 hours, both at once 1.7 hours. Theo's window, Oct 5 to Oct 19: Claude Code 34 minutes, Codex 42.7 hours, both at once 16 minutes. Headline: in the 2 weeks behind the 60% stat, Codex sat under an incident 12.7% of the time. status.claude.com (Claude Code) and status.openai.com (Codex API, Web, CLI, VS Code, Desktop) incident APIs, pulled 10:10 AM ET Oct 19 2026. Now look at Theo's exact window, October 5 to October 19. Claude Code had 2 incidents, 34 minutes total. Codex had 4 incidents covering 2,562 minutes: 42.7 hours, 12.7% of the two weeks. One of them, on October 13, was titled "Degraded performance in GPT-6.1 Sol Codex and Work mode" and ran 8 hours 18 minutes. Of course the majority of Codex users also used Claude Code that fortnight. For one hour in every eight, the Codex status page said something was wrong. The stat was posted as a vibe about the future of harnesses. My read: it is mostly a picture of OpenAI's worst two weeks of the quarter. The API side tells the same story. Claude API: 27 incidents, 45.7 hours. OpenAI Chat Completions plus Responses: 7 incidents, 46.0 hours. Both at once: 19 minutes, one episode. If you run an app on one provider with no fallback, you are choosing to eat about 46 hours a quarter. What two subscriptions actually buy Theo also posted that his most common DM is some version of "which $20 plan is better, Claude or Codex?" (376.8K views). He has a product to sell, so he answers with tier lists and vibes. Here is the answer with a number on it. Claude Pro is $20 a month billed monthly; ChatGPT's entry paid plan is the other $20 in Theo's own question. Average the 80 days out to a month and adding the second plan buys back: - For a Claude Code user: about 15.5 hours a month of incident time. That is $1.29 per hour of not being blocked. - For a Codex user: about 20.8 hours a month. $0.96 per hour. If an hour of your work is worth more than a dollar, the second plan is not a luxury. It is a generator in the garage. That is the whole insight, and you do not need a new app to get it: you need two terminals and the habit of switching the moment one gets weird. The pinned take: "a lot of us are gonna end up where they are" Theo's pinned post, 66.6K views: "Anthropic was so bottleneck'd by GitHub that they rebuilt git itself. This might seem insane, but a lot of us are gonna end up where they are today." !Screenshot of @theo's pinned post, Oct 18 2026 11:19 PM, 66.6K views: Anthropic was so bottleneck'd by GitHub that they rebuilt git itself. This might seem insane, but a lot of us are gonna end up where they are today. It quotes @AnthropicAI: New on our Engineering blog: Creating in-house source control. In May, we merged around 2,500 code changes per day. In September, we merged more than 30,000 per day. @theo's pinned post on X, captured 10:12 AM ET Oct 19 2026, real and unedited. Read the post he is quoting. Anthropic's engineering blog says they went from about 2,500 merged changes a day in May to over 30,000 a day in September, and then says, in plain words: "Scion is by no means a suitable replacement for git for general use." It only makes sense for "truly enormous monorepos, with either tens of thousands of engineers or swarms of agents." Anthropic wrote the warning label and Theo pinned the opposite. Almost nobody reading that post merges 30,000 changes a day. Most of us merge 30. And the bottleneck most people actually hit this quarter was not git. GitHub's own status page lists 4 incidents touching Git Operations, 554 minutes (9.2 hours), across the 50 incidents its API returns (August 3 to October 15). Claude Code had 42.2 hours in a similar window. If you want to rebuild infrastructure because something keeps blocking you, the receipts say rebuild your fallback, not your version control. My own LLM router did 244 for 244 the wrong way Here is where I punch myself before anyone else does. The other half of the "use many models" pitch is routing: send easy prompts to a cheap model and hard ones to an expensive one, automatically. I wrote one of those. llm-router is a single-header C++ library from my llm-cpp toolkit, and its README says it routes "prompts to the right model" with five strategies. So I ran it. First, the repo's own example, untouched, built months ago. Watch the last line: a red-black tree with deletion and step-by-step reasoning scores 13 out of 100 and goes to the cheapest model. !Terminal output of basic_router.exe from the llm-router repo, Balanced strategy. Hi goes to gpt-4o-mini, complexity 2. What is 2+2 goes to gpt-4o-mini, complexity 7. Explain quantum entanglement in detail with examples goes to gpt-4o-mini, complexity 6. Implement a red-black tree in C++ with all edge cases handled, including deletion, with step by step reasoning and code goes to gpt-4o-mini, complexity 13. The repo's own example, unchanged, run 10:15 AM ET Oct 19 2026 (llm-router commit 11f7df6). Then the real test. I took two public prompt sets: all 164 coding tasks from OpenAI's HumanEval and all 80 first-turn questions from MT-Bench (writing, roleplay, reasoning, math, coding, extraction, STEM, humanities). A 30-line C++ harness compiled against the unchanged header ran every prompt through all five strategies, using the model table from the repo's examples/route_prompts.cpp: gpt-4o-mini at $0.15, gpt-4o at $2.50 and gpt-3.5-turbo at $0.50 per million input tokens (the published list prices for those models). !Chart of my llm-router on 244 real prompts across 5 strategies. CostOptimized: gpt-4o-mini 244 of 244. LatencyOptimized: gpt-4o-mini 244 of 244. QualityOptimized: gpt-4o 244 of 244. Balanced: gpt-4o-mini 244 of 244. Budget: gpt-4o-mini 244 of 244. Complexity score median 11, max 43 out of 100; every model is rated for 50 or more, so the score never changed one pick. 164 HumanEval tasks + 80 MT-Bench first turns through llm_router.hpp, model table from examples/route_prompts.cpp, Oct 19 2026. Every strategy picked one model for all 244 prompts. Cost, Latency, Balanced and Budget: gpt-4o-mini, 244 out of 244. Quality: gpt-4o, 244 out of 244. The complexity score had a median of 11 and a maximum of 43 out of 100, and since every model in the table is rated to handle at least 50, the score never changed a single decision. HumanEval, which is literally "write this function", had a median of 12. The reason is two lines of real code from include/llm_router.hpp: // Penalise if complexity exceeds what the model handles well double complexity_fit = (complexity <= m.max_complexity) ? 1.0 : 1.0 - 0.5 * static_cast<double>(complexity - m.max_complexity) / 100.0; ... case RoutingStrategy::Balanced: return complexity_fit * (eq / std::max(0.001, m.cost_per_1k * m.latency_score)); The fit penalty bottoms out at 0.5, but the price gap between the two GPT-4o models is 16.7x. A penalty that can at most halve a score can never beat a 16.7x price ratio. So "Balanced" is a cheapest-model picker with extra steps, and "Quality" is a most-expensive-model picker with extra steps. The keyword scoring (does the prompt contain "explain" or "implement"?) is theater on top. As of today the repo is unchanged; this article is the bug report. I am not the only one. My hot take: most "smart" model routers in the wild do the same thing, they just never get run on 244 prompts in public. A router that guesses difficulty from keywords and length is a coin that always lands on the same side. The only routing I have ever seen pay for itself is the dumb kind: if this provider errors, send it to the other one. That is the 96% from the status pages. That is what Theo's 60% is really measuring. Theo's worst takes this week, ranked Theo says he tends to make enemies, and he has a theory about why. On October 18 he posted "Because I pursue accuracy above all else, I tend to make enemies with those who benefit from people being misinformed," and two minutes later, "The only victims of truth are grifters." Fine. Accuracy above all else. Let's be accurate. !Two screenshots of posts by @theo, Oct 18 2026. First: Just realized something. Because I pursue accuracy above all else, I tend to make enemies with those who benefit from people being misinformed. Second: This is why the overwhelming majority of anti-Theo commentary is trying to discredit me. The more reach I have, the harder life is for those who make a living on lies. The only victims of truth are grifters. @theo on X, Oct 18 2026, captured 10:12 AM ET Oct 19 2026, real and unedited. 1. The 60% stat. Posted with no outage context, during the two weeks Codex spent 12.7% of its time under an incident, by the owner of the product that profits from you running both. Accuracy above all else would have included the denominator. 2. "A lot of us are gonna end up where they are today." The company in question wrote "by no means a suitable replacement for git for general use." Pinning the opposite of the source is a choice. 3. "npm fucked up our release... Time to fuck up npm" (427K views), followed by "I legitimately think I can build something 10x better in a few weeks." Git is a bottleneck, npm is a bottleneck, tsc gets a Rust port. When every tool you touch needs replacing, my read is the product being sold is the replacing. 4. "I know T3 Code is good because the majority of the feature requests are things we already have." That is not a flex. When users keep asking for features you already shipped, they cannot find them. That is a discoverability bug report, read upside down. 5. "I'll pay you $800 over MSRP" for a launch-day phone (334.6K views), from the account that, minutes earlier, complained about paying marked-up prices to a scalper. Posting a public $800 bounty is how the scalper market gets its price. !Four screenshots of posts by @theo, Oct 15 to 17 2026. npm fucked up our release, normally I'd be mad but I'm kind of excited, I needed a new project, time to fuck up npm. I know T3 Code is good because the majority of the feature requests are things we already have. If anyone in SF can get me a 1tb+ iPhone Duo on launch day, I'll pay you $800 over MSRP. Most common format of DMs I get: which $20 plan is better, Claude or Codex? @theo on X, Oct 15 to 17 2026, captured 10:12 AM ET Oct 19 2026, real and unedited. None of this makes him wrong about everything. He is right that Gemini's cache misses cost real money, and he is right that a harness that holds both models is useful. He is wrong about why it is useful, and the wrong reason happens to be the one that sells his product. Do this in your own setup today 1. Keep two coding agents logged in. One Claude plan, one OpenAI plan. At $20 each, the second one costs about $1 per hour of outage it saves you, by the numbers above. 2. Subscribe to both status feeds. Statuspage pages expose RSS and JSON (/history.rss, /api/v2/incidents.json); point them at a Slack or Discord channel so you switch on the first "investigating", not after twenty minutes of retrying. 3. In apps, fail over, do not "route". Wrap each provider in a circuit breaker (the documented example in my tokio-prompt-orchestrator opens after 5 failures in a row and probes again after 60 seconds) and send traffic to the other provider while it is open. Keep prompts portable: no provider-only features in the hot path. 4. If you must route by difficulty, test the router before you trust it. Run it over a public prompt set and count the decisions. If one model wins 100%, you do not have a router. Use a hard rule instead ("anything with a stack trace or over 2,000 tokens goes to the big model") and measure that. 5. Retry smart before you switch. Most blips are rate limits, not outages. My exponential backoff with jitter test shows which retry defaults make a rate limit worse. Related on this site: I benchmarked my own LLM proxy and found a 10x slowdown (the same "test your own tool" habit), and why chat sampling settings break JSON output. Bold prediction By the middle of 2027, "which model is best" stops being the fight, because the labs leapfrog each other every few weeks and nobody can keep a tier list alive (Theo himself said he wants to make a new tier list but knows the next model will drop the second he does). The harness war gets decided on failover: which tool notices the outage first and moves your session to the other model without losing context. The winner will market it as "multi-model". Theo will market it as taste. It will be outage insurance either way. And the next time OpenAI has a 12% fortnight, the Claude share in T3 Code's stats will jump, and it will get posted as a trend. The data, and how to rerun it Pulled October 19, 2026, 10:10 to 10:20 AM Eastern: status.claude.com/api/v2/incidents.json (50 incidents, July 27 to October 7, nothing newer was listed), status.openai.com/proxy/status.openai.com/incidents (105 incidents, July 21 to October 19, per-component start and end times), and githubstatus.com/api/v2/incidents.json (50 incidents). Claude Code intervals use each incident's start and resolve times; Codex intervals use the per-component impacts for Codex API, Codex Web, CLI, VS Code extension and Codex in ChatGPT Desktop. Intervals are merged per product and intersected. Router sweep: llm_router.hpp at commit 11f7df6, compiled with MSVC 2022 /O2, run over HumanEval (164) and MT-Bench first turns (80). Theo's posts were captured from X at about 10:12 AM Eastern on October 19, real and unedited. So, @theo: accuracy above all else. Post the 60% next to the status pages and tell your audience which number explains it. Everyone else, pick a team in the comments. Team Theo: people run both because the harness is the product. Team Receipts: people run both because each one breaks for 40+ hours a quarter. Which one are you, and which agent was down the last time you switched? If you ship apps built with these agents, I run a $49 Launch Check: I test your AI-built app for the failure cases nobody tested, the same way I tested my own router here.