yatinannam

I built Weir: An open-source semantic cache and model router for RAG I built Weir, an open-source gateway for RAG that combines semantic caching, model routing, and per-request cost accounting. The interesting challenge was making semantic caching safer. Two questions can have very similar embeddings but require different answers, so I added an entity guard to catch distinctions like ICU vs. general ward visiting hours. I evaluated four configurations on a fictional hospital FAQ dataset. On a repeat-heavy 300-request replay, full Weir reduced measured cost per 1,000 requests by 94% and median latency from 747 ms to 14 ms. On a cold pass, the cost saving was 20%. These are small synthetic benchmarks, not production results. I also documented an overload mode discovered during load testing, where cache timeouts caused more traffic to reach retrieval and generation. Repo: yatinannam/weir I'd appreciate technical feedback on the cache safety checks, evaluation methodology, and what failure scenarios I should test next.

I built Weir: an open-source semantic cache and model router for RAG I built Weir, an open-source gateway for RAG that combines semantic caching, model routing, and per-request cost accounting. The interesting challenge was making semantic caching safer. Two questions can have very similar embeddings but require different answers, so I added an entity guard to catch distinctions like ICU vs. general ward visiting hours. I evaluated four configurations on a fictional hospital FAQ dataset. On a repeat-heavy 300-request replay, full Weir reduced measured cost per 1,000 requests by 94% and median latency from 747 ms to 14 ms. On a cold pass, the cost saving was 20%. These are small synthetic benchmarks, not production results. I also documented an overload mode discovered during load testing, where cache timeouts caused more traffic to reach retrieval and generation. Repo: yatinannam/weir I'd appreciate technical feedback on the cache safety checks, evaluation methodology, and what failure scenarios I should test next.

I built Weir: An open-source semantic cache and model router for RAG I built Weir, an open-source gateway for RAG that combines semantic caching, model routing, and per-request cost accounting. The interesting challenge was making semantic caching safer. Two questions can have very similar embeddings but require different answers, so I added an entity guard to catch distinctions like ICU vs. general ward visiting hours. I evaluated four configurations on a fictional hospital FAQ dataset. On a repeat-heavy 300-request replay, full Weir reduced measured cost per 1,000 requests by 94% and median latency from 747 ms to 14 ms. On a cold pass, the cost saving was 20%. These are small synthetic benchmarks, not production results. I also documented an overload mode discovered during load testing, where cache timeouts caused more traffic to reach retrieval and generation. Repo: yatinannam/weir I'd appreciate technical feedback on the cache safety checks, evaluation methodology, and what failure scenarios I should test next.

I Built a Semantic Cache for RAG. The Hard Part Was Knowing When NOT to Cache. Every time a RAG application answers a question, it may need to retrieve documents and make an LLM call. But what happens when someone asks essentially the same question again? We could reuse the previous answer. The challenge is knowing when two questions are similar enough to share an answer—and when they only look similar. That's the problem I explored while building Weir, an open-source, cost-aware gateway for Retrieval-Augmented Generation (RAG). GitHub: yatinannam/weir What is Weir? Weir sits in front of an existing RAG service. It decides whether a request can safely use a cached answer or needs retrieval and generation, and chooses a model tier when generation is necessary. It combines three main ideas: - Semantic caching: Reuse answers for sufficiently similar questions. - Model routing: Use a cheaper model for suitable questions and a larger model when needed. - Cost accounting: Track actual request cost, counterfactual cost, and latency so savings can be measured. It also includes request logging, monitoring, evaluation tooling, and a Docker-based demo. The surprising problem with semantic caching Suppose a user asks: "What are the ICU visiting hours?" Later, another user asks: "What are the general ward visiting hours?" The questions are structurally similar. An embedding-based cache might assign them a high similarity score. But serving the ICU answer to the second user would be incorrect. This is why Weir uses an entity guard alongside embedding similarity. It checks distinguishing information such as numbers, negation, and important domain terms before accepting a potential cache hit. The goal isn't to cache as aggressively as possible. It's to save unnecessary work without silently returning the wrong answer. How I evaluated it I compared four configurations: - A baseline that always uses the larger model without caching - Semantic caching alone - Model routing alone - Full Weir, combining caching and routing The evaluation used a fictional hospital FAQ with 40 documents and 158 evaluation questions, including paraphrases, look-alike questions, and unanswerable questions. 1. Cold-pass evaluation Each of the 158 questions was asked once. Full Weir reduced measured cost per 1,000 requests from $0.0988 to $0.0786—a 20% reduction. The evaluation judge score and key-fact score remained unchanged against the baseline. The trade-off matters: median latency increased from 767 ms to 876 ms in this workload. Cost reduction does not automatically mean every workload becomes faster. 2. Repeat-heavy replay I also tested a 300-request replay in which 73.7% of requests were repeats. | Metric | Baseline | Full Weir | | --- | ---: | ---: | | Cache hit rate | 0% | 93.0% | | Cost per 1,000 requests | $0.0970 | $0.0055 | | Median latency | 747 ms | 14 ms | | Judge score | 4.79 | 4.79 | | Key-fact score | 0.947 | 0.948 | On this workload, the measured cost reduction was approximately 94%, and median latency was about 50 times lower. The important qualification is that these are synthetic, repeat-heavy benchmark results—not proof of equivalent savings on a production workload. What about incorrect cache hits? At the chosen similarity threshold, embedding similarity alone accepted some dangerous look-alike matches in the evaluation set. Adding the entity guard reduced wrong accepted cache hits to zero on that test set. That is encouraging, but it is not a guarantee of safety. A small synthetic dataset cannot cover every possible ambiguity or domain-specific distinction. What happened under load? I also tested the system under load, including provider failures and database delays. The tests surfaced a genuine overload mode: when cache lookups began timing out under CPU pressure, more requests bypassed the cache and reached retrieval and generation, adding further load. I documented that limitation rather than presenting the load test as an unconditional capacity claim. Try it yourself Weir includes a one-command demo launcher: uv run demo.py You'll need Docker Desktop and uv. A Groq API key is optional; without one, the demo uses a stub model, so generation is simulated while the cache, retrieval, guard, and routing components remain real. The repository includes the implementation, evaluation reports, architecture, limitations, and instructions for running the tests. What I'd love feedback on I'm particularly interested in feedback from people building RAG applications or LLM infrastructure: - What failure cases would you add to the semantic-cache evaluation? - How would you make the entity guard more robust across domains? - What real-world workload would you use to test whether these savings translate to production? I'd appreciate technical criticism, benchmark suggestions, and contributions.