Why Production Agents Are Ditching Pure RAG for Context Caching
Three months ago, our agentic workflow hit a cost wall. We were running a multi-step research agent that performed iterative document analysis. Every user request triggered between 12 and 18 agent steps. Across 50,000 active daily sessions, our vector database query costs and LLM input token bills were spiraling out of control—and worse, median step latency hovered around 4.2 seconds.
Like most teams building in 2024 and 2025, we assumed the fix was "better vector search." We tweaked top-k parameters, tried hybrid dense-sparse reranking, and chunked documents down to tiny 256-token fragments. It didn't fix the core problem. The bottleneck wasn't retrieval quality; it was retrieval architecture.
In production agentic systems, replacing pure vector retrieval-augmented generation (RAG) with long-context prompt caching (Cache-Augmented Generation or CAG) combined with lightweight local router models reduces median agent step latency by up to 68% and cuts monthly API inference spend by over 80%. This architectural pattern leverages deterministic context reuse over fragment chunking.
TL;DR / Key Engineering Takeaways
- Vector RAG introduces compounding latency in agent loops: Each agent hop requires vector embedding, similarity search, top-k ranking, and prompt assembly, multiplying network overhead across multi-turn sessions.
- Context Caching eliminates redundant prefix processing: Modern API providers allow long documentation, system schemas, and conversation histories to be cached at the edge, reducing token input costs by 75%–90% on cached prefixes.
- Hybrid Local Routing stops token leakage: Routing lightweight classification, routing, and tool-selection decisions to local 8B–30B models reserves heavy frontier models only for final synthesis.
- The 2026 Agent Standard is hybrid: Vector search handles sparse global discovery across millions of items, while context caching handles the dense session working memory.
The Production Bottleneck: Why Vector RAG Struggles in Agent Loops
When we first built our agentic pipeline, we followed the standard textbook pattern. Every agent step extracted state, searched a vector index, pulled the top-5 chunk snippets, and fed them into the prompt window alongside system instructions.
It worked great in prototypes. But once we moved to multi-agent orchestration—where agents delegate sub-tasks, summarize intermediate progress, and call tools repeatedly—the flaws became obvious.
[ Traditional Vector RAG Agent Step ]
State -> Query Gen -> Embed Query -> Vector DB Search -> Top-K Rerank -> Build Prompt -> LLM Inference
Total Step Latency: ~3.8s - 5.2s (Repeated for 15 steps = ~60s total session time)
[ Context Caching / CAG Agent Step ]
State + System Context (Pre-Cached Prefix) -> LLM Inference (Cached Hit)
Total Step Latency: ~1.1s - 1.6s (Repeated for 15 steps = ~18s total session time)
In an agentic loop, context is not static; it evolves incrementally. In our earlier writing on what an agent actually is, we broke down how an agent's harness relies on continuous feedback loops. Re-embedding queries and assembling fragmented chunks at every step creates three major friction points:
- Lost Semantic Coherence: Chunking long technical specifications or codebase trees into 500-token snippets strips away critical cross-document relationships.
- Double-Payment on Tokens: You pay vector database read costs and re-pay LLM input token processing fees for text the model already parsed two steps ago.
- Compounding Latency: Adding 300ms vector index latency across a 15-step agent execution loop adds nearly 5 seconds of pure overhead to every user request.
Enter Context Caching (CAG): How Deterministic Prefix Reuse Works
Context Caching changes the economic math entirely. Instead of searching a vector database for relevant fragments on every turn, we load entire documentation bundles, domain schemas, or codebases directly into the LLM's extended context window (128k to 1M tokens) and cache the KV-cache states at the provider layer.
When the agent executes subsequent steps, the model processes only the new delta tokens (e.g., tool call outputs or user turns). The static prefix—which contains 95% of the total tokens—is retrieved instantly from the cached KV state.
+-------------------------------------------------------------------------+
| CACHED PREFIX (Static - 95% of Tokens) |
| System Persona + API Schemas + Entire Core Codebase / Knowledge Base |
| (Cost: ~90% Discount / Latency: ~50ms TTFT) |
+-------------------------------------------------------------------------+
| UNCACHED DELTA (Dynamic - 5% of Tokens) |
| Current State + Latest Tool Execution Result + User Query |
+-------------------------------------------------------------------------+
As we explored in our breakdown From RAG to CAG: A Smarter Approach to AI, shifting from retrieval-on-demand to context pre-loading simplifies system design. You eliminate vector index maintenance, embedding pipelines, and chunking heuristics.
Empirical Benchmark: Vector RAG vs. Context Caching vs. Local Hybrid
To measure the exact performance difference, we benchmarked three architectural approaches across 1,000 multi-step technical research tasks (averaging 12 agent turns per task, analyzing a 100,000-token technical repository):
| Metric | Standard Vector RAG (Top-K=10) | Pure Context Caching (CAG) | Hybrid Router + Context Caching |
|---|
| Median Step Latency (TTFT) | 3,840 ms | 1,210 ms | 890 ms |
| Total Session Latency (12 turns) | 46.08 s | 14.52 s | 10.68 s |
| Input Token Cost / Session | $0.184 | $0.038 | **$0.021** |
| Task Accuracy / Completion Rate | 78.4% | 91.2% | 92.6% |
| Vector DB Read Cost / 1k Sessions | $12.50 | $0.00 | **$0.00** |
Benchmark Takeaways:
- Latency: Context Caching dropped Time-to-First-Token (TTFT) by 68.4% compared to traditional vector RAG because the prompt prefix didn't require re-computation.
- Cost: Token costs dropped 79.3% under pure CAG and 88.5% under the Hybrid Router approach because cached tokens incur a fraction of standard input pricing.
- Accuracy: Full-context availability eliminated RAG retrieval failures (where vector search missed the relevant chunk due to poor keyword matching or semantic drift).
Architectural Deep Dive: Building the Hybrid Local Router Pattern
While Context Caching is powerful, pre-loading 100k tokens into a frontier LLM on every step can still be wasteful if the agent only needs to make a binary decision or select a simple tool.
To optimize latency further, we introduced a Local Hybrid Router.
How the Hybrid Router Architecture Works
+----------------------------+
| Incoming User Request |
+----------------------------+
|
v
+----------------------------+
| Local Router (8B / 30B) |
| (Tool Choice / State Check)|
+----------------------------+
/ \
/ \
Simple Action / Tool Call Complex Reasoning / Synthesis
/ \
v v
+-----------------------+ +------------------------------------+
| Local Execution Loop | | Frontier LLM with Context Caching |
| Fast & Zero-Cost API | | Cached KV Prefix (100k+ Tokens) |
+-----------------------+ +------------------------------------+
- Step 1: Local Triage: Incoming step state is evaluated by a small local model (e.g., an 8B model running on local GPU infrastructure or edge endpoints).
- Step 2: Simple Execution: If the step is a mechanical tool invocation (e.g.,
git log, read_file, check_status), the local model executes it directly.
- Step 3: Frontier Synthesis: If the step requires deep reasoning or complex synthesis across the codebase, the request routes to the frontier LLM backed by the cached KV prefix.
This hybrid approach mirrors the principles we outlined in our guide on real-time ranking systems at scale: decouple high-frequency lightweight operations from heavy, resource-intensive compute paths.
When Vector RAG Still Wins (And When to Avoid CAG)
Context Caching is not a silver bullet. During our production migration, we identified explicit boundaries where pure CAG fails and vector RAG remains necessary:
1. Corpus Size Exceeds Window Capacity
If your domain knowledge base spans millions of documents (e.g., 500 million tokens of legal case law), pre-loading the entire corpus into context is impossible or prohibitively expensive.
2. Low Prefix Reuse Ratio
Context caching relies on prefix stability. If user queries are entirely distinct and share no common system prompt, documentation context, or conversation history, you pay cache-creation surcharges without reaping cache-hit discounts.
3. Dynamic Write-Heavy Databases
If your knowledge base updates thousands of times per minute, invalidating the cached KV prefix on every write negates the performance benefits of caching.
Lessons Learned & Production Migration Checklist
If you are currently operating a production RAG pipeline and struggling with latency or costs, here is the incremental migration path we recommend:
- [x] Audit your prompt prefix stability: Group your agent prompts into static prefixes (system instructions, tool definitions, reference docs) and dynamic suffixes (user query, latest message).
- [x] Enable provider-level prompt caching: Activate KV-caching on your primary LLM endpoints (Claude, Gemini, or OpenAI) to instantly capture 50%–75% input cost savings on static prefixes.
- [x] Implement a hybrid retrieval gate: Use vector search only for initial broad document discovery (filtering 10M tokens down to 100k tokens), then cache that 100k-token working set for the rest of the agent session.
- [x] Offload simple tool routing to local models: Deploy a small local model to handle state-tracking and simple tool selections without waking up the frontier LLM.
Conclusion
Building reliable agentic systems requires moving past prototype patterns. Vector RAG served us well during the early era of short context windows, but in 2026, forcing multi-turn agents to constantly query vector databases is like making a developer query a database for every line of code they write.
By combining long-context prompt caching with intelligent local routing, you give your agents the entire working memory they need while slashing inference costs and step latency.
