Why unrouted, uncached, unmetered model calls are the real reason your AI feature got expensive
Here's a bug most teams never file. Your support bot answers "what's your refund policy" for the two-hundredth time this week, and every single one of those calls hits the same frontier model, at the same price, as a genuinely hard multi-step reasoning request. Nothing in your stack treats those two calls differently. That's not a budget problem. That's a routing problem, a caching problem, and a metering problem wearing a budget problem's clothes.
If you've shipped an LLM feature and watched the bill climb faster than usage, this is usually why.
Where the money actually leaks
Four patterns account for almost all the waste:
- No routing. Every request, easy or hard, goes to the same model. A one-line classification answer costs the same as a chain-of-thought task.
- No caching. Semantically identical prompts, worded differently, all reach the model. "How do I return an item" and "what's your refund policy" are the same question and two separate bills.
- No limits. A retry loop, a runaway agent, or one overactive user can burn through tokens with nothing to stop it until someone notices the invoice.
- No attribution. Nobody can say which team, app, or feature is driving spend, so there's nothing concrete to fix.
Each of these has a direct fix, and none of them require touching your application code, because the fix belongs at the layer that sits between your app and the model provider: an AI gateway.
The fixes, in order of impact
1. Route by complexity, not by default. A gateway can classify incoming requests and send simple ones to a cheap model while reserving the frontier model for tasks that actually need it. This is the single biggest lever available. The RouteLLM research from LMSYS and UC Berkeley showed a trained routing classifier cutting inference cost by more than half on standard benchmarks without a meaningful drop in output quality, and most production gateways report savings in a similar range once routing is tuned to real traffic.
A minimal routing rule looks something like this:
routes:
- match:
max_estimated_tokens: 200
task_type: [classification, short_summary, faq]
model: gpt-4o-mini
- match:
task_type: [code_generation, multi_step_reasoning]
model: gpt-4o
- default:
model: gpt-4o-mini
The gateway evaluates this before the request leaves your infrastructure. Your app still just calls the gateway endpoint.
2. Cache by meaning, not by string match. A semantic cache uses embedding similarity to catch paraphrases, so two differently worded questions can still return the same stored answer without a second model call. Production systems commonly see cache hit rates in the 20 to 40 percent range, and every one of those hits costs nothing.
3. Set token budgets, not request-count limits. A request-count limit doesn't care if a single call generates 50 tokens or 50,000. A token budget does. Enforce it per user, per app, and per agent session, and a broken loop stops itself instead of showing up as a line item next month.
4. Attribute every request. Tag each call with its source (API key, service, team) and roll it into a per-team, per-model spend report. Attribution isn't a savings mechanism on its own, it's what makes every other fix targetable instead of a guess.
Agents make this worse, fast
A single chat completion is one line item. An agent is a chain of them: tool calls, retrieval steps, sub-model invocations, all stacked before it returns an answer. Without limits on how many tool calls an agent can make per session and how much context it can pull in, a support-ticket agent can quietly rack up a multiple of the expected cost. This is also where cost and security overlap, since an uncapped agent is both a budget risk and an attack surface. AgentSecurity's guidance on agent tool governance is a useful reference if you're building agent permission boundaries alongside cost controls.
Gateways that operate at the Model Context Protocol layer can apply the same budget logic to tool calls that they apply to model calls, which closes this gap without adding code to every agent you ship.
Putting it together
None of these four fixes require rewriting your app. They require a control layer your requests already pass through. TrustGate, NeuralTrust's open-source gateway, implements routing, semantic caching, token budgets, and cost attribution at that layer, so the savings apply to every app and every model in your stack from one deployment rather than being bolted onto each service individually.
If you want the longer version with benchmark numbers and a full breakdown of fallback chains, the original NeuralTrust write-up covers it in more depth.
