Most of us here have shipped something that calls an LLM API, and the first surprise is usually the invoice rather than the model quality. Before you go shopping for a cheaper model, it is worth measuring how much of your traffic you are paying for twice.
The Overlap Nobody Measures
In support-shaped traffic, somewhere between 30 and 60 percent of incoming questions are identical or semantically equivalent to something already answered. Every one of those repeats makes a full API call, pays full price, and waits the full latency. A chatbot handling 10,000 conversations a day can spend a month re-answering tens of thousands of questions it already has a good answer for.
That is not a model selection problem. It is a cache miss problem.
Exact Match Caching And Where It Stops
The simplest version hashes the normalized prompt and stores the completion. Redis, a dict, whatever you already run. It is close to free to implement and it catches the literal duplicates, which in most systems is a thinner slice than people expect.
It stops the moment wording drifts. "How do I cancel" and "what is your cancellation process" hash to different keys, so both hit the model even though the answer is the same paragraph.
How Semantic Caching Changes The Math
Semantic caching embeds the incoming query, searches for a near neighbor above a similarity threshold, and serves the stored answer when one exists. That is where the bulk of the savings lives, because it collapses all the ways a human can phrase the same question down into one paid call.
The threshold is the whole design. Set it too tight and you are back to exact match. Set it too loose and you confidently serve the wrong answer, which costs far more than the tokens ever did.
What Breaks In Production
Invalidation is the hard part, same as it always was. Answers that reference pricing, availability or account state go stale silently, so entries need a TTL tied to how fast the underlying facts actually move.
Two more that bite. Cache entries are model scoped, so swapping models resets your hit rate to zero and week one looks worse than the sticker price suggested. And hit rate on its own is a vanity metric unless you also track what a wrong hit costs you.
The longer guide, covering cache types, architecture decisions, how to measure hit rate properly and the production issues in more depth, is at LLM Caching: Reduce Latency and Cost in AI Applications.
Takeaway
Measure your repeat rate before you optimize anything else. If a third of your calls are questions you have already answered, the cheapest token in your system is the one you never send.