Question

Semantic Caching for LLM Apps Without Serving the Wrong Answer

A customer asks your support assistant: "Can I get a refund if I cancel within 14 days?" Your semantic cache finds a stored question that is almost word-for-word the same — "Can I get a refund if I cancel after 14 days?" — scores it above your threshold, and returns the stored answer before the model is even called. The answer is fluent, confident, and exactly wrong.

That exchange is the whole risk of semantic caching. The idea itself is sound: when a new question means the same thing as one you have already answered, reuse the answer and skip the model — no output tokens, no generation latency. The naive build (embed the question, take the nearest neighbour, serve it above 0.9) is also how you ship wrong answers at scale, because embedding similarity measures how close two questions are in topic, not whether they have the same answer.

You can't tune your way out of that with a better number. The vCache paper (ICLR 2026) found that "correct and incorrect cache hits have highly overlapping similarity distributions, suggesting that fixed thresholds are either unreliable or must be set extremely high to avoid errors." Separate research on universal text embeddings found "a significant lack of negation awareness", with models "often interpreting negated text pairs as semantically similar." Negation, numbers and conditions are precisely the words that decide a support answer.

So this article treats similarity as a way to find candidates, never as the verdict: scoped keys, two thresholds, a deterministic guard for the words embeddings miss, a cheap verifier for the grey zone, invalidation tied to your sources, and a shadow-mode rollout that measures the false-hit rate before one cached answer reaches a user. It is the caching layer we cover when teaching how to ship an AI product to real traffic at Cursuri-AI.ro, written out in code.

Three caches that get confused

	Exact-match response cache	Semantic response cache	Provider prompt caching
Key	Hash of the normalized request	Embedding of the question, inside a scope	Exact prefix of the prompt
What it skips	The whole model call	The whole model call	Re-processing the cached prefix
Saves	Input, output, latency	Input, output, latency	Most of the input cost of the prefix
Can it return a wrong answer?	Only if the key is incomplete	Yes, by design	No, the model still generates
Lives in	Your infrastructure	Your infrastructure	The provider

Prompt caching isn't a competitor. It makes re-reading a stable prefix cheap: on Anthropic's current lineup a cache read costs 10% of the base input price, 5% on Claude Opus 5.5 and 2.5% on Claude Fable 5.1. But the model still runs and still writes every output token. A semantic cache skips the model entirely.

That shifts the business case. When your prompts already cache well, the input side of a request is cheap, and a semantic cache mostly buys you output tokens and latency. Use both layers: a semantic-cache miss still goes to the model with a cached prefix.

First, an expected-value check

A semantic cache has a benefit you can calculate and a cost teams usually leave out: the wrong answers it serves. Put both in one function before you build anything.

def monthly_value(
    requests: int,
    hit_rate: float,              # share of requests answered from cache
    false_hit_rate: float,        # share of cache hits that were wrong
    usd_per_generation: float,    # average cost of a fresh answer
    usd_per_wrong_answer: float,  # support ticket, refund, lost customer: your number
    usd_per_lookup: float,        # embedding call + index query
) -> tuple[float, int]:
    hits = requests * hit_rate
    wrong = hits * false_hit_rate
    value = (hits * usd_per_generation
             - requests * usd_per_lookup
             - wrong * usd_per_wrong_answer)
    return value, round(wrong)

Run it with illustrative numbers: 200,000 questions a month, a 25% hit rate, $0.02 per generated answer, and a wrong answer that costs $5 of support time. At a 2% false-hit rate the cache saves $1,000 of generation and serves 1,000 wrong answers worth $5,000: a net loss of about $4,000 before lookup costs. At 0.2% it serves 100 wrong answers and comes out about $500 ahead. The hit rate sets how much you can save. The false-hit rate decides whether you save anything at all.

Where the arithmetic tends to work: support and FAQ assistants, documentation Q&A, internal help desks, product questions — high repetition, long answers, one correct answer for everyone in a scope. That is the territory of AI-assisted customer support, and it is where the rest of this design pays for itself. Where it doesn't work: coding agents, anything personalized, multi-step tool use, creative generation (users expect variety), and questions about live state such as "is the API down right now?".

The design: similarity proposes, rules decide

Every request goes through the same six steps:

1. Normalize the question (case, whitespace; multi-turn chats are covered further down).
2. Scope it: compute a key from everything that can change the correct answer — tenant, locale, plan, model, prompt version, knowledge-base version. Lookups never cross scopes.
3. Exact match first: a hash lookup catches verbatim repeats with no similarity risk.
4. Nearest neighbour within the scope.
5. Guard: below the low threshold, a miss. Different critical tokens, a miss. Above the high threshold, a hit. In between, verify.
6. On a miss, generate the answer, then decide at write time whether it may be cached at all.

The table

Postgres with pgvector is enough for this; use Redis or a vector store if that is what you already run. Storage is not the hard part.

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE answer_cache (
    id              BIGSERIAL    PRIMARY KEY,
    scope_key       TEXT         NOT NULL,  -- hash of tenant, locale, plan, model, prompt and KB versions
    question        TEXT         NOT NULL,  -- normalized question
    question_hash   TEXT         NOT NULL,  -- exact-match layer
    question_sig    TEXT         NOT NULL,  -- critical tokens, see below
    embedding       VECTOR(1024) NOT NULL,  -- match your embedding model's dimension
    answer          TEXT         NOT NULL,
    source_doc_ids  TEXT[]       NOT NULL DEFAULT '{}',
    created_at      TIMESTAMPTZ  NOT NULL DEFAULT now(),
    expires_at      TIMESTAMPTZ  NOT NULL,
    hit_count       INT          NOT NULL DEFAULT 0,
    UNIQUE (scope_key, question_hash)
);

CREATE INDEX ON answer_cache USING hnsw (embedding vector_cosine_ops);
CREATE INDEX ON answer_cache USING gin (source_doc_ids);

One pgvector detail bites multi-tenant caches. As the README puts it, "with approximate indexes, filtering is applied after the index is scanned". With many scopes, the nearest neighbours the HNSW scan returns can all belong to other tenants, the WHERE scope_key = … filter removes them, and you get zero rows: a miss that isn't one. Turn on iterative index scans (pgvector 0.8.0+, SET hnsw.iterative_scan = relaxed_order), use partial indexes for a few large scopes, or partition when there are many.

The lookup

import hashlib
from dataclasses import dataclass

HIGH, LOW = 0.95, 0.88   # placeholders, not recommendations: calibrate them (see below)

@dataclass
class Candidate:
    id: int
    question: str
    answer: str
    similarity: float

def normalize(q: str) -> str:
    return " ".join(q.lower().split())

def lookup(db, scope_key: str, question: str) -> tuple[str, Candidate | None]:
    q = normalize(question)
    exact = db.fetchone(
        "SELECT id, question, answer FROM answer_cache "
        "WHERE scope_key = %s AND question_hash = %s AND expires_at > now()",
        (scope_key, hashlib.sha256(q.encode()).hexdigest()),
    )
    if exact:
        return "hit", Candidate(exact["id"], exact["question"], exact["answer"], 1.0)

    vec = embed(q)   # normalized vector; pgvector's psycopg adapter registered
    row = db.fetchone(
        """
        SELECT id, question, question_sig, answer,
               1 - (embedding <=> %s) AS similarity
          FROM answer_cache
         WHERE scope_key = %s AND expires_at > now()
         ORDER BY embedding <=> %s
         LIMIT 1
        """,
        (vec, scope_key, vec),
    )
    if row is None or row["similarity"] < LOW:
        return "miss", None
    if row["question_sig"] != critical_signature(q):
        return "miss", None          # same topic, different negation, number or condition
    cand = Candidate(row["id"], row["question"], row["answer"], row["similarity"])
    return ("hit" if cand.similarity >= HIGH else "verify"), cand

<=> is pgvector's cosine distance, so 1 - distance is the similarity you compare against thresholds. The function returns one of three states instead of a boolean, and that third state is the point of the design.

Guard the words embeddings miss

Embeddings are weakest exactly where support answers are most sensitive: negation, numbers and conditions. Within 14 days versus after 14 days, with versus without, EU versus US. A cheap, deterministic check runs before any model is involved: pull out the tokens that flip meaning, and require them to match.

import re

NEGATIONS = re.compile(r"\b(not|no|never|without|except|unless|cannot|can't|don't|doesn't|isn't|won't)\b")
NUMBERS = re.compile(r"\d+(?:[.,]\d+)?")
CONDITIONS = re.compile(r"\b(before|after|within|over|under|more than|less than|at least|at most)\b")
DOMAIN_TERMS = {"eu", "us", "uk", "basic", "pro", "enterprise", "ios", "android"}  # your vocabulary

def critical_signature(q: str) -> str:
    words = set(re.findall(r"[a-z0-9']+", q))
    parts = [
        ",".join(sorted(set(NEGATIONS.findall(q)))),
        ",".join(sorted(set(NUMBERS.findall(q)))),
        ",".join(sorted(set(CONDITIONS.findall(q)))),
        ",".join(sorted(words & DOMAIN_TERMS)),
    ]
    return "|".join(parts)

The refund example from the opening now fails the guard (within ≠ after) and becomes a miss, without any model deciding anything.

This check is deliberately blunt. It will turn some genuine paraphrases into misses ("I can't log in" and "login is not working" carry different negation tokens), and that is the right trade: a miss costs one generation, a false hit costs a wrong answer. Have whoever owns the product catalog own DOMAIN_TERMS too, so a new plan name doesn't quietly become a blind spot.

The grey zone: verify, don't guess

Between the two thresholds, ask a small, fast model one narrow question: does the stored answer fully answer the new question? Structured outputs keep the reply machine-readable.

import json
import anthropic

client = anthropic.Anthropic()
VERIFIER_MODEL = "claude-haiku-4-5-20251001"   # the smallest current model you trust

VERIFY_SYSTEM = (
    "You decide whether a stored answer can be reused for a new question. "
    "reusable is true only if the stored answer fully and correctly answers the new "
    "question: nothing the new question asks is missing, and no condition in the new "
    "question (dates, amounts, negations, products, regions) changes the answer."
)

REUSABLE = {
    "type": "object",
    "properties": {"reusable": {"type": "boolean"}},
    "required": ["reusable"],
    "additionalProperties": False,
}

def still_answers(new_question: str, cand: Candidate) -> bool:
    try:
        resp = client.messages.create(
            model=VERIFIER_MODEL,
            max_tokens=100,
            system=VERIFY_SYSTEM,
            output_config={"format": {"type": "json_schema", "schema": REUSABLE}},
            messages=[{"role": "user", "content": (
                f"<new_question>{new_question}</new_question>\n"
                f"<stored_question>{cand.question}</stored_question>\n"
                f"<stored_answer>{cand.answer}</stored_answer>"
            )}],
        )
        text = "".join(b.text for b in resp.content if b.type == "text")
        return json.loads(text)["reusable"] is True
    except (anthropic.APIError, json.JSONDecodeError, KeyError):
        return False    # any doubt is a miss, including an outage or a retired model ID

At $1/$5 per million tokens, Haiku 4.5 turns a verification into a short prompt and a one-field answer: a small fraction of a full generation on a frontier model, on the model Anthropic lists as its fastest. A verified hit is still slower than a clean one, since it adds a round trip, so narrow the grey zone once your data shows where it is. Keep the verifier's model ID in one constant, because small models get retired like any other, and notice that every failure path returns a miss. A verifier that fails open turns an outage into a stream of unchecked cache hits.

Put together, the request path is short:

def answer(db, request) -> str:
    scope = scope_key_for(request)
    state, cand = lookup(db, scope, request.question)
    if state == "hit" or (state == "verify" and still_answers(request.question, cand)):
        db.execute("UPDATE answer_cache SET hit_count = hit_count + 1 WHERE id = %s", (cand.id,))
        return cand.answer
    result = generate(request)                 # your normal model call
    if cacheable(result):
        store(db, scope, request.question, result)
    return result.text

def store(db, scope: str, question: str, result, ttl_hours: int = 24) -> None:
    q = normalize(question)
    db.execute(
        """
        INSERT INTO answer_cache (scope_key, question, question_hash, question_sig,
                                  embedding, answer, source_doc_ids, expires_at)
        VALUES (%s, %s, %s, %s, %s, %s, %s, now() + make_interval(hours => %s))
        ON CONFLICT (scope_key, question_hash) DO NOTHING
        """,
        (scope, q, hashlib.sha256(q.encode()).hexdigest(), critical_signature(q),
         embed(q), result.text, result.source_doc_ids, ttl_hours),
    )

source_doc_ids records which documents the answer was built from. It looks like bookkeeping until the day a policy page changes; the invalidation section below is where it pays off.

Calibrate thresholds on your own data

HIGH and LOW are not constants you copy from a blog post, this one included. They depend on your embedding model, your domain and your users. Measure them:

1. Take a few weeks of logged questions. For each one, find its nearest earlier neighbour in the same scope and record the similarity.
2. Label a few hundred of those pairs: same answer or different answer. Humans are best; an LLM judge with human spot-checks works if you audit it.
3. Pick the lowest threshold that meets your error budget, and look at how much traffic it leaves you.

def pick_threshold(
    pairs: list[tuple[float, bool]],    # (similarity, same_answer)
    max_false_hit: float = 0.01,
    min_support: int = 50,
) -> tuple[float, float] | None:
    for t in sorted({s for s, _ in pairs}):
        served = [same for s, same in pairs if s >= t]
        if len(served) < min_support:
            return None                  # too little data above this point to trust
        if served.count(False) / len(served) <= max_false_hit:
            return t, len(served) / len(pairs)   # threshold, share of candidates served
    return None

If it returns None on a few hundred labeled pairs, that is a finding, not a bug: no single threshold keeps your error budget on this data, which is exactly the overlap vCache describes. You then need the verifier for everything above LOW, or per-question thresholds, or no semantic layer at all (exact-match only). Treat this the way you would treat any retrieval quality problem: a labeled dataset and a metric, not intuition about what 0.9 "means".

Scope or leak: the cache is a cross-user channel

A semantic cache that serves user B an answer generated for user A is, by construction, a way to move data between users. If A's answer was built from A's account data, A's private documents or A's conversation, that is a data breach delivered with low latency.

Two rules close it.

The scope key includes everything that can change the answer. Tenant, locale, the user's plan (entitlements change answers), model ID, system prompt version, and a version of the knowledge base the answer was built from:

def scope_key_for(request) -> str:
    parts = [request.tenant_id, request.locale, request.plan,
             MODEL_ID, PROMPT_VERSION, KB_SNAPSHOT_ID]
    return hashlib.sha256("|".join(parts).encode()).hexdigest()

Model and prompt versions in the key mean a deploy that changes either one starts a fresh cache automatically, and the old entries simply expire.

Cacheability is decided at write time, from what the generation actually touched. Before generating, you often can't tell whether a question needs private data; afterwards, you know exactly what the model called and read:

def cacheable(result) -> bool:
    return (
        result.stop_reason == "end_turn"
        and not result.used_account_tools        # looked up this user's orders, invoices…
        and not result.read_private_documents    # retrieval hit anything not visible to the whole scope
        and not result.used_conversation_history
        and not result.flagged_by_guardrails
        and bool(result.text.strip())
    )

The same channel runs the other way, too. If someone can get a harmful answer generated once — a prompt-injected link, an offensive reply — a shared cache will serve it to everyone who asks something similar. Cache only answers that passed your output guardrails; consider promoting an answer to the shared cache only after the same question has come from several distinct users with no negative feedback; and show a "cached answer" marker with a regenerate button, which also gives you a false-hit signal at no extra cost. Poisoning, cross-tenant leakage and injection belong on the same threat model when you're securing an LLM application; the cache is just one more component that stores model output.

Invalidation: tie entries to their sources

A cached answer is correct until the thing it was based on changes. Three mechanisms, from coarse to precise:

• TTL by content class. Prices and policies: hours to a day. How-to documentation: days to weeks. Nothing lives forever.
• Versions in the scope key. A new model, prompt or knowledge-base snapshot means a new scope; nothing needs deleting.
• Purge by source. Store the IDs of the documents each answer was built from, and delete the dependent entries when one changes:

def purge_for_document(db, doc_id: str) -> int:
    return db.execute(
        "DELETE FROM answer_cache WHERE source_doc_ids @> ARRAY[%s]::text[]",
        (doc_id,),
    ).rowcount

Hook it to whatever already re-indexes your RAG corpus: the moment a document is re-embedded, the answers built on its old version go away. Of the three, this is the one teams skip, and it is the one that explains why the bot quoted last quarter's refund policy.

Multi-turn conversations

"And for the Enterprise plan?" means nothing without the previous turn, and embedding it on its own will match whatever else was asked about Enterprise. Two safe options: cache only first-turn questions that stand on their own, or rewrite each follow-up into a standalone question with a small model before lookup. The rewrite costs a call on every multi-turn request, which eats into the savings, so measure whether it pays. Never key the cache on the raw last message of a conversation.

Roll it out in shadow mode

Don't find out your false-hit rate from customers. Run the cache in shadow first: on every request, do the lookup, still generate a fresh answer, serve the fresh one, and log both.

def handle(db, request) -> str:
    scope = scope_key_for(request)
    state, cand = lookup(db, scope, request.question)
    result = generate(request)
    if state != "miss":
        log_shadow(request.question, cand, result.text, state)   # judged offline
    if cacheable(result):
        store(db, scope, request.question, result)
    return result.text

Offline, have a judge (with human spot-checks) decide whether each would-be cached answer was equivalent to the fresh one. After a week or two you have a measured false-hit rate per state, hit and verify, at your current thresholds, and monthly_value() stops being a guess. Switch serving on only when the numbers clear your budget, and keep a kill switch. In production, track four things: hit rate, measured false-hit rate (keep sampling), latency for hits and misses, and negative feedback on cached versus fresh answers.

Cached answers are personal data processing too

Anything you store about users' questions and your answers to them falls under the GDPR if it can relate to a person. Three consequences for the cache. Shared entries must never contain personal data, which the write-time rule above enforces. TTLs are your retention policy, so write them down as one. And if you keep any per-user cache, it belongs in your erasure flow: "delete my data" has to reach the cache, not only the main database. Designing data protection into AI systems from the start is cheap; bolting it on after an audit is not.

FAQ

What similarity threshold should I use for semantic caching?
There is no universal value. Similarities depend on the embedding model and the domain, and correct and incorrect hits overlap. Calibrate on labeled pairs from your own traffic, use two thresholds with a verifier in between, and accept that some workloads have no safe threshold at all.

Is semantic caching the same as prompt caching?
No. Prompt caching lives at the provider, discounts re-reading an identical prompt prefix, and still generates a fresh answer, so it can't return a wrong one. Semantic caching lives in your infrastructure and skips generation entirely. They stack: a semantic miss still benefits from a cached prefix.

Should I use Redis or Postgres for a semantic cache?
Whichever you already operate. Both can store vectors and run approximate nearest-neighbour search. The hard parts are scoping, thresholds and invalidation, and none of those depend on the store.

Can I semantically cache an AI agent's responses?
Not end to end. Agent runs call tools, read live state and cause side effects. Cache below the agent instead: exact-match caching of read-only tool results, with short TTLs, carries far less risk.

How do I know the cache is serving wrong answers?
Shadow mode before launch, then continuous sampling: a judge compares a sample of cached answers against fresh generations every week. Add a visible "cached" marker with a regenerate option, and watch whether users regenerate cached answers more often than fresh ones.

The short version

• Embedding similarity tells you two questions are about the same thing, not that they have the same answer.
• Do the expected-value check first: the false-hit rate decides whether the cache saves money at all.
• Scope every lookup by tenant, plan, locale, model, prompt and knowledge-base version. Decide cacheability at write time from what the generation touched.
• Guard negations, numbers and conditions deterministically, and verify the grey zone with a small model.
• Calibrate thresholds on labeled pairs from your own traffic, and expect overlap.
• Purge answers when their source documents change, roll out in shadow mode, and keep a kill switch.

A semantic cache built this way is less aggressive than the one in most tutorials. It serves fewer hits, and every hit it serves has cleared more than a similarity score.

---

I build and teach production AI systems at Cursuri-AI.ro, Eastern Europe's AI education platform — hands-on courses on shipping LLM features that stay correct, secure and affordable under real traffic.