I keep seeing the same failure land on teams that do everything right. The CI eval gate has been green for six weeks, faithfulness at 0.87, task completion at 0.91. Then a support thread surfaces a hallucination on a feature shipped in March. The trace looks clean: well-formed query, plausible retrieval, wrong answer. The failure is not in the eval set. Eight percent of this month's traffic is asking about that feature, and the golden set has zero examples of it. The eval was never wrong about what it covered. It was covering the wrong product.
That is eval dataset drift, and I think it is the most under-managed risk in an LLM eval stack. Your golden set was representative the day you built it. Since then the production query mix has shifted, the prompt template has moved, and the retrieval corpus has rotated, and the eval keeps grading a frozen snapshot of last quarter. It ages in three places, and the score never tells you.
Why the eval score never warns you
A golden set is a snapshot. Production is a distribution. On the day you author it, the snapshot is a representative sample of that distribution. Every day after, the distribution moves and the snapshot does not, and the score stays flat because the eval is still grading the snapshot against itself.
New product surface adds intents you never anticipated. Campaigns and regional rollouts shift the persona and language mix. Users find prompt patterns the curators never imagined. The uncomfortable part is that none of it breaks the API or moves the dashboard. You find out from a customer, which is the most expensive way to learn it.
Drift 1: production prompts move away from the golden set
This is the one you can picture before you measure it. Your golden set has 200 prompts across the intents you knew about when you built it. Production users find an intent you did not anticipate, a campaign shifts the persona, a regional rollout brings a language that tokenizes differently. An edge case that was 0.5 percent of traffic on launch day is now 8 percent.
The signal is embedding distance. Compute one centroid for the golden set on the same embedder you use for retrieval, keep a rolling 7-day centroid over production traffic, and track the cosine distance between them. When it crosses two standard deviations of a 30-day baseline, the distributions have separated and the set has stopped covering the long tail.
For sharper attribution, tag traces with an intent label and compute KL divergence against the golden set's intent mix, or cluster the production embeddings nightly and watch for a new cluster with no golden-set neighbor. Fifty traces in a brand-new cluster is a coverage gap you can file before users complain.
Drift 2: the prompt wrapper changes and the dataset does not
This is the one that hides best, and the one I have watched bite the most teams. Your golden set is 200 curated inputs, but what the model actually sees is those inputs wrapped in a system message, a few-shot block, and a tool schema. That wrapper is the prompt template, and it is not part of the dataset.
Here is how it goes wrong. You ship a system-message tweak on Friday to fix a tool-call issue. The fix is good. The eval set still wraps inputs in last quarter's template, so Monday's CI run grades the dataset under a contract production no longer uses, and the score stays plausible because the rubric was generic enough to grade either one.
Detection is simple: hash the active template every night and compare it against the template hash stored with the eval set, then alert on a mismatch. The fix is dataset versioning. Pin every dataset version to the template hash it was written against, and when the template ships, cut a new dataset version and re-baseline the gate. Keep the previous version around for a release cycle so you can compare the two scores and tell model behavior apart from dataset decay.
Drift 3: the retrieval corpus moves under a RAG eval
This is the RAG-specific one, and it moves the score least while it happens, which is exactly why it is dangerous. Your golden set has fixed inputs and expected outputs, but the retrieval corpus is not in the dataset. The index grows, the chunker re-embeds on a new model, sources rotate, all on a cadence the dataset never sees.
Concretely: the retriever you evaluated in March indexed 12,000 documents at chunk size 800. By May the index has 38,000 documents on a fresh embedder, and the same golden-set query lands on different top-k chunks. The generator grounds faithfully in whatever it was handed, so groundedness still scores 0.94, and the user gets a confident answer from a document that does not apply.
The signal is overlap. Snapshot the retrieved chunk IDs for every golden-set query when you author the dataset. On each refresh, re-retrieve and compare the new top-k against that baseline, and alert when overlap drops past roughly 40 percent. Score the retrieval rubrics (context relevance, chunk attribution) separately from the generation rubrics (groundedness, factual accuracy) so you can bisect: context relevance dropping with groundedness steady means the retriever moved; the reverse means the generator did.
The monthly refresh, in four steps
Detection tells you the set has aged. The refresh is what you do about it, and I tell teams to run it as a standing cadence, not a fire drill: monthly by default, weekly under fast change, and immediately after a model swap or a corpus re-index.
- Pull 200 to 500 recent production traces, stratified by intent, persona, and language, from the last two to four weeks so you do not over-fit to a single campaign.
- Label them against the rubric already in CI, hand-corrected, with SME review where the domain is complex.
- Version the set as a new named dataset (
golden-v4-2026-06), pinning the template hash, embedder version, and retrieval snapshot to its metadata, and keep the previous version live for a cycle.
- Re-baseline the CI gate against the new set. The score delta between the old and new versions on the same model tells you how much of the last "regression" was dataset decay. Most teams find 30 to 50 percent of the movement between refreshes was the dataset, not the model.
The anti-patterns I see most
- Treating the golden set as ground truth instead of a versioned artifact, so it never moves and slowly becomes a snapshot of last quarter's product.
- No input-distribution monitoring, when the embedding compute is already running for retrieval and you can reuse it for the drift signal.
- Versioning prompt templates and eval sets in separate places, so a template ship lands while the dataset still grades the old contract.
- No retrieval-corpus snapshot, so RAG-side decay stays invisible until users complain.
The teams shipping reliable LLM products are not the ones with the biggest golden sets. They are the ones whose sets refresh on a cadence. Input drift moves the distribution, template drift moves the contract, corpus drift moves the grounding. Build the three baselines on day zero (embedding centroid, template hash, top-k snapshot), run the nightly comparison, and version every refresh. The drift will still arrive. That discipline is what decides whether you see it the sprint it shows up or the quarter after, and whether you hear about it from your dashboard or from a customer.
