I have sat in this incident more than once. A user reports a regression on Tuesday. On-call pulls the trace, and the span tree is complete: provider, model, tokens, finish reasons, retrieval chunks, eval scores. None of it carries the prompt version. The registry has 23 versions. The deploy log shows three prompt changes in the last 48 hours. Which one regressed? Nobody knows. On-call rolls back the most recent change at random, the regression continues, and three hours later the right version turns up by hand-checking each release. The trace was wired, the registry was wired, and the link between them was missing.
That link is what I want to talk about: the handful of span attributes that turn a prompt registry and a tracing setup into a closed loop. It works the same whether your registry is a hosted product, an open one, or a Git directory with a thin resolver in front of it. This is one of the cheapest pieces of reliability discipline you can add, and most teams skip it until an incident forces the issue.
The minimum link is a few attributes
Standard OpenTelemetry GenAI conventions cover the model, provider, token, and operation fields. They do not cover prompt-registry semantics. So I keep the gen_ai.* namespace for the spec fields and add a small prompt.* namespace of my own:
prompt.id — the stable identifier from the registry.prompt.version — the versioned tag, semver or sequential.prompt.variant — the A/B variant value, when there is one.prompt.template_hash — a content fingerprint, so drift inside the same version is visible.
Add prompt.locale, prompt.tier, and a fallback boolean if your resolver has those inputs. If you ship one attribute, ship prompt.version. Everything downstream depends on it, and without it the production debug loop is broken.
Why the link is worth the discipline
It closes three gaps I see in every prompt-managed system.
- Regression attribution. With the version on the span, "which change caused the drop" is one query: filter by
prompt.version, slice by score. Without it, you walk the deploy log by hand, which is the Tuesday I opened with.
- Rollback speed. If prompt bodies live in application code, fixing a bad one is a code deploy. With a registry plus version attributes, rollback is promoting an older version to current, and it takes effect as in-process caches expire.
- Eval-production drift. Your eval set runs v17, production runs v19 because someone shipped a flag rollout, and offline scores stop predicting online behavior. Version attributes let the eval pipeline join production traces by version, so the gap is visible instead of a surprise.
Set the attributes at the resolver, and copy them onto the LLM span
The resolver is the function that picks the version, so it is the natural owner of the attributes. It sets prompt.id, prompt.version, prompt.template_hash, and the variant and cohort fields on the current span.
The trap that cost my team time: OTel child spans inherit context, not attributes. The LLM call almost always opens its own child span, so the attributes you set on the resolver's parent span do not flow into the LLM call span on their own. Set them again on the LLM span, directly or with a small wrapper that copies the fixed list of prompt.* keys. Skip that and the span that matters for attribution is the one missing the version, which is the worst place to lose it.
For an A/B test, the flag platform resolves the variant at request time, the resolver consumes it, and the span carries it as prompt.variant. Keep prompt.variant and prompt.version distinct: the variant is which experimental path the user was on, the version is the actual prompt body the model saw. Both belong on the span, and the version is what attribution runs on.
Gradual rollouts work the same way. A move from v22 to v23 across 5, 25, and 100 percent of traffic shows up in the trace store as a shifting distribution of prompt.version values, and the dashboards reflect it in real time.
Eval replay means calling the same resolver
The pattern I have seen fail is evaluating each version in isolation against a static set. That misses the resolver's own contribution to quality. Instead, have the eval set carry the same condition inputs as production (cohort, locale, intent, variant), and have the eval runner call the same resolver with those inputs. Every eval span then carries the same prompt.version, prompt.variant, and prompt.template_hash a production span would.
That buys two things I care about: eval and production traces you can join by version, and a pre-deploy signal, since running the eval set with v24 selected predicts the online change before you promote it.
Drift alerts are per version, sliced by cohort
The signal that matters is the rolling-mean rubric score per prompt.version. When the rolling mean for v23 on citation grounding drops from 0.91 to 0.78 over four hours, the alert fires. A few things I have learned running this:
- Use a short window (around 1 hour) for fast drifts and a 24-hour window for slow ones.
- Set the threshold per rubric. Citation accuracy should be tighter than verbosity.
- Slice by
prompt.tier and prompt.locale. A regression on enterprise plus one locale hides in the global aggregate.
- If you automate rollback on a threshold crossing, keep a human in the loop for the first quarter you run it. I would not hand that to automation on day one.
Once the link is in place, rollback is operational rather than architectural. The alert fires on v23, on-call checks the registry for the previous stable version (v22), the registry's promote action makes v22 current, new requests resolve to v22 as caches expire, and the alert clears within the window. No deploy, no PR, no code review. The one rule I hold to: the promote action needs the same authentication and audit as a production deploy, because a rollback is still a production change.
The mistakes I see most often
- No
prompt.version attribute. Nothing else works without it.
- Setting attributes only at the resolver. The LLM call span loses them.
- Treating
prompt.variant and prompt.version as the same thing. They are not; both belong on the span.
- No
prompt.template_hash. Content drift inside the same version goes undetected.
- Evaluating versions in isolation. Misses the resolver's contribution.
- Rollback by code deploy. Slower than a registry promotion when versioning lives in the registry.
- No per-version drift alerts. Aggregate alerts miss per-version regressions.
- Cohort slicing forgotten. A regression on one tier and locale hides in the global aggregate.
- No documented schema. Attribute names diverge across teams within months.
The point
The registry side of this is common now, and standard tracing is common now. The link between them is the part teams skip, and it is the part that decides whether a Tuesday regression takes one query or three hours to pin down. Pick one attribute schema, own it in the resolver, put the version on every LLM span, and treat the resolver and schema as code that gets reviewed in a PR like anything else. It is a small amount of plumbing for the difference between guessing and knowing.
