Your custom metric is scoring noise unless it has these three parts
A medical agent passes your faithfulness and refusal gates all quarter. A week after launch, a review flags outputs with unverified drug-interaction claims. The faithfulness rubric checked whether the output stayed within the retrieved context, and the context contained the claims. The metric was right. It was the wrong question.
That is when you write a custom metric. Stock evaluators (faithfulness, BLEU, ROUGE, toxicity) cover the common surface. They do not cover your domain: a claim's modality matching a cited row, brand voice on a regulated tier, citations resolving across steps. The trigger is simple. If the stock scores are green and engineering still owns the on-call page, the metric is the problem.
A custom metric is a contract between the rubric and the judge, and it has three parts. Skip any one and you are scoring noise. There is a longer walkthrough with the calibration code if you want it.
1. A tight criterion: one rubric, one behavior
A vague rubric makes the judge invent the criterion on every call, so the score means something slightly different each time. "Rate helpfulness 1 to 5" is the biggest cause of low judge-versus-human agreement I see. Replace it with something behavioral: "Score whether the response answers the question in the first sentence, without restating it, without hedging words, and without asking to clarify when the intent is clear."
- Anchor every scale point to a concrete behavior. A free-form 1-to-5 leaks structure.
- Use the smallest scale that fits the decision. Binary for compliance, 5-point for semantic quality, continuous only when there is a structural rule (fraction of claims cited). Seven points over-resolve the judge and agreement drops.
- Return structured JSON (criterion, short reasoning, score). The reasoning tells you later whether the rubric or the model drifted.
- Add one line against verbosity: "do not reward longer responses." Kills most length inflation for free.
Version the rubric and keep it in source control. Edits are schema migrations.
2. A calibration corpus: proof it agrees with humans
A rubric without calibration is a subjective scorer with a JSON schema. Build 50 to 200 labeled examples per rubric (below 50 is too noisy, above 200 stops helping), stratified across failure modes rather than sampled randomly.
Label each with two humans and compute inter-annotator agreement first. Below 0.6, the rubric is ambiguous, not the labelers, so rewrite before you touch the judge. Then score the judge against those labels with weighted Cohen's kappa. Working bands: above 0.6 for a CI gate, above 0.8 for unattended automation. Hold out 20 percent so the rubric and the corpus cannot co-evolve into something that only fires on the labeled set.
3. A stability check: survive the judge swap
The judge is a prompt, not a function. A rubric calibrated on one model snapshot shifts on the next one, so the dashboard moves while the agent did not.
- Rotate across families (a GPT, a Claude, a Gemini judge) on the calibration set. If agreement is 0.8 on one and 0.55 on another, the rubric is leaking its phrasing into one model's prior. Tighten it until it travels.
- Pin the contract: judge model ID, rubric version, prompt hash. Bump deliberately, never as a side effect of a vendor swap.
- Recalibrate on a one-week SLA whenever the judge updates, the rubric changes, the traffic shifts, or rolling agreement drops for two windows.
Run the same rubric in CI and on live traffic from one definition. That single diff closes most of the offline-versus-production gap.
And put a deterministic layer in front of the judge: a regex catches a missing citation for free, a schema check catches a malformed response in microseconds, and the judge only runs on what the cheap checks cannot decide. On most workloads that cuts the judge bill 80 to 90 percent and never drifts.
- Vague criterion carrying three behaviors at once; agreement never breaks 0.5.
- No calibration corpus, so a 4.2 average gets trusted with no evidence it matches humans.
- Single-judge lock-in; the provider rolls a snapshot and the score moves on its own.
- Reaching for a judge when a regex works.
- One score across multi-dimensional quality; a response can be on-brand and factually wrong, and the scalar averages the failure. Score per dimension.
The rubric is not the moat. A rubric wired into calibration, a deterministic cascade, and quarterly recalibration is what still means something after the judge swap that happens every quarter.
