Most of us have hit this at least once. A test fails, you dig in for an hour, and the bug isn't in your code at all. It's in the fixture. The expected value was copied wrong, or the input was never valid to begin with. The suite is red, your code is arguably right, and nothing in the system can say so.
That failure mode is the subject of one of my IETF Internet-Drafts, Verification Defect Quarantine for Protocol Conformance Systems (draft-watts-verification-defect-quarantine-01). Here's what it proposes, what it isn't, and what's still open.
Status first
This is an individual Internet-Draft. It isn't adopted by any IETF working group, it isn't an RFC, and it doesn't carry IETF consensus or endorsement. Its intended status is Informational, and like every Internet-Draft it's a work in progress that expires (this revision on 3 May 2027). The -01 revision changed only editorial and tooling items from -00. Nobody has independently reviewed or implemented it as far as I know, so please read everything below as a proposal, not a standard.
The problem: when the oracle is wrong
Conformance testing usually treats a published test vector as the source of truth. You feed in the input, compare against the expected output, and pass or fail. The draft's point is that this breaks in two directions when the vector itself is defective:
- A verifier that correctly rejects a bad vector looks non-conformant.
- An implementation that faithfully reproduces the bad value looks conformant.
The second one is the dangerous one, because it turns a bug in the spec into a false PASS that can flow into release, procurement or authorization decisions. The draft notes that test-vector errors have happened in published RFCs too.
The core idea: two statuses, not one
The main move in the draft is to stop collapsing everything into a single result. It keeps a subject status (about the thing being tested: PASS, FAIL, INDETERMINATE, DEFECT_REPORTED, DEFECT_REPRODUCED) separate from a verifier status (about the checker's behavior: PASS, FAIL, INDETERMINATE, DETECTION_PASS).
So when a verifier catches a broken vector, two things are true at once. That vector is negative evidence as a conformance oracle, and the verifier gets narrow credit, DETECTION_PASS, for catching that one specific fault. The draft says these two conclusions must not be merged. It also lists a few "non-collapse" rules I find useful even outside the draft:
- A failing normative vector doesn't, by itself, prove an implementation bug.
- Catching a defective vector doesn't make the subject conformant.
- Catching one defect doesn't certify the verifier as secure or complete.
- A local defect report isn't independent reproduction.
- INDETERMINATE is not PASS.
Defect classes and quarantine
The draft names six defect classes so a dispute gets a label: precondition defect, vector mismatch, ambiguous specification, unsupported expectation (no reproducible derivation), plain implementation failure, and INDETERMINATE when the evidence can't yet locate the fault.
When a normative vector is shown to be defective by a reproducible counterexample, it goes into quarantine. That doesn't delete it. It means the vector can't be used to justify a stronger assurance claim until it's fixed, superseded, or explicitly scoped out by whoever relies on the result. Corrections are append-only: a later revision gets a new receipt that points at the old one, and historical receipts must not be silently rewritten.
The record is a machine-readable receipt naming the subject digest, vector, defect class, both statuses, quarantine scope, the counterexample witness and a timestamp. JSON receipts should be canonicalized with RFC 8785 (JCS) before hashing. The draft is explicit that holding a receipt grants no authorization, deployment approval or standards status, and that logging it in a transparency service proves the statement was included, not that it's true.
A concrete example
The most developer-friendly part is the arithmetic profile. For elliptic-curve vectors, a verifier should check that a claimed point actually satisfies the curve equation, y^2 = x^3 + ax + b (mod p), before running any higher-level group math. If the two sides aren't equal mod p, the input is defective, and those two modular values are a compact witness anyone can recompute. The draft leans on RFC 6090 for the underlying arithmetic and says plainly that it defines no new cryptography. Passing these checks shows only the arithmetic relation, not security or identity.
What it doesn't show
There's no reference implementation or measured result in this post, and I'm not claiming one. The draft says a conforming implementation must reproduce companion positive and negative vectors; I haven't published an independent run of those here, and nobody outside has reproduced them. Separating subject and verifier results, labeling defect types and freezing disputed fixtures aren't new ideas on their own. Errata processes, flaky-test quarantine and test-oracle research cover nearby ground. What the draft tries to add is a small, shared record format and a fail-closed rule set for this one case.
The draft also flags a real downside: quarantine can be abused as denial of service. Anyone can file a "this vector is wrong" report to block a release. Its answer is to scope quarantine to the smallest justified unit, distinguish local reports from independently reproduced ones, and allow human review before release-wide quarantine.
Open questions
- What counts as "independent" reproduction? The draft leaves that to registered requirements per deployment, which may be too loose for interop.
- Should DETECTION_PASS ever count toward verifier qualification, or stay purely informational?
- Is a separate receipt worth the overhead compared with a structured errata report plus a test-suite flag?
If you maintain a conformance suite or have been burned by a wrong fixture, I'd like to hear how you handled it and where this model would get in your way. Corrections to the draft are welcome on its Datatracker page.