A Passing Test Can Still Prove Nothing

Question

Some of the most misleading tests are not flaky, slow, or obviously incomplete. They are fast, deterministic, and green.

They also pass for the wrong reason.

That is especially dangerous in fault-tolerant code. A test may claim that a failed dependency is contained, a fallback runs, or an established workflow continues. Yet its assertion may observe an outcome that can occur both before and after the recovery boundary. The suite reports confidence while the behaviour named by the test remains unproved.

The false-positive shape

Imagine a dispatcher with an optional policy source in front of an established workflow. The intended behaviour is straightforward:

1. Ask the policy source whether the old workflow should stand down.
2. If that optional source fails, contain the fault.
3. Continue into the established downstream path.

The unit test configures a policy source that throws. It also uses a strict downstream test double that throws if reached, because the rest of that workflow is outside the test's concern.

Then it asserts only that dispatch throws an exception.

At first glance, this can feel reasonable. An exception from the strict downstream dependency appears to prove that execution continued. But the same broad assertion also accepts the earlier exception from the policy source. If the recovery logic is deleted, the first exception simply escapes and the test remains green.

The assertion recognizes the shape of the final outcome, not the path that produced it.

Test the claim, not the punctuation

The claim is not “an exception occurs.” The claim is “a fault at this boundary does not stop progress into the established path.”

That distinction changes what the test must observe.

A useful test needs a stable witness on the far side of the boundary. Depending on the design, that witness could be:

• a recorded call to the downstream collaborator;
• a domain result that only the fallback path can produce;
• a distinct exception identity from a deliberately strict dependency;
• a state transition committed after the failed operation; or
• an emitted event that occurs only after recovery has completed.

The specific technique matters less than the causal guarantee. The witness must be impossible—or at least deliberately difficult—to observe if execution stops at the failure being contained.

In the example above, the corrected test distinguishes the downstream failure from the policy-source failure. It therefore passes only when execution crosses the recovery boundary. A recording test double would also work and may be clearer when the downstream call itself is the contract.

Why mutation testing helped

Code coverage could not expose this weakness. Both the recovery code and the test might execute during the normal run. Coverage tells us which lines ran; it does not tell us whether an assertion depended on their behaviour.

Mutation testing asks a sharper question: if a meaningful piece of behaviour is changed or removed, does a test fail?

Removing the recovery behaviour created a surviving mutant. That was valuable evidence. It did not necessarily mean the production design was wrong. It meant the suite could not distinguish the intended implementation from a broken one.

This is one of mutation testing's best uses: finding assertions that observe something adjacent to the claim rather than the claim itself.

You do not need mutation tooling to apply the same reasoning in review. Mentally invert the behaviour:

• If the catch disappeared, would the test fail?
• If the fallback were skipped, would the test fail?
• If execution stopped one step early, would the test fail?
• Could a different branch produce the same asserted result?

If the answer is uncomfortable, look for a witness beyond the boundary.

The trade-off: path awareness without brittleness

Path-sensitive assertions carry a real cost. A test that verifies every internal call order can become coupled to implementation detail and obstruct harmless refactoring. Replacing one collaborator with another should not require rewriting an entire suite if the observable contract is unchanged.

The answer is not to avoid path awareness altogether. It is to choose the smallest stable witness that represents the behaviour promised by the test.

Prefer domain-visible evidence when it exists: a returned decision, a committed state, a durable message, or a recorded recovery status. Use collaborator verification when crossing that collaborator boundary is itself the contract. Use exception type or identity when a strict test double intentionally marks progress, but make that intent explicit so a future reader understands why the exception matters.

This produces a useful hierarchy:

1. Assert the externally meaningful result when it uniquely proves recovery.
2. Assert a durable checkpoint when the final result is produced asynchronously.
3. Assert a boundary interaction when progress across that boundary is the behaviour under test.
4. Use a distinct failure identity as a narrow witness when a strict dependency deliberately terminates the test.

The closer the witness is to the real reliability contract, the less brittle the test will be.

A practical review habit

When reading a test name, underline the verb: continues, retries, falls back, persists, publishes, or rejects. Then find the assertion that proves that verb happened.

If you can only find evidence that something threw, returned, or logged, ask whether the same evidence could appear before the named behaviour occurred.

Green is not the standard. Discrimination is. A useful test must distinguish the system we intended from a nearby broken version.

For fault-tolerance tests in particular, do not merely observe the aftermath of a failure. Prove that the system crossed the boundary it promised to survive.