Most teams check an LLM-assisted decision system for fairness by comparing approval rates across groups. That check can pass cleanly while the model is still changing its mind on individual cases for reasons nobody would defend out loud. Recent fairness-stress work on LLM decision-making points at a better default test - and it's a decision you can make before writing any code.
Aggregate Parity Nets Out; Per-Case Instability Doesn't
The choice is between two measurements. Outcome parity asks: across 1,000 applications, did group A get approved at roughly the same rate as group B? Counterfactual stress asks something narrower and harder: for this one application, if I change only a decision-irrelevant field - the name, the phrasing of the same request, the order of the bullet points - does the answer move?
A model that favors one group on early-career cases and another on senior cases can show perfect parity overall. Parity is cheap and it's what regulators often ask for, but it averages. Per-case stress testing catches that, because it never aggregates across people - it compares a case to itself.
The trade-off is real. Stress testing costs N× the inference calls, and it requires someone to declare which fields are decision-irrelevant. That list is a policy judgment, not an engineering one, which is exactly why a PM should define it.
Running the Test on One Application
base = {"years_exp": 6, "skills": "python, sql"}
names = ["Alex Chen", "Aisha Khan", "John Smith", "Priya Patel"]
votes = [decide(tmpl, {**base, "name": n})
for n in names
for tmpl in PROMPT_TEMPLATES] # 3 paraphrases, same instruction
flip_rate = 1 - max(votes.count(v) for v in set(votes)) / len(votes)
Twelve variants of one application that a human reviewer would call identical. Any flip_rate above zero means the decision moved on something that shouldn't matter.
The number alone isn't the point - the pattern is. Read the flips two ways:
- Clustered - all rejections land on the same name group across all three paraphrases. That's bias. Fix it in the prompt, the retrieval data, or the model choice.
- Scattered - flips appear randomly across names and templates. That's instability. Lower the temperature, constrain the output format, or stop auto-deciding borderline cases.
Both fail the same test and need different fixes, which is why one aggregate percentage can't tell you what to do next. Report worst-case flip rate per application, not the mean - the mean hides the cases that would end up in a complaint.
Key Takeaways
- Group-level parity can look clean while individual decisions flip on irrelevant details; counterfactual stress testing compares each case against itself and catches what averages hide.
- Separate clustered flips (bias - fix prompt, data, or model) from scattered flips (instability - fix sampling, output constraints, or add human review).
- The hard part isn't the code, it's agreeing on which input fields are decision-irrelevant. That list belongs to product and policy, not just engineering.
Which fields in your decision prompt would your team actually agree are irrelevant - and how long would that meeting take?
Sources referenced: FairStress: A deeper evaluation of fairness in LLM decision-making, Hugging Face Blog
