SRA-RiskGate

Question

I quantized my payment-safety model. Payments survived. Disputes didn't.

I maintain SRA-RiskGate, an open-source gate that checks a stablecoin payment before an AI agent signs it: approve, hold or reject. Part of it is a 4B model fine-tuned for two jobs: gating payments, and adjudicating disputes after a payment has settled.

Almost nobody runs the full-precision model. They run the 2.5 GB Q4_K_M build through Ollama on a laptop. Until this week my model card said what most cards say: "quantized builds have not been benchmarked." So I benchmarked them.

The setup

Same 2,000 held-out test cases as the published full-precision run (1,175 payments, 825 disputes), the exact training prompts, greedy decoding, and the benchmark's own scorer. Two builds: Q8_0 (4.3 GB) and Q4_K_M (2.5 GB), served by Ollama on a free Kaggle T4. Every prediction is published, so you can re-score any number below.

The results

	Full precision	Q8_0	Q4_K_M
Payment decision accuracy	99.5%	99.5%	99.3%
Unsafe approvals	0.47%	0.47%	0.83%
Legitimate payments blocked	0%	0%	0%
Dispute outcome accuracy	77.8%	78.6%	67.4%
Wrongful denials (owed refunds refused)	26.0%	23.4%	55.3%

Q8_0 is indistinguishable from full precision. Q4_K_M keeps the payment gate almost intact: it blocks no legitimate payment, and its three extra unsafe approvals are all cases where the payer's risk attestation was missing. But on disputes it falls apart: more than half of the refunds that were owed got denied.

What actually broke

I expected compression to blur everything a little. It didn't. It amplified one specific mistake.

A dispute is only valid if it was filed within the policy's window, say 30 days after payment. The model decides that by reading two Unix timestamps and doing subtraction in its head. Here is a real failure pattern: "filed 26.5 days after payment, outside the 60-day window."

The full-precision model already made this mistake about 90 times on the test set; it caused 59 of its 61 wrongful denials. Q4_K_M made it 196 times. Most of its lost accuracy is this one error: refunds the merchant had already agreed to, or that the evidence supported, dismissed because the model believed the claim was late.

The fix is 12 lines, and it isn't in the model

Whether a dispute is inside its window is arithmetic. Arithmetic belongs in code. The library already had a check for it, so I measured what it does to each build's own predictions:

	Full precision	Q8_0	Q4_K_M
Wrongful denials, model alone	26.0%	23.4%	55.3%
Wrongful denials, with the check	0.85%	0.85%	0.85%

Every build lands at the same number, and no wrongful refund is added. The check never turns a verdict into a refund. If the model refused a claim because it was "outside the window" and the dates say otherwise, the false flag is removed and the case goes to a human:

from sra_riskgate import check_dispute_verdict

checked = check_dispute_verdict(dispute, verdict)   # dispute: policy + payment.paid_at + claim.filed_at
print(checked["outcome"])           # escalate_to_human, if the window claim was false
print(checked["dispute_window"])    # {"days_since_payment": 12.5, "window_days": 30, "within_window": True}

If you use MCP, version 0.15.0 of sra-riskgate-mcp exposes the same thing as a tool, check_dispute_verdict, so an agent can run it on any verdict, from my model, another model, or a person.

A trap worth knowing if you serve GGUF with Ollama

My first Q8_0 run scored 0% valid output. Not low: zero. The model had generated about 196,000 tokens, the same as Q4, yet every answer came back empty.

The Q8_0 file came with Qwen3's "thinking" chat template. Ollama treated the model's entire output as reasoning and returned an empty content field. The fix is to give the new build exactly the template and parameters of the build you trust, so only the weights differ:

ollama show your-trusted-model --modelfile \
  | sed 's#^FROM .*#FROM hf.co/<repo>:Q8_0#' > Modelfile.q8
ollama create my-model-q8 -f Modelfile.q8

If a benchmark result looks impossibly bad, check that you measured the model and not the plumbing.

What I'd tell anyone shipping a quantized model

1. Benchmark the build people run. My full-precision numbers were true and still misleading for most users.
2. Look at which errors grew, not just the average. "Disputes dropped 11 points" sounds like general decay. It was one failure mode doubling.
3. Move arithmetic out of the model. The cheapest fix for a 4B model's date math is not a bigger model. It is the date math.
4. Publish the predictions. Anyone can check these numbers. That matters more than the numbers.

Caveats

The benchmark is synthetic, generated by my own rule engine, and every run here is mine; no third party has reproduced them yet. The 0.85% that remains are cases the window check doesn't cover. Next, I'm training a v2 that receives the computed window as a tool result instead of guessing it.

Links: model SRA-RiskGate-4B · all predictions and the reproduce commands in sra-bench-results · code github.com/sriram1983007-dev/sra-riskgate · pip install sra-riskgate · uvx sra-riskgate-mcp