Last month, my automated software-ingestion pipeline did something terrifying: it rejected 43.8% of all updates in a single run. Forty-three point eight percent. My first instinct was that the pipeline was broken. It wasn't. It was the only thing in the room doing its job.
I run an open software catalog. Every few days, an automated pipeline pulls new releases from vendors, GitHub, and F-Droid, verifies them, and publishes them. That night it pulled the fire alarm on nearly half the catalog — two SHA-256 hash mismatches (Google Chrome and CCleaner, of all things) and twelve signature mismatches across apps like Obsidian, Tor, and Ventoy.
Here’s what I learned fixing it, distilled into the checklist I now run before any binary goes live.
1. START WITH THE HASH — BUT KNOW WHAT IT CAN’T TELL YOU
Every release I publish carries a SHA-256 checksum, recomputed at download time. It’s the cheapest integrity check in existence:
sha256sum installer.exe
# compare against the publisher's published value
If the hash doesn’t match, stop. Don’t install it, don’t “try it anyway.” A mismatched hash means the file changed in transit — or it isn’t the file the publisher shipped. That night, Chrome and CCleaner’s hashes didn’t match their expected values. The pipeline refused them, and rightly so: a hash mismatch is never a rounding error.
But a hash answers only one question: did this file change? It says nothing about who made it. That’s what signatures are for.
2. VERIFY THE SIGNATURE, NOT JUST THE CHECKSUM
A checksum tells you the file is intact. A signature tells you it came from the right key. For Android packages, the tooling is free:
apksigner verify --print-certs app.apk
The critical step most people skip: check that the certificate fingerprint matches the publisher’s known key. The first time you see a new fingerprint on a familiar app, treat it as suspicious until proven otherwise. For desktop software, GPG serves the same role:
gpg --verify installer.exe.sig installer.exe
The pattern is identical everywhere: fetch the artifact, fetch the signature, verify against the publisher’s public key. And if a project ships binaries with no signatures at all? That’s a data point about the project — not a reason to skip verification.
3. ENROLLMENT IS THE HALF OF VERIFICATION EVERYONE FORGETS
Here’s the mistake that cost me a week. My pipeline verified every APK signature correctly — and still rejected twelve apps. The parser wasn’t broken. The cryptography was fine. The problem: I had never enrolled those signers in the first place.
Signature verification has two halves, and I was only doing one:
1. Is this signature cryptographically valid? (the tool answers this)
2. Do I recognize this signer? (only my allowlist answers this)
Twelve apps were signed by keys my system had simply never recorded — new maintainers, rotated keys, corporate renames. My favorite: a publisher whose legal name changed by a single dot (“Dynalist Inc.” vs “Dynalist Inc”), which my strict string match treated as a stranger. The pipeline was doing its job; my trust store was incomplete.
The fix is boring and essential. Keep an explicit registry of known-good signers per package, keyed by certificate fingerprint:
# trust_store.json — enrolled signers, one entry per package
{
"com.example.app": {
"fingerprints": ["AB:CD:EF:12:34..."],
"enrolled_on": "2026-10-13",
"note": "Key rotation announced in v4.2 changelog"
}
}
When a new fingerprint appears, that’s an event, not an error — investigate it once, enroll it deliberately, and move on. Auto-trusting unknown signers defeats the entire purpose.
4. REJECT LOUDLY — WITH A THRESHOLD YOU CHOSE IN ADVANCE
A verification system that silently accepts everything is decoration. Mine fails the entire ingestion run closed if more than 15% of updates get rejected. That number is deliberate: a sudden spike means either the upstream world changed (keys rotated, publishers restructured) or my pipeline broke. Either way, I want a human looking at it before anything ships.
The exact threshold matters less than the principle: decide in advance what “too many failures” looks like, and make the pipeline stop instead of shrugging. Quiet failures are how bad binaries reach users. My 43.8% night looked like a disaster and was actually the system working exactly as designed — it just needed a human to enroll the new signers and re-run.
5. DECIDE IN ADVANCE WHAT TO DO WITH THE UNVERIFIABLE
Not everything is signable. Some legitimate tools ship unsigned binaries — portable builds, indie utilities, projects too small for signing infrastructure. You have three honest options: reject them, allowlist them explicitly, or verify them another way (reproducible builds, publisher-published hashes fetched over HTTPS).
I allowlist them — explicitly, per binary, with a written reason. What I never do is let an unsigned binary through on a technicality. “The check didn’t apply” is not the same as “the check passed,” and conflating the two is how exceptions quietly become the rule. Ventoy’s unsigned binary, for example, sits in my allowlist with a note explaining why — visible, reviewable, deliberate.
THE CHECKLIST
Before any binary goes live:
- SHA-256 matches the publisher’s published value (recomputed at download, not trusted from a mirror)
- Signature verifies against the publisher’s known key
- Signer fingerprint is in the enrolled registry — new fingerprints get investigated, never auto-trusted
- Rejection rate for the run is under threshold, or the whole run stops
- Unsigned artifacts are explicitly allowlisted with a written reason — never silently passed
None of this is exotic. It’s plumbing. But it’s the plumbing between a stranger’s server and your users’ machines, and every piece of it earns its place the first time it catches something. Mine caught something on a random Tuesday night, rejected nearly half the catalog, and was right.