If you have set up a remote build cache and watched the hit rate sit stubbornly low, nondeterministic outputs are one important suspect — but not the only one. Source churn, changing flags, platform differences, toolchain changes, eviction, and incomplete input declarations also cause misses. Nondeterminism is particularly dangerous because it can quietly reduce reuse or make cached results untrustworthy without producing an obvious build error. It is worth understanding exactly how that happens and how to isolate it.
Why determinism is the contract, not a nice-to-have
A build cache works by fingerprinting an action's inputs and using that fingerprint to look up a previously computed output. The entire scheme rests on one assumption: identical inputs produce identical outputs. That is the contract. Break it and one of two bad things happens.
If the cache keys only on inputs — which is how these systems work — and your action is nondeterministic, then the cache will happily hand back an output for those inputs, but not necessarily the one a fresh run would produce. Best case, the variation is cosmetic and harmless. Worst case, you have just served a subtly wrong artifact from cache and you will spend a very confusing afternoon. Either way, the cache is now a source of doubt rather than speed.
Another operational outcome is cascading misses, and that is where determinism stops being abstract.
One bad action poisons everything downstream
Builds are graphs. The output of one action is the input to the next. So consider what happens when a single early action — say, a code generator — embeds the current timestamp in its output. Each time that generator actually runs, the file it produces is byte-for-byte different, so its content hash changes.
Whenever that generator has to execute again — because its entry was evicted, the cache was bypassed, or another declared input changed — it produces a new content digest. Every action that consumes the generated file then sees a different input and misses its previous cache entry. Their outputs feed further actions, so the effect can cascade through the subgraph. You did not merely lose reuse for one action; you invalidated downstream keys even when the semantic content did not need to change.
Where nondeterminism actually comes from
The sources are mundane, which is what makes them easy to miss. The usual suspects:
- Timestamps baked into outputs —
__DATE__ and __TIME__ macros, build IDs, and the modification times stored inside archive formats like .a, .tar, and .zip.
- Absolute paths leaking into outputs — debug info that records the full build directory,
__FILE__ expansions, paths embedded by the linker. Build the same code in /home/alice and /home/bob and you get different bytes.
- Iteration order — generating code by walking a hash map or set, whose iteration order is unspecified, so the output lines come out shuffled run to run.
- Concurrency — actions that assemble output from parallel work without imposing a stable order on the result.
- Randomness — UUIDs, unseeded PRNGs, anything that reaches for entropy at build time.
- The ambient environment — locale, hostname, username, and stray environment variables that the action reads but never declared as inputs.
- Filesystem ordering — relying on
readdir order, which varies across filesystems and runs.
None of these throws an error. Each just makes one action's output wiggle, and the cache does the rest of the damage.
How to hunt them down
The basic diagnostic is to execute the same build twice from clean, isolated directories with cache reuse disabled, then compare the outputs bit for bit. Anything that differs deserves investigation. diffoscope is the right tool here — it understands archives, binaries, and debug information and can show not just that two outputs differ but where.
The fixes are well-trodden, because the reproducible-builds community has been fighting this war for years:
- Timestamps: honor
SOURCE_DATE_EPOCH so embedded dates are fixed rather than "now"; build archives in deterministic mode (tar --sort=name --mtime=..., ar in its deterministic D mode).
- Absolute paths: use
-ffile-prefix-map / -fdebug-prefix-map to rewrite build paths to a stable placeholder, so debug info does not depend on where you built.
- Iteration order: sort before you emit. If codegen walks a map, sort the keys first. Never let an unordered container decide output order.
- Environment: pin the locale (
LC_ALL=C), and run actions in a sandbox that only exposes declared inputs, so an undeclared environment dependency fails loudly instead of silently varying.
- Randomness: seed it, or design it out. A build is not the place for entropy.
Hermeticity is how you keep it fixed
Fixing the determinism bugs you have is half the job. The other half is preventing new ones, and hermetic, sandboxed builds remove many sources of ambient input. A filesystem sandbox can hide undeclared files and tools, and a controlled environment can pin locale and environment variables. Sandboxing alone does not necessarily hide the wall clock, randomness, hostname, kernel details, or every network source; those must be blocked, virtualized, or deliberately normalized. The goal is to make undeclared dependencies fail loudly or become stable rather than silently changing outputs.
Measure it, because it regresses
Determinism is not a one-time cleanup. It rots. Someone adds a code generator, someone embeds a build timestamp "just for diagnostics," and reuse slides without anyone noticing. Treat cache hit rate as a first-class metric, but segment it by action type, platform, branch, and miss reason because determinism is only one contributor. Track reproducibility separately with scheduled rebuild-and-compare tests. When hit rate drops and action keys churn without meaningful input changes, reach for build-twice-and-diff before blaming the cache infrastructure.
That is the real point. Determinism is not purity for its own sake. Along with complete input declarations, compatible platforms and toolchains, and retained cache entries, it is a precondition for trustworthy reuse. Get it wrong and the cache may return questionable outputs or lose reuse across a large subgraph. Get it right and equivalent actions can reliably share results across developers and CI. Determinism is not the whole cache system, but it is part of its correctness contract.
Further reading: the Reproducible Builds documentation