Originally published on bryanhamilton.info - part of an engineering series from production.
Three days of shipped patches vanished from my production homepage in a single click. Not a hack. Not a bad deploy. A code-editor tab I'd left open in cPanel since that morning.
Here's the mechanics, stated plainly because I'd never seen it stated plainly: a browser-based editor holds the file as it was when the tab opened, not what's on disk right now. Hours later, you hit Save. The editor doesn't merge, diff, or ask. It writes the entire stale buffer over the file. Every patch landed since that tab opened is simply gone - replaced by a copy of the past.
The symptom that should stop you cold
Three unrelated fixes broke at the same time: the site's auth navigation block, a JavaScript cache-bust version bump, and a config bridge for a feature I'd wired that same morning. Different features, different days, same file.
One feature broken: debug the feature. Several unrelated things broken in the same file: stop debugging features and suspect the file. Features don't conspire. Overwrites do.
mtime first, always
My first instinct was the wrong one - I nearly started re-debugging the nav patch on its own merits. Instead I shipped a read-only probe that reported two numbers:
size: 32034 bytes
mtime: 2026-11-05 16:29:34 UTC
Then the trap sprang: 16:29 UTC was seven minutes after my last patch reported success. A file can't be newer than your newest patch and missing your newest patch - unless something wrote an old copy over it. One timestamp, and the mystery wasn't a mystery anymore.
The probe also counted marker comments. Every patch I ship embeds a unique one (a convention that started as a double-patch guard). Count them per file and you have an instant inventory of what survived:
tixaff7 bridge marker: 0 # shipped 40 min ago
navauth1 marker: 0 # shipped Nov 3
v25 asset ref: 0 # shipped Nov 5
Three zeros where there should have been three ones. The file was old - whole and healthy, just from before all of it.
The recovery was boring, on purpose
Because every patch is needle-guarded (it only applies if its anchor appears exactly once) and marker-tagged, re-applying all three was a single script: check the anchors exist, apply, lint, verify markers. Diagnosis to live-verified recovery: about twenty minutes.
The only hiccup was a false alarm I should have expected: after the restore ran clean, the live site still served the old page. LiteSpeed caches HTML for five minutes. Disk-clean-but-not-served is not a failed patch - it's a cache aging out. Bust the cache, wait a beat, re-check.
What actually prevents this
- Close editor tabs the moment you're done. An open editor tab is a loaded gun pointed at whatever file it opened - the save is silent and the overwrite is total.
- Prefer upload-once script files over interactive editing. My patches ship as self-deleting one-shot files, so there's no live-editing session to go stale. The full pattern is open source: guarded-deploy-workers.
- Tag every patch with a marker. You can't inventory what you can't count.
- Check mtime before debugging anything. When the file itself is the suspect, everything downstream of the file is a waste of time.
The kicker: none of the lost work was actually lost. The markers made the inventory instant, the guards made the re-application deterministic, and the discipline that felt like overhead on the way in is exactly what made the way out take twenty minutes. The expensive version of this lesson is losing work you have no markers for.
Question for the comments: what's your worst lost-work story - and did you find out the hard way or the lucky way? Collecting these for a follow-up on "boring recovery" discipline.