The page comes in. Your heart rate goes up, you open the laptop, and your hands go straight to the code editor or the logs. That's the most common mistake I see developers make on call, and I made it too.
I work in QA, and a big part of my job is triage and on-call: being the first person to look at a production problem and decide what happens next. After enough nights, I noticed that the incidents that went badly almost never went badly because nobody could find the bug. They went badly because of the first five minutes: someone debugged instead of stopping the damage, nobody said anything in the channel, or two people changed things at the same time.
So here's the routine I use now. It fits on one page, and it deliberately does not start with the code.
Minute 0–1: Acknowledge, then read the whole alert
Acknowledge the page first. Not because the tool wants you to, but because otherwise a second person may start working on the same thing without knowing you're there.
Then read the alert completely: which service, which signal, since when, how far over the threshold. "Error rate > 2 %" is a different night from "error rate > 40 %".
Open the dashboard, not the logs. Set the time range to the last three hours. Logs tell you what one request did. The dashboard tells you whether it matters.
Minute 1–2: Is it real, and how big is it?
Find a second signal that agrees with the alert: the error tracker, a synthetic check, the support channel, a related metric. If nothing else agrees, you might be looking at a broken alert, not a broken product. That's still worth fixing, just not at 3 a.m.
Then answer three questions:
- Who is affected? Everyone, one region, one customer, one endpoint, internal users only?
- Is it getting worse, stable, or recovering?
- Compared to what? Night traffic compared with noon traffic looks like an outage. Compare with the same time yesterday.
Minute 2–3: What changed?
Most incidents follow a change. I check in this order:
- Deploys in the last few hours, for this service and the things it depends on
- Feature flags and config that were toggled recently
- Traffic: a spike, a new client, a batch job, month-end
- Dependencies: the status pages of your cloud, payment, email, and auth providers
- Scheduled work: cron jobs, migrations, backups, certificate renewals
If your dashboards show deploy markers, this step takes ten seconds. If they don't, that's the first thing I'd add after the incident.
Minute 3–4: Decide the path, and mitigate before you understand
This is the part that feels wrong to developers: you don't need to know the root cause to stop the damage. Users don't care why it's broken; they care that it stops being broken.
| If… | Then |
| A recent change lines up with the start | Undo it: switch the flag off, or roll back if it's safe |
| Data might be getting corrupted or exposed | Stop the writers (flag off, maintenance mode) and escalate now |
| A dependency is down | Degrade gracefully, post on the status page |
| You don't know after 4 minutes and users are hurting | Escalate. Asking early is the job |
| Impact is small and stable | Open a ticket and keep watching |
A note on "rollback if it's safe". A rollback is not automatically safe. Before I roll back, I check four things: the previous version is available, the release didn't include a destructive migration the old code can't handle, the old code can read data the new code wrote, and rolling back won't replay side effects like payments or emails. If any of those is a "no", rolling back can turn a bad night into a data-repair week. Switch off the flag, fail forward with a minimal fix, or escalate.
Minute 4–5: Say something
Post in the incident channel, even if you know very little:
Investigating: [what users see]
Since: [HH:MM UTC] · Impact: [who / how many]
Suspect: [recent deploy X / unknown]
Next update: [HH:MM UTC]
Notice what's in it and what isn't. It describes what users see, not what the system does. It promises the next update, not a fix time; you don't know the fix time yet. And it's honest about not knowing.
Four rules that save me every time
- Mitigate before you understand. Root cause can wait. Users can't.
- Write every action in the channel before you run it. At 3 a.m. you will not remember what you did at 2:47. Future-you writes the timeline from these messages.
- One change at a time. If you change two things and it recovers, you don't know which one fixed it, or whether one of them made something else worse.
- No quick fixes by hand in production unless a runbook says so, and then you record it.
A real-ish example
A page at 02:14: API error rate above 2 %. Dashboard: errors jumped from about 0.1 % to 6 % at 02:03. The error tracker shows one new error type, only on the invoice detail endpoint. There's a deploy marker at 02:01. The release added a field to the invoice view behind a feature flag, but the error is in code that runs regardless of the flag, so switching the flag off won't help.
Rollback check: previous version available, migration only added a nullable column, old code ignores it, endpoint is read-only. Safe. Message posted at minute five, rollback started at 02:21, errors back to baseline by 02:40. The root cause took ten minutes to understand the next morning, with coffee.
If I'd opened the code first, users would have kept failing for those extra minutes while I read a stack trace I couldn't fix at 2 a.m. anyway.
Example anonymized and simplified; details are illustrative.
Take the checklist with you
I put this routine on a one-page cheat sheet you can print or pin next to your dashboards: The First 5 Minutes of On-call (free, PDF + Markdown): https://payhip.com/b/CXbTh
If you want the rest of what I use on call (a pre-shift access checklist, a triage decision tree, a guide to reading dashboards under pressure, escalation and handover templates, and ten runbooks for the situations that cause most pages, from 5xx spikes and bad deploys to full disks and expired certificates), it's in the Developer On-call Survival Kit: https://payhip.com/b/DBS1y
What's the first thing you check when the pager goes off? I'm curious whether your order is different.