Postmortem: the DNS change that took us down for four hours

We're publishing this postmortem close to as-written, with only names and some identifying specifics removed, because the mechanics of what went wrong — a DNS record change colliding with a caching chain nobody had fully mapped — are common enough that they're worth more people reading them than just our own team.

The short version: a routine DNS cutover, during a planned migration, turned into four hours of partial downtime because of an assumption about TTLs that turned out to be wrong in three different places at once.

The change, and why we made it

We were migrating our primary API's hosting from one provider to another, a move planned for weeks and intended to be invisible to users. The plan was standard: stand up the new infrastructure, verify it worked end to end against the new IP directly, lower the DNS TTL on the relevant A record 48 hours ahead of the cutover so any caching would clear quickly, then flip the record and watch traffic migrate over.

The TTL lowering step had been done correctly, four days earlier — which matters, because it's exactly the step everyone trusted that turned out not to protect us.

Timeline: what happened, minute by minute

• 14:02 — DNS record flipped from old IP to new IP. Internal monitoring from our own office immediately showed the new infrastructure serving traffic correctly.
• 14:06 — First external error reports start arriving, from users in two specific geographic regions, reporting connection timeouts rather than errors.
• 14:22 — On-call engineer confirms DNS has propagated correctly from multiple public resolvers (Google, Cloudflare) and the new server is healthy. Initial hypothesis: a firewall rule on the new infrastructure.
• 14:40 — Firewall rules confirmed correct. Error reports continue, now clearly clustered around the same two ISPs repeatedly.
• 15:10 — A support ticket from an affected user includes a traceroute, which is the first real clue: their resolver was still returning the old IP, four hours after the TTL should have expired everywhere.
• 15:35 — We identify that both affected ISPs run resolvers that, in some configurations, cap effective TTL honoring at a fixed internal minimum re-check interval regardless of the record's actual TTL — undocumented behavior we hadn't encountered before.
• 16:50 — Old infrastructure, which we had not yet decommissioned, is temporarily restored to full capacity as a stopgap so stale-cached users keep working while we address the resolvers directly.
• 18:05 — Direct outreach to the two ISPs' network operations contacts results in a manual cache flush on their side. Traffic normalizes within fifteen minutes.

Why nobody noticed for twenty minutes

Our own monitoring, running from our own office network and from a monitoring provider's standard regions, resolved the domain correctly within seconds of the change — both were using resolvers that honored the lowered TTL exactly as expected. The gap between "our monitoring says it's fine" and "a meaningful slice of real users can't connect" existed because our synthetic checks happened to not run through either of the two ISPs whose resolvers behaved unusually. That's a monitoring blind spot as much as a DNS problem.

It also meant the first twenty minutes of the incident were spent trusting the wrong signal. Every internal dashboard was green, which made the early error reports feel like they might be user error or a one-off network issue rather than the start of a real outage — a dynamic that's common enough in incident response to have its own name among our team now: "the dashboard is lying by omission, not by commission."

Why the fix took three and a half more hours

Once we understood the actual cause, the fix was conceptually simple — get the two ISPs to flush their caches — but operationally slow, because that requires reaching an actual human with access to that infrastructure, not an API call. Restoring the old servers as a stopgap bought us time but also meant running two parallel environments and double-checking data consistency between them, which added its own overhead during an already stressful incident.

What we got wrong

• We treated "TTL lowered 48 hours in advance" as equivalent to "fully protected from caching," without accounting for resolvers that don't strictly honor TTL.
• We decommissioned planning for the old infrastructure too aggressively — it was still available to restore, which was lucky rather than planned.
• Our synthetic monitoring didn't cover enough distinct network paths to catch a regional, resolver-specific problem quickly.
• We didn't have a pre-existing contact at either affected ISP, which meant the slowest part of the whole incident was simply finding the right person to call.

What we changed afterward

Future infrastructure cutovers now run on a dual-stack approach during the transition window — both old and new infrastructure serving traffic simultaneously behind a load balancer for at least 72 hours post-cutover, rather than a hard flip, specifically to absorb exactly this kind of caching edge case without user impact. We also added synthetic monitoring checks that explicitly route through a wider spread of regional ISPs, not just major cloud and DNS-provider networks, and we now keep a maintained list of network operations contacts at the ISPs with the largest concentration of our users, so the "who do we even call" step doesn't eat an hour next time.

The incident review, lightly edited

The internal review afterward settled on a single sentence that's now pinned in our infrastructure runbook: TTL reduces caching risk, it does not remove it, and any cutover plan that treats a lowered TTL as a guarantee is a plan with an untested assumption in it. We'd rather other teams read that sentence here than learn it the way we did.