DNS TTL cache split

On-call incidentMedium35 min

Your task

Twenty minutes after a planned DNS change repointing api.example.com to a new ingress IP, you get partial outage reports: roughly 40% of users can't reach the API while 60% are fine, and the split doesn't correlate with region. Your new ingress is healthy and serving the 60%. Some clients resolve the old IP (now decommissioned, connections refused) and some the new IP. The old record had a TTL of 3600s. A handful of large enterprise customers behind corporate resolvers are heavily represented in the failures. Triage and stabilize.

It covers

  • Dns
  • Propagation
  • Ttl