DNS TTL causes split-brain writes

On-call incidentHard40 min

Your task

Your primary database endpoint failed over to standby at 09:00, the orchestrator promoted standby and updated the DNS record db-primary.internal to the new IP. Failover is supposed to be 30 seconds. Instead, for ~14 minutes you keep getting partial write failures: roughly 40% of app pods write successfully to the new primary while 60% keep hammering the old (now read-only) primary and fail with 'cannot execute INSERT in a read-only transaction'. Dashboards: the DNS record updated correctly and immediately; promotion completed cleanly; the failing pods are not random, they cluster on certain nodes and certain language runtimes. No deploy. How do you triage and what's the durable fix?

It covers

  • Ttl
  • Failover
  • Stale resolution