DNS split-brain after resolver migration

On-call incidentHard35 min

Your task

Half your fleet suddenly can't reach an internal service db-primary.internal. Affected pods log 'no such host' / NXDOMAIN; the other half resolve it fine. Dashboards: error rate on the orders API is ~50% and bouncing; the unaffected half of pods are healthy. Recent context: 20 minutes ago an infra change migrated internal DNS from a self-hosted CoreDNS to a managed resolver, and someone also bumped record TTLs from 30s to 3600s last week. No application deploy. How do you triage and mitigate?

It covers

  • Resolution
  • Ttl
  • Split horizon