DNS split-brain after resolver migration
Your task
Half your fleet suddenly can't reach an internal service db-primary.internal. Affected pods log 'no such host' / NXDOMAIN; the other half resolve it fine. Dashboards: error rate on the orders API is ~50% and bouncing; the unaffected half of pods are healthy. Recent context: 20 minutes ago an infra change migrated internal DNS from a self-hosted CoreDNS to a managed resolver, and someone also bumped record TTLs from 30s to 3600s last week. No application deploy. How do you triage and mitigate?
It covers
- Resolution
- Ttl
- Split horizon