DNS resolver contention waves
Your task
A previously-fast Node.js payment-callback service starts showing intermittent +200ms to +5000ms latency added to a fraction of requests, clustered in waves every ~30-60s. The slow requests all spend the extra time before the first byte is sent to the downstream payment API, the downstream itself reports fast service times for those same requests. CPU and memory are fine. The only recent change was migrating from a hardcoded IP for the payment API to a hostname. The internal DNS resolver's query rate has jumped. How do you triage and fix?
It covers
- Dns in hot path
- Resolver
- Ttl