Connection pool reconnect storm

On-call incidentHard28 min

Your task

Your database has a planned failover to a standby at 03:00 for maintenance. The DB itself comes back in ~10 seconds, but your stateless API fleet then shows a 60–90 second window of high latency and a burst of errors before fully recovering, even though no app instances restarted. Dashboards show, right after failover: a spike in DB connection-establishment time, app-side connection pools draining and refilling, and a thundering set of simultaneous reconnect attempts from all instances at once hitting the new primary. How do you triage and reduce the post-failover recovery window?

It covers

  • Dependency warmup
  • Connection pool
  • Post failover