Failover connection pool spike

On-call incidentHard35 min

Your task

A managed-Postgres failover just promoted a standby (took ~25s). Right after the new primary came up, your API started returning 500s in waves and Postgres logs are full of FATAL: sorry, too many clients already. You run PgBouncer in transaction mode in front of ~40 app instances. pg_stat_activity on the new primary is pinned at max_connections. The failover itself completed cleanly; the database is up. Connections keep spiking, dropping, and spiking again. Triage and mitigate.

It covers

  • Pgbouncer
  • Connection storm
  • Failover