Replica replay lag growing

On-call incidentHard35 min

Your task

Users report seeing stale data, they update a setting, refresh, and the old value comes back, intermittently. Your app reads from three async streaming replicas behind a load balancer and writes to the primary. pg_stat_replication shows replica-2 with replay_lag of 45 seconds while the other two are sub-second. The primary is healthy. A bulk backfill job started 20 minutes ago on the primary, and replica-2 sits in a different AZ. Disk and CPU on replica-2 look fine but replay_lag keeps growing. Triage and mitigate.

It covers

  • Streaming replication
  • Wal
  • Replica promotion