Replication lag from backfill

On-call incidentHard35 min

Your task

A backfill to populate a new search_vector tsvector column across a 300M-row Postgres documents table was launched at 22:00 as a tight loop issuing large UPDATE ... WHERE id BETWEEN ... batches. By 22:20, read replicas are 8 minutes behind and climbing, user-facing reads (routed to replicas) are serving stale data, and the primary's disk is filling fast. Dashboards: primary CPU moderate, but WAL generation rate is 10x normal and replica apply lag is a straight diagonal line up. No app errors yet. How do you triage and mitigate before replicas fall over?

It covers

  • Wal saturation
  • Replication lag
  • Primary overload
  • Throttling