Code RoomReplica replay lag growing
HardPrep Room Coding #4577

Replica replay lag growing

On-callDatabases & SQLDistributed systemsSenior–Staff~35 min

Users report seeing stale data — they update a setting, refresh, and the old value comes back, intermittently. Your app reads from three async streaming replicas behind a load balancer and writes to the primary. `pg_stat_replication` shows replica-2 with `replay_lag` of 45 seconds while the other two are sub-second. The primary is healthy. A bulk backfill job started 20 minutes ago on the primary, and replica-2 sits in a different AZ. Disk and CPU on replica-2 look fine but `replay_lag` keeps growing. Triage and mitigate.

What a strong answer looks like

Stop the bleeding first (mitigate), then form hypotheses from real signals. Separate root cause from symptom, communicate status as you go, and close with what prevents a repeat.

0:00 of about 35 min
Which questions mattered is sealed until you submit. Telling you now would just be handing over the edge cases.