Writes disappear on replica read-after-write

On-call incidentMedium30 min

Your task

MySQL primary + 3 read replicas behind a read-routing layer. Replication lag is healthy (<1 s on Seconds_Behind_Master) and there's no CPU/IO pressure. But after a deploy this morning, support is flooded with 'I updated my profile and it reverted' and 'I posted a comment and it's not there, then it is on refresh.' Writes go to the primary; the deploy changed the data layer to route more reads to replicas. Triage and fix, and say what monitoring would have caught it.

It covers

  • Read after write
  • Stale reads
  • Replica routing