Read-only transaction after failover

On-call incidentMedium30 min

Your task

A managed Postgres failover (planned maintenance) promoted a standby and updated the cluster's writer DNS endpoint. Most app instances recovered, but a subset kept erroring with 'cannot execute INSERT in a read-only transaction' for ~25 minutes, and a few writes from that subset later appeared to be lost. The healthy instances use the writer endpoint and are fine. The affected subset is an older service. Triage, mitigate, and prevent.

It covers

  • Dns ttl
  • Stale endpoint
  • Writes to old primary