Disk errors after live migration

On-call incidentHard45 minStripe style

Your task

It's 02:10. PagerDuty fires on a spike of ERROR: invalid page in block 4711 of relation base/16384/24576 from the primary Postgres (v14) backing the payments ledger. The dashboards show: query error rate climbed from ~0 to 4% over 20 minutes, only on reads touching the ledger_entries table; replica lag is flat; the host's node_disk_io_errors counter jumped from 0 to 18 in the same window; SMART on the NVMe shows reallocated-sector-count rising. Two hours earlier the host was live-migrated to a new physical node by the cloud provider (maintenance event in the audit log). data_checksums is on. How do you triage this, mitigate the blast radius, and recover the corrupted data safely?

It covers

  • Torn writes
  • Page cache
  • Postgres