Index corruption after OS upgrade

On-call incidentHard40 min

Your task

After your platform team upgraded the host OS on the database fleet (a major distro bump), a few queries start returning wrong results: a WHERE email = ? lookup misses rows that you can plainly see exist when you SELECT with a sequential scan, and a UNIQUE constraint let in what looks like a duplicate. No app or schema change. The tables are fine; only certain text/index lookups misbehave. amcheck on one btree index reports inconsistencies. Triage, mitigate, and prevent recurrence.

It covers

  • Btree
  • Glibc collation
  • Reindex