SSTable checksum corruption

On-call incidentHard40 min

Your task

A Cassandra/RocksDB-style node starts throwing read errors on a subset of keys: queries for certain partitions return CorruptSSTableException / checksum-mismatch errors, and the node is logging block-checksum failures during compaction. Dashboards: read error rate on one node climbed to ~2%; the other replicas serve the same keys fine; SMART data on one NVMe drive shows rising reallocated-sector and uncorrectable-error counts; the corrupted SSTable was written 6 weeks ago and only now read; replication factor is 3. No recent deploy. How do you triage a corrupted SSTable / silent durability fault and recover without losing data or propagating the corruption?

It covers

  • Corruption
  • Checksum
  • Sstable
  • Bit rot