Edits reverting to older values

On-call incidentHard40 min

Your task

You run Cassandra with last-write-wins conflict resolution. Users report that occasionally an edit they JUST made reverts to an older value seconds later. Dashboards: writes for a given key go to coordinators in two different racks; one rack's nodes are ~120ms ahead on wall clock (NTP offset alert firing on 3 hosts since a maintenance window); the older value is 'winning' over the newer edit. No app deploy. How do you triage and mitigate?

It covers

  • Ordering
  • Last write wins
  • Consistency