Hot partition in time-series data

On-call incidentHard35 min

Your task

Cassandra cluster storing IoT sensor readings, partition key = (sensorType, day), clustering by timestamp. As fleet grew, write p99 climbed and some nodes show high load while others idle. nodetool shows a few partitions are enormous (tens of GB) and tombstone/compaction warnings on them; reads of 'latest readings' for popular sensor types time out. Only a handful of sensorType values are common. Triage and redesign the data model so this doesn't recur.

It covers

  • Time series
  • Current partition contention
  • Monotonic key