IoT time-series storage engine

DesignHard45 min

Your task

Design the storage engine of a time-series database optimized for IoT telemetry: 10M devices each writing 50 metrics every 10s (~50M points/sec), almost entirely append-only with monotonically increasing timestamps, occasional late/out-of-order arrivals up to 1h late. Reads are 95% 'latest value for device D' and dashboards over the last 24h for a device group; 5% are large aggregations across all devices for a metric over a month. Design the on-disk layout, write path, compaction, and how you avoid write hot partitions.

It covers

  • Lsm
  • Compaction
  • Hot partition