Design a cold-storage observability system

DesignMedium40 minSnowflake and Databricks style

Your task

Design a log/metrics cold-storage system for an observability platform. Constraints: ingest is tens of millions of events/sec at peak, recent data (hours) must be queryable fast, older data (months/years) is queried rarely and must be cheap, retention is years for compliance, and queries are mostly time-range + filter + aggregate. Cover ingest, the hot-to-cold path, and the query model.

It covers

  • Columnar format
  • Compaction
  • Downsampling