Design a cold-storage observability system
Your task
Design a log/metrics cold-storage system for an observability platform. Constraints: ingest is tens of millions of events/sec at peak, recent data (hours) must be queryable fast, older data (months/years) is queried rarely and must be cheap, retention is years for compliance, and queries are mostly time-range + filter + aggregate. Cover ingest, the hot-to-cold path, and the query model.
It covers
- Columnar format
- Compaction
- Downsampling