Log search with tiering

DesignHard50 min

Your task

Design a log-search system ingesting 2M log lines/sec (5TB/day) where engineers run ad-hoc full-text + structured-field queries over a time range. Recent data (last 24h) must be queryable within seconds of ingestion; older data (up to 30 days) can tolerate slower queries. Cost must stay bounded, so storage tiering is required.

It covers

  • Log search
  • Time series
  • Retention