Log search with tiering
Your task
Design a log-search system ingesting 2M log lines/sec (5TB/day) where engineers run ad-hoc full-text + structured-field queries over a time range. Recent data (last 24h) must be queryable within seconds of ingestion; older data (up to 30 days) can tolerate slower queries. Cost must stay bounded, so storage tiering is required.
It covers
- Log search
- Time series
- Retention