Observability dashboard serving
Your task
Design the query/serving backend for an observability dashboarding product. A popular incident dashboard has 40 panels, each running a metrics query, and during an outage 500 engineers all load it at once, hammering the metrics backend with 20,000 near-identical queries in seconds. Dashboards auto-refresh every 30s. The metrics backend is the bottleneck and starts to brown out. Design the serving layer so dashboards stay fast and the backend survives the thundering herd.
It covers
- Query fanout
- Caching
- Concurrency