Real-time streaming aggregation

DesignHard50 minDatabricks style

Your task

Design a real-time streaming-aggregation system that computes per-minute and per-hour metrics (request counts, p99 latency, error rates) from a 3M-events/sec ingestion firehose, powering live dashboards and alerting. Requirements: aggregates update within a few seconds, events can arrive seconds-to-minutes late or out of order, results must be correct (not double-counted) across worker restarts, and the system must handle a sudden 5x traffic spike. Walk through windowing, lateness, and correctness.

It covers

  • Windowed aggregation
  • Watermarks
  • Exactly once