Real-time mobile analytics

DesignHard40 minDatabricks style

Your task

Design the real-time analytics pipeline for a mobile app that aggregates per-minute active-user and event counts from clients that frequently go offline. Events carry a client-side event_time but can arrive minutes-to-hours late (subway rides, airplane mode, flaky networks); some arrive out of order within a single device. You must publish per-minute aggregates within ~10 seconds for a live ops dashboard, but the numbers must also eventually be correct when the stragglers land. How do you handle watermarks, lateness, and the inevitable correction?

It covers

  • Watermarks
  • Late arriving data
  • Windowing