The app slows to seconds after a release email
Your task
It is 10:25. Your company sells a desktop app. Its API, sign-in and web pages run on 3 VMs behind a load balancer. On each VM, nginx also serves the files under /downloads/ from local disk. Each VM has a 1 Gbit/s network link. At 10:00 marketing emailed 200,000 customers about version 5.0. This alert fires:
ApiLatencyP95: p95 above 1 s for 10 minutes (current 6.4 s)
What you see:
- p95, the time 95% of requests finish within, is above 6 s on every endpoint, even ones that return a few KB. Errors are rare.
- Since 10:03, each VM's outbound traffic has been flat at about 940 Mbit/s. CPU is at 25%, and inbound traffic and memory look normal.
- Since 10:00 the busiest path in the access log is
/downloads/Setup-5.0.exe, 480 MB per request. Those requests take 15 to 40 minutes each. - Earlier release emails linked to
dl.example.com, a CDN in front of a storage bucket. - Support has 60 tickets saying the download crawls or sign-in times out.
How do you make the app fast again without stopping customers getting version 5.0, and what changes before the next release email?
It covers
- Network bandwidth
- Cdn
- Large downloads