The app slows to seconds after a release email

On-call incidentEasy15 min

Your task

It is 10:25. Your company sells a desktop app. Its API, sign-in and web pages run on 3 VMs behind a load balancer. On each VM, nginx also serves the files under /downloads/ from local disk. Each VM has a 1 Gbit/s network link. At 10:00 marketing emailed 200,000 customers about version 5.0. This alert fires:

ApiLatencyP95: p95 above 1 s for 10 minutes (current 6.4 s)

What you see:

  • p95, the time 95% of requests finish within, is above 6 s on every endpoint, even ones that return a few KB. Errors are rare.
  • Since 10:03, each VM's outbound traffic has been flat at about 940 Mbit/s. CPU is at 25%, and inbound traffic and memory look normal.
  • Since 10:00 the busiest path in the access log is /downloads/Setup-5.0.exe, 480 MB per request. Those requests take 15 to 40 minutes each.
  • Earlier release emails linked to dl.example.com, a CDN in front of a storage bucket.
  • Support has 60 tickets saying the download crawls or sign-in times out.

How do you make the app fast again without stopping customers getting version 5.0, and what changes before the next release email?

It covers

  • Network bandwidth
  • Cdn
  • Large downloads