Admin API slows every afternoon on a burstable instance

On-call incidentEasy15 min

Your task

An internal admin API runs on a single t3.large EC2 instance whose credit specification is standard. Since Monday, support agents have complained that it gets very slow every afternoon. Today you're paged at 14:20: p95 latency is 4.8 s against a usual 250 ms. The CPU utilization graph has been flat at 30% since 13:50, while request volume is the same as in the morning, when the instance ran at 70 to 75% CPU with fast responses. CPUCreditBalance fell steadily from 300 at 08:00 to 0 at 13:50. Last Friday a new "export to CSV" feature shipped, and support agents now use it all morning. Nothing else changed. How do you triage and mitigate?

It covers

  • Cpu credits
  • Burstable instances
  • Right sizing