One API key sends a third of all traffic
Your task
At 10:40 latency on the public REST API climbs: p95 goes from 180 ms to 1.4 s and a few requests time out. CPU on the API pods is at 85%. The request rate is up from 1,500 to 2,300 a second, but marketing has nothing running. Grouping requests by API key shows one key, key_int_reporting, sending 780 requests a second to GET /v1/invoices?page=N, when it normally sends almost none. The key belongs to the internal finance-reporting team. The access logs show it requesting pages 1 to 40 over and over, starting at 10:31. The API gateway supports per-key rate limits, but none is set for internal keys. How do you triage and mitigate?
It covers
- Runaway client
- Api keys
- Rate limiting