Downstream rate limit backpressure
Your task
Your async image-processing pipeline consumes jobs from an SQS queue and writes thumbnails to a third-party storage/CDN API. At 13:00 the queue's ApproximateNumberOfMessagesVisible starts climbing steadily, from a few hundred to 250k over an hour, and the oldest-message-age is now 45 minutes (your SLA is 2 minutes). Consumer pods are healthy and CPU is moderate, NOT pegged. There's no poison message, every job eventually succeeds, just slowly. Per-job processing time has quietly tripled. Logs show the third-party storage API is returning a lot of HTTP 429 (rate limited) and your client is retrying with backoff, so each job takes much longer than usual. A marketing campaign drove a 4x upload spike this morning. How do you triage and resolve this backlog?
It covers
- Queue backlog
- Scaling
- Downstream throttle