Backfill throttles live traffic
Your task
A one-off backfill to populate a new risk_score attribute on every account was launched as a fan-out of Lambda workers reading and writing a provisioned-capacity DynamoDB table that also serves live traffic. Within minutes: live API 5xx rate jumps to 25%, DynamoDB ThrottledRequests and UserErrors spike, ConsumedWriteCapacity is pinned at the provisioned ceiling, and the backfill's own DLQ is filling because workers retry-storm on throttles. Live latency-sensitive reads are timing out. The backfill is at ~8% complete and 6,000 Lambdas are running concurrently. How do you triage, protect live traffic, and complete the backfill safely?
It covers
- Job fanout
- Thundering herd
- Dynamodb
- Throttling