Geocoding quota exhausted at 9am

On-call incidentMedium35 min

Your task

Your integration with a third-party geocoding API has a generous daily quota you've never come close to. At 09:00 you start getting 429 'daily quota exceeded', at 9am, with most of the day's traffic still ahead. Dashboards: your outbound call volume to the geocoder spiked 50x starting at 08:30, but your inbound user traffic is completely flat and normal. No quota or pricing change from the vendor. You deployed a small change to the geocoding client last night. How do you triage and mitigate?

It covers

  • Retry loop
  • Self inflicted
  • Quota burn