A customer's nightly stock import fails with 403 errors
Your task
It is 08:40 on Tuesday. Your inventory API runs behind AWS WAF, a firewall for web requests, on a load balancer. Bright Foods, one of your largest customers, pushes its stock levels every night at 02:00. The morning check at 08:30 raises this alert:
PartnerImportMissing: bright-foods import missing 2 nights in a row (threshold 2)
What you see:
- Bright Foods' job log shows
HTTP 403on every call at 02:00 on Monday and again today. The body is a short HTML page,<title>403 Forbidden</title>. Your API's own refusals are JSON. - Your API's access log has no requests from Bright Foods since Sunday at 02:00. Other customers' traffic is normal.
- Bright Foods' IT team says it moved the import job from its own data center to a cloud host on Sunday afternoon.
- WAF logs for those calls show
"action":"BLOCK"and"terminatingRuleId":"tmp-block-scraper". - An engineer added that rule 6 weeks ago, during a scraping incident. Its ticket says "temporary, remove when the scraper stops", and the ticket is still open with no owner.
How do you let Bright Foods back in without inviting the scraper back, and what do you change about temporary rules?
It covers
- Web application firewall
- Partner integrations
- Incident follow up
About this question
A customer's nightly stock import fails with 403 errors
Model answer
Free on every accountFirst move: let Bright Foods' address through the block, and only theirs. Open tmp-block-scraper and read what it matches. Take the source address or addresses from httpRequest.clientIp in the WAF log, and confirm with Bright Foods' IT that they are theirs and fixed. Then pick one of these:
- Add an exception to the rule itself: an AND of its current match and a NOT over an IP set holding that address as a
/32. It then blocks what it matched, except that address. Every other rule still applies to Bright Foods. - Add an Allow rule above it (a lower priority number), limited to that address and the import path. It is quick, but an Allow ends inspection, so later rules such as rate limits no longer apply to that traffic.
Ask for one test call and look for it in your API's access log. A WAF change can take from a few seconds to a few minutes to reach every location, so a call that still fails straight away proves nothing. Wait a few minutes and try again. Then ask Bright Foods to rerun the import.
What this does not fix: 2 nights of missing stock. Ask whether one full push replaces both nights or whether both need loading. Tell the account manager that stock figures have been stale since Sunday. It also does nothing about the rule itself.
Then decide about the rule. Query the WAF logs for what it has blocked over the last few days, grouped by source address. If nothing but Bright Foods shows up, set the rule to Count for a few days, which logs matches without blocking, and then delete it. Its cost: anything it matches gets through in that time, so watch the count. If the scraper is still there, replace the broad block with something narrower, such as specific addresses or a rate-based rule. Do not delete it before you look: if the scraper is still there, deleting the rule lets it back in. Look back over the full 6 weeks too, if the logs reach that far, for any other customer it has blocked, and tell them.
Read the signals. The body is HTML and your API's own refusals are JSON, so something in front of the API answered. The access log has no requests, which agrees. The job's calls worked at 02:00 on Sunday and were refused at 02:00 on Monday, and the one change anyone reports between them is where the job runs. The WAF log names the rule that stopped the calls. So the new cloud host's address falls inside what tmp-block-scraper matches, and a rule meant to be temporary has run for 6 weeks with no owner.
Other causes do not fit. A bad API key would produce your API's JSON refusal and a line in the access log. A network fault would not return a neat 403. Other customers are normal, so the WAF and the load balancer are healthy.
Confirm. The rule's IP set or match statement covers the address in httpRequest.clientIp. After your change, the WAF log for those calls shows "action":"ALLOW", with your Allow rule in terminatingRuleId, or Default_Action if you used the exception and the web ACL's default action is Allow. The access log shows Bright Foods' requests.
Fix. For temporary rules:
- Give each one a named owner and an end date, in the rule's name and in its ticket.
- List rules past their date every week, and send the list to the team that owns the WAF.
- Keep WAF rules in code, so each change has a reason and a reviewer.
- Run a broad new block in Count first when there is time, to see who it would catch.
Follow-ups.
- Change the import alert's threshold to 1 night. Monday's miss would have reached you a day earlier.
- Give the WAF's 403 a body with a request ID and a support address, so a blocked customer can tell you.
- Alert when a block hits a known customer's address.