Orders fail while the API shows no errors
Your task
It is 11:25. The shop's web and mobile apps send every request to one GraphQL API. The API calls the catalog, orders and payments services. This alert has been firing since 11:13:
OrdersPerMinuteLow: orders 50% below the same time last week for 10 minutes (current 47, usual 102)
What you see:
- Customers who tap "Place order" see "We couldn't place your order. Please try again." Many try 2 or 3 times.
- The API dashboard shows a 5xx rate of 0.1%, normal latency and normal request volume. Its error alert is quiet.
- The API's access log shows
POST /graphql 200for these attempts, with response bodies of about 300 bytes. A successful order's response is about 1.9 KB. - Payments deployed version 4.18 at 11:02. Its 4xx responses rose from about 20 a minute to about 160 a minute at 11:03, and it returns no 5xx.
What do you do first, and what do you change so a failed order shows up as an error?
It covers
- Graphql
- Error metrics
- Business alerts