Orders fail while the API shows no errors

On-call incidentEasy15 min

Your task

It is 11:25. The shop's web and mobile apps send every request to one GraphQL API. The API calls the catalog, orders and payments services. This alert has been firing since 11:13:

OrdersPerMinuteLow: orders 50% below the same time last week for 10 minutes (current 47, usual 102)

What you see:

  • Customers who tap "Place order" see "We couldn't place your order. Please try again." Many try 2 or 3 times.
  • The API dashboard shows a 5xx rate of 0.1%, normal latency and normal request volume. Its error alert is quiet.
  • The API's access log shows POST /graphql 200 for these attempts, with response bodies of about 300 bytes. A successful order's response is about 1.9 KB.
  • Payments deployed version 4.18 at 11:02. Its 4xx responses rose from about 20 a minute to about 160 a minute at 11:03, and it returns no 5xx.

What do you do first, and what do you change so a failed order shows up as an error?

It covers

  • Graphql
  • Error metrics
  • Business alerts