Search fails for 40 minutes and no page arrives

On-call incidentHard35 min

Your task

It is 14:42. You are the only on-call engineer for the platform. At 13:40 you were paged for checkout, acknowledged at 13:42 and found a bad config, which you fixed at 14:15. This was the page:

HighErrorRate: checkout 5xx above 5% for 5 minutes (current 12%)

What you see:

  • The search team lead messages you: search has returned errors since about 14:01, and nobody was paged.
  • In Prometheus, HighErrorRate{service="search"} has been firing since 14:07. Alertmanager lists it as active, not silenced and not inhibited, with the same PagerDuty receiver as checkout's alert.
  • PagerDuty shows one open incident, for HighErrorRate, opened at 13:40 and acknowledged by you at 13:42. The PagerDuty service has its own alert grouping turned off.
  • Checkout's 5xx rate has been under 1% since 14:20, but that incident has not resolved.

What do you do in the next 5 minutes, and what do you change so search's next alert reaches someone?

It covers

  • Alertmanager
  • Pagerduty
  • Alert routing