Search fails for 40 minutes and no page arrives
Your task
It is 14:42. You are the only on-call engineer for the platform. At 13:40 you were paged for checkout, acknowledged at 13:42 and found a bad config, which you fixed at 14:15. This was the page:
HighErrorRate: checkout 5xx above 5% for 5 minutes (current 12%)
What you see:
- The search team lead messages you: search has returned errors since about 14:01, and nobody was paged.
- In Prometheus,
HighErrorRate{service="search"}has been firing since 14:07. Alertmanager lists it as active, not silenced and not inhibited, with the same PagerDuty receiver as checkout's alert. - PagerDuty shows one open incident, for HighErrorRate, opened at 13:40 and acknowledged by you at 13:42. The PagerDuty service has its own alert grouping turned off.
- Checkout's 5xx rate has been under 1% since 14:20, but that incident has not resolved.
What do you do in the next 5 minutes, and what do you change so search's next alert reaches someone?
It covers
- Alertmanager
- Pagerduty
- Alert routing