Error alert reports no data after a library upgrade

On-call incidentEasy15 min

Your task

At 14:00 you get a low-urgency notification: the Grafana alert rule "Checkout 5xx rate above 2%" is in state NoData, and has been since 11:40. The checkout dashboard's request-rate panel is also empty since 11:40, but the order count in the database is normal for a Tuesday. At 11:35 checkout deployed a change that upgraded its HTTP instrumentation library to a new major version. The alert query is:

sum(rate(http_requests_total{service="checkout",status=~"5.."}[5m]))
  / sum(rate(http_requests_total{service="checkout"}[5m]))

In Prometheus the checkout scrape target is UP, and /metrics on a checkout pod now shows lines like http_server_requests_total{service="checkout",code="200"} 48211. What do you do?

It covers

  • Alert rules
  • Metric rename
  • No data