Health checks flap

On-call incidentMedium20 min

Your task

After a deploy, a service's instances keep flapping in and out of the load balancer's healthy pool, the dashboard shows hosts being marked unhealthy, removed, then re-added a minute later, over and over. User-facing error rates are spiky as capacity bounces. The new build added a deeper health check that also pings a downstream dependency. How do you triage the flapping?

It covers

  • Health check
  • Load balancer
  • Readiness