Health check flapping

On-call incidentMedium30 min

Your task

An L7 load balancer is flapping backends in and out of rotation every 20-40 seconds. The dashboard shows the healthy-host count for one target group oscillating between 12 and 30 (of 30), and clients see intermittent 503s with elevated tail latency. Per-host metrics look fine: CPU ~55%, memory steady, app error rate near zero. The health check is GET /healthz with a 2s timeout, 2 consecutive failures to mark unhealthy, interval 5s. A deploy 90 minutes ago added a database connectivity probe to /healthz. Walk through triage and the fix.

It covers

  • Health check
  • Flapping
  • Capacity