One pod fails every request but passes health checks
Your task
At 15:20 the error-rate alert for the profile service fires: 12% of requests return HTTP 500, up from 0.1%. There was no deploy today and traffic is normal. The service runs 8 pods behind a Kubernetes Service. Splitting the 5xx metric by pod shows every error comes from profile-7f9c-x2kq; the other 7 pods are clean. That pod logs redis: connection pool timeout on every request. It has stayed Ready the whole time, because its readiness probe calls /healthz, which returns 200 without touching any dependency. The other pods use the same Redis without trouble, and Redis shows no errors. Node events show a brief network blip at 15:14 on the node where the bad pod runs. How do you triage and mitigate?
It covers
- Health checks
- Bad instance
- Triage