Event loop lag climbing

On-call incidentMedium35 min

Your task

Your Node.js notification service started timing out across all endpoints at 08:15. Dashboards: event-loop lag on the pods is 4s+; CPU is only 40%; a downstream template-render call to a Python sidecar over HTTP has p99 of 6s since 08:10; the in-process async queue depth is climbing without bound; memory is rising steadily. No deploy on your service, but the template sidecar got a new model at 08:08. How do you triage and mitigate?

It covers

  • Thread pool
  • Head of line blocking
  • Queue