On-callMediumoc-g053

Subject Deploy incidentsLevel Mid–Senior~30 minCommon in Reliability & on-call interviewsIndustries Technology, Software development

Question

You deploy v205 of a Node.js websocket gateway at 09:00. Everything looks fine for ~40 minutes, then pods begin restarting one by one with OOMKilled, and as each restarts, clients reconnect en masse, briefly spiking load on the survivors which then OOM faster — a rolling brownout. The memory dashboard shows each pod's RSS climbing linearly from deploy time, never plateauing. v205 added a per-connection in-memory cache of recent messages. Request rate is flat versus yesterday. Triage and mitigate.

What a strong answer looks like

Stop the bleeding first (mitigate), then form hypotheses from real signals. Separate root cause from symptom, communicate status as you go, and close with what prevents a repeat.

Learn the concepts

Diagram & narrate the incident

Loading whiteboard…

Run or narrate your approach, then ask the coach.