Prep RoomSign inStart free

On-call & incidents questions.

70 on-call & incidents questions from the bank, ready to practice out loud.

Practice on-call & incidents

Learn the ideas first · free with an account

Checklists, runbooks & the pre-shift briefingRoot cause: five whys, fishbone & causal layersSTAR and CARL: structuring your storyBlameless postmortems & debriefs that change somethingIncident command: severity, roles & the first hourPreserve the evidence, reconstruct the sequencePrevention that sticks: CAPA & the hierarchy of controlsSignal vs noise: is this spike real?

All 70 questions

Investigating unexplained error spikeIt's 2 a.m. Your dashboards show a 30% error rate on your payments service. No recent deploys, no obvious cause. Walk me through your first ten minutes.Incident responseMid–senior
Roll forward or keep diggingYou're 20 minutes into a major outage. The team is asking whether to roll forward a partial fix or keep digging. How do you make that call in the moment?Mitigation vs root cause tradeoffSenior–staff+
Communicating when scope is unclearCustomers are reporting issues but your status page is still green. The incident is real but the scope is unclear. What goes out, and when?Customer communication during outageSenior–staff+
Post-incident recovery and follow-upYour team just resolved a 4-hour incident. What does the next 24 hours look like for you?Postmortem judgmentSenior–staff+
Postmortem after junior engineer's mistakeA junior engineer's push caused a major incident. How do you handle the postmortem conversation in a way that protects them and the team's learning?Blameless cultureSenior–staff+
Responding with outdated runbookYou're paged for high error rates, but the on-call runbook for this service is years out of date. What do you do?Incident responseSenior–staff+
Disabling feature used by executiveMitigation requires temporarily disabling a feature that the CFO uses every Monday morning. You're at 8 a.m. Sunday. How do you handle the trade-off?Mitigation vs root cause tradeoffSenior–leadership
Updating customers without root causeTwo hours into an outage, the engineering team hasn't found the cause. What's in your hourly customer update?Customer communication during outageMid–senior
Taking over mid-investigation handoffYou arrive on-call to find an open incident that the previous on-call left mid-investigation. How do you take over?Incident responseMid–senior
Reframing human error in postmortemYou're writing a postmortem where the root cause was 'human error'. Walk me through how you'd reframe it.Postmortem judgmentSenior–staff+
Managing panicked teammate during incidentDuring an incident, a teammate is making everything worse — guessing, panicking, contradicting. How do you handle them without escalating the chaos?Blameless cultureSenior–staff+
Single customer impact responseAn incident is affecting only a single high-value customer, but most others. How do you weight the response and comms?Incident responseSenior–leadership
Reverting deploy loses other fixesYou can mitigate by reverting a deploy from three days ago, but doing so loses other fixes shipped since. Walk through your decision.Mitigation vs root cause tradeoffSenior–staff+
First status page updateWalk me through the first status-page update you'd write 5 minutes into an outage where you barely know anything.Customer communication during outageMid–senior
Pattern of monitoring gap findingsYour team's last three postmortems all blame 'monitoring gaps'. What does that pattern tell you and what do you do?Postmortem judgmentSenior–staff+
Unresponsive team caused the incidentYou realize mid-incident that the cause is a database change made by a different team. They're not responding. What do you do?Incident responseSenior–staff+
Postmortem with uncomfortable learningWalk me through a postmortem where the most useful learning was something the team didn't want to talk about.Blameless cultureSenior–leadership
Assuring CEO without root causeAn incident is mitigated but you don't know the root cause yet. The CEO wants to know if it could happen again. What do you say?Mitigation vs root cause tradeoffSenior–leadership
Investigating without monitoring visibilityYour dashboards are out and you can't see anything during a suspected incident. What do you do?Incident responseSenior–staff+
All-clear message after false recoveriesWalk me through the final all-clear customer message for an incident that was severe and had two false-positive recoveries earlier.Customer communication during outageSenior–staff+
Stale action itemsAn incident's action items have been open for six months. The original team doesn't own them anymore. What do you do?Postmortem judgmentSenior–leadership
Defensiveness and blame in postmortemHow do you keep a post-incident review productive when one person is clearly defensive and one is clearly looking for blame?Blameless cultureSenior–leadership
Solo escalation and escalation timingYou're on-call solo and an incident is escalating fast. When do you wake people up and who?Incident responseMid–senior
Balancing mitigation against user experienceYour mitigation is working but degrades user experience visibly. How long do you stay on the mitigation before pushing through to root cause?Mitigation vs root cause tradeoffSenior–staff+
Public social media during investigationA customer is publicly tweeting about your outage while you're still investigating. How does that change your comms approach, if at all?Customer communication during outageSenior–leadership
Postmortem substance versus complianceWalk me through what makes a postmortem worth writing versus one that's just compliance theatre.Postmortem judgmentSenior–leadership
Shifting blame culture in postmortemsYou inherited an on-call rotation where postmortems are seen as career-damaging. How do you start to shift that?Blameless cultureSenior–leadership
Alert firing silentlyYou're on-call and an alert is firing repeatedly with no apparent customer impact. What's your read and your next move?Incident responseMid–senior
Incident during critical sales demoAn incident hits during a sales-critical demo for the company's biggest prospect. How do you handle comms differently, if at all?Customer communication during outageSenior–leadership
First hour as sev1 commanderWalk me through your first hour as on-call commander for an active sev1 incident affecting checkout for an entire region.Incident responseSenior–leadership
Responding to unfamiliar service pagesYou're on-call and get paged at 3 a.m. for high CPU usage on the API service. You've never been on-call for this service before. What are your first three steps?Incident responseEntry–mid
Reverting versus fixing during degradationYou've identified the problem: a bad config change. You can revert it in 2 minutes or spend 15 minutes tweaking it to fix the issue properly. The service is degraded but not down. What do you choose?Mitigation vs root cause tradeoffEntry–mid
Customer updates with incomplete informationIt's 10 minutes into an incident and you need to post the first customer update, but you only know 'login is slow for some users' — you don't know why or how many. What do you write?Customer communication during outageEntry–mid
Reframing blame-focused root causesYou're reading a postmortem from another team. The root cause section says 'engineer forgot to test before deploying.' Does this need rewriting, and if so, how?Blameless cultureEntry–mid
Writing your first postmortemAn incident is resolved and your senior asks you to write the postmortem. You've never written one before. What sections do you include and what's the purpose of each?Postmortem judgmentEntry–mid
Mitigation trade-offs with side effectsYou've mitigated an incident by turning off a background job, but users are now missing email notifications. The root cause fix will take 3 hours. Do you keep the mitigation or roll it back?Mitigation vs root cause tradeoffEntry–mid
Customer ETA requests mid-incidentYou're 30 minutes into an incident. A customer emails support asking for an ETA on the fix. Support forwards it to you. What do you tell them to reply?Customer communication during outageEntry–mid
Redirecting blame in postmortem meetingsDuring a postmortem review meeting, a teammate keeps saying 'I should have caught this in code review.' How do you redirect the conversation?Blameless cultureEntry–mid
Investigating low-signal error alertsAn alert fires for elevated 500 errors, but when you check logs, you only see 3 errors in the last hour out of 50,000 requests. What do you do?Incident responseEntry–mid
Identifying vague action itemsYour postmortem has five action items. One is 'improve monitoring,' one is 'add more tests,' and three are code fixes. Your manager says one of these is too vague. Which one and why?Postmortem judgmentEntry–mid
Discovering your own mistake mid-incidentYou're investigating an incident and discover the root cause was a mistake you made two days ago. How do you communicate this to the team during the incident?Blameless cultureEntry–mid
Timing the all-clear messageThe incident is over and service is stable. Your manager asks whether to send an all-clear message now or wait another hour to be sure. What do you recommend?Customer communication during outageEntry–mid
Service restart with in-flight transactionsYou're on-call when monitoring alerts for database connection pool exhaustion. The immediate fix is to restart the service, but that will drop all in-flight transactions. How do you decide?Mitigation vs root cause tradeoffMid–senior
Triaging conflicting incident theoriesThree different teams are pinging you during an active incident, each convinced their theory is correct. You have limited time to investigate. How do you triage conflicting hypotheses?Incident responseSenior–staff+
External comms during long fixesYou've identified the root cause of an outage, but the fix will take six hours to deploy safely. What do you communicate externally in the meantime, and how often?Customer communication during outageMid–staff+
Action items when controls failedA postmortem reveals that an incident was caused by a config change that passed code review and had test coverage. How do you frame the action items?Postmortem judgmentSenior–leadership
Addressing emergency pipeline bypassesAn engineer bypassed the deployment pipeline during an incident to push a hotfix directly to production. It worked, but violated policy. How do you address this in the retro?Blameless cultureSenior–leadership
Investigating unfamiliar services on-callYou're 15 minutes into an incident when you realize the on-call rotation was never updated and you're investigating a service you've never touched. What's your approach?Incident responseEntry–mid
Monitoring gaps in incident timelinesYour incident timeline shows the issue started two hours before the first page went out. Leadership wants to know why monitoring didn't catch it sooner. What do you say?Postmortem judgmentStaff+–leadership
CEO questions during active incidentsDuring an incident, your CEO posts in Slack asking for a detailed explanation. The team is still investigating. How do you respond to them without breaking focus?Customer communication during outageSenior–leadership
Silenced alerts during incident responseYou discover mid-incident that the last three alerts for this same issue were silenced without investigation. How does that change your response now?Incident responseMid–senior
Mitigated but unexplained incidentThe mitigation you deployed stopped the bleeding, but you still don't understand why the incident happened. When do you end the incident call and move to async investigation?Mitigation vs root cause tradeoffMid–staff+
Unresponsive vendor causing outageA vendor's API is causing your outage, and they're unresponsive. You can disable the integration, but it powers a core feature. Walk me through the next 30 minutes.Mitigation vs root cause tradeoffMid–senior
Customer-facing incident report balanceYou're writing a customer-facing incident report. Engineering wants to include every technical detail; marketing wants to say as little as possible. How do you find the middle ground?Customer communication during outageSenior–leadership
Prioritizing postmortem action itemsAn incident postmortem identifies five contributing factors. The team wants to fix all of them. You have budget and time for two. How do you prioritize?Postmortem judgmentStaff+–leadership
Redirecting guilt to systemsDuring a postmortem, an engineer says 'I should have caught this in code review.' How do you steer the conversation back to systems and away from individual guilt?Blameless cultureMid–senior
Repeated false alarm pagesYou're on-call and get paged at 3 a.m. for an issue that turns out to be a false alarm due to a flaky test. This is the fourth time this month. What do you do after acknowledging the page?Incident responseEntry–mid
Complaints after incident resolutionYour metrics show recovery, but customer support is still receiving complaints an hour after you declared the incident resolved. What's your next move?Incident responseMid–senior
Rollback conflicts with product roadmapAn incident was resolved by rolling back a deploy, but the feature that deploy contained is on the roadmap for a major customer demo tomorrow. How do you handle the trade-off?Mitigation vs root cause tradeoffSenior–leadership
Sharing embarrassing root causesYou're drafting an incident update for customers. The root cause is embarrassing—a typo in a critical config file. Do you share that level of detail, and why or why not?Customer communication during outageMid–staff+
Untested failover mechanism failureA high-severity incident occurred because an automated failover didn't trigger as designed. The postmortem reveals the failover was never tested in production. What are your action items?Postmortem judgmentStaff+–leadership
On-call engineer missed critical pageAn on-call engineer missed the initial page because they were in a movie theater. The incident escalated as a result. How do you address this without creating a fear-based on-call culture?Blameless cultureSenior–leadership
Partial outage with unclear patternYou're paged for a partial outage affecting 10% of users, but you can't identify what those users have in common. How do you structure your investigation?Incident responseMid–senior
Maintenance mode during business hoursThe fastest mitigation path requires temporarily taking the site into maintenance mode during business hours. You're the most senior person on the call. How do you make and communicate that decision?Mitigation vs root cause tradeoffStaff+–leadership
Customer reports before internal alertsCustomers are tweeting about an outage before your internal monitoring has alerted. How do you validate the issue and decide whether to escalate?Incident responseEntry–mid
Documentation ownership after incidentYour incident retrospective identifies that better documentation would have reduced time-to-mitigation by an hour. Who is responsible for writing it, and how do you ensure it actually happens?Postmortem judgmentSenior–staff+
Defensive engineer in postmortemAn incident was caused by a well-intentioned performance optimization that introduced a race condition. The engineer is defensive in the postmortem. How do you facilitate the discussion?Blameless cultureSenior–leadership
Reconstructing deployment failureWalk me through how you'd reconstruct a software deployment failure when the rollback destroyed the failed state and logs were not persisted.Postmortem judgmentMid–senior
Preserving cloud system evidenceWalk me through how you'd preserve digital evidence when the failed system is cloud-based and the provider's data retention policy is seven days.Postmortem judgmentMid–senior
Trading algorithm erroneous executionA financial trading algorithm executed erroneous trades for 200 milliseconds before circuit breakers engaged. Walk me through your initial root cause hypotheses.Mitigation vs root cause tradeoffSenior–staff+