On-call & incidents questions.
70 on-call & incidentsquestions from the bank — open to read. Pick one and practice it out loud; a coach note comes back in seconds.
Learn the ideas first
Checklists, runbooks & the pre-shift briefingRoot cause: five whys, fishbone & causal layersBlameless postmortems & debriefs that change somethingIncident command: severity, roles & the first hourPreserve the evidence, reconstruct the sequencePrevention that sticks: CAPA & the hierarchy of controlsSignal vs noise: is this spike real?Crisis communications: what to say before you know anything
All 70 questions
It's 2 a.m. Your dashboards show a 30% error rate on your payments service. No recent deploys, no obvious cause. Walk me through your first ten minutes.You're 20 minutes into a major outage. The team is asking whether to roll forward a partial fix or keep digging. How do you make that call in the moment?Customers are reporting issues but your status page is still green. The incident is real but the scope is unclear. What goes out, and when?Your team just resolved a 4-hour incident. What does the next 24 hours look like for you?A junior engineer's push caused a major incident. How do you handle the postmortem conversation in a way that protects them and the team's learning?You're paged for high error rates, but the on-call runbook for this service is years out of date. What do you do?Mitigation requires temporarily disabling a feature that the CFO uses every Monday morning. You're at 8 a.m. Sunday. How do you handle the trade-off?Two hours into an outage, the engineering team hasn't found the cause. What's in your hourly customer update?You arrive on-call to find an open incident that the previous on-call left mid-investigation. How do you take over?You're writing a postmortem where the root cause was 'human error'. Walk me through how you'd reframe it.During an incident, a teammate is making everything worse — guessing, panicking, contradicting. How do you handle them without escalating the chaos?An incident is affecting only a single high-value customer, but most others. How do you weight the response and comms?You can mitigate by reverting a deploy from three days ago, but doing so loses other fixes shipped since. Walk through your decision.Walk me through the first status-page update you'd write 5 minutes into an outage where you barely know anything.Your team's last three postmortems all blame 'monitoring gaps'. What does that pattern tell you and what do you do?You realize mid-incident that the cause is a database change made by a different team. They're not responding. What do you do?Walk me through a postmortem where the most useful learning was something the team didn't want to talk about.An incident is mitigated but you don't know the root cause yet. The CEO wants to know if it could happen again. What do you say?Your dashboards are out and you can't see anything during a suspected incident. What do you do?Walk me through the final all-clear customer message for an incident that was severe and had two false-positive recoveries earlier.An incident's action items have been open for six months. The original team doesn't own them anymore. What do you do?How do you keep a post-incident review productive when one person is clearly defensive and one is clearly looking for blame?You're on-call solo and an incident is escalating fast. When do you wake people up and who?Your mitigation is working but degrades user experience visibly. How long do you stay on the mitigation before pushing through to root cause?A customer is publicly tweeting about your outage while you're still investigating. How does that change your comms approach, if at all?Walk me through what makes a postmortem worth writing versus one that's just compliance theatre.You inherited an on-call rotation where postmortems are seen as career-damaging. How do you start to shift that?You're on-call and an alert is firing repeatedly with no apparent customer impact. What's your read and your next move?An incident hits during a sales-critical demo for the company's biggest prospect. How do you handle comms differently, if at all?Walk me through your first hour as on-call commander for an active sev1 incident affecting checkout for an entire region.You're on-call and get paged at 3 a.m. for high CPU usage on the API service. You've never been on-call for this service before. What are your first three steps?You've identified the problem: a bad config change. You can revert it in 2 minutes or spend 15 minutes tweaking it to fix the issue properly. The service is degraded but not down. What do you choose?It's 10 minutes into an incident and you need to post the first customer update, but you only know 'login is slow for some users' — you don't know why or how many. What do you write?You're reading a postmortem from another team. The root cause section says 'engineer forgot to test before deploying.' Does this need rewriting, and if so, how?An incident is resolved and your senior asks you to write the postmortem. You've never written one before. What sections do you include and what's the purpose of each?You've mitigated an incident by turning off a background job, but users are now missing email notifications. The root cause fix will take 3 hours. Do you keep the mitigation or roll it back?You're 30 minutes into an incident. A customer emails support asking for an ETA on the fix. Support forwards it to you. What do you tell them to reply?During a postmortem review meeting, a teammate keeps saying 'I should have caught this in code review.' How do you redirect the conversation?An alert fires for elevated 500 errors, but when you check logs, you only see 3 errors in the last hour out of 50,000 requests. What do you do?Your postmortem has five action items. One is 'improve monitoring,' one is 'add more tests,' and three are code fixes. Your manager says one of these is too vague. Which one and why?You're investigating an incident and discover the root cause was a mistake you made two days ago. How do you communicate this to the team during the incident?The incident is over and service is stable. Your manager asks whether to send an all-clear message now or wait another hour to be sure. What do you recommend?You're on-call when monitoring alerts for database connection pool exhaustion. The immediate fix is to restart the service, but that will drop all in-flight transactions. How do you decide?Three different teams are pinging you during an active incident, each convinced their theory is correct. You have limited time to investigate. How do you triage conflicting hypotheses?You've identified the root cause of an outage, but the fix will take six hours to deploy safely. What do you communicate externally in the meantime, and how often?A postmortem reveals that an incident was caused by a config change that passed code review and had test coverage. How do you frame the action items?An engineer bypassed the deployment pipeline during an incident to push a hotfix directly to production. It worked, but violated policy. How do you address this in the retro?You're 15 minutes into an incident when you realize the on-call rotation was never updated and you're investigating a service you've never touched. What's your approach?Your incident timeline shows the issue started two hours before the first page went out. Leadership wants to know why monitoring didn't catch it sooner. What do you say?During an incident, your CEO posts in Slack asking for a detailed explanation. The team is still investigating. How do you respond to them without breaking focus?You discover mid-incident that the last three alerts for this same issue were silenced without investigation. How does that change your response now?The mitigation you deployed stopped the bleeding, but you still don't understand why the incident happened. When do you end the incident call and move to async investigation?A vendor's API is causing your outage, and they're unresponsive. You can disable the integration, but it powers a core feature. Walk me through the next 30 minutes.You're writing a customer-facing incident report. Engineering wants to include every technical detail; marketing wants to say as little as possible. How do you find the middle ground?An incident postmortem identifies five contributing factors. The team wants to fix all of them. You have budget and time for two. How do you prioritize?During a postmortem, an engineer says 'I should have caught this in code review.' How do you steer the conversation back to systems and away from individual guilt?You're on-call and get paged at 3 a.m. for an issue that turns out to be a false alarm due to a flaky test. This is the fourth time this month. What do you do after acknowledging the page?Your metrics show recovery, but customer support is still receiving complaints an hour after you declared the incident resolved. What's your next move?An incident was resolved by rolling back a deploy, but the feature that deploy contained is on the roadmap for a major customer demo tomorrow. How do you handle the trade-off?You're drafting an incident update for customers. The root cause is embarrassing—a typo in a critical config file. Do you share that level of detail, and why or why not?A high-severity incident occurred because an automated failover didn't trigger as designed. The postmortem reveals the failover was never tested in production. What are your action items?An on-call engineer missed the initial page because they were in a movie theater. The incident escalated as a result. How do you address this without creating a fear-based on-call culture?You're paged for a partial outage affecting 10% of users, but you can't identify what those users have in common. How do you structure your investigation?The fastest mitigation path requires temporarily taking the site into maintenance mode during business hours. You're the most senior person on the call. How do you make and communicate that decision?Customers are tweeting about an outage before your internal monitoring has alerted. How do you validate the issue and decide whether to escalate?Your incident retrospective identifies that better documentation would have reduced time-to-mitigation by an hour. Who is responsible for writing it, and how do you ensure it actually happens?An incident was caused by a well-intentioned performance optimization that introduced a race condition. The engineer is defensive in the postmortem. How do you facilitate the discussion?Walk me through how you'd reconstruct a software deployment failure when the rollback destroyed the failed state and logs were not persisted.Walk me through how you'd preserve digital evidence when the failed system is cloud-based and the provider's data retention policy is seven days.A financial trading algorithm executed erroneous trades for 200 milliseconds before circuit breakers engaged. Walk me through your initial root cause hypotheses.