On-callHardoc-g348

Subject Data incidentsLevel Senior–Staff~40 minCommon in Reliability & on-call · Distributed systems interviewsIndustries Technology

Question

A multi-tenant SaaS shards data by `tenant_id`; every write must include it as the partition key. At 11:00 a feature flag enabled a new 'fast path' write handler for 5% of traffic. By 11:40, support escalates: a few tenants see *another tenant's* records appearing in their lists, and some of their own new records are 'missing'. Dashboards: write success 100%, error rate flat, no auth failures. The new handler derives `tenant_id` from a cached request context. How do you triage, stop the cross-tenant contamination immediately, and remediate?

What a strong answer looks like

Stop the bleeding first (mitigate), then form hypotheses from real signals. Separate root cause from symptom, communicate status as you go, and close with what prevents a repeat.

Learn the concepts

Diagram & narrate the incident

Loading whiteboard…

Run or narrate your approach, then ask the coach.