Cross-tenant data leak

On-call incidentHard40 minSalesforce style

Your task

A multi-tenant SaaS shards data by tenant_id; every write must include it as the partition key. At 11:00 a feature flag enabled a new 'fast path' write handler for 5% of traffic. By 11:40, support escalates: a few tenants see another tenant's records appearing in their lists, and some of their own new records are 'missing'. Dashboards: write success 100%, error rate flat, no auth failures. The new handler derives tenant_id from a cached request context. How do you triage, stop the cross-tenant contamination immediately, and remediate?

It covers

  • Tenant isolation
  • Feature flag
  • Wrong partition
  • Cross tenant leak