Questions

Status
Databases in productionLevel 6Open topic
Kafka leader crash data lossHardOn-call incidentDatabricks styleOn-call incidentDatabricks styleHard40 minSchema cache skew silent corruptionHardOn-call incidentDatabricks styleOn-call incidentDatabricks styleHard40 minEphemeral port exhaustion from disabled keep-aliveHardOn-call incidentStripe styleOn-call incidentStripe styleHard40 minPostgreSQL data volume fills overnightHardOn-call incidentOn-call incidentHard40 minConnection limit metastable collapseHardOn-call incidentGoogle styleOn-call incidentGoogle styleHard45 minConnection pool storms after failoverHardOn-call incidentAmazon styleOn-call incidentAmazon styleHard35 minIndex corruption after unclean shutdownHardOn-call incidentOn-call incidentHard40 minNot-null constraint version skewHardOn-call incidentOn-call incidentHard40 minConnection slots reserved errorHardOn-call incidentOn-call incidentHard35 minConnection pool default shrinkHardOn-call incidentOn-call incidentHard35 minUpstream scale-in closes pooled connectionsHardOn-call incidentOn-call incidentHard40 minBlack Friday autoscaling exhausts database connectionsHardOn-call incidentShopify styleOn-call incidentShopify styleHard30 min
Schema migration half-applied across shardsHardOn-call incidentOn-call incidentHard40 minReplication lag from backfillHardOn-call incidentOn-call incidentHard35 minDivergent data after failoverHardOn-call incidentOn-call incidentHard40 minMesh sidecar cannot assign addressHardOn-call incidentOn-call incidentHard35 minQuery plan degrades after autovacuumHardOn-call incidentOn-call incidentHard35 minOrders lost after failoverHardOn-call incidentStripe styleOn-call incidentStripe styleHard40 minIndex corruption after OS upgradeHardOn-call incidentOn-call incidentHard40 minCheckpoint latency spikesHardOn-call incidentOn-call incidentHard35 minPartition pruning regressionHardOn-call incidentOn-call incidentHard35 minLong query blocking transaction poolHardOn-call incidentOn-call incidentHard40 minBackfill incomplete at code cutoverHardOn-call incidentOn-call incidentHard40 minStreaming clients thundering herdHardOn-call incidentOn-call incidentHard40 minEnum value old client rejectionHardOn-call incidentOn-call incidentHard40 minReplication lag read inconsistencyHardOn-call incidentNetflix styleOn-call incidentNetflix styleHard40 minRace condition in refactored order write pathHardOn-call incidentOn-call incidentHard45 minStorage node rebalance cascadesHardOn-call incidentOn-call incidentHard40 minStandby promotion stalledHardOn-call incidentOn-call incidentHard45 minMigration holding exclusive lockHardOn-call incidentOn-call incidentHard40 minTransaction ID wraparound approachingHardOn-call incidentOn-call incidentHard45 minInserts fail on month boundaryHardOn-call incidentOn-call incidentHard40 minQuery model embedding mismatchHardOn-call incidentOpenAI styleOn-call incidentOpenAI styleHard35 minNAT connection timeoutHardOn-call incidentOn-call incidentHard40 minElasticsearch OOM from expensive aggregationHardOn-call incidentOn-call incidentHard40 minPlan cache parameter sensitivityHardOn-call incidentMicrosoft styleOn-call incidentMicrosoft styleHard35 minLogical replication apply lag explodesHardOn-call incidentOn-call incidentHard40 minSequence reached maximum valueHardOn-call incidentOn-call incidentHard35 minIndex larger than tableHardOn-call incidentOn-call incidentHard40 minBilling pods crash on start during a node upgradeHardOn-call incidentOn-call incidentHard25 min
Incident responseLevel 7Open topic
TLS certificate expiredEasyOn-call incidentOn-call incidentEasy15 minOne pod fails every request but passes health checksEasyOn-call incidentOn-call incidentEasy15 minOne API key sends a third of all trafficEasyOn-call incidentOn-call incidentEasy15 minProduct images vanish after a storage cleanup ruleEasyOn-call incidentOn-call incidentEasy20 minHundreds of orders go through at zero dollarsEasyOn-call incidentOn-call incidentEasy15 minCache stampede on aligned expiryMediumOn-call incidentOn-call incidentMedium30 minInventory shows in stock intermittentlyMediumOn-call incidentShopify styleOn-call incidentShopify styleMedium30 minGC pauses after serialization refactorMediumOn-call incidentOn-call incidentMedium30 minJVM warmup tail latencyMediumOn-call incidentOn-call incidentMedium30 minFeature flag cache staleness after kill-switchMediumOn-call incidentOn-call incidentMedium30 min