Questions
Status
Prolonged degradation after Redis failoverHardOn-call incidentOn-call incidentHard35 minTail latency from scatter-gather fan-outHardOn-call incidentGoogle styleOn-call incidentGoogle styleHard40 minFeature flag rollout saturates downstreamHardOn-call incidentOn-call incidentHard40 minCanary session token skewHardOn-call incidentOn-call incidentHard40 minMobile client version skewHardOn-call incidentMeta styleOn-call incidentMeta styleHard40 minIn-flight requests lostHardOn-call incidentStripe styleOn-call incidentStripe styleHard40 minDual-write flag causes read-write skewHardOn-call incidentOn-call incidentHard45 minSplit-brain write conflictsHardOn-call incidentOn-call incidentHard45 minRoot CA certificate expiredHardOn-call incidentOn-call incidentHard40 minSynchronized reconnect stormHardOn-call incidentOn-call incidentHard40 minSkewed partition key starves consumersHardOn-call incidentDatabricks styleOn-call incidentDatabricks styleHard45 minUnscoped delete wipes collectionHardOn-call incidentOn-call incidentHard45 minDual write timeout divergenceHardOn-call incidentOn-call incidentHard45 minOffset replay cascading side effectsHardOn-call incidentDatabricks styleOn-call incidentDatabricks styleHard40 minElasticsearch split brain divergenceHardOn-call incidentOn-call incidentHard45 minDST timezone wall clock divergenceHardOn-call incidentOn-call incidentHard40 minUnauthorized data exfiltrationHardOn-call incidentOn-call incidentHard45 minActive ransomware outbreakHardOn-call incidentOn-call incidentHard50 minShared secret rotationHardOn-call incidentStripe styleOn-call incidentStripe styleHard45 minCI IAM role privilege escalationHardOn-call incidentAmazon styleOn-call incidentAmazon styleHard45 minDouble-extortion ransomware threatHardOn-call incidentOn-call incidentHard50 minSSRF to credential theftHardOn-call incidentAmazon styleOn-call incidentAmazon styleHard45 minJWT signing key rotation after leakHardOn-call incidentOn-call incidentHard45 minInsider data access spikeHardOn-call incidentOn-call incidentHard45 minOut-of-order events post-reshardHardOn-call incidentAmazon styleOn-call incidentAmazon styleHard40 minKafka partition skew from hot customerHardOn-call incidentOn-call incidentHard35 minRabbitMQ retry loop amplifies failuresHardOn-call incidentStripe styleOn-call incidentStripe styleHard40 minEvent ordering violation from producer retriesHardOn-call incidentOn-call incidentHard40 minSQS FIFO queue throughput collapsesHardOn-call incidentAmazon styleOn-call incidentAmazon styleHard35 minLag spikes during rolling restartHardOn-call incidentOn-call incidentHard40 minRecords lost after worker crashesHardOn-call incidentAmazon styleOn-call incidentAmazon styleHard40 minSwap thrashing nodeHardOn-call incidentOn-call incidentHard40 minInotify watch limit exhaustion from file explosionHardOn-call incidentOn-call incidentHard40 minNAT gateway port exhaustion from cluster scaleHardOn-call incidentAmazon styleOn-call incidentAmazon styleHard40 minSwap thrashing during RDB saveHardOn-call incidentOn-call incidentHard40 minInode exhaustion in flat cache directoryHardOn-call incidentOn-call incidentHard40 minRetry amplification from slow upstreamHardOn-call incidentStripe styleOn-call incidentStripe styleHard40 minCache collapse metastable failureHardOn-call incidentMeta styleOn-call incidentMeta styleHard45 minEdits reverting to older valuesHardOn-call incidentOn-call incidentHard40 minInventory oversells after region partitionHardOn-call incidentShopify styleOn-call incidentShopify styleHard45 minP99 latency tripledHardOn-call incidentOn-call incidentHard40 minEtcd cluster writes failingHardOn-call incidentOn-call incidentHard45 minAutovacuum blocked by idle transactionHardOn-call incidentOn-call incidentHard40 minSingle stream partition throttles DynamoDBHardOn-call incidentAmazon styleOn-call incidentAmazon styleHard35 minNoisy neighbor disk latencyHardOn-call incidentOn-call incidentHard35 minQueue buildup hides server latencyHardOn-call incidentOn-call incidentHard40 minRetry storm after brief outageHardOn-call incidentGoogle styleOn-call incidentGoogle styleHard40 minHumongous allocation full GCHardOn-call incidentOn-call incidentHard35 minQueue depth increases tail latencyHardOn-call incidentGoogle styleOn-call incidentGoogle styleHard35 minConsumer lag on few partitionsHardOn-call incidentOn-call incidentHard35 min