Questions
Status
Snapshot bloat fills volumeMediumOn-call incidentOn-call incidentMedium30 minHot partition throttledMediumOn-call incidentAmazon styleOn-call incidentAmazon styleMedium30 minKubernetes node disk full from debug logsMediumOn-call incidentOn-call incidentMedium30 minPredictable traffic spike causes 503sMediumOn-call incidentOn-call incidentMedium30 minBlue-green cutover causes brief spikeMediumOn-call incidentCitadel styleOn-call incidentCitadel styleMedium30 minLLM API token limit exceededMediumOn-call incidentOpenAI styleOn-call incidentOpenAI styleMedium35 minMongoDB CPU maxes after deployMediumOn-call incidentOn-call incidentMedium30 minDisk saturation from debug loggingMediumOn-call incidentOn-call incidentMedium30 minCache hit ratio collapsesHardOn-call incidentOn-call incidentHard35 minMetrics server down breaks autoscalingHardOn-call incidentOn-call incidentHard40 minNo prioritization under overloadHardOn-call incidentGoogle styleOn-call incidentGoogle styleHard40 minScale-in terminates in-flight requestsHardOn-call incidentOn-call incidentHard40 minBackfill throttles live trafficHardOn-call incidentAmazon styleOn-call incidentAmazon styleHard40 minBroker memory alarm blocks publishesHardOn-call incidentOn-call incidentHard40 minLatency spikes with regex validationHardOn-call incidentOn-call incidentHard40 minDirect memory leak in Netty streaming responsesHardOn-call incidentOn-call incidentHard40 minStream consumer OOMKilled on restartHardOn-call incidentDatabricks styleOn-call incidentDatabricks styleHard40 minAutoscaler removal spiral during rampHardOn-call incidentOn-call incidentHard40 minRemote socket memory access latencyHardOn-call incidentOn-call incidentHard40 minSidecar memory climbingHardOn-call incidentOn-call incidentHard45 minOutlier detection cascades pod ejectionHardOn-call incidentGoogle styleOn-call incidentGoogle styleHard45 minHPA oscillates near target thresholdHardOn-call incidentOn-call incidentHard30 minRetry storm locks in saturationHardOn-call incidentGoogle styleOn-call incidentGoogle styleHard32 minFanout amplifies downstream loadHardOn-call incidentOn-call incidentHard30 minLoad balancer registration cappedHardOn-call incidentOn-call incidentHard28 minGarbage collection pauses degradeHardOn-call incidentOn-call incidentHard28 minDisk full truncates log recordsHardOn-call incidentOn-call incidentHard40 minRetry loop after recoveryHardOn-call incidentOn-call incidentHard35 minColocated workload interferenceHardOn-call incidentOn-call incidentHard40 minSidecar CPU throttlingHardOn-call incidentOn-call incidentHard40 minAutoscale metric lag during rampHardOn-call incidentOn-call incidentHard30 minIAM throttling during instance bootstrapHardOn-call incidentAmazon styleOn-call incidentAmazon styleHard30 minWarm pool depleted by deployHardOn-call incidentAmazon styleOn-call incidentAmazon styleHard30 minHPA and VPA thrashingHardOn-call incidentOn-call incidentHard30 minIn-flight requests drop during scale-inHardOn-call incidentOn-call incidentHard30 minRegional quota exhaustion on surgeHardOn-call incidentAmazon styleOn-call incidentAmazon styleHard30 minClient retry amplifies traffic surgeHardOn-call incidentGoogle styleOn-call incidentGoogle styleHard30 minNoisy neighbor starves podHardOn-call incidentOn-call incidentHard30 minShared dependency saturates on surgeHardOn-call incidentNetflix styleOn-call incidentNetflix styleHard30 minFixed-window limiter allows burstHardOn-call incidentStripe styleOn-call incidentStripe styleHard30 minGPU autoscaler CPU metric mismatchHardOn-call incidentOpenAI styleOn-call incidentOpenAI styleHard35 minS3 prefix throttling on burstHardOn-call incidentAmazon styleOn-call incidentAmazon styleHard35 minDDoS volumetric attackHardOn-call incidentOn-call incidentHard40 minTombstone read amplificationHardOn-call incidentOn-call incidentHard35 minGPU memory fragmentationHardOn-call incidentNvidia styleOn-call incidentNvidia styleHard40 minFIFO queue head-of-line blockingHardOn-call incidentAmazon styleOn-call incidentAmazon styleHard40 minGPU memory from large inputsHardOn-call incidentNvidia styleOn-call incidentNvidia styleHard35 minShadow traffic overloadHardOn-call incidentOpenAI styleOn-call incidentOpenAI styleHard35 minAI feature far over its daily budgetHardOn-call incidentOpenAI styleOn-call incidentOpenAI styleHard35 min