Questions

Status
ObservabilityLevel 5Open topic
Synchronous logging blocking request threadsMediumOn-call incidentOn-call incidentMedium30 minSnowflake password in CI logsMediumOn-call incidentOn-call incidentMedium30 minBackup upload silent failureMediumOn-call incidentOn-call incidentMedium30 minSecrets logged at scaleMediumOn-call incidentStripe styleOn-call incidentStripe styleMedium30 minTreatment revenue per user dropsMediumOn-call incidentNetflix styleOn-call incidentNetflix styleMedium30 minModel calibration decays without retrainingMediumOn-call incidentMeta styleOn-call incidentMeta styleMedium30 minIdentity provider token endpoint downHardOn-call incidentOn-call incidentHard40 minRCE exploitation at scaleHardOn-call incidentOn-call incidentHard50 minCardinality explosion in metrics backendHardOn-call incidentOn-call incidentHard40 minStateless JWT allows post-logout accessHardOn-call incidentOn-call incidentHard40 minExfiltration tuned under DLP thresholdHardOn-call incidentOn-call incidentHard45 minLabel freshness reads zeroHardOn-call incidentStripe styleOn-call incidentStripe styleHard35 min
Gray failure disk awaitHardOn-call incidentAmazon styleOn-call incidentAmazon styleHard40 min
Dependencies and deploysLevel 6Open topic
Error rate spike after deployEasyOn-call incidentOn-call incidentEasy15 minInvalid configuration blocks startupEasyOn-call incidentOn-call incidentEasy15 minEmails stop sending silentlyEasyOn-call incidentOn-call incidentEasy15 minShipping rates fail after a vendor retires an API versionEasyOn-call incidentOn-call incidentEasy15 minSome photo uploads fail since this morningEasyOn-call incidentOn-call incidentEasy15 minCache stampede after rolling restartMediumOn-call incidentOn-call incidentMedium30 minCanary blind to currencyMediumOn-call incidentStripe styleOn-call incidentStripe styleMedium30 minTimeout reduction triggers retry stormMediumOn-call incidentOn-call incidentMedium30 minStale personalized content after rollbackMediumOn-call incidentOn-call incidentMedium35 minPayment provider degradationMediumOn-call incidentStripe styleOn-call incidentStripe styleMedium30 minVolumetric DNS amplificationMediumOn-call incidentOn-call incidentMedium35 minJobs transcode multiple times simultaneouslyMediumOn-call incidentAmazon styleOn-call incidentAmazon styleMedium30 minRegional error rate at 100%MediumOn-call incidentNetflix styleOn-call incidentNetflix styleMedium35 minHome screen returns 500MediumOn-call incidentNetflix styleOn-call incidentNetflix styleMedium30 minRead-only transaction after failoverMediumOn-call incidentOn-call incidentMedium30 minDNS resolver contention wavesMediumOn-call incidentStripe styleOn-call incidentStripe styleMedium30 minUpstream returns empty responseMediumOn-call incidentOn-call incidentMedium35 minPayments processed twice from webhooksMediumOn-call incidentStripe styleOn-call incidentStripe styleMedium35 minHealth check thread pool contentionMediumOn-call incidentOn-call incidentMedium35 minSLA recompute retroactively breaches ticketsMediumOn-call incidentServiceNow styleOn-call incidentServiceNow styleMedium35 minFeature flag service outage fallbackMediumOn-call incidentOn-call incidentMedium30 minPayment provider returns 401MediumOn-call incidentOn-call incidentMedium20 minDNS resolution failsMediumOn-call incidentOn-call incidentMedium20 minFeature flag rollout breaks endpointMediumOn-call incidentOn-call incidentMedium30 minModel size GPU saturationMediumOn-call incidentMeta styleOn-call incidentMeta styleMedium30 minModel ranking score collapseMediumOn-call incidentAmazon styleOn-call incidentAmazon styleMedium30 minFeature store timeouts during backfillMediumOn-call incidentMeta styleOn-call incidentMeta styleMedium30 minModel lazy load cold startMediumOn-call incidentOpenAI styleOn-call incidentOpenAI styleMedium30 minDependency latency spikeMediumOn-call incidentOn-call incidentMedium30 minCold storage retrieval timeoutMediumOn-call incidentAmazon styleOn-call incidentAmazon styleMedium30 minCritical dependency CVEMediumOn-call incidentOn-call incidentMedium35 minModel registry outage blocks rolloutMediumOn-call incidentDatabricks styleOn-call incidentDatabricks styleMedium30 minTokenizer version mismatchMediumOn-call incidentOpenAI styleOn-call incidentOpenAI styleMedium30 minDNS TTL cache splitMediumOn-call incidentOn-call incidentMedium35 minDNS resolver latency spike fleet-wideMediumOn-call incidentOn-call incidentMedium35 minAccount field shows wrong valuesMediumOn-call incidentOn-call incidentMedium20 minRetry amplification during vendor outageHardOn-call incidentStripe styleOn-call incidentStripe styleHard35 min