Skip to content

CodingLevel 7

Incident response

Easy

Medium

Hard

Continue Read the guide View as list

Cascading failuresHow a tiny blip in one service can accidentally DDOS your entire system via retries.3 min Cache & queue incidentsWhy one bad payload can block millions of healthy messages.3 min Data loss & corruptionWhat happens when two disconnected databases both think they are in charge.3 min Security incidentsStop the exfiltration first. Do forensics later.3 min Reading the golden signalsWhy your dashboard says "50ms average", but your biggest customers are timing out.3 min Network incidentsThe network is never perfectly reliable; cables get cut and routers crash.3 min Incident response & triageStop the bleeding first. Figure out why it happened later.3 min Thundering herds (cache miss storm)When a single expired cache item takes down your whole backend.3 min Split-brainA network partition tears one cluster into two halves, and if each half still thinks it's in charge, both keep writing and the data diverges.4 min Server-side request forgery (SSRF)If your server fetches whatever URL a user hands it, the attacker borrows your server's place inside the trusted network, and points it at things they could never reach themselves.3 min Data exfiltrationThe break-in is loud; the theft is quiet, data leaving slowly looks almost like normal traffic.3 min Delegating to AI agents safelyHand off the work, not the accountability, bound what it can touch, and check the trace where a wrong turn would compound.3 min Lateral movementThe breach rarely lands where it matters, the attacker hops from a low-value foothold to the crown jewels.4 min CDN time-to-live (TTL)Balancing server load against serving hopelessly outdated data.3 min Cold storageTrading speed for massive cost savings on data you rarely need.3 min Credential stuffingAttackers don't guess passwords, they replay millions of real ones leaked from somewhere else.3 min Inference queue backlogWhy buffering too many LLM requests makes things worse for everyone.3 min CDN invalidation stormWhen telling the cache to forget everything takes down your backend.3 min Model drift and calibrationWhen the world changes faster than your AI's underlying math.3 min OAuth token leakA bearer token is a key that opens any door it touches, so the moment it rides along inside a request it can quietly walk out the front door to a stranger's server.4 min Privilege escalationA low-privilege account quietly gains powers it was never granted, and starts acting as admin.3 min Silent replica divergenceThree copies of the same key quietly drift apart while every read still looks happy.3 min Write amplification and compactionOne write from your app becomes many writes on disk, the storage engine keeps re-copying the same data downhill.4 min A rebalance that halts with hot nodes still hotThe plan was right, even out the load, but the moves never drained, so the overloaded node stayed overloaded.4 min A restore that stallsThe progress bar climbed, then froze, and the volume sat in read-only limbo while the job still claimed to be running.4 min Batch scoring nulls (ML pipelines)When a single missing value ruins predictions for an entire batch.3 min Cache eviction stormsWhen protecting the cache accidentally destroys the database.3 min Cold storage SLAsDeciding how much you're willing to pay when lawyers demand old data.3 min Compaction stallsAn LSM-tree storage engine throttles your writes when the background merge falls behind the incoming flood.4 min Compromised CI tokenYour pipeline holds the keys to everything it deploys, steal its token and you inherit production.4 min Content delivery networksKeep a copy of the answer near the person asking. The first visitor pays the long trip; everyone after them gets it from next door.3 min Durability (bitrot and scrubbing)How storage systems prevent your data from slowly decaying.3 min Embedding store latencyWhy fetching high-dimensional vectors to feed an ML model destroys real-time performance.3 min Flagging bad inputs (ML)Teaching your ML model to explicitly recognize when data is missing or broken.3 min Fsync and commit latencyWhy guaranteeing you won't lose data makes your database slow.3 min Health flappingWhen a server repeatedly crashes, restarts, and immediately receives traffic before it's ready.3 min Late data (streaming ML)When a user's phone goes into a tunnel and events arrive out of order.3 min Model serving recommendation stale cacheA cache makes recommendations cheap and fast, until the model moves on and the cache keeps handing out yesterday's answer.4 min Multipart upload that never finishedA big file uploaded in pieces, then abandoned mid-flight, the pieces linger in the bucket, unseen but still billed.3 min Online learning feedback loopsWhen a recommendation algorithm becomes a self-fulfilling prophecy.3 min Phishing and session hijackingA stolen session cookie lets an attacker walk in as you without ever knowing your password.3 min Pipeline stall (staleness)When your real-time machine learning model is secretly operating on yesterday's data.3 min Point-in-time leakageWhen your ML model accidentally memorizes the future.3 min Point-in-time recoveryRewind a database to the instant before a bad write by replaying its change log onto an old snapshot.4 min Public bucket exposureA storage bucket left readable by anyone on the internet, no login, no token, just a URL away from your private data.4 min Read-after-write consistencyYou just saved a file, ask for it back, and the system says "never heard of it", because your read landed on a copy that hasn't caught up yet.3 min SSTable durability and corruptionHow databases detect bit-rot and silent disk failures.3 min Stale mounts (block storage)When the operating system is lying to the application about the disk.3 min The TLS handshakeTwo strangers shouting across a crowded room agree on a secret nobody else can reconstruct, prove who they are, then whisper privately.5 min Thundering restore stormWhen everyone reaches for the fire exit at once, the door becomes the fire, pace the crowd or nobody gets out.3 min Training job OOM (out of memory)Why you can't just load a 100-Gigabyte dataset into a 16-Gigabyte RAM chip.3 min Vector DB search latencyWhy searching a Vector DB inside an LLM's streaming generation loop is a terrible idea.3 min WAF blocking a SQL-injection spikeA wave of injection attempts hits at 2am. The firewall holds the line while you triage, contain, and chase down why the query was ever exploitable.4 min