BGP prefix withdrawal

On-call incidentHard40 min

Your task

At 02:14 your edge starts dropping ~30% of inbound traffic for users in Europe; the rest of the world is fine. Synthetic probes from EU vantage points time out before reaching your edge IPs, while probes from US/APAC succeed. Your edge servers show a sharp drop in connections from EU ASNs but are otherwise healthy. A network engineer pushed a routing/peering change ~15 minutes before the incident: a BGP prefix announcement was adjusted to shift EU traffic onto a new transit provider. Traceroutes from affected EU networks die a few hops in, in the new transit's network. Walk through triage, mitigation, and prevention.

It covers

  • Bgp
  • Blackhole
  • Routing