How we caused a route leak - Anatomy of a routing incident
Where
Community Stage
Disobey 2023
Duration
19 min
Walks through how a routing mistake at a European ISP leaked routes and disrupted traffic for other networks.
Summary
The speaker, who runs an ISP network across Europe, describes a routing incident in which their network accidentally leaked routes and disrupted traffic for other providers. The talk opens with background on how the company grew from a small network into an IP transit provider that also hosts research sponsorships, and on how customers and upstreams are filtered using route objects and AS sets stored in routing registries.
The middle of the talk follows the sequence of events. A runaway flow-export daemon on a busy router was killed, and a support ticket from a company that was receiving traffic it should not have revealed that the ISP was announcing routes it had no business passing on. Investigation showed that an AS set contained nested entries that pulled in a chain of networks, and that the filtering had failed in several layers at once. These included a router configuration that had been corrupted and fixed by hand, a missing deny rule that let everything through, and an internet exchange route server that accepted a full table without any plausibility check.
The takeaway is that prefix filtering based on registry data is fragile, and a single missing line can expose wider parts of the internet. The speaker argues for plausibility checks based on historical peering data and for better automation of filter generation, while noting that such changes are likely to take years to roll out across the internet.
Related talks
Talks·50 min
Christian Demers
Shows how ransomware in hospitals harms patients, and how emergency medicine ideas could guide recovery of clinical systems.
Talks·39 min
Richard Suls
Walks through a prison's ransomware breach, from a phishing foothold to a public data leak and the response failures behind it.
Talks·36 min
Marina Bochenkova
Recounts a stalled enterprise effort to retire NTLM, with its rollout missteps and the lessons drawn from them.


