This morning, users of Google around the world were unable to access many of the company’s services due to a routing leak in India. Beginning at 08:58 UTC Indian broadband provider Hathway (AS17488) incorrectly announced over 300 Google prefixes to its Indian transit provider Bharti Airtel (AS9498).
Bharti in turn announced these routes to the rest of the world, and a number of ISPs accepted these routes including US carriers Cogent (AS174), Level 3 (AS3549) as well as overseas incumbent carriers Orange (France Telecom, AS5511), Singapore Telecom (Singtel, AS7473) and Pakistan Telecom (PTCL, AS17557). Like many providers around the world, Hathway peers with Google so that their customers have more direct connectivity with Google services. But when that private relationship enters the public Internet the result can be accidental global traffic redirection.
Last fall, I wrote two blog posts here and here about the issues surrounding routing leaks such this one. Routing leaks happen regularly and can have the effect of misdirecting global traffic. Last month, I gave a talk in the NANOG 63 Peering Forum entitled “Hidden Risks of Peering” that went over some examples of routing leaks like this one.
Below is a graph showing the timeline of the incident for one of the 336 prefixes involved. Bharti (AS9498) should never have been seen as an upstream of Hathway (AS17488) for any Google prefixes. As the graph shows, only a portion of the Internet accepted these routes: the providers who peer with or sell to Bharti, and who failed to filter Bharti’s BGP announcements.
trace from Bratislava, Slovakia to 126.96.36.199 (Google) at 09:09 Mar 12, 2015
4 188.8.131.52 te0-0-2-3.nr11.b027220-0.bts01.atlas.cogentco.com 1.95
5 184.108.40.206 te0-0-2-0.agr11.bts01.atlas.cogentco.com 1.908
6 220.127.116.11 te0-3-0-5.ccr21.bts01.atlas.cogentco.com 1.574
7 18.104.22.168 be2222.ccr21.vie01.atlas.cogentco.com 3.552
8 22.214.171.124 be2200.ccr21.muc01.atlas.cogentco.com 9.818
9 126.96.36.199 be2023.ccr21.zrh01.atlas.cogentco.com 14.892
10 188.8.131.52 be2024.ccr21.mrs01.atlas.cogentco.com 27.371
11 184.108.40.206 33.255
12 220.127.116.11 (Airtel Limited, India) 158.796
14 18.104.22.168 (Hathway, Mumbai, India) 283.586
16 22.214.171.124 (Google, Mumbai, India) 282.664
17 126.96.36.199 (Google, Mumbai, India) 294.956
Highly peered content networks such as Google are uniquely vulnerable to this type of accidental traffic misdirection. Once routes are handed off to a peer, that peer can make a mistake and re-route your traffic. Vigilance is critically important: we know that Hathway was a risky peer for Google because just 22 hours previously, Dyn observed Hathway leaking 134 Google prefixes to Bharti for less than a minute. Careful monitoring of global routing is the only way for enterprises to detect these situations before they become front page news.