MENU
bakersdozen-f

A Baker’s Dozen, 2014 Edition

v5

UK traffic diverted through Ukraine

March 12, 2015 Comments (36) Views: 42319 Engineering, Internet, Latency, Performance, Security

Routing Leak briefly takes down Google

Tweet about this on TwitterShare on FacebookShare on Google+Share on LinkedInShare on Reddit

This morning, users of Google around the world were unable to access many of the company’s services due to a routing leak in India. Beginning at 08:58 UTC Indian broadband provider Hathway (AS17488) incorrectly announced over 300 Google prefixes to its Indian transit provider Bharti Airtel (AS9498).

Bharti in turn announced these routes to the rest of the world, and a number of ISPs accepted these routes including US carriers Cogent (AS174), Level 3 (AS3549) as well as overseas incumbent carriers Orange (France Telecom, AS5511), Singapore Telecom (Singtel, AS7473) and Pakistan Telecom (PTCL, AS17557). Like many providers around the world, Hathway peers with Google so that their customers have more direct connectivity with Google services. But when that private relationship enters the public Internet the result can be accidental global traffic redirection.

Last fall, I wrote two blog posts here and here about the issues surrounding routing leaks such this one. Routing leaks happen regularly and can have the effect of misdirecting global traffic. Last month, I gave a talk in the NANOG 63 Peering Forum entitled “Hidden Risks of Peering” that went over some examples of routing leaks like this one.

Below is a graph showing the timeline of the incident for one of the 336 prefixes involved. Bharti (AS9498) should never have been seen as an upstream of Hathway (AS17488) for any Google prefixes. As the graph shows, only a portion of the Internet accepted these routes: the providers who peer with or sell to Bharti, and who failed to filter Bharti’s BGP announcements.


216.58.223.0_24_1426150200-2
Below is a traceroute from one of our servers in Bratislava, Slovakia earlier today showing traffic to Google redirected to India.


trace from Bratislava, Slovakia to 72.14.210.134 (Google) at 09:09 Mar 12, 2015
1  *
2  *
3  *
4  149.11.48.1      te0-0-2-3.nr11.b027220-0.bts01.atlas.cogentco.com   1.95
5  154.25.3.181     te0-0-2-0.agr11.bts01.atlas.cogentco.com            1.908
6  154.54.37.229    te0-3-0-5.ccr21.bts01.atlas.cogentco.com            1.574
7  130.117.1.50     be2222.ccr21.vie01.atlas.cogentco.com               3.552
8  130.117.49.1     be2200.ccr21.muc01.atlas.cogentco.com               9.818
9  130.117.0.250    be2023.ccr21.zrh01.atlas.cogentco.com               14.892
10 130.117.50.165   be2024.ccr21.mrs01.atlas.cogentco.com               27.371
11 149.6.155.182                                                        33.255
12 182.79.237.125   (Airtel Limited, India)                             158.796
13 *
14 202.88.147.66    (Hathway, Mumbai, India)                            283.586
15 *
16 72.14.235.29     (Google, Mumbai, India)                             282.664
17 209.85.255.131   (Google, Mumbai, India)                             294.956
18 *

Highly peered content networks such as Google are uniquely vulnerable to this type of accidental traffic misdirection. Once routes are handed off to a peer, that peer can make a mistake and re-route your traffic. Vigilance is critically important: we know that Hathway was a risky peer for Google because just 22 hours previously, Dyn observed Hathway leaking 134 Google prefixes to Bharti for less than a minute. Careful monitoring of global routing is the only way for enterprises to detect these situations before they become front page news.

Learn More About Dyn Internet IntelligenceLearn More

Not sure how your network is affected by events? Check out the tool our research team uses!

36 Responses to Routing Leak briefly takes down Google

  1. […] were cut off because of what Dyn Research Director of Internet Analysis Doug Madory identified as a “routing leak” from an Indian broadband Internet provider. The leak is similar to a 2012 incident caused by […]

    • Kim BellaGurl says:

      It has also been down in Western WA, (USA) since late last night, this smells like something else from here… Interesting routing late last night… 😉 maybe some interesting trace routes bouncing back out of the back side of the nifty mirror on CO, (US).. 😉

  2. […] were cut off because of what Dyn Research Director of Internet Analysis Doug Madory identified as a “routing leak” from an Indian broadband Internet provider. The leak is similar to a 2012 …read […]

  3. […] were cut off because of what Dyn Research Director of Internet Analysis Doug Madory identified as a “routing leak” from an Indian broadband Internet provider. The leak is similar to a 2012 incident caused by […]

  4. […] were cut off because of what Dyn Research Director of Internet Analysis Doug Madory identified as a “routing leak” from an Indian broadband Internet provider. The leak is similar to a 2012 incident caused by […]

  5. […] were cut off because of what Dyn Research Director of Internet Analysis Doug Madory identified as a “routing leak” from an Indian broadband Internet provider. The leak is similar to a 2012 incident caused by […]

  6. shy says:

    Wondering, how this was possible. If Hathway advertise a route which doesn’t belong to Hathway, it would have learned it through BGP Peering. So when it advertise back to the net, there obviously should have been a BGP loop. As path attribute should have detected that aren’t it ?

    • Doug Madory says:

      Hathway learned it from Google, but the AS path of the leak was as follows:

      … 9498 17488 15169 {prefix}

      So only Google (AS15169) and Hathway itself (AS17488) would have dropped a such route from Bharti (AS9498) based on the AS path loop prevention.

  7. […] were cut off because of what Dyn Research Director of Internet Analysis Doug Madory identified as a “routing leak” from an Indian broadband Internet provider. The leak is similar to a 2012 incident caused by […]

  8. […] traffic to more than 300 network prefixes belonging to Google to be directed to its own network, wrote Doug Madory, director of Internet analysis at Dyn, which studies global traffic […]

  9. […] traffic to more than 300 network prefixes belonging to Google to be directed to its own network, wrote Doug Madory, director of Internet analysis at Dyn, which studies global traffic […]

  10. […] traffic to more than 300 network prefixes belonging to Google to be directed to its own network, wrote Doug Madory, director of Internet analysis at Dyn, which studies global traffic […]

  11. […] traffic to more than 300 network prefixes belonging to Google to be directed to its own network, wrote Doug Madory, director of Internet analysis at Dyn, which studies global traffic […]

  12. […] traffic to more than 300 network prefixes belonging to Google to be directed to its own network, wrote Doug Madory, director of Internet analysis at Dyn, which studies global traffic […]

  13. […] a blog po​st, Madory explained that the mistake was a “routing leak,” which happens when a network provider […]

  14. […] were cut off because of what Dyn Research Director of Internet Analysis Doug Madory identified as a “routing leak” from an Indian broadband Internet provider. The leak is similar to a 2012 incident caused by […]

  15. […] traffic to more than 300 network prefixes belonging to Google to be directed to its own network, wrote Doug Madory, director of Internet analysis at Dyn, which studies global traffic […]

  16. […] traffic to more than 300 network prefixes belonging to Google to be directed to its own network, wrote Doug Madory, director of …read […]

  17. […] oltre 300 prefissi di rete appartenendo a Google per essere indirizzato verso la propria rete, ha scritto Doug Madory, direttore di analisi di Internet a Dyn, che studia i modelli di traffico globale del […]

  18. […] traffic to more than 300 network prefixes belonging to Google to be directed to its own network, wrote Doug Madory, director of Internet analysis at Dyn, which studies global traffic […]

  19. […] traffic to more than 300 network prefixes belonging to Google to be directed to its own network, wrote Doug Madory, director of Internet analysis at Dyn, which studies global traffic […]

  20. […] traffic to more than 300 network prefixes belonging to Google to be directed to its own network, wrote Doug Madory, director of Internet analysis at Dyn, which studies global traffic […]

  21. […] traffic to more than 300 network prefixes belonging to Google to be directed to its own network, wrote Doug Madory, director of Internet analysis at Dyn, which studies global traffic […]

  22. […] traffic to more than 300 network prefixes belonging to Google to be directed to its own network, wrote Doug Madory, director of Internet analysis at Dyn, which studies global traffic […]

  23. […] were cut off because of what Dyn Research Director of Internet Analysis Doug Madory identified as a “routing leak” from an Indian broadband Internet provider. The leak is similar to a 2012 incident caused by […]

  24. […] service for a short period of time due to a technical glitch. Users were cut off due to the routing leak from Indian broadband Internet provider Hathway. The leak is similar to a 2012 incident caused by […]

  25. […] traffic to more than 300 network prefixes belonging to Google to be directed to its own network, wrote Doug Madory, director of Internet analysis at Dyn, which studies global traffic […]

  26. […] were cut off because of what Dyn Research Director of Internet Analysis Doug Madory identified as a “routing leak” from an Indian broadband Internet provider. The leak is similar to a 2012 incident caused by […]

  27. […] => What caused the Google service interruption?. 12/03/2015. «This morning people on twitter reported that they were unable to reach Google services. Businessinsider followed up with a story in which they mentioned that the Google service interruption primarily involved European and Indian users (…).» Source : http://www.bgpmon.net/what-caused-the-google-service-interruption/ Billets en relation : 12/03/2015. Routing Leak briefly takes down Google : research.dyn.com/2015/03/routing-leak-briefly-takes-google/ […]

  28. […] traffic to more than 300 network prefixes belonging to Google to be directed to its own network, wrote Doug Madory, director of Internet analysis at Dyn, which studies global traffic […]

  29. […] “This morning, users of Google around the world were unable to access many of the company’s services due to a routing leak in India. Beginning at 08:58 UTC Indian broadband provider Hathway (AS17488) incorrectly announced over 300 Google prefixes to its Indian transit provider Bharti Airtel (AS9498).” wrote Madory in a blog post.  […]

  30. […] “This morning, users of Google around the world were unable to access many of the company’s services due to a routing leak in India. Beginning at 08:58 UTC Indian broadband provider Hathway (AS17488) incorrectly announced over 300 Google prefixes to its Indian transit provider Bharti Airtel (AS9498).” wrote Madory in a blog post.  […]

  31. […] “This morning, users of Google around the world were unable to access many of the company’s services due to a routing leak in India. Beginning at 08:58 UTC Indian broadband provider Hathway (AS17488) incorrectly announced over 300 Google prefixes to its Indian transit provider Bharti Airtel (AS9498).” wrote Madory in a blog post.  […]

  32. […] were cut off because of what Dyn Research Director of Internet Analysis Doug Madory identified as a “routing leak” from an Indian broadband Internet provider. The leak is similar to a 2012 incident caused by […]

  33. K. Sriram says:

    You were able to trace route to Google address. I wonder what fraction of the overall traffic did make it to Google via Airtel and Hathway while the route leak was active?

    Some work on Route Leaks that is currently in progress in the IETF:

    draft-ietf-grow-route-leak-problem-definition (GROW WG)

    draft-sriram-idr-route-leak-detection-mitigation (IDR WG)

    Sriram

Leave a Reply

Your email address will not be published. Required fields are marked *