MENU
shutterstock_128775836

Renesys Team Launches Dyn Research

leak_scenario1

Use Protection if Peering Promiscuously

November 6, 2014 Comments (34) Views: 44779 Engineering, Internet, Latency, Performance, Security

Chinese Routing Errors Redirect Russian Traffic

Tweet about this on TwitterShare on FacebookShare on Google+Share on LinkedInShare on Reddit

In recent weeks, Russian President Vladimir Putin announced a plan to enact measures to protect the Internet of Russia. In a speech to the Russian National Security Council he said, “we need to greatly improve the security of domestic communications networks and information resources.” Perhaps he should add Internet routing security to his list because, on a number of occasions in the past year, Russian Internet traffic (including domestic traffic) was re-routed out of the country due to routing errors by China Telecom. When international partners carry a country’s domestic traffic out of the country, only to ultimately return it, there are inevitable  security and performance implications.

Last year, Russian mobile provider Vimpelcom and China Telecom signed a network sharing agreement and established a BGP peering relationship. However, as can often happen with these relationships, one party can leak the routes received from the other and effectively insert itself into the path of the other party’s Internet communications. This happened over a dozen times in the past year between these two providers. This is a general phenomenon that occurs with some regularity but isn’t often discussed in BGP security literature. In this blog post, we’ll explore the issue via the lens of this single example. In a follow-on blog, we’ll explore several additional examples.

traceroute-v4

How peering links can go wrong

In the world of BGP parlance, the term peering is overloaded. The same term can refer to a simple BGP adjacency between two autonomous systems (AS) as well as a particular type of relationship between two ASes where traffic is exchanged without money changing hands. This is also referred to as a “settlement-free” relationship and is the definition of peering used in this blog post.

Peering relationships between ISPs are quite common. By exchanging traffic directly without using a transit provider, each ISP saves on transit costs — although as transit prices fall the overhead of maintaining these peering relationships becomes less attractive. Regardless, from a routing standpoint, routes exchanged between two peering ISPs typically stay within the customer cone of each ISP. In the diagram below, ISP A sends routes from its customers to its peer ISP B. As a result, traffic from ISP B’s customers following those routes flows over the peering link to ISP A’s customers.

normal_ops

This arrangement can go wrong in a couple of different ways. ISP B could take the routes learned from ISP A and send them on to its providers and/or peers as illustrated in the graphic below. By doing this, ISP B effectively inserts itself into the path of traffic destined for ISP A from the outside world. We occasionally field questions from customers of Dyn IP Transit Intelligence product (formerly called Renesys Market Intelligence) who want to know why we reclassified one of their peers as a transit provider.

Typically after a little investigation, we determine that the peer is mistakenly passing their company’s routes on to the peer’s transit providers (or its peers) — effectively providing the company with free transit. In fact, one popular use case for IPTI is for ISP operators to monitor how their network’s adjacencies get classified. When there is a difference between what IPTI shows and how the operator thought his network was connected, someone isn’t propagating routes correctly.

leak_scenario1

The other way a peering relationship can go wrong is if ISP B announces routes learned from its providers and/or peers to ISP A. In this scenario (illustrated below), ISP A will then redirect traffic following these routes through ISP B, even if these routes are not normally handled by ISP B.

leak_scenario2

Unlike other routing leak scenarios, such as Indosat originating the entire global routing table or VolumeDrive leaking nearly the entire BGP table from one transit provider to another, the leaks described above occur with much greater frequency and with little fanfare. In fact, typically the parties involved are unaware of the glitch and, as a result, these more limited problems can persist much longer than the larger catastrophic incidents. Nonetheless, the results typically include traffic going where it wasn’t intended with a corresponding immediate hit to performance and the potential for security implications.

The Vimpelcom – China Telecom connection

200px-VimpelCom_logo     505px-China_Telecom_Logo.svg

In September 2013, China Telecom and Vimpelcom (one of Russia’s “big three” mobile operators) announced an agreement to establish a direct network interconnection. We would have expected to see this relationship appear in BGP routing, either as a peering relationship or possibly a limited transit relationship to allow Vimpelcom to reach the Far East. However, on several occasions in the past year, we observed China Telecom mishandling routes it received from (and sent to) Vimpelcom, effectively inserting itself into the path of Internet traffic to and from Vimpelcom.

One of those occasions was the 5th of August of this year, when China Telecom (AS4134) announced routes from Vimpelcom (AS3216) to all of its peers for several hours (leak scenario 1 from above) which forced inbound traffic to Vimpelcom to go through China Telecom routers. During this incident, over 7,000 routes from Vimpelcom’s customer cone were globally announced by China Telecom with AS paths in the following form:

   … {1299,3257,1273,3320,6461,174,…} 4134 3216 …

The following charts illustrate the impact of these incidents by looking at how traceroute measurements into Vimpelcom were affected. As the charts show, latencies spike as traffic is dragged out to China and through the molasses of China Telecom’s under-provisioned international links.

NA_3216_vps01.xrs1 NA_3216_vps01.hnl1
NA_3216_vps01.sjc1 NA_3216_vps01.gdl1

The following example traceroute illustrates an errant path taken by traffic during this routing leak. In this example, the traceroute begins in Reston, Virginia where NTT takes it to California and hands it off to China Telecom. China Telecom then takes it to Shanghai, China and then to its routers in Frankfurt, Germany before entering Russia en-route to Omsk.

trace from Reston, VA to Omsk, Russia at 12:00 Aug 05, 2014
1  *
2  129.250.204.153 (NTT America, Ashburn, US)                1.010ms
3  129.250.4.40    (NTT America, Ashburn, US)                0.934ms
4  *
5  129.250.5.13    (NTT America, San Jose, US)              74.649ms
6  129.250.8.6     xe-0.chinanet.snjsca04.us.bb.gin.ntt.net 70.909ms
7  202.97.90.33    (China Telecom, San Jose, US)            71.451ms
8  202.97.58.237   (China Telecom, Shanghai, CN)           341.819ms
9  202.97.58.66    (China Telecom, Frankfurt, DE)          641.424ms
10 118.85.204.58   (China Telecom, Frankfurt, DE)          608.965ms
11 79.104.245.250  pe-r.Omsk.gldn.net                      632.054ms
12 195.239.162.178 (Vimpelcom, Omsk, RU)                   687.536ms
13 89.179.76.25    vpn2-gi0-0.omsk.corbina.net             682.153ms
14 128.73.155.213  128-73-155-213.broadband.corbina.ru     707.525ms

As seen from within China, latencies to Vimpelcom via the direct link from China Telecom got lower and more stable, but did not improve dramatically. In fact, these traces were still getting directed to China Telecom’s routers in Frankfurt, Germany, and given the fact that overland round-trip latencies from Russia to China are around 120ms, the observed latencies suggest that such traffic may still go across submarine cables to reach Europe.

CN_3216_vps01.pek1 CN_3216_vps01.pvg1

If this routing arrangement is intended to provide Vimpelcom low-latency access to the Far East, it isn’t working that well. The examples below show latencies increasing during a routing leak from locations in the Far East.

3216_Aug5_vps01.sin3 3216_Aug5_bgp02.tyo1

The August 5th event was one of the times that China Telecom briefly announced nearly a full BGP table (326,622 routes) to Vimpelcom, placing itself in the path of outbound traffic from Vimpelcom to the outside world — including Russian routes. (Leak scenario 2 from above.)

Below is a traceroute illustrating the new path to a New Hampshire ISP. Vimpelcom takes the traffic to Frankfurt, hands it over to China Telecom which takes it to Shanghai before handing it over to Cogent in Los Angeles. Cogent takes it across the country to Fairpoint Communications in New Hampshire.

trace from Moscow to Manchester, NH at 12:09 Aug 05, 2014
1  *
2  194.154.89.125 (Vimpelcom, Moscow, RU)                     0.743ms
3  79.104.235.66  mx01.Frankfurt.gldn.net                    40.574ms
4  118.85.204.53  beeline-gw3.china-telecom.net              43.198ms
5  202.97.58.57   (China Telecom, Shanghai, CN)             302.433ms
6  202.97.58.238  (China Telecom, Los Angeles, US)          479.642ms
7  202.97.49.14   (China Telecom, Los Angeles, US)          487.225ms
8  38.104.139.77  te0-7-0-24.ccr21.sjc03.atlas.cogentco.com 380.087ms
9  154.54.6.105   be2000.ccr21.sjc01.atlas.cogentco.com     375.079ms
10 154.54.28.33   be2164.ccr21.sfo01.atlas.cogentco.com     371.727ms
11 154.54.30.54   be2132.ccr21.mci01.atlas.cogentco.com     372.585ms
12 154.54.6.86    be2156.ccr41.ord01.atlas.cogentco.com     370.596ms
13 154.54.44.86   be2351.ccr21.cle04.atlas.cogentco.com     367.498ms
14 154.54.25.89   be2009.ccr21.alb02.atlas.cogentco.com     371.972ms
15 38.104.52.78   (Cogent, Albany, US)                      367.334ms
16 70.109.168.139 burl-lnk.ngn.east.myfairpoint.net         321.980ms
17 64.222.166.166 (Fairpoint Communications, Concord, US)   315.036ms
18 64.223.189.66  static.man.east.myfairpoint.net           321.682ms

But the path from Moscow to New Hampshire was always going to be long. Who’s to say that Los Angeles (or even Shanghai) might not be a reasonable waypoint under certain conditions? Alternatively, we can pick something closer to Moscow to illustrate how crazy this routing is. The traceroute below shows Vimpelcom taking traffic to Frankfurt, handing it over to China Telecom which takes it to Shanghai before handing it over to Chello Broadband, which peers with China Telecom in Los Angeles. Chello then takes it from New York to Frankfurt (again!) and then into the German countryside.

trace from Moscow, Russia to Marburg an der Lahn, Germany at 14:29 Aug 05, 2014
1  *
2  194.154.89.125   (Vimpelcom, Moscow, RU)                  0.908ms
3  79.104.235.74    mx01.Frankfurt.gldn.net                 40.695ms
4  118.85.204.49    beeline-gw2.china-telecom.net           48.799ms
5  202.97.58.61     (China Telecom, Shanghai, CN)          290.810ms
6  202.97.58.202    (China Telecom, Los Angeles, US)       514.115ms
7  202.97.90.62     (China Telecom, Los Angeles, US)       537.933ms
8  213.46.190.217   213-46-190-217.aorta.net (New York)    414.365ms
9  84.116.135.122   (UPC Austria GmbH, Vienna, AT)         420.591ms
10 213.46.160.205   uk-lon02a-rd1-pos-4-0-0.aorta.net      421.530ms
11 84.116.133.65    (UPC Austria GmbH, Frankfurt, DE)      421.557ms
12 81.210.129.234   7111a-mx960-01-ae1.fra.unity-media.net 420.867ms
13 80.69.107.214    7111a-mx960-02-ae0.fra.unity-media.net 418.427ms
14 80.69.107.185    7211a-mx960-01-ae1.gie.unity-media.net 420.454ms
15 88.152.128.0     (Unitymedia, Marburg an der Lahn, DE)  423.105ms

Even Internet paths from Moscow to other parts of Russia were forced through China Telecom’s routers. In the following example, a traceroute from Moscow is taken by Vimpelcom to Frankfurt, handed over to China Telecom’s routers in Frankfurt and, (mercifully) redirected back into Russia via Megafon without getting directed out to Shanghai.  (This diversion of domestic Russian traffic is illustrated in the graphic at the beginning of this blog.)

trace from Moscow, Russia to Yaroslavl, Russia at 13:13 Aug 05, 2014
1  *
2  194.154.89.125  (Vimpelcom, Moscow, RU)             0.542ms
3  79.104.235.74   mx01.Frankfurt.gldn.net            37.006ms
4  118.85.204.57   beeline-gw4.china-telecom.net      39.505ms
5  213.248.84.185  ffm-b10-link.telia.net             41.481ms
6  62.115.137.180  ffm-bb2-link.telia.net             42.227ms
7  80.91.251.159   s-bb4-link.telia.net               42.894ms
8  213.155.133.105 s-b2-link.telia.net                41.528ms
9  80.239.128.234  megafon-ic-151430-s-b2.c.telia.net 42.707ms
10 *
11 78.25.73.42     (MegaFon, Volga,RU)                49.992ms
12 213.187.127.98  ysu1-ccr1036-1.yar.ru              50.301ms
13 213.187.117.230 (NETIS Telecom, Yaroslavl’, RU)    54.769ms

Conclusion

In September, ACM Magazine published a terrific piece by Boston University Professor Sharon Goldberg which asked “Why is it taking so long to secure Internet routing?”. She does a great job covering the issues surrounding routing security and the efforts thus far to address its vulnerabilities. I would add peering leaks like those described in this blog to her list of unresolved issues, problems that are much less well known or even discussed. At present, the best defense is to monitor how your routes traverse the Internet (such as with Dyn Internet Intelligence) and to carefully filter the routes you receive.  Without both measures, your traffic could be easily misdirected, potentially hurting both the performance and security of your Internet communications.

Investors also weren’t too excited about Putin’s Internet security plan, causing Vimpelcom’s stock to drop to a record low, but maybe they were disappointed that it didn’t address routing security.

Learn More About Dyn Internet IntelligenceLearn More

Not sure how your network is affected by events? Check out the tool our research team uses!

34 Responses to Chinese Routing Errors Redirect Russian Traffic

  1. […] finding, reported Thursday in a blog post published by Internet monitoring service Renesys, underscores the fragility of the border gateway protocol (BGP), which forms the underpinning of […]

  2. […] finding, reported Thursday in a blog post published by Internet monitoring service Renesys, underscores the fragility of the border gateway protocol (BGP), which forms the underpinning of […]

  3. […] Errores de enrutamiento chinos redirigen el tráfico ruso fuera del país [ENG] […]

  4. Vasily says:

    Educated guess tells me that it’s not routing error on CT side: CT has peering agreements with both Megafon & Vympelcom, but there is no direct interconnect between Megafon and Vympelcom. So CT announced Megafon AS to Vympelcom, who sent traffic to free/unpaid route.
    It’s completely ok that traffic goes via Europe: there is complicated relationship between RU transit providers, that led to BGP/depeering wars in the past and now many provider has no direct interconnections in Russia.

    • Doug Madory says:

      Yes, I agree it is not that unusual for Russian domestic traffic to “hairpin” through Europe, however China Telecom shouldn’t be the intermediary unless they have leaked routes from a Russian peer.

      Also, there is a direct peering relationship between Megafon and Vimpelcom.

  5. […] finding, reported Thursday in a blog post published by Internet monitoring service Renesys, underscores the fragility of the border gateway protocol (BGP), which forms the underpinning of […]

  6. […] finding, reported Thursday in a blog post published by Internet monitoring service Renesys, underscores the fragility of the border gateway protocol (BGP), which forms the underpinning of […]

  7. […] finding, reported Thursday in a blog post published by Internet monitoring service Renesys, underscores the fragility of the border gateway protocol (BGP), which forms the underpinning of […]

  8. […] finding, reported Thursday in a blog post published by Internet monitoring service Renesys, underscores the fragility of the border gateway protocol (BGP), which forms the underpinning of […]

  9. […] finding, reported Thursday in a blog post published by Internet monitoring service Renesys, underscores the fragility of the border gateway protocol (BGP), which forms the underpinning of […]

  10. […] finding, reported Thursday in a blog post published by Internet monitoring service Renesys, underscores the fragility of the border gateway protocol (BGP), which forms the underpinning of […]

  11. […] finding, reported Thursday in a blog post published by Internet monitoring service Renesys, underscores the fragility of the border gateway protocol (BGP), which forms the underpinning of […]

  12. […] finding, reported Thursday in a blog post published by Internet monitoring service Renesys, underscores the fragility of the border gateway protocol (BGP), which forms the underpinning of […]

  13. […] finding, reported Thursday in a blog post published by Internet monitoring service Renesys, underscores the fragility of the border gateway protocol (BGP), which forms the underpinning of […]

  14. […] finding, reported Thursday in a blog post published by Internet monitoring service Renesys, underscores the fragility of the border gateway protocol (BGP), which forms the underpinning of […]

  15. […] finding, reported Thursday in a blog post published by Internet monitoring service Dyn, underscores the fragility of the border gateway protocol (BGP), which forms the underpinning of […]

  16. […] the security of Russian communications. Internet monitoring service Dyn reported Thursday in a blog post that the apparent networking fault is due to the weakness in the Border gateway protocol (BGP), […]

  17. […] and could compromise the security of Russian communications. The finding, reported Thursday in a blog post published by Internet monitoring service Dyn, underscores the fragility of the border gateway protocol (BGP), which forms the underpinning of […]

  18. […] news has been published by the Internet monitoring service Dyn in a blog post, which reports that also domestic traffic was re-routed  apparently due to a networking fault […]

  19. […] monitoring service Dyn reported Thursday in a blog post that the apparent networking fault is due to the weakness in the Border gateway protocol (BGP), […]

  20. […] news has been published by the Internet monitoring service Dyn in a blog post, which also reports that domestic traffic was re-routed apparently due to a networking fault caused […]

  21. […] the security of Russian communications. Internet monitoring service Dyn reported Thursday in a blog post that the apparent networking fault is due to the weakness in the Border gateway protocol (BGP), […]

  22. […] 由于中国电信的失误,俄罗斯本土的网络流量在不断的向俄罗斯境外路由,这有可能会对俄罗斯的通信安全造成影响。网络监测服务中心Dyn于周四在博客中写到,这次事件是由于边界网关协议(BGP)中存在错误引起的。 […]

  23. […] 由于中国电信的失误,俄罗斯本土的网络流量在不断的向俄罗斯境外路由,这有可能会对俄罗斯的通信安全造成影响。网络监测服务中心Dyn于周四在博客中写到,这次事件是由于边界网关协议(BGP)中存在错误引起的。 […]

  24. […] Chinese Routing Errors Redirect Russian Traffic [Dyn Resarch] […]

  25. […] finding, reported Thursday in a blog post published by Internet monitoring service Dyn, underscores the fragility of the border gateway protocol (BGP), which forms the underpinning of […]

  26. […] finding, reported Thursday in a blog post published by Internet monitoring service Dyn, underscores the fragility of the border gateway protocol (BGP), which forms the underpinning of […]

  27. […] 由于中国电信的失误,俄罗斯本土的网络流量在不断的向俄罗斯境外路由,这有可能会对俄罗斯的通信安全造成影响。网络监测服务中心Dyn于周四在博客中写到,这次事件是由于边界网关协议(BGP)中存在错误引起的。 […]

  28. […] finding, reported Thursday in a blog post published by Internet monitoring service Dyn, underscores the fragility of the border gateway protocol (BGP), which forms the underpinning of […]

  29. componentry says:

    Fascinating research. Keep up the good work guys.

  30. […] happens more than you’d expect!), to the US state of Missouri activating DDoS protection, to Russian Internet challenges and hurting economy, to Cuba’s push for a more usable Internet with the trade embargo with […]

  31. […] finding, reported Thursday in a blog post published by Internet monitoring service Dyn, underscores the fragility of the border gateway protocol (BGP), which forms the underpinning of […]

  32. […] 1 ora e mezza. Secondo la Dyn molto probabilmente non è successo nulla, ma aveva già segnalato simili problemi di routing locali, capaci di far deviare i traffici su […]

  33. […] misconfiguration.  Sometimes, the wonkiness is just wrong, as pointed out in a recent blog post where Dyn/Renesys discovered that some intra-Russia traffic was being routed to China Telecom […]

Leave a Reply

Your email address will not be published. Required fields are marked *