MENU
sales-r

Internet Transit Sales: 2005-10

dns-r

DNS: When Governments Lie (1)

November 18, 2010 Comments (10) Views: 668 Engineering, Internet, Politics, Security

China’s 18-Minute Mystery

Tweet about this on TwitterShare on FacebookShare on Google+Share on LinkedInShare on Reddit

china-f

When the US-China Economic and Security Review Commission released its report to Congress this week, something slightly unusual happened: people read it. And there, buried on pages 236-247, a mystery was revealed, and the media have greedily amplified it.

 
Did China’s government really divert 15% of the Internet’s traffic for eighteen minutes in April, effortlessly intercepting sensitive traffic in flight, and generally creating a massively embarrassing man-in-the-middle attack on vulnerable global communications?

 

Well, yes and no. Mostly no.

 

Yes, this event really happened. No, it probably wasn’t a deliberate attack, or if it was, it wasn’t a very effective one, compared to what might have happened, but that’s where the story tends to bog down in technical detail and lose most readers.

 

Bear with me for a few minutes. There are several fragments of truth floating around this story, but they get more confused with each retelling. Let’s lay out what’s known, and then you can draw your own conclusions.

 

Start over: Was Internet traffic hijacked to China?

 
Yes, it was. Let me very briefly sketch how it happened.

 

Internet routing, believe it or not, still works on the honor system. Any organization on the Internet (call them “the attacker”) can assert that it owns the IP addresses of any other organization (“the defender”), by sending out a single Border Gateway Protocol (BGP) message to its neighbors. The neighbors can choose to believe it, or not.

 

If they are good and diligent neighbors, they will check the new assertion against published policies, find a mismatch, and ignore it entirely. But if they believe it, they immediately pass it on again to their neighbors, who pass it on to their neighbors, and so on, potentially passing the false assertion on to the entire planet in less than 30 seconds.

 

Now the stage is set for traffic redirection. When you need to send Internet traffic to the defender (for example, to send him email or read his website), it’s passed towards the “closest” organization that asserted ownership. A large fraction of all the defender’s inbound traffic is potentially redirected straight into the waiting arms of the attacker. And until they withdraw their BGP route assertion, or their neighbors start filtering it out, there’s no way to stop it. It’s that simple.

In fact, it’s so simple, that it happens every year to somebody through sheer accidental misconfiguration. It’s been happening like this, periodically, at varying levels of severity, for over a decade. Sometimes it happens to just a network or two, as in Pakistan’s global hijacking of YouTube. Sometimes it happens to tens of thousands of prefixes, as someone briefly asserts ownership of huge swaths of the Internet. Sometimes it’s China, and sometimes it’s Con-Ed. We’ve seen it happen so many times, to so many people, that when it happened again in April, we didn’t even feel like investing the time to blog about it.

 
Ok, now we’re getting somewhere. So, did the April 8th event target the US Government?

 

No, almost certainly not.

 

On April 8th, starting at 15:50 UTC, China Telecom incorrectly asserted ownership of more than 50,000 different blocks of IP addresses. This is the source of the “15% of the Internet” factoid that you’ll hear floating around. One small part of China Telecom (autonomous system number 23724, used for operations in Beijing, not their primary countrywide ASN 4134) made this assertion, and nobody disbelieved them. Within a few minutes, they “grew” to more than 1,000 times their normal size, and started to receive some of the traffic bound for these 50,000 networks.

 

The media have reported that the hijacked addresses included victims in the US Government, sites in .mil, and so forth. Which is true.

 

China-hijack.png

Impacted countries by network count during the April 8th event.
 

In fact, it was such a broad shotgun blast of address space that it included networks from 170 different countries, including 16,000 from the USA. It also included 11,500 hijacked networks… from China! Asian networks were disproportionately affected (China, Korea, India, Australia, Japan), because they were closer to the source. Several different governments had networks among the victims, as you’d expect by pure chance, out of such a large sample.

 

In summary, the scattershot nature of the hijack suggests a random mistake, not a deliberate attack on anyone in particular. Of course, it’s impossible to know for sure.

 

Okay, but they intercepted twenty minutes of Internet traffic!

 

True.
But how much traffic were they really in a position to intercept? As the hijacking propagated, ten minutes passed, and then twenty. Nobody seems to have noticed anything amiss during the event itself, which seems strange, since it was lunchtime on a busy Thursday in the USA. No increased delays? No closed connections? No tweets about the Internets being broken?

One clue lies in the nature of hijacking. If two US networks are talking to each other, even if both are hijacked, traffic will probably continue to flow normally between them. Why? Both are closer to each other than they are to the hijacker. They both choose the legitimate routes, because they’re closer than the fake routes. No traffic is diverted. (There’s an interesting corner case where one of two US participants is at a US Internet exchange point where the attacker is also present, and therefore artificially “close enough” to take the traffic, but let’s skip that for the moment.)

 
How much data could have been intercepted, then? Basically, the packets in flight at the time of the route hijack .. a few kilobytes of effectively random content in the middle of each TCP window, times (at a guess) millions of redirected conversations between largely unknown participants. It would be pretty hard to plan such an attack so that you ended up with anything useful to read!

 

Got it. So, you’re telling me that this was nothing to worry about, then.

 

Oh, no. You’ve missed my point. Indeed, that’s what we thought when we first approached the April event, based on our previous experience with big-bore route hijackings.

But there’s one critical difference this time out, and it has to do with the size of the hijacker. Usually, if a small operator “hijacks” a large part of the Internet, they simply get buried under the weight of all the random traffic that suddenly come their way. The traffic goes down a hole and dies, and that quickly alerts people to the problem, and it gets fixed.

 
But this is China Telecom: the largest retail-customer-facing Internet provider on Earth, and the 11th largest Internet provider, period. They have the capacity to deal with flows like these, and their global network is large enough that some strange internal routing effects become possible.

 

For example, it’s also true that some traffic was diverted to China, passed through China Telecom’s network, and then was sent back into the Internet towards its intended destination, with no visible effects to the end users except an increase in packet delay. People have suggested to us that this re-routing creates the ideal conditions for a traffic-archiving man-in-the-middle attack, but cooler heads have observed that there’s absolutely no evidence that such a thing actually took place.

 
Redirected Packets: A Long March Around the World

 

So during the event, you’d see paths going from the US to Turkey by way of China. Or London to Frankfurt by way of China. What we think happened is that one China Telecom router (with the false route from ASN23724) would attract the traffic (“come here, little packet…”) and send it to Beijing. In Beijing, another China Telecom router, this one without the false route, would realize the mistake and send it along to the intended destination (“hm, this seems to be going to America..”). Large autonomous systems are like that: they don’t always have a perfectly consistent routing picture all along their global border.

 

As a concrete example, here’s one of the typical traceroutes we saw during the incident, between the London Internet Exchange and a host in the USA, passing through China Telecom. This trace was collected at 16:03 UTC, about 13 minutes into the event. Total time in transit is 525ms (this trace typically takes no more than 110ms under normal conditions).

 

1. <our host> 0.785ms # London
2. 195.66.248.229 1.752ms # London
3. 195.66.225.54 1.371ms # London
4. 202.97.52.101 399.707ms # China Telecom
5. 202.97.60.6 408.006ms # China Telecom
6. 202.97.53.121 432.204ms # China Telecom
7. 4.71.114.101 323.690ms # Level3
8. 4.68.18.254 357.566ms # Level3
9. 4.69.134.221 481.273ms # Level3
10. 4.69.132.14 506.159ms # Level3
11. 4.69.132.78 463.024ms # Level3
12. 4.71.170.78 449.416ms # Level3
13. 66.174.98.66 456.970ms # Verizon
14. 66.174.105.24 459.652ms # Verizon
[.. four more Verizon hops ..]
19. 69.83.32.3 508.757ms # Verizon
20. <last hop> 516.006ms # Verizon

 
Traceroutes are notoriously hard to interpret, but here the fact that we consistently see 400+ms latencies from hops approaching the destination is a strong indicator that these packets have, in fact, taken a Long March through China on their way to this particular US host.

 

china-trace.png

 

It’s a story that was repeated over and over, as this image of selected traceroute paths during the event suggests. Chinese routers are in red, non-Chinese routers in other colors. Traffic flows that dip into China and re-emerge on the other side to reach their original destinations are clearly visible.

 
Conclusion: Watch Your Backs, People

 

There you have it. The route hijacking took place, pretty much as described by the Congressional report and the media. Once you dig into the details, the conclusion you reach is up to you. On one hand, Internet routing is an exceedingly blunt instrument with which to attack an organization or capture man-in-the-middle traffic. It’s about as subtle as a firecracker in a funeral home — the effects are visible for all to see, planetwide.

 

On the other hand, if you were so inclined, I suppose you could plausibly interpret the April 8th incident as a bit of technological muscle-flexing … maybe even a capability demonstration. The problems with the Internet’s trust-based infrastructure are real, they are serious, and they are trivially exploitable by state- and non-state actors who want to influence world affairs.

 

Is there anyone left on the planet by now who’s (a) in charge of a large chunk of address space, (b) not monitoring the BGP routing of that space, and (c) not petitioning their service providers to implement best common practices for route filtering? If the US Congress, Fox News, and the Drudge Report all know about these threats, then your business continuity insurance provider and SAS-70 auditors probably aren’t far behind.

10 Responses to China’s 18-Minute Mystery

  1. Peter Marks says:

    Now, if China had some really fast supercomputers they could decrypt the traffic that they intercepted. Oh wait, they do.

  2. Tyler says:

    Thanks for making more sense of this than the networks could. A sober examination of an event is always refreshing compared to “OH MY GOD CHINA!” histrionics.

  3. Chris says:

    When I first read about this I thought “man, this sounds like another one of those false route advertisements that Rensys blogs about a lot. Bet they’ll get to the bottom of this”. I am not disappointed. Thanks for the sleuthing guys, great work as usual. =)

  4. IGP Blog says:

    Congress should heed U.S.-China Economic and Security Review Commission’s Advice

    Early yesterday the U.S.-China Economic and Security Review Commission released its 324 page annual report to Congress. The lead story …

  5. Andy says:

    I tend to defer to Renesys, Arbor, and Schneier on issues like this. Well reported, guys, as usual.

  6. IanPJ says:

    Good article, well explained.
    But the possibility that you didn’t mention was that by hoovering up the haystack, you also get the needle (.mil or any other target traffic?) that you wanted without drawing attention to it.

  7. Goege says:

    The same scheme like we kill the Toyota, this time who is the victim

  8. Chinese BGP hijack, putting things in perspective

    China denies hijacking a huge chunk of US net traffic Internet Traffic from U.S. Government Websites Was Redirected Via Chinese Networks…

  9. Hey, China! The Internet called. It wants its traffic back.

    Last Wednesday, Nov 17, the US China Economic and Security Review Commission released a wide-ranging

  10. IDR issues says:

    U.S. Commission accuses China of data hijacking…

    …is the title of an article published yesterday on Spiegel Online (German), one of the biggest news-websites in Germany (an article discussing this topic may also be found on cnn.com). Referring to a report published…

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>