MENU

Mediterranean Cable Break – Part IV

You can’t get there from here

February 24, 2008 Comments (29) Views: 4145 Engineering

Pakistan hijacks YouTube

Tweet about this on TwitterShare on FacebookShare on Google+Share on LinkedInShare on Reddit

Late in the (UTC) day on 24 February 2008, Pakistan Telecom (AS 17557) began advertising a small part of YouTube’s (AS 36561) assigned network. This story is almost as old as BGP. Old hands will recognize this as, fundamentally, the same problem as the infamous AS 7007 from 1997, a more recent ConEd mistake of early 2006 and even TTNet’s Christmas Eve gift 2004.

Just before 18:48 UTC, Pakistan Telecom, in response to government order to block access to YouTube (see news item) started advertising a route for 208.65.153.0/24 to its provider, PCCW (AS 3491). For those unfamiliar with BGP, this is a more specific route than the ones used by YouTube (208.65.152.0/22), and therefore most routers would choose to send traffic to Pakistan Telecom for this slice of YouTube’s network.

I became interested in this immediately as I was concerned that I wouldn’t be able to spend my evening watching imbecilic videos of cats doing foolish things (even for a cat). Then, I started to examine our mountains of BGP data and quickly noticed that the correct AS path (“Will the real YouTube please stand up?”) was getting restored to most of our peers.

The data points identified below are culled from over 250 peering sessions with 170 unique ASNs. While it is hard to describe exactly how widely this hijacked prefix was seen, we estimate that it was seen by a bit more than two-thirds of the Internet.

This table shows the timing of the event and how quickly the route propagated (this is actually a fairly normal propagation pattern). The ASNs seeing the prefix were mostly transit ASNs below, so this means that these routes were distributed broadly across the Internet. Almost all of the default free zone (DFZ) carried the hijacked route at least briefly.

18:47:00 uninterrupted videos of exploding jello
18:47:45 first evidence of hijacked route propagating in Asia, AS path 3491 17557
18:48:00 several big trans-Pacific providers carrying hijacked route (9 ASNs)
18:48:30 several DFZ providers now carrying the bad route (and 47 ASNs)
18:49:00 most of the DFZ now carrying the bad route (and 93 ASNs)
18:49:30 all providers who will carry the hijacked route have it (total 97 ASNs)
20:07:25 YouTube, AS 36561 advertises the /24 that has been hijacked to its providers
20:07:30 several DFZ providers stop carrying the erroneous route
20:08:00 many downstream providers also drop the bad route
20:08:30 and a total of 40 some-odd providers have stopped using the hijacked route
20:18:43 and now, two more specific /25 routes are first seen from 36561
20:19:37 25 more providers prefer the /25 routes from 36561
20:28:12 peers of 36561 start seeing the routes that were advertised to transit at 20:07
20:50:59 evidence of attempted prepending, AS path was 3491 17557 17557
20:59:39 hijacked prefix is withdrawn by 3491, who disconnect 17557
21:00:00 the world rejoices; Leeroy Jenkins online again.

Since BGP relies on a transitive trust model, validation between customer and provider is important. In this case, PCCW (3491) did not validate Pakistan Telecom’s (17557) advertisement for 208.65.153.0/24. By accepting this advertisement and readvertising to its peers and providers PCCW was propagating the wrong route. Those who saw this route from PCCW selected it since it was a more specific route. YouTube was advertising 208.65.152.0/22 before the event started and the /24 was a smaller (and more specific) advertisement. According to usual BGP route selection process, the /24 was then chosen, effectively completing the hijack.

Because of the fast detection and reaction of the YouTube staff and cooperation with other providers, service for their (sub-) prefix was interrupted for about an hour and forty minutes for some lucky customers and, at most, a bit more than two hours. The exact duration of the outage depends on your vantage point on the Internet.

When these sorts of events occur, there is renewed interest in a variety of solutions to this problem. BGP is fundamental to provider relationships and will not be going away anytime soon. Cryptographic extensions to BGP have been suggested (Pretty Good BGP, Secure Origin BGP and SBGP). These may be too taxing for router CPUs. Of course, after any sort of hijacking event (whether inadvertent or malicious) prefix and AS monitoring is suggested (e.g., the Internet Alert Registry, the Prefix Hijack Alert System, RIPE’s MyASN and
Renesys’ Routing Intelligence).

Ultimately, though, the problem remains one of transitive trust. A provider can and should limit the advertisements it will accept from a customer. The mechanics can be arranged manually or can be configured using Routing Policy Specification Language (RPSL) to communicate the policy and drive configuration. In the case of Pakistan Telecom, they originate or transit fewer than 1000 prefixes.

So, it’s heartwarming to know that two things are still true. It is still trivially possible to hijack prefixes (whether maliciously or inadvertently). I can go to sleep knowing that my neighbors are happily watching their LOLCATS.

29 Responses to Pakistan hijacks YouTube

  1. exact says:

    Pakistan hijacks YouTube

    Bookmarked your post over at Blog Bookmarker.com!

  2. billso.com says:

    Pakistan blocks YouTube, breaks trust

    Earlier today, we noticed that YouTube was not available. An ISP in Pakistan, PieNet, single-handedly blocked global access to the popular video site for two hours, according to multiple reports on the Times of London, ZDnet, ReneSys, OpenDNS and Data …

  3. Telco 2.0 says:

    Ring! Ring! Hot News, 25th February 2008

    Concentrated links, every Monday.

  4. YouTube Offline, Pakistan Telecom Blamed

    YouTube was offline for about two hours Sunday, sparking a debate about whether the outage was caused by an effort by Pakistan to block the site.

  5. lamib says:

    Wait: the order PDF mentions a specific video. It has been deleted due to a “terms of use violation”: what was it?

  6. How to Avoid Another Major IP Hijacking

    YouTube isn’t the first site to have its IP space hijacked. Some history, and a look at existing preventive measures.

  7. xan says:

    You said that lucky folks only noticed a 30 minute outage. However, in the timeline you posted there is a 1hr20min gap between action and the first reaction. (18.49 to 20.07)
    Could you clarify what fixed the bad route problem for any affected parties after about 30 minutes, and when was that countermeasure taken?

  8. Trogdor says:

    I’m hearing rumblings that the block of YouTube had more to do with videos showing how the elections were rigged, and less to do with the “blasphemous” videos. The latter was simply an easy excuse to block the former.

  9. The fragility of the Internets – as demonstrated by Pakistan / Youtube

    I love how fragile the Internet really is. This is demonstrated from time to time and when it is – I’m

  10. ebw says:

    Really nice post, and better than slogging through the NANOG hijack thread.

  11. Pakistan Telecom Hijacked Youtube

    It could have been completely accidental but Pakistan Telecom, in trying to comply with a Pakistan government censorship order, hijacked part of Youtube’s internet routing last night.
    Renesys blog tells us:
    Just before 18:48 UTC, Pakistan Teleco…

  12. Martin A. Brown says:

    Thanks, xan!
    My explanation did indeed say thirty minutes, though it should have said “about an hour and thirty minutes”. (It was actually about 1h42m, but I say an hour and forty minutes in the corrected text above.)
    I appreciate the correction!
    -Martin

  13. Pakistan hijacks YouTube… [Spyware Sucks]

    Those of you with a technical mindset may find this explanation about what happened, and the timeline

  14. Daniel says:

    Cryptographic BGP extension can’t help in this case. It only tell who announce this, not who can announce this.
    The upstream provider (AS 3491) don’t filter any route. Just knowing who help nothing

  15. So, in fact it was NOT Pakistan or the Pakistan Telecom Authority that blocked YouTube, but a technician at PCCW who did not verify the PTA’s routing advertisement.
    I’m no fan of political censorship, and think that trying to prevent people from seeing a cartoon is self-defeating and wrong.
    But as a journalist, I am a fan of the truth, which is that PCCW caused the problem, which it can correct by implementing a manual verification procedure before complying with customer requests.
    Is that right, Earl?
    ———-
    Earl: If I light my neighbor’s house on fire and burn it to the ground, do you place blame solely on the fire department for not seeing the smoke and putting out the fire in time? All providers need to be good net citizens, which including not injecting garbage into the routing tables and also guarding against it from others – when possible. Both parties are responsible, but the source of “the fire” bears the greater responsibility.

  16. Rattle says:

    And here is an lolcat just for for this occasion…
    http://nicklevay.net/misc/bgpcat.jpg

  17. Pakistani Hijack Of Youtube: The HK Connection

    Going through the RSS feed for the day, it seems that a Pakistani government order to ban Youtube resulted in a temporary hijack of Youtube’s internet routing information. (via Wampum) But it seems there is a Hong Kong connection to the hijacking that pro

  18. Simon Leinen says:

    Here’s a BGPlay link (using RIPE RIS data) that nicely shows the propagation dynamics for the /24.
    http://www.ris.ripe.net/cgi-bin/bgplay.cgi?prefix=208.65.153.0/24&start=2008-02-24+18:46&end=2008-02-24+21:05

    Simon.

  19. Nick Barnes says:

    Thank you for this detailed technical account. The mass media accounts have, as usual, been an unintelligible mishmash.

  20. Martin A. Brown says:

    Nathaniel,
    The point is that there were two technical errors. First, Pakistan Telecom was advertising a route they had only intended to blackhole. Second, PCCW didn’t have prefix filters installed to limit the reach of this advertisement.
    Also routing advertisements are usually subject to a series of checks–it’s unfortunate that PCCW did not have prefix checks to prevent this entire situation.
    -Martin

  21. La ragnatela

    ha dei buchi! Scherzi a parte, non so quanti di voi abbiano approfondito l’hijacking dell’address space di YouTube occorso Domenica e tutte le discussioni nate dopo, a questo proposito una buona fonte può essere la mailing list nanog.
    Ho creato un elenc

  22. Just a note: for those who want a lower-latency way to discusss this event, we started a few discussions over at Babbledog, Renesys’s personalized social news project.

    Babbledog supports live discussion without moderation or waiting for your posts to show up.

    Take a look at this this discussion or search for related related stories

  23. YouTube outage *updated* (caused by routing filter mistake in Pakistan)

    youtube had their ip’s hijacked. Pakistan was advertising an invalid route announcement which not only blocked youtube for Pakistan but other networks for some reason accepted this as a valid route and blocked youtube for other networks as well….

  24. Glen Bowes says:

    How Pakistan Hijacked YouTube

    On February 24, 2008 in response to a government order, a Pakistani ISP (Internet Service Provider, a business that provides access to the Internet such as Bell, Cogeco, and IAW) PieNet, began blocking access to a YouTube video that apparently containe…

  25. Der gekidnappte Youtube IP Prefix – das Protokoll

    von Fredy K�nzler
    Die Netzwerk-Analysten von Renesys haben ihren Datenhaufen durchw�hlt und auf ihrem Blog eine genaue Analyse ver�ffentlicht, wie der Youtube-IP-Prefix von Pakistan Telecom gekidnappt worden ist. Oder, salopper ausgedr�ckt: so machen d

  26. hina says:

    Its open again, i can view youtube here in Islamabad

  27. World Views says:

    Pakistan Blocks YouTube Access

    Last week, the Pakistani ordered all Internet service providers to block the YouTube website for containing

  28. World Views says:

    Pakistan Blocks YouTube Access

    Last week, the Pakistani Government ordered all Internet service providers to block the YouTube website

  29. Lilu says:

    Hmmm… I’m living in China now and here, in Beijing, I often collide with site blocking. To prevent that, I’m using http://strongvpn.com. It’s a VPN account with strong and reliable service. I haven’t use a new proxy every day after its blocking.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>