MENU

Hotmail Systems Engineering

What Really Caused the Panix Outage

January 22, 2006 Comments (7) Views: 72 Engineering

Con-Ed Steals the ‘Net

Tweet about this on TwitterShare on FacebookShare on Google+Share on LinkedInShare on Reddit

Well, not the whole Internet, but Con Edison (AS27506) “stole” several important prefixes on the Internet earlier today, probably by mistake. Earlier this afternoon, I saw a message on the NANOG mailing list claiming that Con Ed was “stealing” routes to Panix, the venerable New York ISP, who had previously been hit with another outage beyond their control. Looking quickly into this with Renesys Routing Intelligence, it’s far worse than that.

coned5.png

Con Edison apparently spent the better part of last night and today pretending to be a fair number of other people’s networks ranging from Martha Stewart Living to NYFIX, from The New York Daily News to Walrus Internet. This is bad. While some of these networks were customers of Con Edison, many were not. Did anyone else notice or care that all of their traffic was being misrouted or is Panix the only one of these people who isn’t asleep at the switch? Read on for significantly more detail about what we saw happen and who was affected.

[This is thrown together fairly quickly, so please expect corrections and addtions. Any questions that aren't answered here can be posted in the comments and I'll do my best to look at them and post replies.]

So, what happened? At 05:05:33 UTC 22 Jan 2006 Con Edison begins to announce a number prefixes (networks) owned by their customers. Announcing routes is the way that an Autonomous System introduces Internet addresses into the global routing table. It’s what allows the rest of us to send packets to the IP addresses contained within those networks. The normal way that service providers route traffic to their larger customers is that they wait for those customers to announce networks and then they just repeat the announcement. In this case, Con Edison started pretending to be their own customers. It’s probably not service-impacting, assuming they still know how to get the traffic to the customers, but it is odd.

They spew out huge numbers of announcements for the next several minutes. One of the first customers we see them announcing is Martha Stuart Living Omnimedia. It’s worth noting that this event is three and a half hours earlier than the problem that Panix complained about. So something strange was already at work at Con Edison.

Con Edison is lying (or making a mistake, but it’s definitely not true), but who is believing them? Some people call events like this network identity theft and although there’s no crime, the metaphor is somewhat apt. Just as in the case of real identity theft, it starts by someone pretending to be you, to own all of your networks. But what good is it if someone doesn’t believe them and spreads the lie? In this case, it’s UUnet (AS701). They keep believing the first set of lies until 05:22:29 UTC when the networks start moving back to their rightful owners.

But we’re not done! At 8:23:12 UTC Verio (aka NTT America, AS2914) starts believing some of the same lies that UUnet was already believing and Con Edison starts telling more lies. 8:31:15 UTC is the first instance of Con Edison announcing Panix’s route, and that one only ever appears through Verio. And here’s the worst part: as of about 4:00 UTC some of these networks are still being hijacked by Con Edison, for example, the more specific prefixes of Advanced Digital Internet listed below.

Here is a list of what appear to be honest-to-gosh hijackings of networks not belonging to or associated with Con Edison in any obvious way. I would assume that these would produce outages for the real owners of these networks:

  • 65.164.53.0/24, 65.169.15.0/24 – previously originated by NYFIX, INC (AS20282) (but registered to Trinitech Systems, Inc.), via UUNet (AS701) and Level(3) (AS3356). NYFIX hasn’t appeared downstream of Con Edison any time in the last month.
  • 65.209.31.0/24,65.206.109.0/24 – previously originated by Claren Road Asset Management, LLC (AS19277) via Cogent.
  • 66.194.137.0/24 – previously originated by MacKay Shields,LLC (AS31860) via Time Warner Telecom.
  • 66.207.32.0/20 (including 66.207.36.0/24, 66.207.37.0/24, 66.207.39.0/24, 66.207.46.0/24, 66.207.47.0/24) – previously originated by Advanced Digital Internet, Inc (AS23011) via eLink Coms (AS12006).
  • 68.236.224.0/24 – previously originated by Insurance Information Institute (AS30359) via Verizon Internet Services (AS19262) (also connected to AT&T).
  • 160.79.45.0/24 – previously originated by RHODES ASSOCIATES EXECUTIVE SEARCH, INC. (AS33313) via US Cybersites (AS6221) and AT&T.
  • 166.84.0.0/16 – previously originated by Panix Public Access Internet (AS2033) via Cogent and Level (3).
  • 198.6.16.0/20 (including 198.6.16.0/24, 198.6.17.0/24,198.6.18.0/24, 198.6.19.0/24, 198.6.20.0/24, 198.6.21.0/24) – not previously originated, registered to Track Data Corporation and covered by UUNet 198.6.0.0/16.
  • 208.28.1.0/24 – not previously originated, owned by Sprint and covered by 208.0.0.0/11.
  • 208.196.138.0/24 – not previously originated but registered to World Dot Com, Inc.
  • 209.166.192.0/18, 209.166.192.0/19, 209.166.224.0/19 – previously originated by Walrus Internet (AS7169) via Level (3) and Cogent.

Here is a list of customer-owned networks that Con Edison announced during the event. It’s impossible to say for sure, but it’s likely that these did not cause outages for most of the real network owners:

  • 12.173.227.0/24 – previously originated by Martha Stewart (AS11570) via AT&T (AS7018) and Con Edison
  • 63.145.132.0/24 – previously originated by View Trade Securities (AS23004) via Qwest (AS209), Cogent (AS174) and Con Edison
  • 63.165.71.0/24 – previously originated by Folksamerica (AS26913) via Sprint (AS1239) and Con Edison
  • 64.20.163.0/24 – not previously originated, inside Con Edison Communications (64.20.160.0/20)
  • 64.20.164.0/24 – not previously originated, inside Con Edison Communications (64.20.160.0/20)
  • 64.132.55.0/24 – previously originated by Overseas Media (AS33477) via Time Warner Telecom (AS4323). Overseas is downstream of Con Edison.
  • 65.115.240.0/24 – previously originated by VIEWTRADE SECURITIES (AS23004) (although the prefix is borrowed from Qwest) via Cogent (AS174). View Trade is downstream of Qwest, Cogent and Con Edison.
  • 65.173.134.0/24 – previously originated by STERLING NATIONAL BANK (AS19758) via UUNet, Time Warner Telecom and Con Edison.
  • 65.209.93.0/24, 204.13.184.0/21 (including 204.13.184.0/24, 204.13.185.0/24, 204.13.186.0/24, 204.13.187.0/24, 204.13.188.0/24, 204.13.189.0/24, 204.13.190.0/24, 204.13.191.0/24) – previously originated by Lava Trading, INC. (AS35967) via UUnet and Con Edison.
  • 66.77.142.0/24 – previously originated by Folksamerica (AS26913) via Con Edison (although Folksamerica is also connected to Sprint).
  • 160.79.67.0/24,160.79.107.0/24 – previously originated by TheStreet.Com (AS14732) via US Cybersites and Con Edison.
  • 192.149.224.0/24,192.251.15.0/24,192.251.16.0/24 – previously originated by Tullett & Tokyo Forex, Inc. (AS20179) via Cogent and Internap (AS10910) (although AS20179 is downstream of Con Edison).
  • 204.13.72.0/24 – previously originated by FTEN, Inc. (AS33584) via Yipes (AS6517) and Con Edison.
  • 208.51.94.0/24 – previously originated by New York Daily News, L.P. (AS33535) via Global Crossing (AS3549) and Con Edison.
  • 208.255.106.0/24 – prefix borrowed from UUNet, previously originated by Kohn Pedersen Fox (AS30430) via UUNet and Con Edison.
  • 209.191.224.0/20, 209.191.228.0/24, 209.191.246.0/24 – previously originated by Knight Capital Group, Inc (AS22190) via UUNet, although Knight Capital Group is also downstream of Con Edison.
  • 216.223.46.0/24 – previously originated by Securities Dealing Systems, Inc. (AS12265) via Internap and Con Edison.

Why did this happen? Probably someone at Con Edison made a mistake, although this series of events is messy and complicated enough that it’s hard to imagine exactly what kind of mistake it could have been. Perhaps they’ll comment here or on the NANOG mailing list. Probably not.

Although distressing, this kind of event is not that uncommon. The core routing of the Internet, as it is deployed, has few automated mechanisms to prevent this kind of foolishness. Certainly, UUNet and Verio should have known better. They should have refused to propagate the misinformation being spread by their customer. And the rest of us should have known better as well. Why does the entire internet believe these kinds of lies every time they are spread into the global routing table? That’s rhetorical, I know the answer. It’s right above in this very paragraph: there are few automated mechanisms to prevent it. With service providers running on zero, resources don’t get allocated to fix this stuff.

It makes one a little bit philosophical. The openness and flexibility of the Internet is what has allowed it to grow so quickly. It is what has allowed so many people to do so many wonderful things with it. But the legacy of that openness and flexibility remains to this day at the core of the Internet. And on days like today, that core looks a little bit rotten.

Update 2006-01-23 12:07: Manish Karir of Merit has posted a nice writeup of some of the raw data involved in this event. For those of who who are quibbling to me in private about the formatting and level of detail in this write-up, please see that one. :-) You can also use the BGP Inspect Tool to look at the Panix prefix and the other prefixes involved in the event.

It has been suggested that these prefixes may all be from former Con Edison customers and that failure to update the relevant Internet Routing Registry may be to blame. I haven’t had time to look at that theory yet. It certainly would explain why Verio (NTT America) would accept prefixes that don’t rightfully belong to a customer, since Verio is known for being strict about building filters from routing registries.

7 Responses to Con-Ed Steals the ‘Net

  1. RobL says:

    Hahahaha. Very ugly. It has to be the internet equivalent of this:
    http://news.bbc.co.uk/1/hi/magazine/4545714.stm
    RobL

  2. Ben Rosengart says:

    Panix is a former ConEd customer.
    (I am a former Panix employee and still consider myself a friend of Panix.)

  3. Ed Ravin says:

    I’m an admin at Panix and want to correct and clarify a few things:
    *) “asleep at the switch” is a bit strong – I know Walrus Internet staff were ringing ConEdCom’s NOC phone regularly until a human being finally answered. The admins at the other sites that were having problems may well have been busy fixing their network but didn’t have time to post to NANOG about it.
    *) My impression is that ConEdCom’s network configuration was heavily dependent upon static routes – rather than this being an inadvertent typo like the Tokyo Exchange incident RobL refers to, I think it was a natural outgrowth of their poor network planning.
    * ConEdCom has upstreams via Verio and MCI/UUnet. Verio tells me that they build their route filter from the as-27506-transit object in RADB, which thankfully no longer lists Panix or Walrus. I will be following up with Verio to confirm that ConEdCom is no longer able to announce Panix’s routes.

  4. I have now heard from 3-4 sources that Panix (and maybe all of the affected networks) were former customers of Con Ed Com. But looking through Renesys’s routing data back to the beginning of 2002, I see no evidence of Con Ed ever providing service (or peering or being in any other way network-adjacent) to Panix.

    I very much appreciate the feedback from Panix-associated admins that I’ve received (some of it public and some private). It helps to flesh out their perspective of this event. I would love to know what prompted Con Ed Comm to add Panix routes to their RADB entries. And whether this explains the other, apparently unrelated, ASes as well.

  5. Ed Ravin says:

    Panix used ConEdCom as a backup ilnk, but only on a contigency basis – we announced our routes through them only when our primary link had problems. The longest use was for 2-3 days during the Cogent/Level3 spat last year.

  6. Max Clark says:

    Con-Ed Steals the ‘Net

    An interesting event took place last week in New York. Basically an ISP started telling the Internet that traffic for another ISP should be sent to them instead. This is the equivilent of Cingular telling the world to direct all calls to Verizon to the…

  7. Note that Merit has ‘improved’ their website, and (likely every one ever of) your mailing list archive links are now broken.
    <sigh>

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>