For Sale: Clean, Lightly Used IP Address?


Fiber To The Home: Ideal Economic Stimulus?

November 18, 2008 Comments (13) Views: 454 Business, Economics, Internet

Will Work For Bandwidth

Tweet about this on TwitterShare on FacebookShare on Google+Share on LinkedInShare on Reddit

The Internet is in for interesting times. Previously, I wrote about the engineering issues and about the policy issues facing us over the next five years. But there is at least one large issue still lurking. Most of you will not be surprised to learn that almost all of these issues are outgrowths of a single factor: money. The core of the Internet still doesn’t have a sustainable business model.

Many people are getting rich on the Internet, and almost none of them are spending money to keep the interconnection infrastructure (the “Inter” in “Internet”) growing and expanding. Look at it from a massively oversimplified perspective: Google make their money from the advertising they sell to search audiences. Comcast make their money by offering TV and Internet access on their local cable infrastructure. Amazon make money selling books and other stuff (including servers and storage space). Most datacenter companies make their money selling space and power inside of their buildings. Spammers make money filling up your inbox with useless crap. Organized crime makes money by launching attacks against profitable companies if they don’t pay extortion. DNS squatters make money registering thousands (or millions) of domain names and sitting on them until someone else is willing to pay. And almost none of this helps the core of the Internet.

Look to the wholesale carriers if you want to see an income statement wasteland. Level 3 lost $1.1b last year. They lost $120m in the most recent quarter alone. Cogent is thrilled because they reported a tiny, tiny positive net income last quarter on top of a yearly loss of $30m in 2007. Global Crossing lost $300m in 2007 and $88m in the last quarter they’re reporting, which doesn’t include much of the recent downturn. Other wholesale networks are in the same boat. Dan Golding suggested that it’s more important to look at net cash flows rather tha income, but the result is pretty much the same: almost no one is making any money. The only wholesalers who do make money make it on other service offerings: wireless service, metro Ethernet services, VPNs, local phone service, video services and so on. Are there sustainable Internet backbone business models? Does anyone have one?

Just to be clear, the networks I am referring to here are those that make some significant amount of their money by selling to other providers of networking services. They sell in very large capacities to other companies who then sell in smaller capacities and provide support and service and billing. There are very few networks that are pure wholesalers anymore, but to some extent all of the largest networks in the world are wholesalers in at least part of their business. These are networks like Level 3, Sprint, AT&T, Verizon Business, Global Crossing, Abovenet, Savvis, Tiscali and others.

These networks’ business models are in trouble because of price erosion driven by vicious competition. Level 3 and Cogent are routinely blamed for the sharp decline in prices (to levels below $3 per megabit per second in gigabit and above speeds). The first DS3 of Internet that I purchased was $28k/month for 45mb/s of transit, or a whopping $622/Mb/s. This was a new UUNet (AS701) connection that offered a sharp savings off of the $45k/month ANS connection we were replacing (yes, the ANS connection cost $1k/Mb/s). Many people reading this blog paid far more for far slower speeds (because they’re older!).

Low prices are nice and as a consumer of Internet bandwidth I don’t mind them one bit. But the current prices are unsustainable. I don’t say this because I’m some industry shill who wishes prices would go back up (as do many of the financial analysts that I talk to). The proof is in the financials. It does not look like it is possible to run a network of a reasonable size selling bandwidth on a high capacity basis and make money selling full Gig-Es of Internet transit for $3000/month or below.

It’s possible that people who focus on the backbone and on wholesale IP will go away. Edge networks who get value by selling to content providers, selling ads or selling access to consumers will all interconnect directly. And the “backbone provider” will go away. It’s also possible that the backbone provider can make a comeback by developing a business model with some differentiation that might prop up plummeting prices long enough for them to depreciate some equipment and make a little bit of money. It’s not likely, but it’s possible. And I have some thoughts on how it might happen.

The biggest problem that wholesale networks have right now is the complete commoditization of IP service. There is almost nothing, other than price, to differentiate these service providers from each other. Some have bigger brands (AT&T, Sprint), some are more focused on providing Ethernet-based services (Cogent), some are reputed to be more reliable, some are not. But no one has any real data on service provider quality So everyone shops on price. There’s no good reason not to.

Commodity markets are good for most of us. They provide necessary goods at a great price. They provide strong and direct incentives for companies to align investment with market demand, and keep costs down for everyone. Commodity markets tend to misbehave when some number of players can afford to lose money for very long amounts of time. This keeps prices below costs and has the potential to ultimately raise prices (and margins) significantly, provided many competitors go out of business. I believe that this is essentially what several of the low-price leaders in the marketplace are hoping will happen: that they can drive several competitors out of business with unsustainably low costs and afterwards can capture significantly more market share and make money through a combination of economies of scale and raising prices. It’s not a bad strategy provided they have a lot more money than everyone else, or a nearly infinite ability to lose investors’ money on their way to profitability. But it’s risky.

There may be another way forward here: quality differentiation. My colleagues Jim Cowie and Alin Popescu have started researching long-term trends in service provider quality, especially with respect to outages and instability. Preliminary results show that networks have significantly different levels of quality with respect to these two metrics. This is based on routing data at the moment, but one can imagine incorporating lots of other, related data. The results are interesting and potentially quite significant.

As transit prices have come down, the real cost and value of one provider or another has shifted. It is not the reliability, responsiveness, features, and service that matter much more than price. If I can buy a gigabit of transit from one network at $5/Mb/s and another for $3, that can save me $2000. But it’s likely that any outage, misconfiguration, or problem with that circuit would cost me much more than $2000. If there were a way to seriously predict which network was likely to be the most reliable over time, or which network would be the easiest to deal with, offer the best response to a denial of service attack, or the lowest-hassle provisioning, those could all get factored into my decision-making. And odds are good that I wouldn’t just blindly pick the cheapest network.

In addition to competing on quality, there’s another way that IP networks may choose to compete: differentiated cost. Right now everyone pays per bit per second thresholded in some way regardless of where those bits come from or are going to. This billing model completely ignores the reality that carrying 1 Mb/s from Los Angeles to Karachi is dramatically more expensive than carrying that same traffic flow to San Jose. People who are carrying cheaper traffic are subsidizing those who have more expensive traffic, because the market has always worked that way and carriers don’t have any good way of accounting for the difference.

But as the cost of Internet transit drops, it has crossed the cost of long-haul transport on many routes. Anyone paying $3/Mb/s for Internet transit that crosses the Pacific is getting a really good deal right now. And the rest of us are paying for it in marginally higher costs. Carriers may try to charge “expensive” customers more for their traffic (or give “cheap” customers even lower prices). This is already done to a very coarse extent on a per-customer basis but carriers may try to do it on a per-connection basis. This would have the huge advantage of aligning revenue with costs in a way that the market does not currently accomplish at all right now. But it would have the huge disadvantage of making the Internet that much more like the phone, with differentiated costs and “long distance” tolls. Flat rate billing with low (or zero) marginal costs creates enormous incentives to experiment and create new services. It’s difficult to imagine what the Internet might look like with those conditions gone.

I don’t pretend to know what is going to happen with the market dynamics of the Internet any more than I know what would happen with the engineering or policy problems that we are facing. But I know for certain that we are in for an interesting 3-5 years as these challenges coincide and we all muddle our way forward.

This is my last post for Renesys. I was skeptical of blogging at the beginning. I suspected that no one would read it. I decried the navel-gazing character of most “blogging”. Heck, I even hated the word. I prefer the xkcd-inspired “blag” to this day. But something interesting happened along the way: lots of people started reading these posts and it has become an= interesting place for people who care about the Internet to get news and analysis. You did stuff to the Internet. We analyzed that stuff and posted objective data about it. And then you came and commented. And in the end, we all understood more about how things were put together and where the cracks in the foundation might lie. The Renesys blog will continue to be an excellent place for an exchange of ideas and information with Earl Zmijewski, Jim Cowie, Martin A. Brown and other colleagues continuing to observe what is happening on the net and analyze what they see.

As long as the Internet keeps working well enough, I expect to be a regular reader (and perhaps even occasional commenter) of this blog now, too.

13 Responses to Will Work For Bandwidth

  1. Zed says:

    Great blog, mate! Sad to hear you are passing on the torch. Best of luck in you new ventures.

  2. LUF says:

    Thanks for everything Todd. I guess we all hope to see you comment some posts from now on!

  3. George Cornachini says:

    Good luck in your next challenge if possible remain in contact,
    Best Regards,
    George J. Cornachini – Sr.Director IP Planning
    T-Systems Inc
    472 Wheelers Farm Road
    Milford CT, 06461 3rd floor
    Phone: 203-306-5231 (new number)
    Fax: 203-783-9616

  4. Aaron Glenn says:

    thanks for all your insights, Todd.

  5. Francis says:

    There is a lot of commentary around how the USA is behind in broadband uptake. And there you are commenting on the profitability of various broadband (and other) service providers.
    How do the wholesalers in Korea or Finland survive? Their market must have different costs, if for no other reason than interconnect costs to US based carriers strongly favour US companies.
    How can Japan have such high rates of broadband uptake if it’s so unprofitable?

  6. Thanks Todd for your posts, this last it’s very very good….. good luck for your next ventures :)

  7. The Internet is not a business — it’s the way we use availble infrastructure. Let’s not confuse connectivity with railroads.

  8. Bill Valaski says:

    Todd, thanks for your insights into the industry and good luck in your future ventures. I ran across the Renesys Blog a couple years ago when the Internet2 Blog was detailing the process of lighting up the backbone for that network, and I’ve been privileged to read the commentaries from Earl, Martin, and yourself about (not only) the technical aspects of how everything fits together, and the political and business-personal influences. Again, thank you, and good luck!

  9. JZP says:

    Thanks for saying what needs saying. Once upon a time, (3) tried to offer automatic (still coarse-grained) destination sensitive billing… any relevant lessons from that? In my previous life, we tried to characterize customer types and predict costs on that. Sales folks never wanted to gather the data, were not incented properly to do so, and GIGO. It is *very* difficult to characterize providers in a way to defeat simple spreadsheet accounting, and I constantly had to fend off “stupid cut-rate-provider of the week/month” who was pitching spot-pricing to some pinhead higher up the executive food chain. It is tiring, and the only fix IMO will be more technocrats and less bean-counters at the executive level & the ability to have a bloody spine for everything from pricing to abuse-handling. For the irst 4 years in the startup zone, the company I helped start had no “all you can eat” offering for residential, commercial, or storage. We were in the black within our first year. Cause and effect?
    I do think that the fundamental problem is not restricted to packet-switched networks. Looking at long-haul rail, air transit, etc I see nothing but failures & lossses. Those industries have an easier way to set distance-sensitive pricing, and still get it wrong. So it is no wonder to me that the less-visible distance of data is even harder to get right.

  10. La banda sarà il problema del futuro

    Todd Underwood scrive su uno dei blog in assoluto più interessanti del panorama tecnologico, in particolare di quello legato alle discussioni su Internet e sul suo sviluppo, andando a toccare spesso tematiche che rientrano pienamente nel settore hostin…

  11. Jay Borden says:

    Thoughtful post, thank you. If your theory that transit providers are playing chicken with capital exhaustion, then it’s pretty simple to predict who wins and who loses — the larger diversified players (to the extent that they are playing the same game) will always have greater capacity to outlast the specialized wholesalers. Although the specialized folks may simply be betting that the big guys will lose appetite to compete, and focus on their higher margin wireless and ethernet businesses…so maybe it’s not so simple after all!
    Come back and post a guest blog now and again…

  12. Jeff Turner says:

    Great post! I see the lack of money that pure IP transit providers offer as an evolutionary stepping stone in the Internet’s overall economic model. Profits will accrue to those ISPs who can offer unique value. IP transit, as a commodity service with some level of service guarantee, has naturally evolved into the low margin service it is today. New enhanced bandwidth services must be offered in order to right this trend. Those ISPs who can sell premium service across their broadband networks will have the most to gain in the future.
    Good luck in your new ventures…

  13. Estetik says:

    There is a lot of commentary around how the USA is behind in broadband uptake. And there you are commenting on the profitability of various broadband (and other) service providers.
    How do the wholesalers in Korea or Finland survive? Their market must have different costs, if for no other reason than interconnect costs to US based carriers strongly favour US companies.
    How can Japan have such high rates of broadband uptake if it’s so unprofitable?

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>