The revelation that Comcast (News - Alert) is using certain controversial practices to limit peer application traffic across its Internet service has led to calls for further regulation. To many observers, it’s a clear violation of that fuzzy concept of “network neutrality.” Comcast was caught using a system that jammed bogus “reset” packets into BitTorrent uploads, limiting the ability of its users to “seed” that popular file-sharing system, though it didn’t impede downloads. In their defense, Comcast claims that it is a “network management” practice that improves the performance of their network for vast majority of users. Companies dependent on peer file transfer of large video files, such as Vuze, naturally see it differently.

 

This leads to a rather interesting set of alliances. Arch-rivals Comcast and Verizon (News - Alert) are in agreement that the FCC should not step in and directly regulate Internet services. Neutrality purists and peer content providers are aligned against them. One side wants the right to control its cost of providing Internet bit transport; the other side wants the right to create and use new applications without any restriction.

 

This sounds like a motherhood-and-apple-pie kind of debate. Who wants their ISP to tell them what they can and cannot do? But it’s really not so simple. The Internet is fundamentally broken, and it’s getting worse. Throwing money and Moore’s Law at it has helped, but it can’t go on forever. The deep dark secret of the Internet’s business model is finally coming home to roost. Economics is sometimes called the Dismal Science, and here’s a case where the term really seems to apply.

 

The Internet is not, as some might think, defined by a set of protocols. Sure, it uses the TCP/IP protocol stack, but that’s really just an experiment from the 1970s. It worked better than it should have, and got put into production, but the protocol is just a tool, subject to change. The Internet is really defined by its business model. It’s an agreement among many network operators to exchange packets with one another. At the wholesale level, it’s a free market: The biggest backbone ISPs, the so-called Tier 1s, exchange packets with one another at no cost, and provide each other with transit to smaller ISPs behind each other. Smaller ISPs, retail ISPs, and large enterprise users purchase upstream capacity from backbone ISPs. This is usually priced by the size of the connection — per megabit of capacity, per month — F.O.B. a site on the backbone ISP’s network.

 

Retail subscribers — you and me — purchase retail Internet service, which is almost always unmetered: Consumer end users, at least in the United States, almost never pay for Internet usage. Bits are priced at precisely zero. This works, usually, because the average consumer doesn’t consume so many of them. ISPs sell to a lot of users, and purchase a pool of upstream Internet backbone capacity that is shared among them. For the average ISP, upstream capacity is a relatively small share of their costs; “last mile” expenses dominate. But that is only because the average usage is so low.

 

Consumer “broadband” services are usually priced based on their peak download bit rate. A DSL service might be good for 3 Mbps, a cable modem set to 6 Mbps, and a fiber-to-the-home service rated at 50 Mbps. The ratio of peak to average rate is known as the oversubscription ratio. This ratio can be very high in practice. Today, a typical consumer’s average usage might be in the 50 kbps range. (That’s about an order of magnitude higher than the average dial-up modem user’s usage.) So the oversubscription ratio could well be over 50 or even 100 to one! That has a big impact on the retail ISP’s costs.

 

How much does upstream capacity cost? The price of backbone ISP service has fallen over the years; in large quantities (hundreds of megabits at a time), it is in the range of around $20/month per Mbps, more or less. So an average (50 kbps) user might use a buck’s worth. But that’s at the backbone site. If the ISP is in an NFL city, and owns its own fiber optic plant to get to a backbone site, then that might be the whole cost. But for an ISP serving an outlying area, the path to the Internet backbone — the so-called “middle mile” — can be frightfully expensive!

 

The price of middle-mile capacity in the United States varies, quite literally, all over the map. On some routes, competitive providers make high-capacity circuits available at reasonable prices. Dark fiber, with huge capacity, is even available hither and yon, though it has become considerably rarer in recent years. So a smaller-market ISP with, say, a good electric-company network provider might be able to get a DS3 circuit (45 Mbps) for a few thousand dollars per month. But in some places, especially rural areas and the smallest markets, there’s nobody to turn to but the old incumbent telephone company (ILEC). And with the decline in competition over the past seven years or so, such markets are becoming more, not less, common.

 

The ILEC tariff for raw bit capacity (leased lines) is called Special Access. Pricing for Special Access in rural areas is capped, with the base rate set based on 1992 tariffs. That conveniently predates the public Internet; in those days, most Special Access was used for private voice networks, a tiny fraction of today’s capacity. So ISPs are basically paying to get to the backbone at rates that are based on how many toll calls would be displaced if the capacity were used for voice! In fact, those rates don’t even reflect the economics of primitive fiber optic networks, which came to dominate local telephone networks in the 1980s. Typical Special Access rates are over $100/mile/month per DS3, atop a couple of grand in fixed charges. In most non-rural areas (Metropolitan Statistical Areas), ILEC rates are not even capped any more. The FCC (News - Alert) fatuously assumes that competition in these areas holds down the price, but in practice, it means that ILECs charge even more.

 

So an ISP who’s 100 miles away from a backbone city may have to pay over $10,000/month — over $200/Mbps/month — just to get a 45 Mbps circuit to the Internet. A really tiny ISP might pay about $20/mile for each Special Access circuit at the “T1” rate of 1.5 Mbps. This makes oversubscription ratios rather important.

 

Commercial Internet services — such as T1 connections — are much more expensive per peak megabit precisely because they anticipate less oversubscription. A business might purchase a T1 (1.5 Mbps) connection to connect its LAN, with dozens or even hundreds of users, to the backbone. That will cost it several hundred dollars per month. It pays much more for 1.5 Mbps than the kid next door pays for 6 Mbps, but that’s because it isn’t peak rate, but total usage, that really creates the costs. Similarly, urban fiber providers such as Cogent can provide small businesses with “100 Mbps” Internet services at a very competitive-sounding rate because they are counting on some level of oversubscription.

 

The main application that threatens the ISP business model is video. Television broadcasts require high bandwidth. I don’t mean YouTube (News - Alert) flash video, in its cute little window — that’s relatively chicken feed at about 200 kbps, downloaded in brief chunks. It’s real broadcast-type video, of the type that Vuze and others are starting to distribute, that potentially breaks the bank.

 

Standard-definition (480 line) video can typically be compressed down to about 1 Mbps, though good DVD quality requires a few times as much. HDTV needs more like 10-20 Mbps, depending on quality. So if 1% of an ISP’s subscribers were watching HDTV at a time, then they alone would raise the average load by at least 100 kbps. A user who uses the Internet to view TV for 10% of the busy hour (not less-costly off-peak downloads) is adding 1 Mbps to the load. That’s about $20/month in the city, but can add $200 in the sticks. Download a DVD movie and it’s probably 4 to 8 gigabytes. That would hold an awful lot of Web pages!

 

While Special Access circuits and LANs are generally symmetrical, cable modems and DSL are not. In the case of DSL, the division of capacity between upstream and downstream is basically arbitrary; there are symmetric flavors of DSL, but Asymmetric DSL is the typically sold to consumers. This is assumed to follow demand; consumers are not seen as “producers” of capacity. That may be inaccurate, especially with the growth of peer applications, but it’s a design choice. It also keeps cheap DSL from cannibalizing business sales of costly T1s.

 

Cable modems are a bit more constrained. The same coax is used for both upstream and downstream. An old FCC rule says that TV stations have the right to demand that they be carried “on channel,” so a channel 2 station can demand to be carried on channel 2 (54–60 MHz). The Digital TV transition makes the rule obsolete, but essentially all domestic cable systems were built with the downstream video beginning at 54 MHz. (Higher splitting frequencies are used in Japan and Europe.) Given the need for a guard band, upstream ends at 42 MHz. Frequencies below about 20 MHz are not really usable for broadband services. So all of the upstream capacity is squeezed between 20 and 42 MHz. That includes cable modems, telephony, video-on-demand signaling, network management, and any other special applications. Downstream, including TV channels, occupies 54 to (as high as) 860 MHz. The original cable modem standard, DOCSIS 1.x, had a limit of 10 Mbps upstream in a 6 MHz-wide channel. DOCSIS 2.x, which is widely deployed, can theoretically fit 30 Mbps into a channel, but it’s rarely achieved. Long-term network upgrades may be able to improve things, but cable operators have a valid reason for not wanting file servers at subscriber locations.

 

Cable operators have always accommodated this to some extent by providing Web hosting service. This keeps the users’ Web pages on servers located upstream, at well-connected locations, not on servers at customers’ premises. Cable (and some DSL) terms of service usually prohibit the operation of file servers on consumer premises. Peer applications technically violate this, as they consist of both client and server. And even DSL has bandwidth limits: While it doesn’t have the same upstream limit as cable, the link between the DSLAM (the system that serves up to hundreds of subscribers in a local area) and the rest of the Internet is shared — and usually heavily oversubscribed.

 

If video distribution were to become a major application on the Internet, how might it be best handled? In the real world, Internet backbone capacity is not free. Backbone ISPs do not charge for distance. But it still has a cost. So if users download movies from a local server, the ISP saves money. In practice, this translates to performance: A connection to a local server is generally faster and more reliable than to a distant one.

 

This leads to the various content distribution networks (CDNs) that exist to make local copies of popular content. The best-known is probably Akamai (News - Alert) Technologies, who maintains servers at many ISP backbone sites, but there are other CDNs, such as Limelight, CDNetworks and Level 3’s. These operate transparently: The user requests a popular site, such as MySpace, and it’s automatically routed from the nearest CDN node. Of course this isn’t free; the content provider pays for this service. CDNs are actively marketing to video distributors.

 

Another approach for delivering high-volume content without a vast server farm is to use peer network protocols, such as BitTorrent. These do not depend on centralized servers. Instead, individual users copy segments of the target file from other users who have already downloaded it. Users computers are both clients and servers. A BitTorrent tracker directs user requests to a swarm of other users who already have the desired content, and multiple parallel downloads can commence. Once a user receives a segment of a file, it can join the swarm and feed that segment to other users. It all sounds very warm and fuzzy, share and share alike.

 

But a peer network is not an optimized CDN. It doesn’t necessarily know where the peers are. So you may end up downloading bits from half way around the world, even if there’s a more local copy somewhere. What is more clear is that it substitutes end user computers, and their ISP’s upstream capacity, for a CDN. This basically shifts costs to retail ISPs, and the ISPs have a good reason not to like it.

 

This leads to the specific problem that got Comcast into hot water. Comcast installed a system made by Sandvine that claimed to do some kind of magic with BitTorrent such that connections would usually stay within the provider’s network, rather than go to the backbone. This would save a lot of upstream capacity if it worked. But in practice, it reduced uploads by forging Reset packets, pretending to be the other side of the connection. Under ideal circumstances, it results in the upload being sent from somewhere else, not Comcast, thus reducing Comcast’s upstream load. But packet forgery is bad form, and it’s far from clear that Sandvine’s trick does what it claims.

 

By singling out BitTorrent, Comcast was behaving in a non-neutral manner. Other high-bandwidth applications, after all, would get through unscathed. Is this a bad thing? Not necessarily — neutrality sounds like a good idea, but as noted, Torrent-type applications move costs to retail ISPs. That is eventually reflected in either higher prices or inferior service. ISPs, even when they’re big telephone or cable companies, are not charities. And they eventually pass along their costs. Even on the Internet, lunch isn’t really free; the cost is just averaged in to the package.

 

Some purists want all ISPs to carry all packets equally, but that has practical problems. Every network needs some kind of tool to manage its traffic. Ideally, it would only be used for clearly-defined, obvious purposes. Spam, for instance, should be throttled if possible. So should virus-infected machines that “jabber” or belong to spam-sending botnets. Sites that exist to receive orders from spam, or to control botnets, should be shut down. Traffic management is also used to enforce user rate limits.

 

But that does not justify the use of Deep Packet Inspection (DPI). What DPI does is look inside the application layer payload, at the user traffic, to decide how to carry, or what to charge for, that particular use. DPI discriminates between applications, and even within applications. A DPI’d network can block new applications, provide discriminatory access to some Web sites while blocking others, and even bill users based on the value of their e-commerce transactions. DPI vendors (such as Sandvine) essentially try to get their customer ISPs out of the real Internet business (routing packets), and into the business of selling selective applications. That hurts everyone. It interferes with basic freedoms, while dramatically reducing the value to the user of the “broadband” connection. It stands in the way of progress and innovation. DPI explicitly manages the flow of voice, data, and video across an IP network, but does so by violating users’ privacy. It is really a form of wiretapping, carried out without warrant by the ISP.

 

Peer file transfers and video downloads may create high expenses for ISPs, but DPI is not needed to solve the problem. A better approach is to simply cap “free” usage. A typical monthly user should not have to worry about hitting the cap, but a heavy user might. At that point, service might be drastically slowed down; the subscriber could then be offered a higher-priced service tier with higher usage caps. In other words, pricing should move from today’s peak-rate basis to one based on total usage. That would correspond more closely with costs, and allow networks to recover their full costs without violating core “neutrality” principles.

 

The telephone network business model is very different. It’s all about billable events. It’s about maintaining scarcity, so that raw bandwidth can be sold at rates based on inflated 1992 norms. What people often fail to realize is that the traditional telephone network, as a common carrier, was the paragon of neutrality. What you said on the phone was no business of the phone company. The equivalent of DPI, literally wiretapping, really did require a warrant (before the current regime took control in Washington). They made their money by charging top dollar for billable calls.

 

A corollary of the telephone model was overcharging for non-voice bandwidth services. High Special Access rates are a vestige of the old model of voice-based charges. But 20 years ago, the telecom industry was working on an ambitious plan for a network that would carry voice, data, and even high-definition video to the home, at speeds of 155 Mbps. Called Broadband ISDN, it was based around the ATM (asynchronous transfer mode) protocol which lives on, quietly, in most DSL systems. It probably failed for two related reasons. One was the inability of phone companies to come up with a viable charging model. If they charged for 64 kilobit phone calls, a 10 megabit video connection would have to cost more, lest it be used to avoid phone charges. But then nobody would use it to watch TV. As common carriers, they couldn’t use DPI, so they couldn’t discriminate with how bits were used. They could provide different Quality of Service options for different prices, but the numbers never added up. Problem two: The Internet happened. And with no billable events for the average end user (other than the per-hour charges prevalent in the early dial-up era), it seemed like a bargain. The telco business model just couldn’t stand up to the competition.

 

But the Internet’s business model depended on rationing, not precise traffic engineering. ISPs bought as much capacity as they felt they needed to. If they didn’t keep up with demand, they’d lose customers, but they didn’t want to spend too much either. This worked well when there were many ISPs to choose from. Besides, TCP adapts to available network capacity; there’s no such thing as “enough.” It’s streaming that breaks that part of the model.

 

Recent FCC policies, though, did away with the ability of independent ISPs to lease telephone company DSL. The result was usually a two-ISP duopoly, cable and telco. And that breaks the other part of the model. Who can determine what is enough, what is worthwhile, what is valid network management and what is excessively intrusive? If the end user could choose from among any number of ISPs, the market would decide. But duopolies only protect against the most egregious abuses; they are not free markets. There really should be no debate about “neutrality.” Telephone companies should carry any ISP’s bits for a tariffed fee (common carriage), and ISPs would figure this out with their customers. Some ISPs might allow video and peer applications, while others might not but would have a lower price. Some would experiment with bucket-of-bits pricing. Until common carriage and a free market for ISP services is restored, trying to come up with a proper rule for Internet management is an exercise in well-intentioned futility, and its unintended consequences are likely to make matters worse, not better.