European vs U.S. Internet Exchange Point Models


Internet Exchange Points are just ethernet switches, aren't they?

This article spotlights a few of the key differences between U.S. and European IXes.

European vs U.S. Internet Exchange Point Models

Introduction & definitions

Definition: An Internet (IX) is a layer 2 service to facilitate the interconnection between Internet Service Providers and Large Scale Network Savvy Content Providers.

These interconnects are either Peering or Transit interconnections, but the mission of most IXes is the facilitate Internet Peering wthin the Internet Region.

Definition: Internet Peering is the business relationship whereby companies reciprocally provide access to each others’ customers.

This definition applies equally to Internet Service Providers (ISPs), Content Distribution Networks (CDNs), and Large Scale Network Savvy Content Providers.

To illustrate peering, considerthe figure below showing a much simplified Internet; an Internet with only three ISPs: WestNet, MidNet, and EastNet.

Peering Diagram

WestNet is an ISP with green customers, MidNet is an ISP with blue customers, and EastNet is an ISP with red customers.

WestNet is in a Peering relationship with MidNet whereby WestNet learns how to reach MidNet's blue customers, and MidNet reciprocally learns how to reach WestNet's green customers.

EastNet is in a Peering relationship with MidNet whereby EastNet learns how to reach MidNet's blue customers, and MidNet reciprocally learns how to reach EastNet's red customers.

Peering is typically a free arrangement, with each side deriving about the same value from the reciprocally arrangement. If there is not equal value, sometime one party of the other pushes for a Paid Peering relationship.

Important as well is that peering is not a transitive relationship. WestNet peering with MidNet and EastNet peering with MidNet does not mean EastNet customers can reach WestNet customers. WestNet only knows how to get to blue and green customers, and EastNet knows how to reach only blue and red customers. The fact that they both peer with MidNet is inconsequential; peering is a non-transitive relationship.

Peering can be Public or Private, and both are supported at most European and U.S.-based Internet Exchange points.

How are European Internet Exchange Points different from U.S. based Internet Exchange Points? The models are a bit different.

European IXes and U.S. IXes tend to differ in:

  • Organizational Structure
  • Scale
  • Pricing
  • Business Model
  • Colocation
  • Neutrality
  • Cooperation

Other than that, they are identical. The technology and the business rationale is essentially the same; facilitate the interconnection of ISPs.

Over the years, there are a few things that ISPs have found surprising as they extend their networks from Europe into the U.S. These differences between U.S. and European Internet Exchange (IX) points can be traced back to an evolutionary split that took place around 15 years ago. The European model for IXes is largely based on the successful model initiated by the London Internet Exchange (LINX), while the modern U.S. IX model is largely based on the carrier-neutral IX model initiated by the Palo Alto Internet Exchange (PAIX).

We will sketch the most notable distinctions graphically in this article.

In Europe, as shown in the figure below, Internet Exchange points have a handful of distinguishing features when compared with their U.S. cousins.

U.S. vs European IXes

The European Peering community would point with pride at the European IXes saying that they are both carrier AND colocation neutral. This means that the IXes in Europe are generally available at multiple colocation facilities operated by different companies, none of which is the IX operator. ISPs can then decide which colocation operator provides the facilities services needed, and can connect to the IX whichever site they choose.

European IXes don't usually have customers, they have members. They are often formed as a not-for-profit associations (44%) founded by members that are the ISPs that first connect to the fabric and seed its operation. The other common form of European IX organization is that of a service offered by an academic or regional development organization (42%). In either case, the focus is on providing mutual value, sharing the costs, and having some degree of community oversight over what they see as their organization. Even when they are not formal associations (e.g. NetNod), they tend to operate as if they were. Many European IXes also have activities outside the mere operation of the IX, including political, legal, and community outreach and research activities. Results are generally shared with the broader community.

Sidebar. It is worth noting that the PAIX was itself late to the game, having been launched after the commercial IXes ('NAPs') were already running. These were operated by large scale ISPs. Over time, these non-carrier-neutral IXes fell out of favor and carrier-neutral IXes (PAIX, EQIX, etc. )replaced them as the dominant IXes in the U.S. See the "How Equinix beat MAE-East" article for how this was one.

Soapbox. This association structure for IXes has proven to be both a blessing and a curse. It helps get the IX launched and to critical mass; since the founding association members share in the benefits and the costs equally they all can exert some pressure to get additional ISPs into their IX. However, the 'same pricing for all structure' also limits the use of sales promotions and subsidies which a commercial IX could use to strategically grow its customer base. This is especially important during the early stages before an IX reaches critical mass, and the value of peering there is less than the cost of peering there.

Exceptions: There are three or so commercial IX operations in Europe. Netnod in Sweden is actually run as a nto-for-profit trust, but tends to act as a European membership organization.

Pricing in Europe for IX services tends to approximate a cost-based pricing model. This is reflective of the not-for-profit nature of the operation; if the not-for-profit makes too much excess profit its not-for-profit status is in jeopardy. The pricing decline then makes it easier for other to justify joining, increasing the economies of scale and allowing the operator to decrease the prices further. European IXes are distinct from the U.S. counterparts also in that they cooperate. There is a long history of European IXes sharing operations notes, growth trends, additional service offerings, pricing information, traffic statistics and customer status updates at public Internet operations fora like RIPE, APRICOT, and NANOG. Now let's compare the U.S. IXes across the same dimensions.


The U.S. Internet Exchange Point Model

Internet Exchange Points in the U.S. are largely for-profit commercial operations run by the colocation companies that house them.

US IX Model

This has several implications.

First, for-profit commercial IX companies generally have the ability to do strategic differential pricing. This means a U.S. IX can offer free colo and free switch ports for IXPs that will help lure in other ISPs into the building. For example, when Equinix Ashburn was launched, and there were no tenants, WorldCom was incented to come in with stock warrants in the IX. In Los Angeles the Any2 IX was free for tenants in 1 Wilshire, and in Miami the NOTA IX offered free ports to all tenants. The ability to incent the right ISPs into the IX is a tool that the U.S. IXes have that the European IXes generally do not have.

Second, since they are for-profit, the pricing in the U.S. for IX services tends to approach the price the market will bear, as opposed to the European prices which tend to approach the cost of operations.

Third, in the U.S. you don't tend to see many IXes interconnect with other IXes. The whole question of which IX benefits more comes into play as discussed in the previous Ask Dr Peering article on How Equinix beat MAE-East. Ultimately, these IXes compete aggressively on the colocation and IX fronts and, unless both sides think the deal is unfairly balanced in their favor, the inter-IX deals don't go forward. Especially within the same building that houses multiple IXes, there is much tension surrounding inter-building cross connects between different IXes customers.

Fourth, the focus within commercial IXes tends to be more towards revenue generating activities. While there is much outreach done here, there is a commercial ultimately revenue generating motivation behind most activities.

There are no members of commercial IXes - peers are customers. Customers pay, they don't vote. Decisions are made by the commercial IX operator, not by the population. Next, a soft point. You tend to see that U.S. IXes tend to compete with one another more than cooperate. The exception to this is when the customer base insists that the different IXes work together. This occurred for example when each U.S. IX started running their own peering forums. The customer base, faced with constant travel to so many peering events, insisted that the IXes pool resources to run a single group event which became the Global Peering Forum. It has turned into a singular must-attend event for U.S. peering coordinators that were otherwise splintered across many different events, so everyone won with this evolution.

Finally, U.S. IX public peering traffic volume is much lower than Europe - tens of Gbps vs hundreds of Gbps on each peering platform. Also, in the U.S. there is much more private peering than public peering. As DrPeering has documented in the past, there are valid arguments on both sides of the Public v Private Peering debate. The U.S. IXes are essentially venture capital-backed purpose-built secure 100,000 sq. ft. meet-me rooms facilitating thousands of private peering connections.

These are the differences that folks point to the most often. I've held back some of the more controversial points surrounding contract negotiations and terms, quality of colo and SLAs.

-- DrPeering

Thanks to Stephen Wilcox, Colin Corbett and John Souter for helping edit these articles, and to ianai and ras for their observations as well.