With the low price of Internet Transit, does Internet Peering still make sense?
This white paper is intended for the Chief Financial Officer with little time; it is intended to directly answer this fundamental and recurring business question in the peering community.
We answer this question with financial proof using real-world numbers from the field.
We demonstrate the "Peering Break Even Analysis Graph," a financial model for the cost of peering as compared with the cost of transit.
Over two hundred ISP Peering Coordinators were interviewed over the past decade to determine the processes of and motivations for peering, and chief among the motivations was reducing the cost of “transit”. To describe this motivation in brief let’s first introduce a couple of definitions.
First, the most basic definition is the definition of the Internet Service Provider:
By definition therefore, ISPs must somehow themselves connect to the Internet. For most ISPs this means purchasing a service called “Internet Transit” from an ISP that is already attached to the Internet.
Definition: Internet Transit is the business relationship whereby one ISP provides (usually sells) access to all destinations in its routing table .
Figure - Transit Relationship - selling access to the entire Internet.
Internet Transit can be thought of as a pipe in the wall that says "Internet this way".
In the illustration above, the Cyan ISP purchases transit from the Orange Transit Provider (sometimes called the "Upstream ISP"), who announces to the Cyan ISP reachability to the entire Internet (shown as many colored networks to the right of the Transit Providers). At the same time, the Transit Provider propagates the Cyan routes across the Internet so that all attached networks know to send packets to the Cyan ISP through the Orange Transit Provider. In this way, all Internet attachments know how to reach the Cyan ISP, and the Cyan ISP knows how to get to all the Internet routes.
Some Characteristics of Internet Transit
Transit is a simple service from the customer perspective. All one needs to do is pay for the Internet Transit service and all traffic sent to the upstream ISP is delivered to the Internet. The transit provider charges on a metered basis, measured on a per-Megabit-per-second basis using the 95th percentile measurement method.
Transit provides a Customer-Supplier Relationship. Some content providers shared with the author that they prefer a transit service (paying customer) relationship with ISPs for business reasons as well. They argue that they will get better service with a paid relationship than with any free or bartered relationship. They believe that the threat of lost revenue is greater than the threat of terminating a peering arrangement if performance of the interconnection is inadequate.
Transit may have SLAs. Service Level Agreement and rigorous contracts with financial penalties for failure to meet service levels may feel comforting but are widely dismissed by the ISPs that we spoke with as merely insurance policies. The ISPs said that it is common practice to simply price the service higher with SLAs, with increased pricing proportional to the liklihood of their failure to meet these requirements. Then the customer has to notice, file for the SLA credits, and check to see that they are indeed applied. There are many ways for SLAs to simply increase margins for the ISPs without needing to improve the service.
Transit Commits and Discounts. Upstream ISPs often provide volume discounts based on negotiated commit levels. Thus, if you commit to 10Gigabits-per-second of transit per month, you will likely get a better unit price than if you commit to only 1gigabit-per-second of transit per month. However, you are on the hook for (at least) the commit level worth of transit regardless of how much traffic you send.
Transit is a commodity. There is debate within the community on the differences between transit from a low cost provider and the Internet transit service delivered from a higher priced provider. The higher price providers argure that they have better quality equipment (routers vs. switches)
Transit is a metered Service. The more you send or receive, the more you pay. (There are other models such as all-you-can-eat, flat monthly rate plans, unmetered with bandwidth caps, etc. but at the scale of networking where Internet Peering makes sense the Internet Transit services are metered.)
The Cost of Traffic Exchange in a Peering Relationship
The cost of peering at an Internet Exchange Point usually involves the following cost components:
Transport of the traffic into the exchange point,
Colocation space,
Equipment, and
A port on the exchange point shared fabric.
Figure 7 - Components of the Public Peering Cost Model
Definition: Transport refers to a physical/data link layer media interconnection (e.g. circuits, gigE switching fabric, gigE over fiber cross connects).
Unlike transit service, transport is typically not metered. Transport is sold as a fixed capacity circuit that costs the same regardless of the amount of traffic exchanged over it.
Likewise, colocation fees, monthly amortized equipment costs and peering port fees are typically the same monthly recurring cost regardless of the amount of traffic that is exchanged over the infrastructure.
Traffic exchanged over a peering relationship itself is typically free. Both peers mutually benefit from the interconnection so neither party charges the other for the peering relationship. Both parties can then send and receive as much traffic as can fit across the transport circuit and peering fabric for the cost of the interconnection.
Question: When does it make sense to Peer?
Answer: When is it less expensive to send traffic over a Peering Interconnection, as compared with simply sending all traffic to Upstream ISP(s) in a Transit relationship(s).
In order to compare Peering and Transit we need to normalize peering costs into the same terms as transit costs ; we need to compare both Peering and Transit on a per-megabit-per-second basis.
The Business Case for Peering in 2010
Example: the Cost of Peering vs Transit in 2010
To illustrate, let's apply some market price points based on 2010 pricing model for peering costs.
Transport Cost Data Points. At the DE-CIX 2010 meeting, Edward Punt (KPN) and Sven Engelhardt (TiNet) shared with the author that the costs for 10G transport circuits into an IX tend to cost
about $2000 per month for a local connection,
about $4000 per month for adjacent countries, and
about $6000 per month for Pan-European 10G circuits.
A specific data point came from a Slovenia ISP who said during Peering Personals that the 10G circuit from Slovenia into the DE-CIX in Frankfurt was quoted as high as $7000 per month, and that from Budapest to the DE-CIX was about $6000 per month.
Let's assume one can purchase a 10 Gigabit transport circuit into an Internet Exchange Point for $6,000/month. At the Internet Exchange Point, let's assume there is a 10G port for $2,000 per month and a powered half rack for $1000/month as pictured in the figure below. We will throw some equipment costs into the mix as well; perhaps a Juniper 320 will amortize to $8,000 per month.
Sample Costs of Internet Peering - Source: Discussions at the DE-CIX Meeting Sept 2010
With all of these costs taken into account, we see a monthly cost of peering from afar of $17,000 per month as shown below.
Peering Costs assumptions
Question: What happens if we peer only 100Mbps on peering infrastructure capable of handling 10G?
Answer: We see in the table below that the cost of peering will work out to be $170 per Mbps when 100Mbps is peered.
Mbps ExchangedPeering Cost100$170.00 per Mbps200$85.00 per Mbps300$56.67 per Mbps400$42.50 per Mbps500$34.00 per Mbps600$28.33 per Mbps700$24.29 per Mbps800$21.25 per Mbps900$18.89 per Mbps1000$17.00 per Mbps
Peering Costs allocated over Traffic Volume
As we peer more and more traffic we also see that the unit cost effectively drops. The more traffic we can peer for free over this infrastructure, the lower the average cost of traffic delivery. Effectively, the unit cost per Mbps of traffic exchanged will vary based upon how many Mbps are exchanged at the peering point using this infrastructure.
Key Internet Peering Metrics
The analysis so far leads to four key peering metrics for determining when peering makes sense from a purely financial perspective.
Peering Break Even Point
If we continue plotting the cost per Mbps across the size of the peering bandwidth we get the Peering Break Even graph as shown below. We are also highlighting the alternative means of offloading that same traffic, which is simply sending that traffic to a transit provider at a known metered rate (Transit Price).
To the right of the break even point (see figure above), ISPs completely cover their peering costs with reduction in transit fees. This savings is proportional to the amount of traffic exchanged with the peering population.
To the left of the Peering Break even Point is the ”Peering Risk”, the possibility of not realizing sufficient peering traffic volume to offset the cost of peering.
The conclusion is clear - if an ISP can exchange more traffic than indicated at the “Peering Break Even Point”, then the ISP should prefer to peer instead of solely purchasing transit from an upstream ISP.
Formulaically, we can calculate the unit cost of peering (per Mbps) as the monthly cost of peering divided by the amount of traffic that can be peered across that infrastructure.
In 2010, the price of transit is around $5 per Mbps. From a strictly financial position, peering makes sense when the unit cost of peering is less than the cost of transit, that is, when more than $17,000/$5= 3,400 Mbps will be sent to this Exchange Point. We call this the “Peering Break Even Point".
What is the best price for traffic exchange that we can hope for?
Effective Peering Bandwidth
For modeling purposes we will assume that the Effective Peering Bandwidth is only 70% of the capacity is available for traffic exchange. This takes into account framing overhead, the need for burst ability, and the sinusoidal nature of Internet traffic. So we will assume that we can use 7Gbps of the 10Gbps capacity. This assumption seems to resonate with the Peering Coordinator community.
In the best case scenario we can exchange the Effective Peering Bandwidth worth of traffic for the monthly recurring cost, yielding the best price for traffic exchange that we can hope for:
If we plug in the numbers we can see that in this best case scenario, traffic exchange with peering can cost as little as $2.43 per Mbps. When we compare this cost for traffic exchange against the "market price for transit at the origin", we can prove that it makes sense to build into the remote Internet Exchange point.
Effective Peering Range
With the analysis so far we can prove that peering financially makes sense if you can peer more traffic than the Peering Break Even Point, and can peer up to the Effective Peering Bandwidth. The Effective Peering Range is the range of peering traffic exchange that makes sense financially.
The table above also covers the local and near cases where the transport costs less.
Other
Summary: Generalized Peering Break Even Analysis Graph
Let’s summarize in the form of a generalization of the examples we have seen so far in a “Peering Break even Analysis” graph .
The graph below (figure below) generalizes the cost of traffic exchange in a transit relationship against the cost of traffic exchange in peering relationships across an exchange point.
Generalized Peering Break Even Point Analysis Graph
The unit cost of traffic exchange is on the Y-axis in cost-per-Mbps. The X-axis shows the volume of traffic exchanged in Mbps. The cost of transit is shown as a relatively flat unit cost line . The sloped line shows the cost of traffic exchange in a peering relationship.
Peering costs are fixed and include the cost of transport into an exchange, the cost of a partial rack for routing equipment, and the cost of a port on a switch for peering with the exchange population. To compare Peering and Transit, this peering cost is allocated across the amount of traffic exchanged between the ISP and the population of ISPs that are peering with the ISP at the exchange point.
The more traffic exchanged at the peering point, the lower the unit cost of traffic exchanged.
There is a “Peering Break even Point” where ISPs are financially indifferent between peering and simply sending all traffic through its upstream ISP . Once traffic volume between the ISP and the peering population reaches the break even point, ISPs start saving money by peering. The amount of money saved is proportional to the amount of traffic sent to the population of ISPs at the exchange point. The “Peering Risk” is the range of traffic exchange where an ISP fails to exchange enough traffic with the other ISPs at the exchange point to offset the cost of peering.
The amount of traffic sent to the exchange is capped by the minimum of the transport bandwidth and the port bandwidth, termed the “Effective Peering Bandwidth”.
The minimum cost of traffic exchange can be calculated to be the cost of peering divided by the Effective Peering Bandwidth. This is the best price we can hope for when peering over this infrastructure.
Acknowledgements
The author wishes to express appreciation for the following individuals who assisted and provided valuable feedback during the creation of this document: Frank Orlowski (DE-CIX), Harald Summa (DE-CIX), Job Witterman (AMS-IX), and historically Jeffrey Pappen (Yahoo!), Mitchell Rose (Digital Island), Dennis L. Shreve (FACET Networks), Jon Castle (Comdisco), Jeff Swinton (Conxion), Jalil Sanad Halim (9 Telecom), Glen Larwill, Nick Monjo (PSINet Australia), Tim Griffin (AT&T Research), Rene Hatem (CA*net3), Todd Braning (Level3), Joe Klein (Cogent), Ren Nowlin (Carrier1), Joe Provo (RCN), Vince Fuller (MFN), Mike Leber (Hurricane Electric), Brokaw Price (Yahoo! & Broadcast.com), Waqar Khan (Qwest), John Osman (rigozsaurus.com), Mike Hughes (LINX), Ted Hardie (Equinix), Lane Patterson (Equinix), Jason Starr (Equinix), Nic Lewis (AduroNet), Wolfgang Tremmel (via Net.Works), John Zweibel (Procket networks), James Rice (BBC Internet Services) and Beau Sackett (Telus).
Appendix A: Non-Financial Motivations for Peering
Finally, it is important to highlight some non-financial motivations for peering that lead to less easily quantifiable motivations for peering:
Peering provides the lowest latency path between ISP customers. Peering has been found to improve performance by as much as 40-50 milliseconds .
Peering gives ISPs more control over routing, and have more flexibility to route around congested paths that could cause packet loss.
Peering provides redundancy. If peering sessions fail, the transit services provide backup connectivity to the peer networks. If the transit connectivity fails, the peering connectivity is unaffected.
Some exchange points support both Peering and Transit traffic exchange, allowing for the aggregation of transit traffic and peering traffic back to the ISP network. This reduces local loop costs for access to transit services .
These can have a significant financial impact, but there are also factors that are more difficult to model and quantify. For example, packet loss causes data transfers to time out, and retransmissions cause the data transfer window size to decrease, ultimately resulting in lower aggregate data transfer rate. This means that customers not only have a degraded experience, but the resulting decrease in data transferred, results in the ISP not make as much money either! To maximize revenue therefore, ISPs should seek to minimize packet loss and latency. Peering gives the ISP lower latency (direct path) and lower loss (assuming at least one ISP is motivated) with greater control over the routing.
Conversations with ISPs highlighted a few other subtle advantages to peering as well.
By peering, ISPs build and maintain a relationship with other ISPs and as a side effect get a better sense of the competitive environment in which they operate.
Peering was cited as substantial marketing collateral, particularly for content heavy or hosting ISPs. The implied assumption is that more peering means more redundancy, and larger capacity peering with the largest ISPs yields greater performance.
ISPs can improve their network reliability by peering at multiple points. Internet routers are very good at detecting failed paths and automatically rerouting traffic around breaks. Wide scale peering decreases the affect of any one failed network component.
Peering requires greater network expertise than simply purchasing from a single upstream transit provider.
There are administrative startup costs associated with peering. Peering often requires contracts and negotiations iterations between ISP legal departments.
Peering is not always granted and is sometimes impossible to obtain. Beyond the difficulty in finding the contact person to initiate peering discussions, some ISPs have unpublished peering prerequisites that prevent all but the largest ISPs from peering with them.
There is greater operational overhead with peering than if one managed only a few transit relationships. Some ISPs cited transit (customer) outage trouble tickets get addressed more quickly than peering tickets. A couple ISPs mentioned that they prefer the “teeth” of a customer-based contract over the softer peering assurance that both ISPs will work diligently to fix peering-related issues.
The process of peering is slow. It may take months to get peering up and operationally passing traffic .
All in all, peering can offer substantial benefits for ISPs and large-scale content players that exchange a lot of traffic. The larger the traffic volume and the greater the difference between transit costs and peering costs, the greater the motivation to explore peering as a cost savings strategy.
Appendix B: Why Peer and Why Not Peer?
When it came down to it, most peering coordinators had common motivations for peering and many similar motivations for not peering. These motivations go beyond the business realm but are worth mentioning here. Here is what the peering coordinators said.
1) Lower Transit Costs. Choices made by Internet Service Providers (ISP) are often dominated by telecommunications cost issues. Highest among these costs is Internet Transit service that provides the ISP with connectivity to the global Internet.
To reduce these transit costs, ISPs seek peering (zero or reduced cost paid peering) relationships with other ISPs. These peering relationships provide a more direct path for traffic and reduce the load on these expensive transit services. By contrast, the equivalent peering transport (interconnection circuit) is typically less expensive. This is shown pictorially in the figure below.
Figure 3 – Migrating traffic from Transit to Peering Interconnection
2) Lower Latency. As a side effect of interconnecting directly with peers, ISP customers experience lower latency to the other ISP’s customers. Traffic destined for a local competitor’s customers may need to traverse a couple of transit providers and potentially across great distances (with high latency) before reaching the other customer. We heard of circuitous paths for example between the United Arab Emirates and Saudi Arabia that traversed an overloaded exchange point (MAE-East) at the time located across the Atlantic Ocean in Washington DC. Through direct peering interconnections (using direct circuits or regional exchange points) ISP customers realize better performance.
3) Usage-based traffic billing. Some ISPs charge customers based upon metered traffic. Since packet loss and latency slows traffic consumption, they benefit from a lower latency, lower packet loss Internet. It is in their best interest therefore to assure that customers use as much bandwidth as possible by minimizing loss and latency through effective traffic engineering . This assertion was first shared with the author by Dave Rand, CTO of AboveNet but is a source of debate among peering coordinators.
4) Marketing Benefits. Some ISP pointed to the benefits of a network diagram rich in interconnections as a proxy for quality as seen by customer prospects.
During the conversations with ISP and Content Player Peering Coordinators, several reasons for not peering were uncovered. The following represents the most common reasons that ISPs have given for not peering:
On the surface, peering appears to be an obviously good idea from a business (financial) and technical perspective. But not all ISPs see peering as automatically providing benefits. Here are the top ten reasons ISPs gave for not peering.
1) Traffic Asymmetry and Investment Asymmetry means that one party bears more of the cost as a result of peering. (See "The Folly of Peering Ratios" for a more complete discussion of this issue. )
In brief, consider the figure below where a Content Heavy ISP wants to peers with a global Internet backbone. Web traffic (the dominant traffic flow on the Internet) is inherently asymmetric so the Content Heavy ISP is a net source of content,and therefore uses more of the Global backbone provider's resources (bandwidth across the entire path of the network) than the Content Heavy ISP who simply dumps the traffic onto the peer at the peering point.
"I don't want to haul your traffic around the world for free, when I only gain a few peer routes in return."
In some cases ISPs will peer without settlement up to a certain traffic ratio (for example 4:1 traffic out to traffic in) and then de-peer or negotiate on a usage basis beyond that on a Mbps basis . In the Exodus-GTEI depeering-negotiations-repeering, rumor has it that the solution was to peer at more locations and to engineer cold-potato routing to reduce the distance the traffic had to spend on the GTEI backbone .
2) Transit Sales are Preferred. There may be the potential for transit sales if they don’t peer . Some ISPs will not peer if there is any existing or pending customer-provider relationship between the parties, even if the sale is completely unrelated to interconnection (i.e. fiber sale or colo sale) . Ren Nowlin (Onyx/Level 3) described the Level 3 policy here as "Once a customer, never a peer".
3) Ports are for revenue. Peering consumes resources (router interface slots, circuits, staff time, etc.) that could otherwise be used to revenue generation. Router slots, cards, interconnection costs of circuits or Internet Exchange environments, staff install time are incremental expenditures. Further, operating costs, particularly for peering sessions with ISPs without the necessary on-call engineering talent, may require increased processing power for filters, and otherwise take staff time that could be focused on paying customers.
4) Keep Transit Pricing from Sliding, the Motivation not to commoditize IP transit. Tier 1 ISPs compete on the basis of better performance. They accomplish better performance by leveraging a large customer base of direct attachments and high-speed interconnections with other Tier 1 ISPs. Since peering with other ISPs improves the performance of the “peer”, it effectively makes your competitors into a more powerful competitor. Therefore, there is a strong disincentive to peer and increase the number of top tier competitors.
5) Prefer to be a Customer with SLAs. As a “peer” there are no Service Level Agreements (SLAs) to guarantee rapid repair of problems. Both parties are assumed to benefit equally from the interconnect and therefore benefit equally from the reparation of an interconnect outage. There may be clauses in the peering agreements that both parties agree to work diligently to repair the problem. However, a customer relationship (with or without SLAs) generally has more contractual teeth (financial repercussion for failure to perform). There is a category for companies with this Peering Policy: we call it a No-Peering Policy.
6) Traffic Ratio denial. Peering with a Content Heavy ISP will adversely affect peering ratios with other peers that the ISP cares about. See "The Folly of Peering Ratios" for a complete discussion of this issue.
7) Transit is Cheaper. Every year the price of transit has declined as shown in the table below. Why bother with peering when transit is so cheap? This is the perennial question in this sector.
8) Personality Conflicts. Oddly enough, the personalities of the Peering Coordinators, the engineering staff, the executives at the ISPs can have a polarizing effect on the peering process. The Peering Simulation Game even plays this out - often, ISPs that should peer will not peer for non-technical reasons.
9) We aren't true Peers. How ISPs measure themselves can be approximated by their "peers". We are not "true peers" was the reason given as to why ISPs rebuke peering requests from other seemingly good fit peers.
10) We don't have the cycles. Even when there is a good reason to peer, some ISPs simply don't have the people time to set up peering, so they put off peering requests.
