African Peering Ecosystems

Question

DrPeering - 


I learned of your travels to the African Peering and Interconnect Forum (AfPIF-5) in Senegal. How are the African Internet Regions shaping up these days?


Dr. Heywood R. Floyd

Clavius Base, TMA-1 Project

Answer

Dr. Floyd -


Thanks for the question.


ISOC put on another terrific AfPIF event. After five years, they really have this stuff down now and the event has become a mainstay for the community. The program was relevant, the right people were into the room to learn, discuss, and share their experiences. In this case they were hosted in a French-speaking nation and so attracted a lot of francophones that may not have attended AfPIFs before. They had a full house with 175 participants in attendance. The translators did a masterful job, and this role is especially essential for establishing strong Internet Peering Ecosystems across a continent with so many languages.


The Internet Regions across Africa are growing slowly but surely, like a crock pot instead of a pressure cooker as we have seen in the U.S. and Europe evolutions. In the U.S. for example we the Internet Peering Ecosystem is like that of a living organism responding to stimuli placed upon it. Changes were almost immediate. Across Africa there are many more moving parts so things take longer, and we can all learn about these complexities as we work together to overcome them at these forums.


From the AfPIF-5 meeting I walked away with five observations.



#1 - Choose your IXP colocation center carefully


We learned from the Kenyan Internet Exchange (KIXP) that it is very difficult to move an existing Internet Exchange Point, so one should be extremely careful and thoughtful when choosing a colocation center to host the IXP. They described six months of weekly conference calls, the very thorough detailed plan for the cutover, and the coordination required to get the participants onboard. You could tell that the speaker lived through the pain of managing this process and people because he said “Do not move the IXP!” 3 times during his talk. Moving logistics were much harder than pulling the plug on one switch and plugging in the new switch; the larger the IX, the more people are involved and the longer it takes.


We learned that the Internet is growing across Africa, and that some of the heroes that enabled this include Seacom, Liquid Telecom, Main One, and others that are laying fiber terrestrially or over the ocean between countries. Kenya, Nigeria, South Africa are some of the emerging regional hubs, but some in the room believe Senegal has a good chance of emerging as a regional hub as well. Getting these folks into the IXP as partners/customers/etc. is very helpful.


#2 - Regional IXP or National IXP


Success of an African IXP starts with the location and proximity to international fiber.


There must also be a local reason for the IX to exist in the first place, and if there is enough “peerable” traffic volume, or enough eyeballs or interesting content, then there is a gravitational pull for outsiders to build in.


So why care about having a regional IXP in your country?


Popular regional IXPs lead to lower Internet Transit prices


These price declines start when an IXP is attractive enough for international ISPs to build in for peering. If these international ISPs can also sell transit, they bring an open market for Internet Transit. As a result,  the local population build into the IXP and buy transit for maybe 30% less than outside the colocation center. And so, the colocated network operators get lower cost Internet access, and therefore offer lower priced Internet access to the local population. This is why Internet Transit is $1/Mbps across western Europe with strong IXPs. We have seen this around the globe - Countries with strong IXPs with strong peering populations enjoy lower transit prices.


The further away you get from those peering epicenters, the higher the transit prices. This is due to the fact that ISPs offering Internet access have to pay more money for transport their bits into these IXPs for offloading in peering and transit relationships. 


So it is in the national best interest for each nation to emerge as the regional IXP.


#3 - Media companies are effective at scaring us


I was not going to include this one but I found I repeatedly reflected back to it.


Some people did not attend the Senegal-based meeting because of the ebola outbreak in nearby countries. Before the trip to AfPIF, the media bombarded us all with “crisis! crisis! crisis!” messaging. What we were shown was very scary. The images of the scared and desperate masses looking through a police line into the media cameras. We were all inundated with images that demonstrated that the people could not leave Liberia, etc. and were forced to live among the potentially other infected people. Sentenced to death in effect. Scare us CNN, and keep the flow going every 30 minutes, and it will work.


Some people saw that Senegal had 0 cases of the ebola virus. It turns out that the ebola virus outbreak was largely uncontained because of the failed health care infrastructure that never recovered after their civil war. This led to health care workers quickly running out of gloves and masks, etc. and having to wash (instead of dispose of) their haz mat suits and reuse gloves. Add to that the cultural practice of washing cadavers with water after death, and you have an effective context for the unabated virus spreading.


Unfortunately we did not hear the messages about the reality on the ground in Senegal, so some people did not attend the conference.


So the lesson is that the media has an incentive to sensationalize things because it keeps people watching TV, and it can scare us off. We should not let the media scare us into not living our lives, and we should make a more proactive effort to inform conference attendees about the reality on the ground where the conference is being held.



#4 - There is a difference between an Internet Transit Exchange and an Internet Exchange Point


There are several cases where a network operator running an Internet Exchange Point and bundling in Transit with the service. The EMIX in the United Arab Emirates does this and the Singapore Internet Exchange (STIX) does this for example. There are others. These exchanges purchase international transit in bulk for their customers and allocate the cost of the service across all participants.


One problem with the transit exchange model is that those that want to peer their local traffic locally end up subsidizing those that also make use of the Internet Transit service. This leads to a more expensive IXP as well.


Another problem is that the exchange purchases the transit and locks in the price, when the participants may be able to get a better price on the open market at some time during the duration of the upstream transit contract(s). This will at least skew the types of customers that participate, but it also make other transit provides stay away from the exchange because they don’t want to support a competitor (the one who won the transit contract).


Finally, the transit exchange that purchase the transit in bulk is paying whether that purchased bandwidth is used or not. There is therefore at some point a disincentive to assist in local peering relationships, since those peering relationships may decrease the use of the pre-purchased transit. 


What is needed instead is an IXP with people employed to grease the skids to a large peering critical mass.


#5 - The challenges facing Africa are more complex than in the rest of the world


Consider an opening question: How does one get the continental African network operators to negotiate and agree on cross border business terms when there are 1000 languages in use across Africa?


Some more challenges...There is great difficulty getting fiber across the borders between two countries. The regulators have not harmonized the interconnect regimes across the continent, and some of them may not understand how peering is different from telephony interconnection.  The international fiber is now coming into the countries, but the local transport costs are still very high. And 98% of all African Internet traffic is coming from Europe and the U.S., so the desirability of the local market for peering is pretty small. These are just a few of the obstacles we learned about at these AfPIF conferences.


Continuing, it is expensive to get network kit into some countries in Africa because of duty, licenses, and fees. Some countries have poor educational systems too, and a certain level of education and literacy is required to take advantage of the global information economy. 


As a result, growing the Internet Peering Ecosystems continues on its long hard ride.


At the same time, savvy investors recognize that there are over 2 billion people in Africa, and many are technology seeking! They have embraced mobile phones for their Internet needs. I observed people using their mobile phone payment systems for buying fruits and vegetables on the street as easily we in the western world have embraced ATM cards. Over the next few years, technology companies will have the opportunity to address a 2 billion person market with new technology, and once proven, a massive market share of one of the world’s largest emerging markets for Internet services. It is hard to get into China but comparatively easy to address the African market.


My prediction. The future of peering in Africa will be driven by community building. The Tier 2 ISPs will peer with one another, the international ISPs will sell access to the European priced Internet, and the drop in prices will encourage usage of the Internet and will grow the local peering markets. Traffic will grow into the Tbps, and the local peering populations will grow towards 100, and the price of transport into these countries will drop from $200/Mbps to $25/Mbps. At this point, there will be more gravitational pull to get the European and US content and content distribution companies to build into Africa.

Summary

PS Notes from my notebook:


On Monday Irina Trentea (Lyon-IX) ran the Peering Simulation Game with the group. 


After the break we had some really good Q&A about the game, about the way the game played out, and the implications. We discussed the regulatory role if any to encouraging the growth of the peering ecosystems. And there was interest in what the IXPs can do to encourage foreign network operators to build into their country. 


We tried something new this time - we had the players wear headphones so they could not hear the other players negotiations. It added a dimension of reality to the game, that peering negotiations are not done within earshot of competitors. Bottom line, as usual, the Peering Simulation Game was a crowd pleaser and a lot of fun to run.




On Wednesday we did the book signing. The Internet Peering Playbook was translated into French by Irina Trentea (Lyon-iX) and released as “Le Manuel du Peering.” This photo is the crowd waiting for their books and the one above is the book signing line.


I have to admit that I really enjoy book signings. It is good for the ego, and it provides a connection, even if a short duration connection, between the author and the people that are going to read the book. And peering is a game of relationships - what better way to start a relationship than by sharing information with folks just now coming into this peering world?


On Thursday I gave the keynote : “The Future of Peering.”  (I believe ISOC posted the slides.) Mostly my talk was about how the U.S. Peering Ecosystem evolved like a living organism responding to the stimuli applied onto it. Since the transit price curves in the US decline along a similar path to the price curves in Africa, it stands to reason that Africa transit price declines might follow a similar trajectory. I made a few specific suggestion about how one might grow an ecosystem. All of these ideas led to building  community, growing the market, identifying impediments and documenting them along with their target values.