What others are saying...

“Increasing an understanding of Internet economics and peering fundamentals is an essential part of our work to help developing countries grow their networks. We put copies of Bill’s book into the hands of everyone from network administrators to government officials.”

– Karen Rose, Michuki Mwangi, Internet Society

“I recommend this book for anyone who wants to understand how the Internet is built.”

– Scott Landman

“I think that the author has done a great service here to a very extended set of networking professionals, that go way beyond the peering community, by writing this book.”

–Nickolaous Laoutaris

“This book is destined to be the peering bible.”

– Jeff Turner, InterStream

“Essential Reading.”

– Martin Sanne, SEACOM

“Bill Norton is one of the rock stars of Internet peering. The Internet Peering Playbook lays the groundwork to help you make peering an integral part of your network strategy.”

– Steve Ginsberg, Pandora

“For someone who wants to know more about peering or is thinking of taking the steps into peering this book is a must have!”

– Jon Waldron

“The book is a great start to understand peering, the leverage it provides, and the clever ways it is used in the ecosystem. We hand it out to parties needing understanding and guidance and to new personnel.”

– Job Witteman, AMS-IX

“Mr. Norton is one of the most knowledgeable people in the world on the subject of Internet Peering – this is a must-read for network operators.”

– Ari Benowitz, HighWinds

“Great foundation in understanding the basis for ISP peering, interaction, and insight into where those relationships are heading (upcoming challenges).”

– David Mindel, Cisco Systems

“I walked away from this workshop with a clear understanding of the underlying principals of Internet peering, the business rationale for how decisions are made, and a deep appreciation of the evolution that is underway as video transport begins to dominate the Internet.”

– Mark Canha

“William Norton really has ‘written the book’ on peering!”

– Bob Bender, CTS Telecom

“Your workshop is still being referred to as the benchmark of most useful technical workshop ever.”

– Jaco Muller, Head of Technical Operations, MWEB

“The Internet Peering Playbook: Connecting to the Core of the Internet is an authoritative guide for understanding the complex business relationships inside the Internet and their implications for Internet users. Written by a highly experienced practitioner, the accessible guide is essentially a picture book combining clear verbal and visual narratives. While simple in presentation, the book offers deep insights into the state and evolution of Internet economic affairs. Both descriptive and analytic, the writing incorporates numerous anecdotes and historical perspectives to highlight its crucial points. This unique book is an indispensable read for scientific researchers of the Internet.”

– Sergey Gorinsky, Senior Researcher, IMDEA Networks

“Even if you are not responsible for peering for your ISP, involved in data center operations, or an expert in the Internet, if you are interested in how the Internet works, you will enjoy this book.”

– Zen Kishimoto

This book is dedicated to the next generation of peering coordinators.

Foreword to the 2014 Edition

I once gave a presentation on an Internet peering tool that I built for a LinkedIn “Hackday.” The judges were impressed with the automation capabilities of the tool, but had no idea what this “Internet Peering” thing was. One of the judges told me “I have heard about peering, but it is one of those ‘dark arts’ that I don’t understand.” I replied, “You obviously haven’t read Bill Norton’s book.”

The world knows of the Internet as a global system of interconnected networks, but very few know about the core of the Internet where networks peer. Fewer people still understand how critical peering is to the stability of the Internet. For the Internet to continue to scale, it is essential to educate the communities of interest (network operators, content companies, government officials, international regulators, etc.) on important interconnection topics such as peering and transit.

Over the years peering has attracted the world’s finest network engineers and negotiators, but no one had documented this “dark art of peering” until Bill Norton started writing white papers. Bill has spent most of his career helping people connect their networks together to exchange traffic in a mutually beneficial way. The greatest gift Bill has given to the Internet community is the detailed documentation of all things peering, through his series of white papers, now rewritten and updated in the form of this book.

I firmly believe that this book is a must-read for anyone who wishes to gain knowledge about Internet Peering, or practitioners seeking to participate in peering. The reader will find it really isn’t a “dark art.”

I first met Bill through a mutual friend over lunch. As someone building networks for a major content provider, I was stumped on explaining the business case for peering and demonstrating when peering made sense. Since then Bill and I have had many lunches, not only developing a provable business case for peering, but also exchanging data and exploring new frontiers such as remote peering and quantifying the performance benefits of peering. Bill also showed me the ropes at peering forums globally and introduced me to peering negotiation tactics both at a technical and social level.

The Internet Peering Playbook is an important contribution to the Internet community. This book begins with explaining the fundamentals of Internet Peering with clear pictures that will stick with you, along with historical and future trends of transit pricing. This book also provides detailed information on current peering architectures as well as new frontiers such as remote peering. Peering is always evolving and Bill ensures that readers are informed during these evolutions with each updated edition.

If you are curious about Internet Peering or wish to apply peering to your business model, then you will find The Internet Peering Playbook to be THE comprehensive resource on the subject. If you are already involved in peering, then this is a must-have book on your bookshelf… One that you will both recommend and reference.

Zaid Ali Kahn

Director

Network Engineering, LinkedIn

Preface – Note to the Reader

With the publication of The Internet Peering Playbook two years ago, three thousand people have been exposed to the inner workings of the Internet. The popularity of this book has been credited to the application of the terminology to real-world situations in the “Notes from the field”, and the “tricks-of-the-trade” described in the playbook chapters. With the emergence of remote peering and international peering, Internet Peering remains a highly strategic competitive weapon for the largest network operators in the world. This is an exciting time to be working in the Internet infrastructure arena.

Video continues to dominate the Internet from a traffic perspective, and the Internet has evolved its shape around these new services. These emerging video services require something better than commodity Internet services purchased at the edge of the Internet. To continue the wave of mass adoption, these services require flawless performance at massive scale, achievable only by connecting directly to the core of the Internet. Massive volumes of Internet traffic are exchanged at the core of the Internet in a sort of open Internet transit marketplace. This area is where the largest Internet Service Providers (ISPs) in the world interconnect, where the largest content providers interconnect with the largest regional ISPs, where the content delivery networks offload their traffic directly onto the broadband networks. The information and strategies described in this book will enable the next generation of Internet services companies to connect to and leverage the performance benefits at the core of the Internet as well.

This book is the assimilation of thousands of discussions with hundreds of the smartest peering coordinators in the world. I spent over a decade on the road, traveling to almost every continent, racking up over 500,000 air miles and spending over $500,000 in travel budget. I attended every Internet Operations conference so I could understand how Internet interconnection worked. I documented what I learned, citing the sources when allowed, and then walked others through the resulting white paper to see if I got it right. After about 100 walkthroughs, I would have a white paper that documented a particular aspect of Internet interconnection. I went through this process to produce 12 peering white papers over the course of 10 years. This base research is the raw source material for this book.

The information contained in this book is valuable. As a consultant, I help executive teams build effective peering and transit strategies. I also advise investors on the core Internet industries (ISPs, carriers, colocation centers, IXPs, CDNs, etc.). About 70% of the information I provide during these consulting engagements is information contained within this book. The rest of the information is provided on-site as the teams develop their strategies and investment theses. On-site consulting costs a lot of money, but to these teams, collectively, their time is worth a lot more. They prefer a two-day on-site consulting engagement where they learn together and develop a strategy based on the information I collected during the last couple of decades. Internet interconnection and colocation is a highly specialized area of expertise, and there are only a few of us that can deliver this type of on-site consulting. This book provides this proven valuable information in a much more cost-effective package.

My hope and expectation is that this material will help the next generation of peering coordinators, network engineers, network architects, and business leaders understand and leverage the power of peering at the core of the Internet.

About This Book

This book is not about technology. There are many great books about the protocols, hardware, and algorithms. One can look at the protocols, the hardware, and the algorithms, but that doesn’t tell the story about how the Internet works at the core.

This book is about the core of the Internet, the relationships between the companies that operate the central infrastructure, the “interconnection machinery” that makes the Internet work. It describes the business relationships that simultaneously demonstrate symbiosis and grudging interdependence. This book models the Internet as a Global Internet Peering Ecosystem, and describes the players, their power positions, the corresponding motivations and, indeed, predictable behavior.

This book includes everything you need to know about the Global Internet Peering Ecosystem to make the critical decision: when and why does your company need to connect to the core of the Internet? Some parts of this book will be of interest to you immediately, and some parts will be of interest to you later on. All of it is essential for business people, executives, and technical staff focused on the Internet.

Together, we will take an outward-in view of the Internet.

Section I Connecting to the Edge of the Internet

Section II Connecting to the Core of the Internet

Section III The Global Internet Peering Ecosystem

Section IV The Tricks of the Trade: The Playbooks

Section V Advanced Peering Topics: Remote and International Peering

We will start by looking at how most companies attach to the edge of the Internet, and learn about the industry terms. We then dive into the core of the Internet, where the largest ISPs and content companies in the world peer their networks. Then we take a 30,000-foot view on the Global Internet Peering Ecosystem, and recognize the commonalities across the globe. With this holistic view of the Global Internet Peering Ecosystem, we are able to continue on to examining the “Tricks of the Trade” – the playbooks collected from some of the smartest peering coordinators and Internet Exchange Point Operators in the world. These playbooks document how ISPs obtain peering where they otherwise would not have been able to. The IXP playbook documents how the most sophisticated IXPs build, grow, maintain, and attack or defend their critical mass of peering participants. The book finishes up with “Advanced Topics,” an exploration of the emerging Remote Peering and International Peering strategies.

This book takes the reader beyond “The Internet is a network of networks” and documents the internal workings of Internet interconnection and the strategic nature of these interconnections. This information is essential for any Internet services company building out at scale.

Who Should Read This Book

This book is for anyone who wants to learn how the core of the Internet works.

Internet Service Providers and network operators in general will benefit from understanding the methods and clever maneuvers surrounding interconnection used around the globe. This book highlights some of the issues and clever tactics that seasoned peering professionals see and develop after years in the field.

Internet Exchange Point Operators (IXPs) will find this book helpful for building a peering ecosystem in their facilities. They will also better understand their customers’ mindset and motivations. This book presents the financial value proposition that Internet Exchange Point Operators provide to customers. The business case for peering at an Internet Exchange Point is clearly explained. Most important to IXPs, this book documents the tactics that the smartest IXPs in the world have used to build and grow, attack and defend a critical mass or peers. This material has been used as training material for IXP staff as well as for their customers and investor base.

Hardware vendors have found this material of practical value for better understanding their customers’ deployment environment, their applications, and their motivations. While this book was not written explicitly for this audience, the larger network equipment manufacturers are now becoming keenly interested in this material.

Students have used the previously released white papers in their classes for years. This book, rewritten from this earlier material, provides a good starting point for understanding the Internet core. The book provides a pragmatic industry view on interconnections that are essential for identifying research projects that are relevant.

Investors in colocation, data centers, IXPs, or large-scale Internet services will find this book to be an invaluable resource in understanding how the Internet actually works at the core.

This book is an assimilation of thousands of discussions with some of the brightest peering coordinators in the world between 1994 and 2014. A subset of these people is listed in the acknowledgements. Many more of them provided valuable data and insights but asked to remain anonymous. What you hold in your hands is based on over fifteen years of primary research.

Notes from the field.

Stories and Anecdotes to Make a Point

Throughout this book I include stories and anecdotes that illustrate a particular point. I will format them as I do here to set them apart from the generalizations made in the book. When I consult with clients and when I present peering workshops, I have found that these stories help bring to life the points presented in the material.

In The Grand Design, Stephen Hawking says that a model is a good model if it is “1) elegant, 2) contains few arbitrary or adjustable elements, 3) agrees with and explains all existing observations, and 4) makes detailed predictions about future observations that can prove or disprove the model if they are not borne out.”

While the modeling of the Internet core as an ecosystem is not of the scale or importance of his work, I believe the Global Internet Peering Ecosystem model is a good model. The model is 1) simple, 2) contains few variables, 3) agrees with the observations made around the world, and 4) provides detailed predictions on the behavior of the parties involved at the Internet core.

The Global Internet Peering Ecosystem has many characteristics of a living ecosystem. There are individual identifiable species that hold power positions within the ecosystem, complete with motivations and predictable behaviors consistent with their position. The ecosystem itself has morphed over time to adjust to the stimulus placed upon it by the customers and their applications at the edge. The scope of this Internet ecosystem is global, so the stakes are very high, and information has been hard to gather. This book uncovers perhaps the most important topics on the Internet today – the evolution of the Global Internet Peering Ecosystem, and are we going in the right direction?

/wbn

Section I – Connecting to the Edge of the Internet

Almost all attachments to the Internet are made at the edge. We will begin our exploration here.

In section I, we introduce the “Internet Transit” service, how it is offered, priced, and metered. We exercise these definitions with a set of “Internet Transit Playbook” tactics that some of the smartest peering coordinators in the world have used to derive the maximum value from the Internet Transit service for the minimum cost. While many of these tactics are not recommended, they have been used in the field, they demonstrate the Internet Transit service, and they exercise the lexicon. In my peering workshops and consulting engagements, the Internet Transit Playbook chapter has proven to be very educational.

By the end of this section you will understand the Internet Transit service.

Section I includes the following chapters:

Chapter 1 – Introduction

Chapter 2 – Internet Transit

Chapter 3 – The Internet Transit Playbook

Chapter 1 – Introduction

Context Is Important

I was waiting in the gate area for my flight from Detroit to Los Angeles. I had already been upgraded, so the flight was starting very well! The gate agent called me forward to tell me that my already-delayed flight would most likely be cancelled, and that she would like to rebook me on the next flight to LA, about 10 gates down the terminal. “Your flight will leave in 15 minutes.” I asked that my work colleagues be rebooked as well and she obliged, so the three of us walked to the new gate and boarded the plane, just in time for it to depart. Or so we thought.

The flight didn’t take off on time. For 30 minutes or so we witnessed some very stressed people getting on the plane, disheveled and complaining. These people were clearly distressed. After a dozen or so people headed back into coach, several came back to the front of the plane saying that their assigned seat was taken and they plopped down in first class. The flight attendant said, “OK, just sit anywhere. We have to leave now!”

If you have ever flown first class, you know that you never hear a flight attendant say, “just sit anywhere.” In any case, the door closed and we took off. I learned the story from the agitated people, after we were in the air.

As it turns out, shortly after we had left the original gate, the gate agent announced the cancellation of that first flight. She made the following announcement:

“Ladies and gentlemen, we regret to inform you that flight 234 to Los Angeles has been cancelled. We will work to rebook you on a later flight, but in the meantime, we have a flight to L.A. leaving in 10 minutes at gate 34, about a 5-minute walk from here. We can take the first 15 of you. If you would please progress to gate 34 in a calm and orderly fashion we will handle boarding or rebooking there. Please proceed to gate 34 in a calm and orderly fashion.”

Given this context, can you guess what happened? Isn’t it predictable what would happen?

Sure enough, the 154 passengers rushed to gate 34, many in a slight panic. The line developed at gate 34 and the gate agent processed people in order until people started circling around the gate agent to plead their special case. “My band plays tonight,” “I am already late to my sister’s wedding,” “My business meeting starts an hour after I land,” “I cannot be late to my next flight,” etc.

The orderly line turned into a “make-your-case” mob fest. Those who were the most convincing and/or aggressive were given a boarding pass, perhaps because the agent just wanted to get rid of them. Of course rewarding this behavior led to louder and more aggressive acts to get to the front of the line. Imagine the situation. The notion of fairness, engraved in our minds, kicked in and led to the airport police and the Detroit police being summoned to the gate to calm things down and restore order.

Who bears responsibility for this ugly scene? Before you state the obvious, that the airline bears the responsibility, do you place any blame on the passengers? Do they not have control of their actions? Or are we wound up like an involuntary muscle forced to act upon stimuli without applying higher-level thought?

Since that experience I have asked many people for their opinions about responsibility in such situations. In the U.S., most believed that the airline was 100% responsible for that mob scene, for the pushing and shoving, for the incivility and selfishness that resulted. They argue that the gate agents rewarded the bad behavior, and the people were justified in playing by the gate agents’ rules. Those who were aggressive and convincing with their cases would be rewarded. This was the game and those who would win would adopt this winning strategy. It is a rational strategy. It would be irrational not to be aggressive. If you were not aggressive, you would have no chance to board this next flight.

Others made the case that the airline was 90% responsible for the mêlée, but the people were responsible for their own actions regardless of the context they found themselves in. To say that the airline was 100% responsible is to allow people to dismiss civility and responsibility for their own actions when they are placed in a bad situation. If true, then such situations would provide a “free pass” for whatever behavior we choose, knowing that someone else would be blamed. The people in this camp believe we are responsible for our actions regardless of the context we find ourselves in.

As the 1980s rock band Devo so eloquently put it,

“Are we not men?”

I would like to think that if I were placed in that situation, I would sit in line, legs crossed in yoga pose, breathing deep breaths while chanting the words “calm blue ocean.”

It was interesting to find that while many in my European audiences also believed the airline was 100% responsible, a large other percentage of them believed that the airline was maybe 60% to 90% responsible for the riot. Some said the Europeans are more inclined towards personal responsibility, while others said that they are just more accustomed to waiting in lines.

In any case, everyone agreed that the outcome (the riot) was predominantly driven by the context (the way the airline had set up the situation).

Just as context is the most important part to this airplane story, the most important thing to understand about the Internet is its context.

The Internet Operations Context Is Important

In this book, I will share a generalized model for the “Global Internet Peering Ecosystem.” All species within an ecosystem react to their environment. We will learn how Internet Service Providers are driven by their position within their Internet Peering Ecosystem. Their positional power determines their motivations, which in turn drive their behavior. Much as the passenger behavior in the airplane story is predictable and driven by context, so too are the behaviors of the players in the Internet Peering Ecosystem. A decade of research has gone into documenting the Internet operations context described in this book.

The Playbooks Are Important

The first time you do anything, you make mistakes. The second time, perhaps you make different mistakes. After a few dozen more times, it is rare to make a mistake, and the process becomes systematic. With even more experience, you master a systematic process. In this book I will present three “playbooks” that demonstrate some mastery in the peering domain.

The tactics in the playbooks are not necessarily recommended tactics, or best practices. Rather, they are actual tactics (both good and bad, ethical and otherwise) that are in use in the real world. They are presented here to enable you to understand the interconnection paradigm by exploring the various manipulations of the system. Understanding the manipulations of the system helps us better understand the system.

The Lexicon Is Important

After leading hundreds of peering workshops and consulting engagements, I have learned that people don’t truly understand terminology until they use it themselves in actual applications.

To this end, the first few chapters will get you comfortable with the interconnection lexicon. First I will introduce the Internet Transit model and terminology used when attaching at the edge of the Internet. Then we will explore the model with some problem sets and manipulations of the system in the “Transit Playbook.” The playbook uses these definitions to describe the manipulations.

Then we introduce the local routing optimization called “Internet Peering.” We examine the terminology, the processes typically used in peering, and the business case for peering. The problem sets at the end of each chapter can help you determine if you absorbed the material.

Everyone will misuse the terminology a few times before they understand it. During the workshops the “ah-ha” moment generally occurs as we apply the terminology correctly.

Let’s dive into the heart of Internet operations and explore.

Chapter 2 – Internet Transit

In this chapter you will learn about Internet Transit, a service that provides access to the global Internet. We introduce the definitions and the service model, and then describe how Internet Transit service is offered, priced, and metered.

Introduction to Internet Transit

The Internet is a network of networks.

To get connected to the Internet, an entity must attach itself to an entity that is already connected to the Internet. It usually accomplishes this connection by purchasing a service called “Internet Transit.”

Definition: Internet Transit is the business relationship whereby an Internet Service Provider provides (usually sells) access to the global Internet.

From a high-level perspective, Internet Transit can be thought of as a pipe in the wall that says “Internet this way.” Customers connect their networks to their Transit Provider, and the Transit Provider does the rest.

Definition: An Internet Service Provider (ISP), also called a “Transit Provider,” is an entity that provides (usually sells) access to the Internet.

The primary business of an Internet Service Provider is to market, sell, and operate the Internet Transit service described next.

The Internet Transit Service Model

The Internet Transit service has two primary functions:

1. to announce Internet routes to the customer, and

2. to announce the customer routes to the rest of the Internet

Internet Service Providers need to get connected to the Internet in order to sell access to the Internet, so they too need to purchase Internet Transit from a transit provider. To illustrate how this would work, let’s assume that ISP A is purchasing the Internet Transit service from Transit Provider B as shown in Figure 2-1.

Figure 2-1. The Internet Transit Service.

We will use graphical notations throughout this book. The different colored clouds represent networks attached to the Internet. The ‘T’ next to a line represents an Internet Transit relationship. The routing advertisements box describes which routes are “announced” between the Internet Transit customer (ISP A) and the upstream Transit Provider B. The arrows signify the direction of the routing announcement.

These routing announcements are propagated across the Internet. Transit Provider B announces to the ISP A the routes to reach the entire Internet (shown as many colored networks to the right of the Transit Providers). The ISPs across the Internet propagate this ISP A route so that all networks know how to reach the ISP A.

In summary, through the Internet Transit service, all Internet attachments know how to reach ISP A, and ISP A knows how to reach the rest of the Internet.

For convenience, we will also use the following equivalent notations (Figure 2-2) to indicate a transit relationship between a customer and the upstream transit provider.

Figure 2-2. Equivalent Internet Transit service notations.

The black circle is shorthand for a default route to reach the rest of the Internet. This typical and default arrangement can be inferred by identifying the two parties in the relationship with the “T” and the “$” symbol describing which party is being paid for the Internet Transit service.

The Internet Transit Pricing Model

Internet Transit is typically a metered service outside of the residential market. The unit price for Internet Transit services varies widely, but the service itself is typically priced on a per-megabit-per-second (Mbps) basis, metered using the 95th percentile traffic sampling technique.

Definition: The 95th Percentile Measurement Method (also called 95/5) identifies a single measurement (the 95th percentile 5-minute sample for the month) to determine the transit service volume for monthly transit fee calculation.

The 95th Percentile Measurement Method is typically a three-step process, as follows (diagrammed in Figure 2-3):

1. Every 5 minutes the meter on the service is sampled.

2. The delta is calculated between the two adjacent samples and the result is stored. (It is in this step that corrections are made for wrapped counters, counter resets from reboots, anomalies in the measurement on the metering device, etc.)

3. At the end of the month, the 5-minute deltas are converted into Mbps and stacked lowest to highest. It is the 95th percentile value that is used to calculate the traffic volume for the month.

Figure 2-3. Calculating the monthly cost of Internet Transit using the 95th percentile measurement methodology.

The basic Internet Transit billing model is simple:

Monthly Bill = Internet Transit Vume * Internet Transit Unit Price

Example:

If you signed an Internet Transit contract with a negotiated price of $5/Mbps (measured at the 95th percentile), and at the end of the month you sent 500Mbps (measured at the 95th percentile), what would be your transit cost that month?

Answer: 500Mbps * $5/Mbps = $2500 per month

The Origin of the 95th Percentile Measurement

How did the 95th percentile come into existence?

The Internet Transit pricing model went through three distinct pricing phases:

• Phase 1 - Internet Transit was sold by access capacity. In the early Internet days (circa 1995), Internet Transit was billed on a circuit-capacity basis. You would purchase a T1 circuit (1.544 Mbps) to the Internet and pay as if you used the entire circuit capacity 24 hours a day. But if you didn’t use very much of this capacity, you were still paying as if you did. At the time, this reality made Internet Transit tough to sell to some low-volume prospects.

• Phase 2 - Internet Transit sold by average usage. One of the early ISP pioneers started charging based on average use, but this measure ended up being skewed by the burst nature of Internet traffic. Customers who used only a little transit were paying as if they used much more because the average was skewed upwards by these occasional peaks.

• Phase 3 - Internet Transit sold by 95/5 model. To address this situation, one ISP adopted the 95th percentile measure to prevent overly punishing a customer for the occasional spike in traffic volume, while still allowing the ISP to bill based on the actual load placed on its network. This approach seemed palatable to the market and sold well. The rest of the industry adopted this pricing model, and within a year or so 95/5 became the pricing paradigm for the Internet Transit service.

Measurement Direction

One question always comes up at this point in the peering workshops:

“Is 95/5 measuring traffic coming into the network or out of the network?”

The 95th percentile measurement uses the greater of in versus out, as shown in the following equation:

Monthly Bill = max(inAt95th,outAt95th) * Internet Transit Unit Price

But for the sake of simplicity, when we introduce equations from here forward, we will not clutter the equations with this in-vs.-out nuance. When we say 95th percentile, we know that what we really mean is the 95th percentile of the larger of in vs. out.

The Internet Transit pricing model gets only slightly more complicated when adding “commits.”

Internet Transit Pricing with Commits

To encourage the use of the Internet Transit service, and to compete more aggressively in the market, most ISPs provide pricing discounts for pre-committing to certain volumes of traffic (Table 2-1).

Table 2-1. Sample Internet Transit with Commits Pricing Table.

Here we see an ISP that offers a price of $12 per Mbps for a monthly commit of 10Mbps, and a price of $5 per Mbps with a monthly commit of 100Mbps. These price points guarantee that the monthly transit fee will be at least $120 and $500 per month, respectively.

What happens if customers commit to 100Mbps but don’t use all of the 100Mbps in a particular month? They are still required to pay as if they had used the full 100Mbps.

What happens if they use more than the 100Mbps in a particular month? They generally will have to pay the metered rate on the volume of traffic they actually used. This rate is sometimes referred to as the “burst rate,” but it is generally the same price as the commit rate.

We can describe this model formulaically as:

Monthly Bill = max(TransitVolumeAt95th*Price, Commit Volume*Price)

The Internet Transit with commits model typically involves contracts that state the price point for the agreed-upon commit level and the duration of the agreement. So, for example, a 1Gbps commit at $3.50 per Mbps might have a 1-year term. This is how the Internet Transit service is sold now.

Internet Transit Price Declines

I have been tracking the price of Internet Transit since 1998, and as you can see from Table 2-2 and Figure 2-4, the prices for Internet Transit have dropped every year.

Table 2-2. U.S. Internet Transit Prices (1998–2015) Historical and Projected.

It is worth pointing out that these data have always generated heated debate in the peering coordinator community.

Notes from the field.

Internet Transit Pricing Data and Controversy

The controversy over these numbers is related to the wide variability in service offerings and pricing. Further, pricing schedules are often protected under a nondisclosure agreement (NDA), and generally not publicly available to systematically analyze. Some ISPs consider many variables when determining pricing. They might, for example, account for the number and size of the ports requested, type of the customer traffic (inbound vs. outbound), how that traffic would impact the ISP, the region(s) and market conditions at the location of the interconnection, etc. For these reasons, many would argue that there is no single market price for Internet Transit, that the data are inherently flawed.

And they have a valid point.

My data are anecdotal and based on a variable and small sample size, polling maybe 30–50 people three to five times a year. The transit price points are collected on an ad-hoc basis at operations conferences that I attend every year. I ask ISPs and transit customers, “What is the going price for transit these days?” From the dataset I throw out the outliers and select the middle from the mass of numbers that fall within a narrow range.

However flawed this manual measurement system is, it remains one of the very few longitudinal measures of the market price for transit. Some of these measurements were based on or validated by contributed ISP pricing sheets. One can say “according to the conversations Bill has had with ISPs over the years, the transit prices have dropped roughly like this line.” I would not recommend taking these numbers as anything but very rough indications – and a better mechanism is on its way as we start measuring in $/Gbps.

Besides, to focus on the individual data points is to miss the point.

The point is that the trend is unmistakable, and no one would disagree – because transit prices have dropped every year.

The other interesting thing I noticed is that every year the ISPs say to me things like, “Transit prices can’t get any lower,” and “No one is making any money at these prices.” And every year, the prices drop again.

Figure 2-4. Internet Transit prices historically decline every year.

The good news for ISPs is that Internet traffic volumes have always grown as well, and all indications are that this trend will continue for some time. Even though the price of transit has declined by around 30% per year, Internet traffic typically increases by more than 50% per year.

Variability in Internet Transit Service

There is also some debate within the community on the qualitative differences between transit from a lower-priced provider and the Internet transit service delivered from a higher-priced provider. Higher-priced providers would argue that they have better-quality equipment (routers vs. switches, etc.), and better and more sustainable operations procedures that simply cost more to operate. They would argue that “you get what you pay for in the transit market.” They could be right, but I would also point out that I have not met very many paying customers complaining about the quality of service of lower-priced providers.

Internet Transit is a Customer-Supplier Relationship

Some content providers prefer a transit service (paying customer) relationship with ISPs for business reasons. They argue that they will get better service with a paid relationship than with any free or “special deal,” bartered, or “peering” relationship.

Internet Transit May Have SLAs

Service-Level Agreements (SLAs) are contracts with financial penalties for failure to meet specified service levels. They are intended to punish an ISP for failure to meet the explicit customer expectations. Many Internet Transit customers I have spoken with have said that they felt comforted by these financial penalties, these “teeth.”

Internet Transit SLAs, however, are widely dismissed by the network operators as merely insurance policies. The ISPs I spoke with said that it is common practice to simply price the Internet Transit service higher when it includes SLAs, and with the increased pricing proportional to the likelihood of their failure to meet the SLA requirements. Further, the customer has to notice the SLA failure, file for the SLA credits, and check to see that credits are indeed applied. Some ISPs see SLAs as a way to simply increase margins without needing to improve the service.

Implementation Model for the Internet Transit Service

While a complete exploration of the ISP business models is beyond the scope of this book, a few commonalities across ISPs are worth highlighting. Every year the underlying technologies get better, cheaper, and faster. These advancements tend to come in waves. As a result, most larger ISPs have an Architecture-Engineering-Operations workflow, with responsibilities being transitioned between these disciplines as shown in Figure 2-5.

Figure 2-5. The Architecture-Engineering-Operations new technology waves and responsibilities.

When new technologies are considered for a network, the ISP Architecture team evaluates the applicability and designs a solution. Once accepted, the solution moves through a joint Architecture/Engineering development phase, during which the Engineering team increasingly takes over responsibility for the deployment of the technology. The Operations group takes over responsibility from the engineering team as the technology hardens and the logistical transition challenges are worked through. The Engineering team eventually is removed from the day-to-day operations activities of the technology, and the Operations team runs the technology in the production network.

This cycle continues as the next wave of technology becomes available and the Architecture team looks at its applicability. These technology waves apply to backbone infrastructure (routers, transmission gear) as well as edge devices (routers, switches, auxiliary services) and support systems.

These waves of technology are important because they enable the ISPs to compete more effectively in the Internet Transit market. Those with old equipment simply can’t compete with the economies of scale realized by the large players.

Observations about Internet Transit

To finish this chapter, let’s highlight a few important characteristics about Internet Transit:

1. Internet Transit is a simple service from the customer perspective. All you need to do is pay for the Internet Transit service, and all traffic sent to the upstream ISP is delivered to the Internet.

2. Internet Transit is typically a metered service. The more you send or receive, the more you pay.

3. Internet Transit has commits and discounts. Upstream ISPs often provide volume discounts based on negotiated commit levels. Thus, if you commit to 10Gbps of transit per month, you will likely get a better unit price than if you commit to only 1Gbps of transit per month. However, you must pay for (at least) the commit level worth of transit regardless of how much traffic you send.

4. Internet Transit contracts have a term, a duration.

5. Internet Transit prices drop every year.

That is what you need to know about Internet Transit. In the next chapter we will discuss the “tricks of the trade” that some have used in an attempt to improve the efficiency of their transit service purchase.

Peering Workshop Practice Questions

Here are a few practice questions from the Internet Peering Workshop:

1. I am purchasing Internet Transit from ISP A for $5 per Mbps with no commits. At the end of the month I send 500Mbps and receive 800Mbps at the 95th percentile. What is my monthly bill for Internet Transit?

A) $5/month B) $2500/month C) $4000/month D) $6500/month

2. I am purchasing Internet Transit from ISP B for $5 per Mbps but I am considering buying its 1Gbps commit transit product at a price of $3/Mbps. I still expect to send 500Mbps and receive 800Mbps at the 95th percentile. Should I commit to 1Gbps?

Answers to these questions are in the answer key in the back of the book.

Chapter 3 – The Internet Transit Playbook

This chapter presents clever manipulations of commodity Internet Transit services.

As previously mentioned, these “tricks of the trade” are not “recommended” tactics. A few, you may consider unsavory or even unethical. They were gleaned from conversations with hundreds of peering coordinators, and are included here because they clarify the Internet Transit paradigm so well, and they give us an opportunity to apply the terms and definitions learned in the previous chapter.

Introduction

When I lived in Ann Arbor, Michigan, the local newspaper ran a yearly article featuring how the locals get around town efficiently. These tricks included rarely used routes and secret ways to bypass congested paths using side streets, one-way streets, and alleys, and included even illegally using parking lots as avenues between two roads. Whether you used them or not, it was enlightening to read about these tricks.

In this chapter we will present a dozen tactics that have been used effectively to manipulate simple commodity Internet Transit services.

Edge “Tricks of the Trade”

One challenge that we pose during the peering workshops is to create your own “manipulation.”

Put yourself in the shoes of a purchaser of Internet Transit. Based upon the information you have learned so far, what clever maneuvers could you use? How would you optimize your transit purchase? How would you minimize your transit costs? You have all of the information you need to answer these questions.

Sometimes it helps to start by enumerating some assumptions.

Let’s assume that you are the purchaser of Internet Transit in a market with the following typical characteristics:

• The price declines yearly.

• The market for Internet Transit services is open and competitive.

• Pricing varies widely (say plus or minus 20 to 50%).

• 95th Percentile zero-commit services are available for $20/Mbps.

• You may be wildly successful (70Gbps) or have only 5Mbps of total traffic.

• There are tiered transit services as discussed in Chapter 2 and shown in Table 3-1.

Table 3-1. Sample Internet Transit Pricing with Commits.

It is very important that, before you continue reading, you think about and write down a few strategies that reduce your monthly transit expense and/or improve its performance.

Answer: Here are the tricks of the trade collected from the field.

Transit Tactic 1 – Optimal Internet Transit

This tactic seeks to minimize the cost of Internet Transit by committing early to the next higher tier of commitment. By carefully analyzing the pricing sheet, we can identify the optimal point to commit to the next higher level of traffic. Seasoned peering coordinators have learned that it is often cheaper to commit to a higher traffic volume level, even when there is little chance of actually using that much traffic in any given month.

For example, if we allocate our tiered pricing model in Table 3-1 across the number of Mbps we expect to send, we can see that it is cheaper to sign up for the lowest commit level (10Mbps) until we start using more than 41Mbps (see Table 3-2).

Table 3-2. Internet Transit Pricing Table with Commits.

The highlighted cells show the lowest monthly Internet Transit fee.

Even though you may not use the full 100Mbps commit that month, it is still less expensive to commit to 100Mbps if you expect to send more than 42Mbps. This “early commit” is shown graphically in Figure 3-1.

Figure 3-1. Tactic 1 – Early commits lead to optimal unit prices for Internet Transit.

The monthly transit price for this “Optimal Transit Heuristic” can be described formulaically as:

Optimal Transit Bill = min(T * P, C * H)

where:

T = monthly traffic volume

P = unit transit price at commit level

C = next higher traffic volume commit level

H = next higher traffic volume commit level price

This heuristic can result in a reduction in monthly Internet Transit fees, but it is worth pointing out that the cost savings depend on the ability to predict the future traffic volume. The savings will disappear if for some reason the traffic volume isn’t there. For example, if you commit to 100Mbps based on the assumption that you will hit 45Mbps this month but you reach only 40Mbps, this tactic will actually cost you more money. With this tactic the company must track and project its Internet Transit traffic volumes.

Notes from the field.

Cable Companies Pay Too Much for Internet Transit?

A few ISPs said that the large access networks were not very savvy negotiators because they were paying much more than market prices for Internet Transit. They said the cable companies were paying about double the market rate. I asked a few cable companies about this, and they explained that it might be true. They said that their inclination was to commit well below the traffic projections to gain flexibility in adjusting traffic and transit purchases across providers based on market conditions. What might be assumed to be poor negotiating might actually be the exercising of a different strategy.

For those that track traffic volume very carefully, the next tactic is particularly clever (and dangerous).

Transit Tactic 2 – Gaming the 95th Percentile

In this tactic, an ISP can (theoretically) create a $0 cost Internet Transit service by leveraging two facets of the Internet Transit pricing model discussed so far:

• The 95th percentile measurement methodology allows you to burst above the 95th percentile mark without any additional cost.

• Some ISPs sell a zero-commit Internet Transit service.

From these two facets, can you guess what this tactic entails?

This tactic involves “purchasing” Internet Transit from about 25 transit providers, each with a zero-commit Internet Transit service and enough capacity to carry the entire offered traffic load. The ISP running this tactic “dumps” all of its traffic for just under 36 hours to each transit provider in turn, sending zero traffic for the rest of the month to that upstream transit provider (see Figure 3-2).

Figure 3-2. Tactic 2 – Gaming the 95th percentile spreads load across 25 or so ISPs, bursting for 4% of the time and idle 96% of the time.

This tactic takes advantage of the fact that 36 hours is five percent of the month – the amount of time per month that is above the 95th percentile (where the traffic is free!). So when you look at the stack of 5-minute samples for the month, the 95th percentile measure will be zero! Starting at the 96th percentile and on up, the traffic load will be the entire offered load.

While many derivatives of this tactic have been run against Internet Transit services and bandwidth exchanges, the practitioner of this tactic must be very careful. There are several stories of ISPs using this tactic only to find that they calculated the 95th percentile measure differently than their upstream providers. The result was that multiple ISPs billed them for the full offered load! In their case, instead of paying $0, they paid double.

This tactic in its pure form is rarely seen in the Internet Peering Ecosystems anymore Just the same, every ISP should understand this tactic and recognize it.

The key to making this 95th percentile gaming work is the zero traffic commit and the 95th percentile billing – one of the reasons why most ISPs require minimum commits these days.

Transit Tactic 3 – Multi-Homing

Most companies and ISPs start off singly homed. They pay the metered rate and they get access to the Internet and it just works – until it doesn’t.

Multi-homing is the practice of connecting to two or more upstream ISPs. Most large-scale companies spread their traffic across two or more ISPs in order to improve performance and resiliency and reduce costs in a competitive market (see Figure 3-3).

Figure 3-3. Tactic 3 – A multi-homed Internet Transit customer.

But there are some hurdles they need to overcome.

Many of these companies don’t have the internal networking resources to explore multi-homing. While not difficult, it may be challenging for a systems administrator to learn enough about networking to configure multi-homing the first time.

Notes from the field.

Content Companies and the Networking Conceptual Hurdle

I consulted with a content provider that described its content-creation operation. The provider had writers, graphic designers, marketing and advertising, sales, etc. The tech staff were primarily web site folks and a couple of systems administrators. One systems administrator shared that he had the most networking experience of his team, so he was appointed to get multi-homing operational. He described the process as “harrowing.”

He configured the secondary Internet Transit service and “held his breath” as he cut over the offered load to the backup. It didn’t work, and the company was off the air! Scary. He cut back over to the primary service and went back to the books to see what he had done wrong. When he finally had the router configured to the backup service provider, he configured the primary backup and used the primary. After everything was up and working properly (he hoped), he locked everything down and no one dared to touch the configuration again!

“This is a dangerous on-the-job-training activity – perhaps multi-homing is best done by those who have done it before,” he said.

One flavor of this multi-homing tactic is to build into a well-populated co-location center – a data center housing an open market for Internet Transit services. You will pay for the cost of getting there, but the open-market effect may result in transit pricing 30% or more lower than if the transit provider has to build into your location.

One final benefit of multi-homing is that it makes the path to free traffic exchange (“peering”) much easier. The trick here is to identify the appropriate co-location center for transit and future peering, and the appropriate upstream ISPs for multi-homing. We will discuss this in a later chapter.

Transit Tactic 4 – Renegotiate Multi-Year Contracts

Transit purchasers employing this tactic renegotiate their transit agreements before the end of their terms, as shown in Figure 3-4.

Figure 3-4. Tactic 4 – Renegotiating Internet Transit deals mid-contract.

In this tactic, the customer argues (mid-contract) that a renegotiation is needed to bring the unit transit price in line with the current market price. If there is resistance, the customer usually publicly declares the ISP as overpriced, denigrates its service, and accepts that there might be a court settlement years from now. Not wanting the damage to its reputation in the market, the upstream ISP acquiesces. The ISP sales team internally rationalizes this renegotiation as a new sale with extended contract terms.

Of course, this new multi-year sale will be renegotiated the next year as well.

Transit Tactic 5 – Play the Market

In this tactic, the ISP plays one transit provider off the other to get the lowest price, but takes the process to an unethical extreme.

For example, verbal agreements and handshake deals start the process of playing each ISP against each other. Parallel rounds of negotiations are in play (see Figure 3-5) all the way to the signing of the contracts. The price points, even though protected under a nondisclosure agreement (NDA), are used (or bluffed) during this process to negotiate better terms with the other ISPs. In some cases, even after signing, the price negotiations continue as alternative offers from the parallel negotiations chime in.

Figure 3-5. Tactic 5 – Playing the market participants against each other.

In this tactic, the buyer leverages the open market with parallel negotiations, information asymmetry, and bluffing to achieve the lowest possible transit price points.

Transit Tactic 6 – Resell Transit

Even back in the early days of the Internet (1990), many companies committed to more Internet Transit than they needed to obtain the lower unit price. They were then free to sell the excess to others to reduce their average unit cost (see Figure 3 6).

Figure 3-6. Tactic 6 – Selling Internet Transit to Friends.

Museums and small community colleges would formally or informally group-buy Internet access. Since Internet operations was never their primary activity, the customer support available to customers was minimal. Still, this tactic was used to lower the aggregate cost of transit.

Transit Tactic 7 – Secret Sauce Transit

The Border Gateway Protocol (BGP) uses a variety of metrics for route selection, but it doesn’t account for the cost or performance of sending traffic along a particular path. Several specialty ISPs over the years have created a “secret sauce” of routing adjustments that account for these variables (see Figure 3-7).

Figure 3-7. Tactic 7 – Route to minimize cost and improve performance.

This tactic can provide the cost-performance routing optimization very quickly, and it solves a real-world problem. However, implementation requires a high degree of expertise. The service solutions in this space, while attractive, require an educated and network-savvy customer prospect to recognize their value. As a result, the sales cycle for these services is long; this solution has not been highly successful in the marketplace.

Transit Tactic 8 – Build into Cheap Transit Internet Region

Some Internet Transit markets are expensive. For example, in 2010 the price of Internet Transit in South Africa and Kenya was close to $400/Mbps, whereas Internet Transit could be purchased in London for $2-$4/Mbps. The question is, can you save enough money to cover the cost of building into London from these places?

This dynamic is generalized in Figure 3-8.

Figure 3-8. Tactic 8 – Buy Internet Transit in a better Internet region.

For an emerging set of ISPs from the newer ecosystems in Africa, Eastern Europe, and the Middle East, the answer is “Yes!” especially when they have the goal to reduce the dependency on the transit providers in their home Internet region.

A company called SEACOM, for example, has been lighting transoceanic fiber into each country around the perimeter of Africa. This fiber replaces the high cost and complexity of acquiring cross-border circuits between neighboring countries, and also heads up to Europe. The reason I mention this reality is that this oceanic fiber deployment enables this play for many ISPs in Africa.

Similarly, one European IXP said that this is a popular tactic for ISPs coming to Western Europe from Eastern Europe or the Middle East. Cheap transit ($5-$10/Mbps) is a powerful lure for those who are paying $400/Mbps.

Transit Tactic 9 – Internet Transit Troughs

Internet usage patterns are mostly predictable. An Internet Transit service load typically follows a sinusoidal curve if it is used primarily during working hours. Other loads exhibit a dual-hump curve because people use the service first in the morning when they wake up, and then again when they return from work in the evening. In this tactic, Internet services companies use the low points in the network load curve – the troughs.

For example, many residential access networks in the U.S. sell Internet Transit to business customers as well now (see Figure 3-9). Business customers use the network primarily during the daytime hours when the residential customers are away at work, allowing the access networks to make additional money without overloading the network during peak hours. Adding this business traffic may increase revenue while not impacting their Internet Transit expenses.

Figure 3-9. Tactic 9 – Traffic shape to use the off-peak times.

Content providers leverage these traffic patterns as well. One content provider that sells access to video content that “transcends the language barrier” said that it tends to see a dual-hump usage pattern with peaks around noon and 10 p.m. Since the provider has extended periods where the network is very lightly loaded, it is looking for clever ways to utilize its wide troughs of low usage.

One way to spread the traffic load into the troughs is with a “slow-bleed” tactic. The content provider anticipates what content will be requested and preemptively sends and pre-caches the content on (or topologically close to) the end-user machine. These video objects are distributed several weeks before the “airing,” but only during the trough periods when there is low load and practically no incremental cost at the 95th percentile. A small distribution key is then sent out to “activate” the video object so people can view it. This optimization uses the troughs in their transit load to reduce the cost of distribution.

Transit Tactic 10 – Capture Content and Access Customers

In this tactic, ISPs seek to keep the traffic on their network with a strategic customer-acquisition strategy. They acquire both the content and eyeballs customers.

There are three possible scenarios, shown in Figure 3-10.

Figure 3-10. Tactic 10 – Three ways for traffic between content and eyeballs.

1. Only the content side pays you. Scenario #1 is where your content customers’ traffic goes through your upstream transit provider. If this direction is the dominant direction of your traffic flow (i.e., you are “content heavy”), then you are paying for that traffic to be delivered. At the same time, you are being paid by your content customer for that content to be delivered. You get to keep the difference in prices.

2. Only the eyeball side pays you. Scenario #2 is where your eyeball customers are requesting content that comes through your upstream provider. If you are “access-heavy,” then you are paying for this traffic. At the same time, you are being paid by your eyeball customers for that content to be delivered to them. Again, you get to keep the difference between the prices.

3. Both content and eyeball networks pay you. Scenario #3 is the best-case scenario since you are being paid by your eyeball customers to reach content that you are also being paid to deliver. You are being paid on both sides and simultaneously reducing your transit fees. The only costs are the costs of the infrastructure to carry this traffic across your network. Economists call this scenario a “two-sided” market.

Many ISPs have offered cheap transit simultaneously to the access and content-heavy networks. Around the turn of the century, for example, an ISP called Level 3 had most of the North American cable companies as downstream customers and simultaneously acquired some of the larger content customers that wanted to deliver to those eyeballs. Obtaining customers by “charging the other side” is a common tactic in the industry.

Transit Tactic 11 – Short-Term Transit

With this tactic the customer keeps ratcheting down the negotiated price every month, loosely tracking the dropping market prices for transit. With this strategy there is little chance that the customer will be stuck with a contracted transit price that is significantly higher than the market price for Internet Transit (see Figure 3 11).

Figure 3-11. Tactic 11 – Purchase with little or no commits for short-duration contracts.

Some of the very large content companies have negotiated no-commit month-to-month pricing. They have the luxury of deciding where to send their traffic without the burden of watching their traffic volumes against commits.

Transit Tactic 12 – Deploy Trial Gear into Operational Environment

One ISP deployed trial “loaner” gear into its production network. In this way the ISP leveraged free equipment, free technical support, and free delivery by an enthusiastic sales person.

Notes from the field.

Deploying Trial Gear Into Production

I visited a (now bankrupt) large-scale tier 2 ISP at its headquarters. I saw its deployment staging area in a strip mall near Detroit. For months, this ISP had fended off a Cisco salesperson who had been trying to reclaim a large core router that was supposed to be in the ISP’s remote test lab. Actually, this router was installed and had been operating in the production network for 6 months. There were claims of card failures that required retesting before accepting for purchase, claims of test-lab delays, budget cycle issues, etc. Before this router was returned to Cisco it had been in full service for over a year. Such a scenario would certainly keep one’s costs down.

Summary

As we have seen, even a simple service like Internet Transit can be manipulated. This chapter enumerated a variety of tactics that illustrate the Internet Transit paradigm and some of the market dynamics at play:

• Optimal Internet Transit, where you commit early to the next higher commit level

• Gaming the 95th Percentile, where you carefully orchestrate traffic across many ISPs

• Multi-Homing for performance and economic benefits

• Renegotiating Multi-Year Contracts every year

• Playing the Market to leverage information asymmetry

• Resell Transit to allow you entry into a higher commit and lower price point in a tiered market

• Secret Sauce Transit to create additional routing controls based on performance and cost

• Building into a Cheap Internet Region to leverage the wide market disparity around the globe

• Internet Transit Troughs, which can be filled with no cost

• Capturing Content and Access Customers to allow feeding from a two-sided market

• Short-term Transit, which enables the participation in the dropping transit market without commitments

• Deploying Trial Gear into Production to minimize the cost of operating a network

As a reminder, these tactics are not necessarily ones that I recommend. Rather, they are useful to illustrate the Internet Transit paradigm.

Peering Workshop Practice Questions

Here are a few practice questions from the Internet Peering Workshop:

1. Given the assumptions in this chapter, what is the maximum cost savings realized by the successful execution of the:

A) Optimal Transit tactic

B) Gaming 95th percentile

2. What other tactics were not mentioned?

3. You are an eyeball-heavy ISP and your buddy is a content-heavy ISP. Therefore, you are not paying for traffic in your outbound direction, and your buddy is not paying for traffic in the inbound direction. How might you work together to reduce costs?

Answers to these questions are in the answer key in the back of the book.

Section II – Connecting to the Core of the Internet

Now that you understand how to connect to the edge of the Internet (the way 90% of connections are made) it’s time to learn how to do it better. The rest of this book is about Internet Peering, connecting your network at the core of the Internet. This section provides the necessary foundation for understanding the Internet Peering Ecosystem and Internet Peering Playbooks.

We will explore the

• Internet Peering Model

• Business Case for Internet Peering

• Top Ten Internet Exchange Point Selection Criteria

• Public vs. Private Peering Tradeoffs

By the end of this section you will understand the concept of Internet Peering. You will understand the Internet Peering model enough to prove – not convince – but prove when it makes sense to pursue peering and when it doesn’t, from a purely objective financial perspective. You will learn the processes of Internet Peering, the financial value derived from peering, and the processes proven most effective to find, contact, and negotiate with peers. We will walk through the ten criteria that ISPs have claimed were the most important to them when selecting among Internet Exchange Points. We will finish with a discussion of the benefits and drawbacks of Public Peering and Private Peering.

Section II includes the following chapters:

Chapter 4 – Internet Peering

Chapter 5 – The Business Case for Internet Peering

Chapter 6 – The Internet Exchange Point Selection Process

Chapter 7 – Public vs. Private Peering

Examples of peering fabrics from Equinix (above), CoreSite’s Any2 exchange (right) and the Toronto Internet Exchange (below).

Photos courtesy of Equinix and Matt Peterson.

Chapter 4 – Internet Peering

In this chapter we introduce the other dominant form of interconnection, called Internet Peering. Internet Peering is a local routing optimization, a way to exchange some of your traffic with another party, with neither party incurring Internet Transit fees. We describe the process of peering, introduce (a lot of) terminology, and enumerate the top five motivations ISPs give for peering. The chapter closes with a brief discussion of Paid Peering.

Introduction

Some people in the peering community see peering as like dating, finding a suitable partner where each party derives about equal value.

Notes from the field.

Peering is Like Dating

C: I am trying to meet these people at this conference to see if they want to peer, and this feels like looking for a date.

DrP: Peering is a grudging interdependence. Both parties need to agree that it is better for them to interconnect than the next best alternative.

C: Like dating.

DrP: And then after peering is established, both parties continue to agree that the relationship is worth keeping in place, as that they both receive about equal value from the interconnection.

C: Like going steady.

DrP: Well, some peering requires a contract, like a yearly renewable marriage contract. ...Maybe the analogy is a bit stretched here. If the value is not there, the interconnection is removed, a process referred to as “de-peering.” This de-peering gives the community something to talk about... De-peering is sometimes a very public thing.

C: Does name-calling occur, ascribing of evil intent? And are there any of those uncomfortable social situations?

DrP: Sometimes. And sometimes, the peering is reestablished but with some additional conditions.

C: So peering is like dating?

DrP: Somewhat. However, unlike dating monogamy, peering monogamy would be just weird.

Internet Peering

One of the most common discussions within the peering coordinator community is

“Internet Transit is so inexpensive, why do we need anything else?”

The answer is related to transit traffic volume.

As stated earlier, as the price of Internet Transit has dropped an average of 30% per year, the average volume of Internet traffic has historically grown by 40–50% per year. This growth leads to the growth of the transit fees paid by and paid to the upstream ISPs.

For example, while the price for Internet Transit drops from $3/Mbps down to $2/Mbps, the volume of traffic during the same period might grow from 6Gbps to 10Gbps. The net result of these simultaneous forces is a 25% increase in monthly transit fees, from $16,000 per month to $20,000 per month. The percentages vary, but every year the Internet packet transmission market could grow as shown in Figure 4-1.

Figure 4-1. The unit cost of Internet bandwidth drops, but the volume increases. The result is a monthly Internet Transit bill that continues to rise.

So even though the unit cost of transit drops, the monthly transit bill increases. Decreasing the absolute cost of Internet Transit is one of the principal drivers for pursuing an Internet Peering strategy.

Definition: Internet Peering is the business relationship whereby two companies reciprocally provide access to each other’s customers.

Internet Peering is typically settlement-free, meaning that neither party pays the other for access to each other’s customers, reflective of the underlying notion that peering is a relationship of approximately equal value to each party. Since both parties benefit about the same from the relationship, there is no need to bother with the overhead of measurement and settlement.

There is also no standard way to calculate and monitor the absolute value derived from a peering relationship. Is the value of the peering relationship proportional to the volume of traffic freely peered bi-directionally? Or is it proportional to the desirability or uniqueness of the routes? Or is the value the number of people reached?

For these and other reasons, the dominant form of peering is settlement-free. When you see the term “peering” from this point on, it means settlement-free peering.

To illustrate Internet Peering, consider the mini-Internet Peering Ecosystem shown in Figure 4-2 with only three ISPs: WestNet, MidNet, and EastNet.

Figure 4-2. Internet Peering provides reciprocal access to each other’s customers.

Using graphical peering notation, we see that:

• WestNet is an ISP with customers shown as clouds below it, MidNet is an ISP with its own downstream customers, and EastNet is an ISP with its own downstream customers.

• WestNet is in an Internet Peering relationship with MidNet in which WestNet learns how to reach MidNet’s customers, and MidNet reciprocally learns how to reach WestNet’s green customers.

• EastNet is in an Internet Peering relationship with MidNet in which EastNet learns how to reach MidNet’s customers, and MidNet reciprocally learns how to reach EastNet’s customers.

After these two peering sessions are established, the routing tables are in place (as graphically shown as colored circles in the “routing table” beneath the ISP clouds). This diagram shows that MidNet peers with both EastNet and WestNet, and therefore MidNet customers can reach both EastNet and WestNet customers.

It is important to observe that the routing announcements go in the opposite direction as the traffic to that destination. The Internet routing scheme separates the control plane from the data plane.

For convenience, we will use the simplified and equivalent notation shown in Figure 4-3 to indicate a settlement-free peering relationship between two parties.

Figure 4-3. Equivalent graphical notations for an Internet Peering relationship between ISP A and ISP B.

These notations concisely convey that the two parties (ISP A and ISP B) are in a peering relationship.

Three Key Points about Internet Peering

Three key points often get lost when one is first introduced to Internet Peering. It is worth reading the following points a few times:

• Internet Peering is not a transitive relationship. The fact that WestNet is peering with MidNet and MidNet is peering with EastNet does not imply that EastNet customers can reach WestNet customers. WestNet knows how to get to only its own and MidNet’s customers, and EastNet knows how to reach only its own and MidNet’s customers. The fact that they both peer with MidNet is inconsequential; peering is a nontransitive relationship.

• As such, Internet Peering is not a perfect substitute for Internet Transit. Internet Transit is a service that provides access to the global Internet, while Internet Peering simply provides a more direct path for a subset of the traffic.

• Internet Peering is typically settlement-free, with each side deriving about the same value from the reciprocal arrangement. If either party perceives that the benefit derived from peering is asymmetric, one party or the other may deny peering or suggest an alternative paid arrangement.

The Top Five Motivations to Peer

Discussions with the peering coordinators highlighted several dominant motivations for Internet Peering:

1. Transit costs are reduced. Internet Transit is often a large component of the cost of operating an Internet service. Peering provides a more direct traffic path between the parties while simultaneously reducing the load on these expensive transit services (as shown in Figure 4-4). If the cost of exchanging traffic in peering relationships is less than the cost of sending that traffic through a transit service, then peering can be proven to be a financially rational decision.

Figure 4-4. Internet Peering bypasses metered Internet Transit.

We will work through a detailed mathematical calculation of this reduction of transit costs in “The Business Case for Peering” chapter.

2. End-user experience is better. By interconnecting directly with peers, the ISP’s customers experience lower latency to the other entity’s customers. Transit usually provides a more circuitous path than peering – a path through potentially many networks. (It is not uncommon to find more than 30 router hops from eyeballs to content.)

Notes from the field.

Without Peering, Packets Travel the Ocean. Twice!

In the late 1990s, traffic between the United Arab Emirates and Saudi Arabia traversed two international ISPs and across an overloaded exchange point in Washington, D.C. By peering this traffic in the Middle East instead, these ISPs reduced their transit costs and improved the performance for both customer bases.

3. Control over routing is strategic. The performance of applications like gaming and video require special attention because of the adverse affect poor performance has on the application. For example, the Hollywood studios that I spoke with said that the “suspension of disbelief” is destroyed with just one or two “artifacts” such as pixilation, audio garbling, or freeze frames. For some of these companies, monitoring and managing the network traffic is too important to casually select the cheapest way to deliver video to the Internet.

Notes from the field.

End-User Experience Key Driver for Content Provider Peering

It is interesting to note that the content providers that started peering all said that the end-user experience was the primary driver for peering, and that control over routing was the means to accomplish a good end-user experience.

Yahoo! for example said that when it sees packet loss on a flow it adjusts its routing to route around the troubled path. The company could not do this if it simply allowed its upstream ISP to handle its traffic. For Yahoo! and most other content providers that are peering, the end-user experience – and therefore control over their traffic – is strategic.

4. Traffic billing is usage-based. Most ISPs charge customers based upon how much traffic they send or receive. Since packet loss and latency severely restrict traffic consumption, these ISPs strive for the lowest-latency, lowest-packet-loss Internet Transit service for their customers. It is in their best interest to ensure that customers use as much bandwidth as possible. Minimizing loss and latency through effective traffic engineering helps them make more money.

Notes from the field.

ISPs Make More Money by Peering

Dave Rand first highlighted this effect, and conversations with European ISPs showed increasing adoption of the usage-based billing model.

5. ISPs enjoy marketing benefits. ISPs market their extensive peering capabilities in their marketing literature. They market their well-peered backbone, their shorter paths, lower latency, etc. Wouldn’t you buy from an ISP that cares enough about performance to peer its traffic on a large scale?

Notes from the field.

Hurricane Electric Markets Broad Peering

A large-scale ISP and hosting company called Hurricane Electric highlights all of the interconnect points across the globe in its marketing literature. The marketing message is that the company takes responsibility for traffic all the way to the access networks, and that it leverages peering to provide better performance for customers.

Now let’s talk about the process of peering.

The Internet Peering Process

Internet Peering has essentially three phases:

• Phase 1 – Identification of Peering Target

• Phase 2 – Contact, Qualification, and Peering Negotiation

• Phase 3 – Peering Implementation

We will examine each peering negotiation phase in turn. This section speaks from the perspective of an ISP exploring whether peering makes sense. Most ISPs go through these processes, and later on we will see that they apply to content providers as well.

Phase 1 – Identification of Peering Target

The first process is identifying the set of potential peering targets. ISPs determine where their traffic is going to and coming from. They are looking for peering partners that should be similarly incentivized to directly interconnect their network, to save money, to improve performance, etc., as shown in Figure 4-5.

Figure 4-5. Traffic analysis helps identify potential peers.

For example, in the figure, ISP A would look at its traffic and determine that a large volume of traffic goes to or comes from ISP B. Since both ISP A and ISP B are paying transit fees for that traffic exchange, it might be in both parties’ interest to “peer” with each other.

This traffic analysis is done with NetFlow or other traffic management software such as PeakFlow provided by Arbor Networks. The result of this analysis is a top 50 list of networks sorted by traffic volume. When sorted, a typical transit traffic histogram looks something like the graph shown in Figure 4-6.

Figure 4-6. Traffic distribution follows a power law.

The bad news is that the top traffic sources and sinks are typically the larger ISPs that do not peer openly. These top destinations represent a large chunk of traffic that is ultimately not peerable; if you are a typical small or medium-sized ISP, these ISPs simply will not peer with you (we will discuss why later). The remainder of the traffic, however, is potentially “peerable.”

Once the top traffic destinations are identified and associated with specific ISPs, these ISPs are targeted for potential peering relationship discussions. Figure 4-7 shows a sample “Peering Top 50 list” template – peering coordinators use some form of this template to help them systematically track their interactions and progress with new potential peers.

Figure 4-7. Worksheet for prioritizing and tracking peering discussions.

These target peers are ordered by traffic volume and sometimes color-coded to reflect the likelihood of obtaining peering.

Phase 2 – Contact, Qualification, and Peering Negotiation

ISPs typically have a person or group specifically tasked with peering and traffic engineering issues. For example, Verizon has a “Peering Steering Committee” to evaluate peering requests, a structure now common in many of the larger ISPs.

Interviews I have had with ISPs have highlighted a key challenge: Finding the right person to speak with at the target ISP is more difficult than you might expect; in fact, it is a time-intensive process.

Peering coordinators change jobs, and there is no single standard way to find out who handles this task. Even if the name is known, peering coordinators are often traveling, way behind in e-mail, and prioritizing e-mail based on the subject or the sender. In this area, “people networking” helps a great deal, and hiring experts for their contacts speeds this initial contact process up significantly.

Next we will enumerate the most effective ways to put target ISPs in context.

The top 10 ways seasoned peering coordinators contact target ISPs follow:

1. Face-to-face at informal meetings in an “open” Internet Operations forums like NANOG, IETF, RIPE, APNIC, AFNOG, etc. (Here “open” means anyone can attend.)

2. Face-to-face at “selective” commercially sponsored peering forums like the Global Peering Forum (Here “selective” means there are some prerequisites to attend, like being a customer of a GPF IXP sponsor.)

3. Face-to-face at “selective” IXP Member Meetings like DE-CIX, LINX, or AMS-IX member meetings

4. Via introductions made through an IXP Chief Technical Liaison or a peer who knows the right contact

5. Via electronic mail, using the pseudo standard peering@ispdomain.net or a personal contact

6. From contacts listed on an exchange point participant list, or peeringdb registrations

7. With tech-c or admin-c from DNS or ASN registries

8. Through a search for “peering contact $ASN peering”

9. From the target ISP sales force, at a trade show or as part of the sales process

10. From the target ISP NOC

Notes from the field.

Tier 1 ISP Operations Meetings

There are also “restrictive” operations meetings, such as the Tier 1 ISP operations and interconnection meetings. These meetings are not advertised anywhere since the Tier 1 ISPs all know each other, and the sole focus is on planning for interconnection growth among those Tier 1 ISPs who are already peering. I was invited to one of these meetings to present my first white paper, “Interconnection Strategies for ISPs,” which documented the math that justified building into an IXP with dark fiber to decrease the cost of interconnection. I don’t include these meetings in the list above, since they would not be an effective avenue for acquiring peering.

A short aside about e-mailing peering requests:

Peering request e-mails are among the easiest e-mails to ignore. Many ISPs pass peering e-mail duties around on a weekly basis, with many engineers viewing the peering activity as “not real engineering.” Some sneeringly view peering work as secretarial in nature. Therefore, e-mail messages that are poorly written or missing information are much less likely to receive a reply. It is worth your time to write the peering request carefully and completely.

Peering requests need to include the relevant information. Peering request e-mail messages that fail to include the necessary information will delay or prevent peering from occurring. Including too much information is also a problem; the request may be perceived as too burdensome. Figure 4-8 illustrates the point with a real-world example of a poorly worded peering request.

Figure 4-8. Example of a poorly worded peering request.

A Better Peering Request

The template shown in Figure 4-9 is based on a sample peering template from Serge Radovcic (from Euro-IX at the time):

“I put this together a few years ago. I did this by first looking through several peering coordinators’ e-mails that they had passed on to me, and after putting together a first draft I sent it back to three or four coordinators and I ended up with the draft I sent you.”

Figure 4-9. A template for a better peering request.

Serge goes on to say, “My original idea was based on a conversation that we had a few years ago where we would ask some peering coordinators from different countries to translate this letter and we would then automate the languages to what the Requestor and Requestee needed.”

The full template is available at: http://scripts.drpeering.net/pr3.html

Peering and Interconnection Contracts and Negotiations

Next, if a discussion is scheduled, mutual nondisclosure agreements (NDAs) may be negotiated and signed, and a discussion of Peering Policy and prerequisites then follows.

Note that NDAs are an optional step, and many ISPs do not require signed NDAs prior to discussions.

Traffic engineering discussions and data disclosure may be needed to justify the peering relationship. Each ISP typically has a set of requirements for peering. During these discussions both sides explore the peering prerequisites such as peering at some number of geographically distributed locations, or peering at specific public exchange points.

Traffic volume is often a key determinant. As a rule, the decision hinges upon whether or not there is sufficient value from peering to justify spending time and money. A Bilateral Peering Agreement (BLPA) is the legal form that details each party’s understanding of acceptable behavior, and it legally defines the arms-length (loosely coupled) relationship that each side agrees to.

In some cases the effect on the business is also examined. Might this “peer” be a customer? How will peering impact the network?

After this initial discussion, either party may decide to walk away from the peering discussions until certain criteria are met. If both parties agree that their requirements are sufficiently met, they discuss interconnect methods as described next.

Phase 3 – Implementation of Peering Session

Regional interconnection has two dominant forms: The Direct-Circuit Peering model and the Internet Exchange Point peering model.

The Direct-Circuit Peering Model

Definition: Direct-Circuit Peering is peering using a point-to-point circuit.

This peering model doesn’t require additional Points of Presence (POPs), nor does it require the deployment of additional hardware; the two peers simply purchase a circuit between their existing POPs. If there is a failure in the interconnection, only three parties are involved: the two ISPs and the circuit provider.

The Direct-Circuit Peering model has two dominant uses:

First, Metro-area Direct-Circuit Peering is used between two parties that seek interconnection with only a few other parties within the metro area. If more than one interconnect region is required, the two parties often split the interconnect costs by alternating who pays for each additional regional interconnection (Figure 4-10).

Figure 4-10. Metro-area Direct-Circuit Peering.

The second place you will see the Direct-Circuit Peering model is with “transoceanic half-circuits,” interconnections between the large incumbent ISPs that seek interconnection without the foreign peer having a local presence in their home markets. This interconnection model is sometimes called “half-circuits” because each party pays for half of the cost.

Notes from the field.

Benefits of Peering < Benefits of Protecting Home Market

John Milburn brought up an excellent point about half-circuit peering. He said, “the benefits of peering are dwarfed by the desire to protect one’s home market.” By this he means that it may cost more money to build into another region and peer there. It may cost more money to pay for half of a transoceanic cable and peer in the middle of the ocean. However, the primary motivation for these ISPs is to “own” the home market and not make it easy for other ISPs to build in to their home market and compete with them. If they peer abroad, a peer could still build into their home market and offer Internet Transit services, but traffic to the incumbent’s customers would have to traverse an ocean twice before being delivered to the incumbent’s customers. This offering is not a very compelling one in a market where most traffic is between customers in the home market. If the ISP peered in its home market, then the peer could easily undercut the price and provide a reasonably low-latency service within the home market.

The Direct-Circuit Peering cost model scales linearly; the more interconnections one has, the more it costs to interconnect. Every additional peer incurs the cost of an additional point-to-point metro circuit.

The Internet Exchange Point Peering Model

The alternative to the Direct-Circuit Peering model is to peer at an Internet Exchange Point.

Definition: An Internet Exchange Point (IXP) is a place where multiple ISPs interconnect their networks together (Figure 4-11).

Potentially many peering sessions can be established across a single well-populated IXP peering fabric.

Figure 4-11. The IXP model.

Notes from the field.

Tier 1 ISPs Migrate From Metro Circuits to Carrier-Neutral Exchange Points

In the early days of the U.S. Internet Peering Ecosystem, the largest ISPs in the U.S. interconnected their networks together using direct circuits. They ordered circuits between their POPs in a region from a local carrier. These circuits were sometimes delayed up to 18 months past the due date. Traffic management for the two parties became a very tough challenge, as they had to route their peering traffic along a more circuitous, uncongested path until the new regional circuits were in place. This situation led to a large-scale build into carrier-neutral data centers and explains why the IXP model is now the method predominantly used for interconnection at the core of the Internet.

The IXP Peering Cost Model

The cost of peering at an IXP usually involves the following cost components (Figure 4-12):

• Transport fees for getting the traffic to the exchange point

• Colocation fees

• Equipment expenses

• Peering port fees on the exchange point shared fabric

Figure 4-12. The costs of the IXP model.

Definition: Transport Fees refer to the monthly recurring expenses associated with a physical/data link layer media interconnection into a peering location.

Unlike transit service, transport is not metered; it is sold as a fixed-capacity circuit that costs the same regardless of the amount of traffic exchanged over it.

Notes from the field.

Ray the Hoster

A counter example came up during the hard times of the last economy collapse. During the downturn, an entrepreneur (“Ray”) purchased a data center for pennies on the dollar. He “un-mothballed” the generators; Uninterruptible Power Supply (UPS); Heating, Venting, and Air Conditioning (HVAC) systems; security systems; etc. and was able to negotiate a metered transport pipe, something very rarely made available. He explained to Qwest that there was no traffic yet, but the gear was already in the building, and if a deal could be arranged he would start paying Qwest for his usage of the transport pipe. The deal was struck. This situation is the only time I have heard of the circuit provider allowing the customer to meter itself.

Definition: A Colocation Facility refers to a specialized data center that houses telecommunications equipment for multiple network operators.

Definition: Colocation Fees are the expenses paid to the colocation facility operator for housing the telecommunications equipment and facilitating interconnections.

Not only do colocation facilities provide the operations environment necessary for the equipment, but the better ones also make it easy and cost-effective for their population to interconnect with each other. They understand their customers’ businesses and seek to establish and grow a community of participants. A handful of these colocation centers provide much more than space, power, and cross-connects. They facilitate peering.

Definition: Equipment Fees are the amortized costs of the networking equipment used for Internet Peering.

(In modeling the Internet exchange later on, we will calculate the total cost of peering at an exchange, including equipment, and make assumptions about pricing, depreciating the deployed router gear.)

Definition: Peering Port Fees are the monthly recurring costs associated with peering across a shared peering fabric.

Together, these fees are the monthly cost of peering. These fees are typically the same monthly recurring cost regardless of the amount of traffic that is exchanged over the infrastructure.

For the cost of the interconnection, both parties can then send and receive as much traffic as can fit across the transport circuit and peering fabric.

Public and Private Peering

The interconnections at the IXP take one of two forms: Private Peering or Public Peering.

Definition: Private Peering is peering across a dedicated layer 2 circuit between exactly two parties, typically using a fiber cross-connect or a VLAN between two parties at an IXP.

Private Peering is the same as the Direct-Circuit Peering model but within a building or set of interconnected buildings. There is typically a nominal cost to Private Peering (a few hundred dollars per month for a fiber cross-connect, for example), whereas circuits are typically more expensive (a few thousand dollars per month for a 10G circuit, for example).

Notes from the field.

Pricing of Colocation Cross Connects in U.S. and Europe

In the U.S. the Private Peering cross-connects are run by the colocation provider with a nominal (~$300/month) recurring monthly charge. In Europe it is somewhat more common for ISPs to run the cross-connects themselves with perhaps a one-time fee. After tours of dozens of IXPs around the globe, I have seen some of the very messy results of decades of ad hoc cross-connect runs. Increasingly I am seeing the European colocation cross connect model migrating towards the U.S. model with recurring monthly fees for cross connects.

For a colocation center to charge ~$300/month for a fiber cross-connect that costs $40 to deploy may seem excessive. It is a large-margin product, but it is also a high-value product. This topic is discussed more in “Chapter 13 – The IXP Playbook” – you will see that this high-margin cross-connect product can motivate the IXP operator to build that population of peering customers.

Definition: Public Peering is peering across a shared fabric such as an Ethernet switch.

Public Peering is the dominant method of peering in the peering ecosystems we studied, although many support both Public and Private Peering.

A basic and important question emerges:

Question: When does it make sense to peer?

We will make the business case for peering in the next chapter, but to make this chapter independently complete we will answer this question in brief.

Answer: From a strictly financial view…

Peering makes sense when it is cheaper to send traffic to peers than through a transit provider.

A Paid Peering Aside

Some companies offer a derivative service called Paid Peering.

Definition: A Paid Peering relationship is a peering relationship with an exchange of compensation from one party to the other (Figure 4-13).

Figure 4-13. The graphical notation for Paid Peering.

The compensation may take the metered form of $/Mbps. In other cases, one side might cover more of the peering costs than the other.

When does peering become Paid Peering?

My litmus test: If the peering is not a settlement-free and no-strings-attached peering relationship, then it is Paid Peering.

Notes from the field.

Paid Peering Examples

Here are a few examples of Paid Peering observed in the field:

Paid Peering for Cash. Comcast offers a Paid Peering service for those who do not meet their settlement-free Peering Policy. The metered rate is rumored to be in the $2-$4/Mbps price range, approximately the same as the market price of transit. Why would you pay the market transit price when you receive only the Comcast routes? Peering generally provides a better performing path to the eyeballs, and if it costs the same and performs better, why wouldn’t you prefer to buy Paid Peering?

Paid Peering via Barter. PSINet in the early 1990s freely peered, but you had to meet PSINet at its data center; you paid for all of the costs of peering, including building into its location. The fact that there was an asymmetric allocation of the costs of peering makes this arrangement Paid Peering by my litmus test.

Paid Peering via Bundling/Barter. Both AOL and Comcast have worked out complex business relationships with U.S. Tier 1 ISPs in which other services such as fiber, colocation, large-volume transit relationships, etc. were contingent upon the inclusion of settlement-free peering. This peering is not true settlement-free peering since there appears to be some form of broader business arrangement (strings attached) involved in the transaction.

Note that Paid Peering can in fact be created using transit by filtering route announcements and being selective in the routes that you accept. Since there is no indication in the global Internet routing table for “paid routes” and “free peering” routes, most people would not be able to infer the relationship from the routing tools. At the same time, from a practical perspective, the ISP community all knows who is peering and who is most likely paying. There is a long tradition of bar stories between people who are paid to talk with each other in order to keep current with this information.

Peering Workshop Practice Questions

Here are a few practice questions from the Internet Peering Workshop:

1. “I am ISP A. If I peer with B and C, won’t B and C start sending their traffic to each other through my network?”

2. After delivering a speech, “Why Telstra won’t peer,” I was approached by an Australian policy maker who suggested that peering sounded a lot like bartering; that things of similar value (routes) are exchanged without the exchange of money. He pointed out that “Barter is something we tax here in Australia.” How is Paid Peering like bartering? How is Paid Peering unlike bartering?

3. Draw the network diagram using the peering and transit notation for the following scenario: ISP A and ISP B purchase transit from ISP C, which peers with ISP D and ISP E. ISP D sells transit to ISP X and ISP E sells transit to ISP Y.

4. In #3, would ISP C likely be interested in peering with ISP A?

5. Should Paid Peering be priced the same as Internet Transit? What is the case for its being priced cheaper than transit, and what is the case for pricing it higher than the price of transit?

Answers to these questions are in the answer key in the back of the book.

Chapter 5 – The Business Case for Peering

This chapter is written primarily for a Chief Financial Officer at an ISP who asks the question: “Tell me why we should burn a port that might otherwise be generating revenue from customers.”

I will present a systematic analysis for determining if and when peering makes financial sense. This topic is one of the most popular, and the analysis I present is done by just about every entity looking to attach at the core of the Internet. This chapter is an essential one for all interested in Internet Peering.

The Cost of Peering

To illustrate the Peering vs. Transit decision, let’s enhance our previous peering model with some specific market price points collected at the DE-CIX Customer Meeting in Frankfurt, Germany, in late 2010.

Transport Cost Data Points. At the DE-CIX 2010 meeting, Edward Punt (KPN) and Sven Engelhardt (TiNet) shared pricing estimates for three categories of 10G transport circuit into the European IXP. They said that pricing tends to vary based on how close one is to the IXP:

Local: $2,000 per month for a local 10G connection into an IXP

Nearby: $4,000 per month for a 10G originating in a nearby country

Far: $6,000 per month for a Pan-European 10G circuit

At the meeting, a Slovenian ISP validated these price points. He shared with the group that the 10G circuit from Slovenia into the DE-CIX in Frankfurt was quoted as high as $7000 per month, and that from Budapest to the DE-CIX was about $6000 per month. So these numbers appear to be in the right ballpark for our analysis.

Peering with 10G Ethernet

We will first determine the unit cost of peering at 10G.

Let’s start with the assumption that the ISP is coming from far away and will purchase a 10G transport circuit into an IXP for $6,000/month.

At the IXP, let’s assume the 10G peering port costs $2,000 per month, and a powered half rack costs $1000/month, as shown in Figure 5-1.

We will add some equipment costs into the mix as well. A Cisco 6509 might amortize to about $2,000 per month.

Figure 5-1. Sample cost of peering at an IXP.

With all of these costs accounted for, we see a monthly cost of peering of $11,000 per month, as shown in Table 5-1.

Table 5-1. The cost of peering using cross-continent transport.

Calculating the Cost of Peering

Question: Given these figures, how much does it cost to peer 1Gbps over this 10Gbps infrastructure?

To answer this question, we need to normalize peering costs into the same terms as transit costs. We need to compare both peering and transit on a Megabit-per-second basis. To do this we divide the fixed monthly cost of peering by the amount of traffic we will peer across the infrastructure.

For example, since peering costs $11,000 per month, and we will peer 1Gbps (1000Mbps) over that infrastructure, we will realize a cost of $11,000/1000Mbps= $11.00 per Mbps when that 1Gbps of traffic is peered.

As shown in Table 5-2, when we peer away 2.2Gbps of our traffic, the unit cost for traffic exchange is $5/Mbps.

Table 5-2. Allocating the costs of peering across the amount of traffic peered.

Now let’s generalize the analysis.

Generalized Analysis

Four key peering metrics describe the effectiveness of peering:

• The Peering Break-Even Point answers the question, “How much traffic do I have to peer for free to save enough money to cover the cost of peering?”

• The Effective Peering Bandwidth answers the question, “What is the maximum amount of traffic I can realistically push through the peering infrastructure?”

• The Effective Peering Range answers the question, “Where is the peering sweet spot – the peering bandwidth range where peering is financially rational?”

• The Minimum Cost for Traffic Exchange answers the question, “What is the best possible unit cost for peering?”

I will walk through each of these peering vs. transit calculations in turn.

The Peering vs. Transit Economics

The Peering Break-Even Point

To determine if peering makes sense, we need to identify the point at which the cost of peering equals the cost of transit. At this point, an ISP is financially indifferent between peering and simply sending that traffic through a transit provider. To calculate this metric, we first add the costs of peering and allocate those costs across the capacity of the peering infrastructure. The result is the curve in the Peering Break-Even graph shown in Figure 5-2.

Figure 5-2 The Peering Break-Even Point.

In this graph we also plotted the cost of the next-best alternative for offloading that same traffic, the market price of Internet Transit. (We assume for simplicity that the price of transit is fixed, not tiered.)

The point at which these two lines intersect is called the Peering Break-Even Point.

Definition: The Peering Break-Even Point is the point at which the unit cost of peering exactly equals the unit price for Internet Transit.

To calculate the Peering Break-Even Point, we allocate the monthly costs of peering across the market price for transit.

PeeringBreakEvenPoint = MonthlyCostofPeering/UnitPriceofTransit

Beyond the Peering Break-Even Point (see Figure 5-2), the ISP peers more than enough traffic to completely cover the cost of peering. The monthly transit bill is lowered more than the total cost of peering.

Savings = (MbpsPeered-PeeringBreakEvenPoint)*MarketTransitPrice

To the left of the Peering Break-Even Point is an area I call “Peering Risk.” The risk here is of not peering enough traffic for free to offset the monthly cost of peering. When the unit price of transit is less than the unit cost of peering, peering does not provably make financial sense. This situation is where many of the smaller ISPs find themselves today.

The Peering Break-Even Point helps make the business case for peering objective and clear – if an ISP can peer more traffic than indicated by the Peering Break-Even Point, then the ISP should prefer to peer instead of solely purchasing transit from an upstream ISP.

Application of the Peering Break-Even Point

Let’s continue with our previous example and further assume that the market price for Internet Transit is $3.50 per Mbps.

From a strictly financial position, peering makes sense when the unit cost of peering is less than the cost of transit; that is, when more than $11,000/$3.50 = 3,143Mbps can be peered at this exchange point (Table 5-3).

Table 5-3. The break-even analysis with different transport costs into the IXP.

Remember the context – in this example, the prospective peer is building into the IXP from across a continent. The Peering Break-Even Point is lower for those who are “near” or “local” to the IXP, as shown in the other columns in the spreadsheet.

We are also assuming that the ISP is not already present in the colocation space. If the ISP already has a Point of Presence (POP) there, the business case for peering is much easier to make.

Effective Peering Bandwidth

How much traffic can I send over the peering infrastructure?

Definition: The Effective Peering Bandwidth is the maximum amount of traffic that can be safely transported (without packet loss) across peering infrastructure.

The general rule is that the Effective Peering Bandwidth is about 70% of the smallest-sized physical layer capacity in the path to the peer. This rule accounts for framing overhead, some need to burst, and the sinusoidal nature of Internet traffic. So for these sample calculations, we assume that we can maximally use about 7Gbps of the 10Gbps capacity. This assumption seems to resonate with the peering coordinator community.

Effective Peering Bandwidth = 70% * 10Gbps = 7000Mbps

So when modeling using the service, one should not expect to be able to use more than 70% of the peering capacity.

Minimum Cost for Traffic Exchange

What is the lowest cost for peering?

Definition: The Minimum Cost for Traffic Exchange is the lowest unit cost obtainable in a peering infrastructure deployment.

To calculate the lowest cost for traffic exchange, we simply divide the monthly cost of peering by the Effective Peering Bandwidth.

MinimumCostofTrafficExchange=MonthlyCostofPeering/Effective PeeringBW

In the best-case scenario we can use all of the effective peering bandwidth for our peering traffic exchanges with peers. Some would call this “running the peering interconnect hot.”

At this maximum utilization point, peering can cost about $1.57/Mbps:

MinimumCostofTrafficExchange=$11,000/7,000Mbp = $1.57/Mbps

This metric helps us identify if peering is even interesting. Will peering save enough money to be worth the effort?

Effective Peering Range

With the analysis we have done, we proved that peering makes sense financially if you can peer more traffic than the Peering Break-Even Point, and that we can peer up to the Effective Peering Bandwidth.

Definition: The Effective Peering Range is the range of peering traffic exchange that makes sense financially.

The Effective Peering Range is the “Sweet Spot,” or to use an aircraft analogy, the landing strip that a peering coordinator must land on to save the company money by peering. It is measured as a range (in Mbps) between the Peering Break-Even Point and the Effective Peering Bandwidth.

2014 Peering vs. Transit Calculations

In September 2013 I spoke with Edwin Punt and others at the European Peering Forum in Reyjavik, Iceland and assimilated the following 2014 estimates for the European market:

Table 5-4. The Cost of Peering in 2014.

Challenge: Given these 2014 price points above, and Internet Transit pricing of $1/Mbps, create the business case for peering spreadsheet highlighting the Peering Break-Even Point, the Effective Peering Bandwidth, the Minimum Cost for Traffic Exchange, and the Effective Peering Range.

Tracking Peering Effectiveness

After the business case has been approved and executed, this story suggests that the peering analysis should be on-going and adjusted based on market criteria.

Notes from the field.

Periodically Review Peering Analysis

At an early Gigabit Peering Forum, Bryan Garrett (BellSouth at the time) shared a good process tip. After the business case is made and accepted, it is a good idea to bring forward quarterly updates applying the same logic, terminology, and metrics. Peering can be a provably valuable alternative to simply purchasing transit, and it is important to track whether it remains a good strategy given the market price for transit.

Peering Workshop Practice Questions

Here are a few practice questions from the Internet Peering Workshop:

1. What is the monthly bill if you buy 1.5Gbps of transit at $2/Mbps and also peer 1Gbps at an IXP with a monthly peering cost of $5000/month?

2. You have 5Gbps of video traffic to push and your price of transit is $2/Mbps with a 2G commit. Does it make sense to build into the Michuki Internet Exchange (aka “The MIX”) if the circuit in costs $3000/month, colocation costs $1000/month, the 10Gbps peering port costs $3000/month, and you think you can peer away 2Gbps for free at the IXP?

3. You have 6Gbps of video traffic to push, and you have just started a two-year contract in which you pay transit fees of $2 per Mbps with a 2G commit. Does it make sense to build into the K-Rose Internet Exchange (aka “The KRIX”) if you can peer 2Gbps, with the only cost being port costs of $2500/month for a 10G port?

4. What is the effective peering range for question 1 if the peering infrastructure is a 10G Ethernet?

Answers to these questions are in the answer key in the back of the book.

Chapter 6 – Selecting an IXP

During my years working with the peering coordinator community, I asked them about peering, their peering processes, and in particular, what they looked for in Internet Exchange Points (IXPs). Naturally, given my role as Co-Founder and Chief Technical Liaison for Equinix, I wanted to understand when and why they used IXPs. I asked for the types of questions they asked of IXPs and I tried to understand their decision processes. After hundreds of conversations and taking lots of notes, I had documented the collective peering community view of IXPs. To the ISP and IXP community, the information in this chapter is valuable. In this chapter I enumerate the top ten selection criteria that ISPs use when choosing an IXP, along with the types of questions they said that they ask the IXPs to determine their strengths in each of these critical areas.

Introduction

The first thing to point out is that the ISPs generally have the following goals with respect to setting up peering:

• Get peering set up as soon as possible

• Minimize the cost of the interconnection and transit costs

• Maximize the benefits of a systematic approach to peering

• Execute the regional operations plan as strategy dictates

• Fulfill obligations of a larger business arrangement

The ISPs said that they selected IXPs based on the following criteria:

1. How will I get transport into the IXP? (Telecommunications access)

2. How will I get equipment into the IXP? (Ease of deployment)

3. Am I and my target peer both in the IXP already? (Current presences)

4. How easy will it be for me to operate at the IXP? (Operations support)

5. Will a competitor benefit from my participation at the IXP? (Business alignment)

6. How much will it cost to peer at the IXP? (Cost issues)

7. Will the IXP be successful? (Credibility issues)

8. Whom else can I peer with at the IXP? (Exchange population issues)

9. Does the IXP exist today? (Existing vs. new IXP?)

10. What routes will I pick up at the IXP? (Regional route strength)

Let’s expand briefly on each of these criteria by presenting the types of questions the ISPs ask during the selection process.

1 – Telecommunications Access

One of the first issues ISPs explore is how they will establish their telecommunications services in the exchange. Here are the types of questions they ask:

What carriers service the IXP? Some ISPs have volume deals with certain carriers and therefore prefer exchanges where their carrier of choice can quickly provision circuits as part of an existing agreement.

How fast can my circuits be brought into the colocation center? This question goes to the goal of fast deployment once the IXP selection is made.

How many carriers compete for my business for circuits back to my local POP? Multiple competing carriers lead to lower costs and faster deployment.

For facilities-based ISPs, what is the cost of building into the exchange (how far away are they and what obstacles present themselves)?

Do dark-fiber providers need to go through a meet-me room or can they connect directly to the ISP equipment? Every cut in the fiber causes some signal loss.

What fiber services are available from which vendors? Is there a vendor that will sell or lease dark fiber? Wavelengths? Circuits? Some ISPs have preferred transport models. These issues are shown graphically in Figure 6-1.

Figure 6-1. Telecommunications Access Issues.

Notes from the Field.

Unique Challenges Deploying Fiber Across Africa

Fiber deployment has been a challenge across some parts of Africa.

The first problem described at the African Internet Peering and Interconnect Forum (AfPIF) meeting was that the local scavengers were digging up the fiber under the false impression that the cables held copper, which could be melted down to make bracelets and sold on the side of the road. The fiber providers responded with signs indicating that the cable buried contained fiber, not copper, and digging it up was not only a crime but also would be useless for making anything. This solution failed because most of the thieves digging up the wires were illiterate.

This fiber provider then buried the fiber with a Kevlar sheath that looked different from the usual sheathing encasing the legacy copper cables. The reasoning was that the scavengers would surely learn that cutting the cable would do them no good once they could differentiate the cable types. However, the cables were being dug up at an even greater rate! The thieves learned that the Kevlar sheathing could be cut and woven into bullet-proof vests to be sold on the streets for much more money than the copper bracelts. They also learned that the Kevlar sheathing held up well during extraction when attached to the trailer hitch of a pickup and driven down the roadway. The Kevlar helped speeding the harvesting of the cable.

In response the fiber providers encased the fiber in concrete in areas where the fiber was being dug up. In some countries the local teenage street gangs would harass the workers deploying the concrete-encased fiber until they paid off the street gangs. In some cases the local police were hired to accompany the workers through dangerous stretches of deployment. The other human threat was from competitors who would sabotage telecommunications infrastructure stations.

The continent of Africa hosts massive national wildlife refuges and parks, and much of the fiber across Africa traverses these reserves and parks. In one case the fiber provider traced a fiber break to an elephant that snapped the fiber along a relatively shallow ditch fiber run. The workers are supposed to bury one meter deep but occasionally they would hit rock. If they could not bury one meter deep, the workers are supposed to bury the fiber in concrete, but they failed to do so in this case. Deploying repair technicians into the national parks in the middle of the night was not allowed for concern over being attacked by lions and other nocturnal predators. This fiber outage lasted days.

Every Internet Peering Ecosystem seems to have different set of telecommunications industry threats, and the ones described here are by far the most interesting.

Access into the IXP data center is one of the most important issues that peers explore when selecting an IXP.

2 – Deployment Process

Deploying telecommunications equipment into an IXP has proven to be difficult in some instances. Questions that highlight these issues include:

How do I get my equipment into the exchange (assuming it supports colocation)? Some IXPs used to allow only circuits to connect remotely – colocation was not supported.

Can I drop-ship equipment in or do I have to bring it with me for an install? Some equipment manufacturers allow you to ship from manufacturer directly to the colocation provider.

Will someone act as remote hands and eyes to get the equipment into the racks or do I have to do the installation myself? The installation process, called “rack and stack,” can take days or months to complete. Some ISPs deploy equipment preloaded in a rack and simply need someone to plug it in and turn it on, and perhaps help debug things when they break. Some IXPs offer and facilitate these services and some do not.

Does the exchange have sufficient space, power, air conditioning, etc. for my current and future needs? Running out of power remains a significant problem as equipment power requirements continue to increase. Many colocation providers sell racks with power equivalents, so if you need more power you have to buy more racks whether you use them or not!

In summary, when comparing exchange environments, what processes and costs are associated with deployment (shipping, installation, travel, staff time, etc.) into the IXP? The answers to these questions impact the speed of deployment and the costs of peering.

3 – ISP Current Presences

The most expedient (and inexpensive) peering arrangements are the ones made between ISPs that are already located at the same exchange (Table 6-1).

Table 6-1. ISP Current Presences Preferences.

For peers within the same exchange, cross-connects and switching fabric connections can be made within a few hours. Peering sessions can be established a few hours after that.

This fact may seem obvious, but ISPs tend to prefer to peer at IXPs where one or both ISPs are already present.

4 – Operations Support

The ISPs said that when evaluating IXPs they also consider the ongoing operations environment within the exchange. They ask questions like these:

How is the real estate protected? How is access and security handled at the facility? Some IXPs run by telephone companies allowed access only between 9am and 6pm and required escorts during the installation.

How are data center operations handled? The reliability of the infrastructure and the time to repair failures was cited as a critical issue.

Is a 24/7 NOC monitoring the switch? For those that depend on the peering fabric for the value they derive from the IXP cited, this criterion is a critical one.

How will I be notified for outages or issues related to the IXP or underlying services? ISPs can forgive the occasional failure provided proper notification and updates are made.

The rings of data center operations support services are shown in Figure 6-2. We will discuss this data center operations model in more detail in Chapter 11 - Taxonomy of Data Centers.

Figure 6-2 Internet Data Center (IDC) operations.

Beyond the basic data center operations, ISPs are interested in the interconnection services and ask questions such as the following:

Do the colocation facilities allow private network interconnections? How long will it take to add private interconnections? Fiber cross-connects in some IXPs in Japan require the approval of the ISP that owns the exchange. It may take months to process a cross-connect in these cases. Facilities in the U.S. provision cross-connects within 24 hours.

Are there requirements to connect to a central switch? At the Hong Kong Internet Exchange (HKIX) customers must connect to the switch fabric and sign a Multi-Lateral Peering Agreement (MLPA) agreeing to peer with all participants. Some get around this requirement to peer with everyone by announcing only a subset of routes to everyone.

Is there sufficient power, HVAC, capacity at the switch, space for additional racks, real-time staff support, and legal time for contract review? For a variety of reasons, it may take time to provision new racks, or there may be a long lead time for the allocation of internal resources to install new customers.

Will I have a Right-of-First-Refusal, the ability to lease adjacent available racks should others want to lease them? During times of limited supply of racks, this option is a valuable one to have. Expanding into another area of the colocation center or building into another colocation center results in a more costly and difficult-to- maintain deployment.

ISPs prefer operations-friendly exchanges over those with operations restrictions. In any case they seek to understand the processes used to maintain and grow their presence at the IXP.

5 – Business Alignment

One important nontechnical issue is business alignment.

Will using this exchange directly or indirectly support a competitor? When the NAPs were owned by carriers, there was significant concern about handing over money to competitors. Building into a competitor’s exchange point supported that competitor.

A carrier-neutral exchange (defined as an IXP that is not owned or aligned with any particular carrier, fiber provider, or ISP) provides an open, distortion-free marketplace for carrier and ISP services.

Market distortions often result when an exchange is owned by one of its participants. This ownership manifests itself as a constraint (can’t interconnect to a particular party, for example) or as a requirement (a particular carrier or ISP service is required). These business clashes effectively constrain the market for services within an exchange.

Since it is difficult and disruptive to move equipment out of an exchange, ISPs prefer a neutrally operated exchange environment that will not suffer from market distortions or limitations due to business conflicts of interest.

Notes from the field.

Why is Carrier-Neutrality Important?

Here are a few examples of business alignment clashes:

A large network-savvy content provider installed at the JPIX peered openly. The provider requested a cross-connect to another ISP in the building. The colocation provider refused since it might adversely impact its business by bypassing it. (KDDI owns the building and KDDI also sells transit.)

Another example of a business alignment issue is MAE-East. In the late 1990s, WorldCom, a large ISP at the time, owned MAE-East, a large IXP on the east cost of the U.S. Building into MAE-East supported a competitor (WorldCom) and required purchasing WorldCom circuits. Peering at MAE-East meant paying money to your competitor, supporting its peering business, and reinforcing its position as a central component of the Internet in North America.

Finally, in the days of Exodus (the company), upper management sign-off was required for any in-building cross-connects within its data center. Too many hosted companies were bypassing the Exodus network, so Exodus felt it was missing out on a transit revenue opportunity with every cross-connect run within its building.

The fact that the colocation company can see who comes and goes was also raised as a concern to some ISPs. If your competitor gets to see all of your prospective customers, that competitor is in a position to offer its services directly. These issues are the types of issues that come up in discussions on the topic.

ISPs prefer an IXP that is not owned or operated by a competitor.

6 – Cost

As discussed in Chapter 5 - The Business Case for Peering, the cost of an exchange point can affect whether peering makes sense.

Discussions along this dimension generally include the following questions:

What are the rack fees, power fees, in- and inter-building cross-connect fees, peering port fees, and installation fees?

What are the parameters surrounding these fees?

What is the cost and value of this IXP?

All else being equal, ISPs seek to minimize the peering costs, particularly the initial costs, associated with the interconnection for peering. Waiving initial (nonrecurring) fees is common in this industry.

7 – Credibility

The credibility issue is twofold.

First, credibility goes to the survivability of the exchange. Does the IXP have the financial backing to exist in the future? During the early stages of the exchange, ISPs are asked to make a leap of faith that the IXP will be valuable to them. ISPs therefore prefer an exchange that has strong financial backing and will likely survive.

Second, credibility goes to the current and future value of the exchange.

Will the IXP deliver the anticipated peering value? To explore this “peering value” notion, we first take a detour and discuss network externality.

Network externality is the economist’s term for when the value of the product or service is proportional to the number of users of the product or service. The classic example is the fax machine – the first fax machine has no value, but when two fax machines exist, they can send and receive faxes with one another. The more fax machines in the world, the greater the potential value you can derive from having one.

IXPs exhibit network externality characteristics as well. The value of an exchange point is proportional to the number and type of other potential peers at the exchange.

The primary question is:

Does this exchange operator have the financial backing and credibility to attract the peering candidates that are valuable?

Additional questions include:

Does this exchange have the peering community support?

Who in the community champions this exchange?

Is there a peering facilitator who helps me find peers?

What potential peers are building in, and why?

The answers to these questions signal the credibility and survivability of the exchange. ISPs prefer an exchange that is financially stable and likely to attract the most desirable peering candidates.

8 – Exchange Population

The Internet Service Providers said that one of the most important selection criteria is the exchange point population.

Exchange points often have a publicly accessible participants list. After all, the primary value delivered by the exchange is the peering population. Look at the Belgium Neutral Internet Exchange (BNIX) web site for a very good example of a participants list (http://www.bnix.net/participants). BNIX includes all relevant information for each peer and makes it very easy to become a member.

The traffic volume statistics are also a good source of information about the peering population. A successful exchange point is like a well-attended party. If you hear people having a good time inside, chances are that it is a good party. There are no guarantees, of course, but when a large amount of traffic is peered across a public switch fabric, other peers are deriving value there.

To some participants, perhaps more important is the question of sales potential. Are transit sales possible at the exchange? Sales potential is a particularly strong motivation. Stated more eloquently,

Cost savings are cool, but revenue generation is sexy.

All things being equal, ISPs prefer an IXP populated with a large base of potential peering partners and transit sales potential.

9 – Existing vs. Emerging Exchange

Simply put, ISPs generally prefer an existing exchange to an emerging one.

The U.S. has dozens of successful exchange points where the value derived from participating exceeds the cost of participating.

There are many emerging exchange points that may someday be successful, but there is no guarantee that the value derived will exceed the cost of participating.

ISPs prefer an existing exchange to an emerging one.

10 – Regional Route Strength

IXPs tend to be strong in some routes and less strong in other routes. For example, if you want Eastern European or Middle East routes, you might prefer to connect to the DE-CIX, a large exchange in Frankfurt, Germany. If you want to reach Western Europe and the U.S., the LINX in London and the AMS-IX in Amsterdam are particularly strong.

Some ISPs prefer to peer at exchange points with a regional coverage matching their traffic profile.

One Final Note

The ISPs tend to value the criteria differently. The most important criteria to some ISPs is cost, while others care most about strategic business alignment. Still others are most concerned about telecommunications access, deployment, operations issues, and the cost of power and the effect of latency. How these criteria are weighted tend to be different across all peers.

Peering Workshop Practice Questions

Here are a few practice questions from the Internet Peering Workshop:

1. What are some of the challenges that a carrier or ISP faces when they decide to launch their own IXP in their data center?

2. Under what circumstances might an ISP prefer a more expensive IXP?

3. Assume you are a salesperson for a brand new IXP. During a sales call, which selection criteria might you want to talk about with ISP prospects and which would you want to avoid?

Answers to these questions are in the answer key in the back of the book.

Chapter 7 – Public vs. Private Peering

In the peering coordinator community, the Public vs. Private Peering issue has generated much debate. In this chapter we will present the strongest arguments on each side of the debate.

This information was collected at “A Great Debate,” which I held at a NANOG Peering BOF. The more detailed points were collected from hallway discussions after the debate.

Introduction

When Internet Service Providers build into an IXP, they have two interconnect options. They can connect to the public switching fabric for the chance to interconnect with all of the participants connected to the switch. At some IXPs you can also interconnect privately, across a dedicated piece of copper or fiber called a “cross-connect.” Both of these methods are illustrated in Figure 7-1.

Figure 7-1. An IXP often supports both Public and Private Peering.

This chapter highlights some of the strong points of both approaches.

When comparing Public and Private Peering, we assume that both are taking place within a IXP. In particular, we assume that in both cases the cost of transport, colocation, power, etc. is identical. Only the equipment and peering services costs are considered along with the operations issues associated with each model.

We further assume that Private Peering involves interconnecting peers using Gigabit Ethernet over dedicated cross-connects. The Public Peering is accomplished by aggregating these same peering sessions across a 10G port on an IXP switch platform.

The Top 3 Reasons Public Peering Is Better Than Private Peering

Here are the strongest arguments presented from Public Peering advocates:

1. Public Peering Is Better Because of Aggregation Benefits.

a. A network can easily aggregate a large number of relatively small peering sessions across a single fixed-cost peering port, with zero incremental cost for each peer.

(Private Peering requires additional cross-connects and potentially an additional interface card for each peering session, so there are real costs associated with each incremental peering session.)

Small peering sessions often exhibit a high degree of variability in their traffic levels, making them perfect candidates for aggregation. Since not all peers peak at the same time, multiple peers can be multiplexed onto the shared peering interface card, with one peer’s peak traffic filling in the valleys of another peer’s traffic, helping make peering cost effective: “I can’t afford to dedicate a whole Gigabit Ethernet port to Private Peering with this guy, but Public Peering is a no-brainer.”

b. Public Peering ports usually have very large gradations of bandwidth: 100Mbps Ethernet upgrades to 1Gbps Ethernet, which upgrades to 10Gbps Ethernet and now 100Gbps.

With such large gradations, it is easier for smaller peers to maintain several times more capacity via Public Peering than they are currently using, reducing the likelihood of congestion due to shifting traffic patterns, bursty traffic, or uncontrolled Denial of Service (DoS) attacks. “Some peers aren’t as responsive to upgrading their peering infrastructure, nor are they of similar mind with respect to the desire for peering bandwidth headroom.” The large gradations of Public Peering bandwidth help reconcile these two issues.

2. Public Peering Administration Is Easier.

a. Public Peering is the easiest and fastest way to both turn up and turn down a peering session, since zero physical work is required. Peering is “soft configured” by the two parties on the router, and then the peering session is up. (Private Peering cross-connects must be ordered and provisioned/tested, and who pays for it has to be determined, etc.)

b. It is common for a network to set up a trial peering session to determine the amount of traffic that would be exchanged should a session be turned up. If Public Peering capacity is available, no incremental cost or extra administrative work is required to turn up a trial peer, and the information gathered may prevent choosing an incorrect Private Peering port size if the traffic is moved to a private peer later.

c. Many peering coordinators must work within a budget, and do not have decision-making authority for purchases within their company. Once the Public Peering switch port is ordered, there is no additional cost and, therefore, no additional hurdle to peering for the peering coordinator.

d. Public Peering provides financial predictability. The hardware requirements and monthly recurring costs of peering are the same every month, making planning and budgeting much easier.

e. 10G Public Peering scales large peering sessions (those greater than 1Gbps) seamlessly, while Private Peering beyond Gigabit Ethernet capacities requires Private Peering at 10G (more expensive), or link aggregation (connecting 10Gs together), which can be tricky. The same logic goes as the industry jumps gradations (today the peering port are 100G ports).

3. Customers Use Public Peering as a Selection Criterion.

a. Some corporate and enterprise customers continue to ask to see the list of the ISP’s Public Peering points.

b. Public Peering may be the only cost-effective way to peer across multiple colocation facilities.

c. Across Europe, where Public Peering across multiple collocation centers is the norm, Private Peering is often a much more expensive solution. Purchasing Private Peering circuits within a metro is potentially very expensive, while the same traffic can traverse a shared peering fabric for much less.

The Top 5 Reasons Private Peering Is Better Than Public Peering

Here are the strongest arguments that Private Peering advocates presented at the debate or shared with me in person:

1. Private Peering Sessions Are Easier to Monitor.

a. Private Peering monitoring is cheap and easy. SNMP counters can be easily collected on each peering port to monitor the utilization of the peering session resources. There is no time- or CPU-intensive Netflow data collection and processing, nor is any expensive network analysis software required to sort through routing data to determine per-peering-session traffic volume.

b. Private Peering provides greater visibility with no blind oversubscription problem. With Public Peering, the remote peer could be congesting his port with the other peering sessions, and you have no visibility into their Public Peering port utilization. Packets could be dropped because of port oversubscription, resulting in poor peering performance. Since Private Peering involves only the two parties, when the port reaches an agreed-upon utilization (say 60% utilization, for example), both parties can see that it is time to upgrade the peering session.

c. Some parties mandate migration to Private Peering should either side reach 40% utilization on its Public Peering port. Since this direction is the migration direction anyway, why not start there?

2. Private Peering Is Very Cost-Effective.

a. This conclusion is based on the fact that router ports are expensive and switch ports are cheap. Even if an expected interface card port and cross-connect costs were $800 per month and the parties were expected to send 4Gbps to each other, the traffic exchange cost would be only $800/4000Mbps=$0.50/Mbps – an attractive price compared to today’s transit market.

b. For those who exchange traffic with a few large peers, the 80-20 rule applies; the majority of peering benefits can be derived by peering with the 20% of potential peers that deliver 80% of your traffic, suggesting fewer larger peers is preferable over many smaller peers across a Public Peering fabric.

3. Private Peering Is More Reliable and Easier to Debug.

a. Private Peering involves fewer network components that could break. It should be noted that this argument weakens when the “private” peering sessions are provisioned across VLANs, through optical interconnects, telco provisioned SONET services, or other active electronics.

b. An architecture of Private Peering removes the variability of support processes across Public Peering IXPs. Across Europe, each IXP is different, and a NOC operator may need to understand the processes and the levels of support and debugging capabilities of the switch support staff on call at the IXP, and may even need to craft NOC scripts to navigate through the IXP operations tasks. A Private Peering architecture provides consistency that helps the NOC debug and fix things more rapidly.

c. One Public Peering fear is that layer 2 fabrics could be connected through other layer 2 fabrics, perhaps without the knowledge or consent of the peer, resulting in a very difficult debugging and diagnostics situation if a peering failure occurs.

4. Private Peering Sessions Are More Secure.

a. A Private Peering network that is directly connected only with those with whom there is an explicit peering arrangement is more secure than a network that connects to a Public Peering fabric that includes participants with whom there is no relationship with the company. There is some history here; early exchange points were places where “traffic stealing” was accomplished by pointing default at unsuspecting and poorly secured public peers. Other problems included peers tunneling traffic across the ocean across a peer’s network. These things are disallowed in most peering and IXP terms and conditions, and can be further secured through filtering, but they are still seen as potential hazards reduced by privately peering.

b. An architecture that solely privately peers is less likely to be compromised. With a switch or other active electronics in between peers, there is the possibility that traffic can be mirrored and captured at the peering point without detection. It is relatively easy to mirror a Public Peering port as compared with tapping into Private Peering fiber cross-connects without the detection of the peers involved.

5. Private Peering Inclination Signals a More Attractive Peer.

a. The “Big Players” privately peer with each other, and some even loath Public Peering fabrics for historical reasons. Adopting this attitude puts one in the company of the largest Tier 1 ISPs in the world. “For certain very large networks, Public Peering makes no sense at all. For certain very small networks, Public Peering may make perfect sense.” Or put more harshly, “if you think that Public Peering is a good idea, you’re just not large enough yet.”

Hybrid Approach (Public + Private Peering)

A combination of Public and Private Peering is common, where ISPs peer publicly and “peel off” peering sessions to Private Peering as the volume of traffic to and from those peers increases.

One ISP I spoke with primarily uses Private Peering but does maintain Public Peering for reserve and emergency interconnect capacity. The ability to scale Public Peering quickly and seamlessly was seen as a key attribute here. History has shown that traffic generally grows, incrementally or sporadically, from emergencies and spot events.

Vijay Gill shared his belief that ISPs go through a Peering Policy lifecycle as shown in Figure 7-2.

Figure 7-2. The Peering Policy Lifecycle.

When the networks first enter the peering ecosystem, they tend to be more open and willing to peer with anyone that can help decrease their transit expenses. During this time, Public Peering is preferred as a means to aggregate large numbers of small peering sessions.

As networks grow they tend to implement a hybrid of public and Private Peering, and adopt a more selective Peering Policy.

The largest ISPs in every Internet region studied tend to migrate exclusively to Private Peering and are very restrictive with respect to whom they are willing to peer with. Since there are few peers, dedicated Private Peering ports scale very well.

We will discuss peering policies and the broader Internet Peering Ecosystem dynamics next.

Peering Workshop Practice Questions

Here are a few practice questions from the Internet Peering Workshop:

1. Based solely on the interconnect costs of peering (cross-connects at $300/month) and the cost of Public Peering ports (a 10G port at $1400/month), how much peered traffic would make an ISP indifferent between the two?

2. Why might a large ISP prefer Private Peering?

3. Why might a small ISP prefer Public Peering?

Answers to these questions are in the answer key in the back of the book.

Section III – The Global Internet Peering Ecosystem

In this section we will explore the Internet operations context – the environment, the participants, and the dynamics at the core of the Internet.

We will start by quickly examining how we got here. We will learn about the first peering and how the government-funded NSFNET backbone evolved into the commercial model of today.

The Global Internet Peering Ecosystem model is introduced as a collection of Internet Regions, each with its own Internet Peering Ecosystem. We spotlight the categories of participants, their positions, and corresponding motivations. You will learn why their behavior is predictable in the ecosystem.

The section then brings us to the present day by walking through the addition of several key players in the ecosystem. These players embrace peering on a massive scale, creating a “fat middle” in the ecosystem. We finish with the emerging access power peering dynamic. This section contains the following chapters:

Chapter 8 – The 20th Century Internet Peering Ecosystem

Chapter 9 – The Global Internet Peering Ecosystem

Chapter 10 – The 21st Century Internet Peering Ecosystem

Here are examples of ad hoc interfloor wiring within legacy European colocation centers. Anyone could run their own wiring in these buildings.When telecom firms go out of business, their space is re-leased to someone else, but their cables remain in place, leading to the raised floor tile mess shown to the left. A better (but openly accessible) interfloor riser is shown on the top and bottom.

Photos taken by author.

Chapter 8 – The 20th Century Internet Peering Ecosystem

In this chapter we travel back in time to briefly examine the origin of Internet Peering and the transition to the commercial Internet.

The First Peering

Lyman Chapin traced the first peering session back to the early 1980s when two distinct government-funded networks wished to interconnect but nontechnical issues presented obstacles. The Advanced Research Project Agency Network (ARPAnet) and the Computer Science Network (CSNet) were operated by different organizations that had their own administrative structure and policies for who could connect to their network. It was easy to connect to organizations like you but difficult to connect different organizations:

“This disconnect persisted as both sides assumed that any agreement to exchange traffic would necessarily involve the settlement of administrative, financial, contractual, and a host of other issues, the bureaucratic complexity of which daunted even the most fervent advocates of interconnection – until the CSNet managers came up with the idea that we now call “peering,” or interconnection without explicit accounting or settlement.”

[Source: http://www.interisle.net/sub/ISP Interconnection.pdf ]

Peering was invented as an “interconnection without explicit contracts.” The motivation was to interconnect using a model without administrative complexity to interconnect networks as simply as possible.

Internet Peering can also be described as grudging interdependence. Inherent in the model is mutual self-interest balanced with an underlying tension of mistrust. Both networks are operated by two autonomous organizations, so neither knows what the other is doing. The networks mutually benefit from access to each other’s networks using protocols that communicate only the reachability information necessary to provide connectivity. Internet Peering from the earliest days accomplishes interconnection as an arms-length distance, no-contract relationship.

NSFNET-Era Internet Peering, 1987-1994

The National Science Foundation (NSF) created the NSFNET backbone network to interconnect the NSF-funded regional networks across the U.S., allowing researchers at NSF-funded institutes to share data and network resources. Merit Network was awarded the contract to deploy and operate the NSFNET backbone. I worked for Merit during this period and was fortunate enough to get to work on the NSFNET.

The original 13 regional networks connected directly to the NSFNET core backbone as shown in Figure 8-1. The network was later expanded to connect 17 regional networks, with a couple of additional spur attachments added in Europe and Asia. These additional network interconnections were added only when the U.S. government-funded NSF authorized and paid for incremental connections to the NSFNET. Much like a planned economy, the funding agency (NSF) determined which networks were allowed to connect to the backbone – and under what terms.

Figure 8-1. The NSFNET as the core of the Internet (1987-1994 era).

As part of the NSFNET project, Merit Network pulled the Internet operations community together with periodic “Regional Techs Meetings.” These meetings provided a two-way dialog. We shared our NSFNET operations reports and the regional network representatives shared what they were seeing from their vantage point. The community was open to anyone, and we all shared information freely.

NSFNET Transition Ecosystem, 1992-1996

Around 1992 the NSF determined that the operation of the core of the Internet could and should be operated by the private sector. The NSF worked with the Internet community to craft a NSFNET Transition Plan, an evolutionary approach to migrate from the government-funded backbone model to a private sector backbone composed of a set of competing commercial backbones. This way, the regional networks could then choose from these providers in a competitive bidding process.

This peering ecosystem consisted of three types of operators that collectively replaced the NSFNET Internet core.

• Network Service Providers (NSPs) replaced the NSFNET backbone.

• Network Access Points (NAPs) carried traffic between the NSPs at specific places across the country.

• The Routing Arbiter (RA) collected and propagated routing information at the NAPs.

The Routing Arbiter operated a routing database for the community, much like Merit had for the NSFNET backbone. The RA also operated route servers to ensure reachability information would be propagated between and across peers at the NAPs.

With this model, regional networks were empowered to use their NSF funds to purchase access to the Internet from an NSP that interconnected its networks at three of the four “priority NAPs.”

The Early Commercial Internet Model

The NSF awarded the designation of “priority NAP” to Sprint, PacBell, AADS, and eventually MAE-East. Merit Network won the Routing Arbiter contract, and a handful of NSPs (internetMCI, SprintLink, etc.) were selected by the regional networks for their Internet connectivity.

The transition from the NSFNET to the commercial model started around 1994, and it took longer than anyone anticipated. For the cautious (and cheap), it was more comfortable to delay the transition and stay on the NSFNET as long as possible. By mid-1995 though, all regional networks had migrated to the commercial Internet (Figure 8-2).

Figure 8-2. The post-NSFNET model Internet core (1996-2000).

The Regional Techs Meetings were renamed the North American Network Operators Group (NANOG), and Mark Knopper and Elise Gerich developed a charter. Part of my role was to develop a business plan to make the NANOG meetings self-sustaining (not requiring any NSF funds) and reoriented to serve the commercial Internet. I developed a business plan and created the role of NANOG “chair,” coordinating a team to manage the meeting logistics and agenda (speakers and topics) and introduce the speakers at the event, etc. I wasn’t sure if anyone would come since there was a registration fee ($175), but sure enough they did and NANOG grew. I chaired the NANOG meetings from 1995 to 1998.

Post-NSFNET Internet in the Growing-Pains Era, 1997-1998

It was during this time that I first noticed that it was much more difficult to find speakers who would agree to speak at NANOG meetings. No speakers would share that the commercial Internet had problems during this transition – and certainly not with the part that they managed. My old regional techs friends who went to work for these commercial companies apologized but shared that their company had business interests here, and that anything bad might be used against them in the marketplace. They instead brought marketing messages that they were supposed to communicate, and any other message would dilute the important marketing messages.

The context of the commercial Internet put at odds the community desire to understand this new, evolving commercial Internet and the corporate messaging interests. This conflict continues today. It is very difficult to get operators to speak about operations issues, lessons learned, or cooperative solutions.

New interconnections went from being regular announcements at the Regional Techs Meeting to being a private matter between two parties. Commercial ISPs were not interested in letting anyone know who attached and at what capacity. Sharing this information would make it easier for competitors to gauge them and perhaps “cherry pick” the largest customers. All interconnections activities became proprietary – not to be discussed on stage. At the same time, peering was certainly a heated topic discussed with vigor in between formal talks and during socials.

By 1997, the commercial Internet was growing exponentially, chaotically, and organically (Figure 8-3).

Figure 8-3. The core of the Internet grows organically.

The NAPs became a problem. The fact that the NAPs were operated by competitors (Sprint, PacBell, AADS, MFS/WorldCom) was a powerful motivator for the NSPs to move out of the NAPs. Further, the NAPs had some scaling issues – some of them suffered from serious packet loss (Figure 8-4). Finally, the largest ISPs wanted to peer only with the other large ISPs and had little interest in peering for free with the masses of ISPs that populated the NAPs.

Figure 8-4. The post-NSFNET commercial Internet model suffers growing pains.

For these reasons, many of the larger backbones migrated their interconnections from the NAPs to private point-to-point circuits in multiple geographies across the U.S. The commercial model included a separate interconnect regime for the large backbone ISPs that peered with each other, and the masses of smaller ISPs that peered openly and publicly. The significance of the notion of NSF-blessed “NSPs and NAPs” went away as the NSF regional networks became an increasingly insignificant part of the Internet landscape.

The Internet continued to grow exponentially. More Internet Service Providers entered the market and grew as resellers of bandwidth from these larger ISPs at the “core” of the Internet. Around this time, large-scale content companies like Yahoo! had enough traffic to dwarf the traffic loads seen during the NSFNET days.

Carrier-Neutral IXPs Replace NAPs

In early 1999, I spent a lot of time with the Tier 1 ISPs trying to understand why they pulled out of the NAPs, what pain points they faced with the point-to-point model, and under what circumstance IXPs would make sense.

What I learned was that point-to-point circuits were being delivered 18 months late! Internet traffic at the time was doubling every year, and upgrades were planned and ordered well in advance. Still, the local telephone companies could not deliver the needed capacity on a timely basis, presenting a significant problem.

To accommodate the delay in interconnect bandwidth, the largest ISPs had to take great pains to route traffic to avoid congesting their interconnect circuits.

The other issue with circuits was the cost. The point-to-point circuits were very expensive, and they scaled linearly.

In my discussions with people from these ISPs, I discovered that they all experienced the same pain points. In a white paper called “Interconnection Strategies for ISPs,” I documented how Private Peering at an Internet exchange was more economical than point-to-point circuits if these large ISPs could interconnect with at least five others. Further, these interconnects could be run in less than 24 hours. As long as five of them agreed to install, the cost to each would be less than they paid for point-to-point circuits. The Tier 1 ISPs agreed with my analysis and moved into peering at carrier-neutral IXPs across the U.S. Carrier-neutral IXPs thus became the dominant model for peering in the U.S.

Summary

In the 20th century the Internet went through several significant transitions. The Internet model evolved from the government-funded NSFNET to the transitional commercial model, and then from peering at carrier NAPs to peering at carrier-neutral exchange points.

We will next examine more closely the next stage of the transition of the modern Internet – the morphing of the U.S.-centric Internet by organic growth into the Global Internet Peering Ecosystem.

Peering Workshop Practice Questions

Here are a few practice questions from the Internet Peering Workshop:

1. Attachment information was shared openly in the NSFNET days. Why might network attachments be considered proprietary information in the commercial Internet?

2. If the NSPs peer openly with everyone, does that remove their value to the market?

3. What motivated the large ISPs to pull out of the NAPs?

A. Congestion problems at the NAPs

B. NAPS were run by competitors

C. The NSF carried little weight in the commercial Internet

D. all of the above

Answers to these questions are in the answer key in the back of the book.

Chapter 9 – The Global Internet Peering Ecosystem

In this chapter we model the Internet as a Global Internet Peering Ecosystem consisting of Internet Regions, each of which contains an Internet Peering Ecosystem. The categories of participants in each ecosystem are identical across the globe. Their corresponding power positions dictate their motivations and drive their behaviors.

The base model presented in this chapter exists in every Internet Region studied. The names of the players may be different, but the behavior is predictably the same.

The chapter that follows will enhance this base model by identifying additional key players that participate in the ecosystems.

The Global Internet Peering Ecosystem

While the commercial Internet has grown organically, it has developed with a consistent and identifiable structure. This structure can best be described as the Global Internet Peering Ecosystem.

Definition: The Global Internet Peering Ecosystem is the set of loosely coupled Internet Regions, each of which contains an autonomous Internet Peering Ecosystem.

Let’s walk through the Global Internet Peering Ecosystem model in Figure 9-1.

Figure 9-1. The Global Internet Peering Ecosystem model.

Definition: An Internet Region is a portion of the Internet contained within the boundaries of a country.

Each Internet Region presents a different operations context to the ISP offering services there. An Internet Region will have different:

• Internet Service Providers (ISPs)

• Internet service offerings with different pricing models

• Internet Exchange Points (IXPs)

• Transport options and price points

• Point-of-Presence options

• Business contexts

• Regulatory environments

• Interconnect regions

For example, some Internet Regions like Singapore are small enough that all traffic can be exchanged at a single IXP. Other Internet Regions like the U.S. have large amounts of traffic to distribute across a geographically vast area. Many interconnect regions are required there.

These are just a few areas where Internet Regions differ. The Internet Region abstraction allows us to compartmentalize the operational context in which ISPs need to operate if they wish to participate in the Internet Peering Ecosystem there.

The Internet Peering Ecosystem

Definition: The Internet Peering Ecosystem is the collection of Internet services providers that interconnect their networks in various business relationships within an Internet Region.

One common characteristic of every Internet Peering Ecosystem is that each has these two methods for participant interconnection:

• Internet Transit

• Internet Peering

Another common characteristic is that every Internet Peering Ecosystem has at least three categories of Internet participants: Tier 1 ISPs, Tier 2 ISPs, and Content Providers.

[Note: Since I am using the ecosystem metaphor here I will use the terms “species” and “category” interchangeably.]

Let’s look at each species in the Internet Peering Ecosystem.

The Tier 1 ISP

In each Internet Peering Ecosystem the Tier 1 ISP is at the top of the pyramid.

Definition: A Tier 1 ISP is an ISP that has access to the entire Internet Region routing table only through its settlement-free peering relationships.

The term “Tier 1 ISP” has been an overloaded and misused term in the industry. The term has been used to communicate everything from scope (“global”) to quality (“top-notch”).

In this book, however, the term “Tier 1 ISP” has an objective, functional definition, which includes a litmus test that can be used to determine a participant’s position, corresponding motivation, and expected behavior.

The Tier 1 ISP litmus test is whether an ISP pays to reach any destination within a particular Internet Region. If the ISP requires anything more than its free and reciprocal peering arrangements, it is not a Tier 1 ISP in that Internet Region.

So that you may identify this species in the wild, Figure 9-2 shows a model of the Tier 1 ISP.

Figure 9-2. The Tier 1 ISP model.

The Tier 1 ISP model has three elements:

• Transit customers links

• Peering interconnections

• Backbone links to the ISP’s other core routers

We will discuss each of these interconnections in turn.

Transit Customer Links. Shown as attached lines underneath the ISP core routers, transit customers connect either directly to the core routers or, more commonly, to tributary routers that feed into the core router.

Peering Interconnections. Each interconnect link needs to have enough capacity to handle the aggregate traffic to and from the other Tier 1 ISPs.

Backbone Links. The lines on the left side of the Tier 1 ISP model represent the backbone connections linking the Tier 1 ISP’s core routers together. The core routers and links together make up the core of the Tier 1 ISP’s transit network. These links must have the capacity to carry the aggregate traffic across the core and on to the end customers.

The Tier 1 ISP Interconnection Mesh

Figure 9-3 shows the full-mesh nature of Tier 1 ISP interconnections. For simplicity, not all Tier 1 ISPs in the U.S. are included in the figure.

Figure 9-3. Tier 1 ISPs interconnect in a full mesh topology.

By definition, Tier 1 ISPs enjoy settlement-free access to all destinations within the Internet Region. Some in the peering community refer to the set of Tier 1 ISPs as the “Tier 1 Club.” This area of the Internet Region is called the “default-free zone” since the Tier 1 ISPs do not have a default route to any of the other Tier 1 ISPs for any destination within the Internet Region. They merely announce their customer routes to each other and they have access to the entire Internet Region routing table.

There are few (1–15) Tier 1 ISPs within an Internet Region and, with so few peers, they tend to prefer Private Peering.

The Tier 1 ISPs sell wholesale transit services primarily to resellers, but may also sell directly to the largest content companies.

Notes from the field.

Why the Tier 1 ISP Distinction is Important

The existence and nature of the business relationships between ISPs is typically confidential information, making it difficult for an outsider to verify that an ISP has only “free” peering relationships in an Internet Region. As a result, some would say this “Tier 1 ISP“ litmus test is almost impossible to execute. At the same time, most of the active players in the Internet Region have a pretty good idea about who is (and who is not) part of the “Tier 1 Club.” From a practical perspective, who is and who is not a Tier 1 ISP is not a tightly held secret. The “Tier 1 ISP” is an important distinction because it is directly related to the ISP’s motivations and predictable behaviors in the Internet Region. Tier 1 ISPs are not motivated to peer with anyone else to reduce the cost of transit since, by definition, Tier 1 ISPs don’t pay for transit.

Tier 1 ISP Interconnect Regions

The Tier 1 ISPs in large Internet Regions full-mesh peer with each other in multiple geographically diverse locations. In the U.S. for example, there are eight Interconnect Regions, as shown in Figure 9-4.

Figure 9-4. Tier 1 ISPs interconnect their networks in each of the eight interconnection regions.

There are several reasons for spreading the load like this. First, no single interconnect can handle all traffic between the Tier 1 ISPs. Second, customers look at network maps during purchasing, and they want to see redundancy and resilience. Third, spreading the traffic provides a lower-latency path between the ISPs’ customers. Finally, spreading the load can decrease the likelihood of network congestion and retransmissions, which further overload an already congested network. Spreading the load not only decreases the bit-miles the network has to carry the traffic, it also improves the performance of the network.

Most Internet Regions distribute enough traffic to require multiple interconnect regions.

To understand the behavior of the Tier 1 ISP in the peering ecosystem, we must first take a diversion to talk about Peering Inclinations and peering policies.

Peering Inclinations and Peering Policies

What are the Tier 1 ISPs’ motivations? To answer this question, we need to separate the Tier 1 ISP Peering Inclination from its articulated Peering Policy.

Definition: A Peering Inclination is a predisposition towards or against peering in a particular Internet Peering Ecosystem.

Definition: A Peering Policy is an articulation of the Peering Inclination; it documents and defines (for others) the prerequisites to peering.

The Peering Policy is what one might find on an ISP web site, or it may be received after signing a nondisclosure agreement. But to predict the peering behavior, it is important to understand the underlying Peering Inclination.

There are roughly four categories of Peering Inclination observed in the field, as shown in Figure 9-5.

Figure 9-5. The spectrum of Peering Inclinations.

An “Open” Peering Inclination is a willingness to peer with any other player in the ecosystem.

A “Selective” Peering Inclination is the willingness to peer with any other player, but with a few requirements. A common requirement is that there must be a minimum traffic volume for peering to be worth setting up.

A “Restrictive” Peering Inclination reflects a desire not to peer with anyone other than their existing peers.

A “No Peering” inclination reflects the desire not to peer at all. Purchasing transit is the preferred method for these players.

Peering Position and Peering Inclination Predict Behavior

We can predict peering behavior by identifying the player position and its corresponding Peering Inclination.

One might ask “Why model both a Peering Inclination and a Peering Policy? Aren’t they the same?”

Consider this common (composite) story heard in the field.

Notes from the field.

Peering Policy vs. Peering Inclination

A Tier 1 ISP has a posted Peering Policy indicating that it will peer with ISPs of “similar size and scale.” The ISP may require the peer to meet it in a large number of locations, in a large number of regions around the world, and with a large traffic volume to peer with a balanced in:out ratio. Many Tier 2 ISPs have expressed frustration because as soon as they met these stated requirements, the Tier 1 ISP immediately adjusted the requirements upwards – just out of their reach.

This situation demonstrates the more important underlying motivation – the Tier 1 ISP inclination not to peer. The Peering Policy is to “restrict.” There is a big difference between the stated policy and the inclination.

Stated most eloquently by Waqar Khan (Qwest, a Tier 1 ISP in the U.S.),

“We have all the peering we need.”

And he is correct. Almost all Tier 1 ISPs across the globe we studied exhibited the same motivations and exhibit the same behavior – they simply do not need, nor do they want, any more peering. In the field they call it a Restrictive Peering Policy, but everyone knows that means that they are not interested in peering with anyone else in their Internet Region.

Definition: A Restrictive Peering Policy is an articulation of an inclination not to peer with any more entities.

Few Tier 1 ISPs will publicly admit that they have a Restrictive Peering Policy. They will argue that they have a Selective Peering Policy by these definitions because the requirements are objective and articulated, and they would claim that some have found the requirements reachable.

As demonstrated in the notes from the field, most peering policies can be updated at any time. The threat of regulation has made the Tier 1 ISPs systematically keep track of peering requests and the processing of those requests through their systems. They want to be able to demonstrate how they are consistent about the application of their publicly stated policies. Ultimately however, the decision to peer is an internal business decision. The important thing to understand is the underlying motivations. That will ultimately drive behavior, regardless of what the Peering Policy indicates.

Tier 1 ISPs see the rest of the participants in the Internet Peering Ecosystem as potential customers, and do not seek peering with them. In fact, they are averse to peering – they have all the peering that they need, and perceive additional peering as a loss of paid transit revenue.

Notes from the field.

Peering Inclination to Make It Long and Difficult

James Spenceley (Comindico) went through the process of negotiating peering with Telstra, the Tier 1 ISP in Australia and encountered the MILD peering negotiation tactic: Make It Long and Difficult. This reflects an ISP’s underlying restrictive Peering Inclination, articulated as a “Selective” Peering Policy. Ultimately, after building out massive infrastructure to meet the Telstra prerequisites, and after many rounds of negotiations, meeting rescheduling, network map sharing, and legal team replacements, Telstra adjusted its peering prerequisites upwards and Comindico no longer qualified. Comindico had wasted many millions of dollars, and it went out of business. The MILD approach is a common tactic and is included in this peering playbook as a defensive peering tactic.

To summarize, Tier 1 ISPs generally have a Restrictive Peering Inclination, which can be articulated clearly as a Restrictive Peering Policy, or sometimes less clearly as a “Selective Peering Policy.” Since they already have by definition free access to every destination within the entire Internet Region, and they forego revenue by peering, there is no incentive to accept additional peers.

In every Internet Region around the world, Tier 1 ISPs do not want to peer with anybody else. But there is always an exception to the rule, and I found it in Italy.

Notes from the field.

The Exception: A Tier 1 ISP That Peers Openly!

I found a Tier 1 ISP in an Internet Region that peered openly. Thinking this claim was a trick, I probed a little further asking (with suspicion), “how many regions are required for interconnection?” The ISP answered “One, but if the peer needs two then we will do two. We don’t want any more than two interconnects though.” I explained my research and why I was puzzled. I asked what the ISP’s motivations were.

Telecom Italia pointed to a couple of reasons. First, the company didn’t want the government regulators to become interested in regulating its peering practices. (This motivator is a common one across all of the Internet Regions I studied.) Second, Telecom Italia owns all of the copper and fiber in the ground, so it makes more money selling fiber and copper when the other ISPs in the ecosystem succeed. Enlightened thinking? Finally, this peering was only for routes received within the Italian Internet Region, so the cost to Telecom Italia to peer openly was relatively low. This case remains the only case in which we found a Tier 1 ISP peering openly – in every other case around the world the Tier 1 ISPs did not want to peer with anybody else.

Update: In June 2013, Telecom Italia sent de-peering (disconnect) notificiations to all of its peers across Italy. At the same time they introduced a new paid peering product to the market. The underlying peering inclination is that Tier 1 ISPs don’t need or want any more peering.

Notes from the field.

I am a Tier 1 ISP in My House.

Some have said that a weakness in the model is the imprecise notion of the Internet Region. They say something like, “By your definition, I am a Tier 1 ISP in my house because I can reach any destination within my home without paying anyone.” This comment is an accurate application of the definition, and their statements are correct; they are indeed entitled to all of the power and prestige associated with being the Tier 1 ISP in their home.

At the same time, an emerging trend in Asia is the notion of a “Tier 1 Club” across Asia, not restricted to a single country. In this way, the loose definition of Internet Region is helpful.

The Tier 2 ISP

Roughly speaking, all non-Tier 1 ISPs in the Internet Region are Tier 2 ISPs.

Definition: A Tier 2 ISP is an Internet Service Provider that purchases transit to reach some destination(s) within an Internet Region.

The Tier 2 ISP litmus test is as follows: If an ISP is paying to reach any destination within the Internet Region, it is a Tier 2 ISP.

The model is shown graphically in Figure 9-6.

Figure 9-6. The Tier 2 ISP model.

We see that the Tier 2 ISP model is identical to the Tier 1 ISP model with one important distinction: The Tier 2 ISP purchases transit from an upstream ISP, shown graphically as lines coming out of the top of the model.

The Tier 2 ISP Motivations and Behaviors

Tier 2 ISPs are motivated to reduce their monthly transit fees, so they tend to peer openly or selectively. As a result of this broad Peering Inclination, Tier 2 ISPs tend to prefer Public Peering. Many use Private Peering as well, particularly for the larger peers.

Tier 2 ISP Peering Policies

Tier 2 ISPs tend to have matching Peering Policies and Peering Inclinations.

Many Tier 2 ISPs have an Open Peering Policy and will peer with anyone.

Notes from the field.

Pulse Peering

Brokaw Price (Yahoo! at the time) coined the term for open peering as “pulse peering; if you have a pulse we will peer with you!”

Some ISPs are slightly more selective. They may require interconnects deployed across three time zones or require a certain traffic level and a 24/7 Network Operation Center (NOC). The underlying inclination, however, is to peer broadly.

Tier 2 ISPs are a social species. There is heterogeneity and competition across the Tier 2 ISP population, but somehow there is also great cooperation. This cooperation is not purely altruistic; it is also driven by broadly aligned interests. Tier 2 ISPs want to offload as much traffic as possible in free peering relationships. They are, therefore, also interested in encouraging others into the peering ecosystem so they can offload more traffic.

To this end, many of the active players in the ecosystem help their IXP operators evangelize Internet Peering. You will find the Tier 2 ISPs actively participating, even leading the community peering forums. This behavior is not surprising from players looking to increase the amount of peering that they have. Some in this group have been able to peer away 70% of their Internet traffic!

The Tier 2 ISP Interconnect Mesh

The Tier 2 ISPs generally interconnect with each other in one or more Interconnect Regions within an Internet Region.

Unlike Tier 1 full-mesh peering, Tier 2 ISP peering is generally a partial mesh (Figure 9-7), for several reasons. Some ISPs are regional and don’t peer in every Interconnect Region, while others are geographically distributed and require multiple points of interconnect. This setup removes some interconnection possibilities from the Tier 2 mesh. Most ISPs won’t peer with a peer’s customer, and this unwillingness to peer also removes some Tier 2 ISP interconnection possibilities. Some content-heavy ISPs have no interest in peering with other content-heavy ISPs. And so on.

The Tier 2 ISPs are disappointed when peering is denied, but they are not too upset about it. Any traffic not peered is sent through a transit provider, so peering in the Tier 2 ISP space is thought of as a routing optimization. The failure to obtain peering is merely an inability to exercise a particular optimization.

Figure 9-7. A Tier 2 ISP sparse peering mesh in an Internet Region. There are different meshes in each interconnect region.

Relationship of Tier 2 ISPs with Tier 1 ISPs. Tier 1 ISPs are seen as suppliers and also as competitors for the really big customers. Since the Tier 2 ISPs know that they will be referred to as the “middle man” in competitive situations, they tend not to mention that they purchase transit at all. As one industry expert put it, “In a competitive bid situation, everyone is a Tier 1 ISP.”

The Content Provider

The last species of player that exists in every Internet Peering Ecosystem is the Content Provider.

The basic Content Provider does not peer. A separate type of Content Provider, called a “Large-Scale Network-Savvy Content Provider,” has a strong network engineering team and does peer. We will discuss these peering content providers in the next chapter.

Definition: Content Providers are companies that operate an Internet-based service but do not sell transit within the Internet Peering Ecosystem.

The Content Provider’s core competence is the creation and managing of content and the relationships with those who use, enhance, or support the content. They are generally happy to purchase transit because operating a network is not strategic or core to their mission. These interests are shown graphically in Figure 9 8.

Figure 9-8. The Content Provider model.

Content Provider model. The Content Provider model shows that this player simply purchases transit from an upstream provider and creates content as its core business.

Content Players include companies like Neopets, Visa, Staples.com, Avon.com, Hertz, Agilent, General Electric, etc.

Relationship of Content Providers with each other. Generally speaking, content providers act as independent players, with their only interplay with the ecosystem being the purchase of Internet Transit. There is little or no reason for them to exchange traffic.

Relationship of Content Providers with the other players. Content Providers generally see ISPs as vendors, and they select vendors along widely varying parameters, but increasingly they are looking at the underlying network and peering infrastructure of ISPs.

Motivation and Behavior of Content Providers

Most Content Providers do not peer – they have a “No Peering” policy. Their focus is on their core competence, and they pay others to handle the rest. Content providers I interviewed uniformly said that the most important thing was the end-user experience, although they also are motivated to reduce their Internet Transit costs, among others.

Notes from the field.

A No-Peering Policy

I visited a content company based in Los Angeles that ran a web site that went viral, and we discussed peering. The CTO said that NTT Verio provided a perfectly good transit service and that there was no strategic need to explore Internet Peering. The company didn’t want to hire network engineers. It didn’t want the added head count, or the additional complexity. It was not the company’s core competence and it was not critical to the delivery of the company’s service. For this content company, Internet Transit was just fine. In effect, it made the strategic choice to have a No-Peering Policy.

Now we can construct the most basic form of an Internet Peering Ecosystem.

The Base Internet Peering Ecosystem

When we pull together these three species, we can see the skeleton of every Internet Peering Ecosystem in the world (Figure 9-9).

Figure 9-9. The base model Internet Peering Ecosystem.

This figure summarizes the interconnection between the categories of players. The Tier 1 ISPs within the circle at the top of the figure interconnect with a full mesh of settlement-free peering. The Tier 2 group interconnect with each other in a sparse mesh of open and selective peering relationships. While there may be some transit relationships between some of the Tier 2 ISPs, most of their transit is purchased from participants in the Tier 1 Club. The Content Providers purchase transit from the Tier 1 or Tier 2 ISPs. Recall that the “$” labels on the transit lines show which side of the relationship is getting paid.

In the next chapter we will see the morphing of the peering ecosystem into the 21st century Internet Peering Ecosystem.

Peering Workshop Practice Questions

Here are a few practice questions from the Internet Peering Workshop:

Let’s assume that there is a simple Internet Peering Ecosystem with the players interconnected as shown in Figure 9-10.

Figure 9-10. A simple Internet Peering Ecosystem.

1. What will the answer be when ISP B requests peering with ISP Y?

2. What will the answer be when Content Provider C requests peering with ISP Y?

3. What will the answer be when ISP A requests peering with ISP Y?

Answers to these questions are in the answer key in the back of the book.

Chapter 10 – The 21st Century Internet Peering Ecosystem

In this chapter we will show how the basic U.S. Internet Peering Ecosystem has morphed into the 21st century Internet Peering Ecosystem of today.

The following six evolutions demonstrate how the players react to the changes in the environment. We will first highlight the evolutions that occurred during the economic downturn in the year 2001.

Evolution #1 – The Cable Companies Peer with Each Other

Up until 2001, a company called @Home provided essentially all Internet access services to the cable companies. But in 2001, @Home unexpectedly went bankrupt.

Suddenly, the cable companies were forced to in-source broadband Internet services. The cable companies were given only 30 days’ notice to establish high-capacity transit relationships and launch their own Internet services. They entered the ecosystem as independent broadband access network providers as shown graphically in Figure 10-1.

Figure 10-1. The Cable Company model.

The cable company player model is similar to that for Tier 2 ISPs but is a little different in that the cable companies are regional and don’t tend to compete in overlapping areas, and they tend to pull enormous volumes of traffic.

When the cable companies attached to their new upstream ISPs, each cable company experienced the same thing: Internet Transit circuits were congested almost immediately. Each of the cable companies ordered upgrades to these circuits, and almost immediately the Internet Transit connections were again congested!

What was going on here?

After some analysis, the cable companies found that about 40% of all cable company traffic was peer-to-peer (file sharing) traffic. Importantly, this was traffic that was ultimately destined for other cable company networks. Peer-to-peer traffic is different than other kinds of traffic, in that it will expand to consume all available bandwidth. At that time, I had been working with the cable companies on the benefits of peering, and as a result, many of them built into multiple regional IXPs and started peering openly with each other and with the other Tier 2 ISPs in the ecosystem.

These new players and their peering relationships are shown graphically in Figure 10-2.

Figure 10-2. The cable companies emerge as independent players.

One might ask, why is there a separate species of player; isn’t the cable company model identical to the Tier 2 ISP model? Why is this evolution significant?

This evolution represented a significant change in the U.S. Internet Peering Ecosystem because:

• The volume of traffic being exchanged directly in a peering relationship, among the cable companies and between the cable companies and the Tier 2 ISPs was huge – second only to the Tier 1 ISPs’ traffic exchange volume estimates at the time. Therefore, the amount of traffic that bypassed the Tier 1 ISPs, and therefore the amount of money that peering saved the cable companies, could be measured in millions of dollars.

• In 2003, the cable companies had an Open Peering Policy. The cable companies were not generally interested in selling transit to ISPs or Content Providers, and they generally worked in non-overlapping regions, so they didn’t compete against each other. As a result, there was no reason not to peer openly.

• Peer-to-peer traffic consumed all available bandwidth. The typical ISP sinusoidal demand curve was replaced with a flatter demand curve as peer-to-peer file-sharing software now preferred the higher-bandwidth, lower-latency paths to peered cable company customers.

It was also noted that the peer-to-peer traffic volume grew immediately after the cable companies peered with each other. Peering caused the file-sharing service Kazaa to prefer to fetch files across the now peered network path. The Kazaa selection protocol at the time used latency to determine which Kazaa file sharer was “more local,” and automatically selected that file sharer. The result was that when cable companies peered with each other, there was an immediate 20% growth in Kazaa-originated peering traffic volume.

Notes from the field.

The Grandma Story.

A grandmother in Australia invited her grandkids over for the holidays. In Australia, end users are charged for Internet access on a per-megabytes-downloaded basis. Grandma just received a computer and the grandkids helped her set it up. The grandkids were excited to demonstrate their expertise, and wanted to hear the latest Britney Spears album on grandma’s new computer. So, they installed some peer-to-peer software and started the download.

The download did not complete during the visit, they never got to hear the album, and the kids forgot about it. However, the download did eventually complete, and the grandmother was not tech-savvy enough to know to shut down the connection. By default peer-to-peer software shares all files it downloads, and soon grandma’s computer became the preferred source for the Britney Spears album for the entire continent! Grandma’s bill from Telstra was thousands as opposed to the usual $25 she paid per month. The issue was compounded since the pricing point in Australia was twice as high if you were a net source of content as opposed to a net sink of traffic.

So grandma called Telstra and said its billing machine was broken. Telstra reversed the charges and told her to turn off her computer until someone could uninstall the peer-to-peer software. So grandma thanked them, hung up the phone, and turned off the monitor to the computer. Why not – that works with the TV set. The next month the bill was double; again she disputed it, but Telstra didn’t reverse the charges and the story became public, enabling me to write about it. In contrast, in the U.S. broadband is typically a flat fee, and there is no such financial disincentive to use all the bandwidth available.

Evolution #2 – Large-Scale Network-Savvy Content Providers Peer

Around the same time, a small group of Large-Scale Network-Savvy Content Providers emerged as large volume peers.

Definition: A Large-Scale Network-Savvy Content Provider (LSNSCP) is a content provider that sees networking as strategic enough to build a backbone and peer.

Most of the LSNSCPs peer at least bi-coastally, and some peer more broadly across the country. The dominant three motivations to peer are similar to the Tier 2 ISP motivations to peer:

• To improve the end-user experience. Yahoo! for example sees the end-user experience as mission-critical. The company applies considerable resources monitoring (in real-time!) its application network performance characteristics. It identifies congestion, notes the network path, and alters its interconnections so that traffic traverses an alternative network path to improve the end-user experience. Around the world, this motivation is the #1 motivation that leads Content Providers to peer.

• To reduce transit costs. Any traffic that can be sent directly to the access-heavy (also called eyeball) network is traffic that doesn’t have to go over the metered transit connection. In a related way, e-mail service providers such as MSN HotMail and Yahoo! can exchange e-mail without incurring any transit charges.

• They needed to move to a new data center anyway. Many colocation and data centers were going out of business. Given the opportunity, some content players chose a data center where they could peer as well as buy transit in an open market.

Large-scale content players have always purchased transit to deliver their content to the end users on the Internet. Therefore, their role and behavior in the Peering Ecosystem is similar to the Tier 2 ISPs with the exceptions that Content Players:

• do not sell transit

• focus on content creation, and if they do operate a network it is for exactly one customer: themselves

• have visibility into the end-to-end performance characteristics (Tier 2 ISPs see packets, while LSNSCPs can see packets and flows)

The model for the Network-Savvy Large-Scale Content Provider is shown in Figure 10-3.

Figure 10-3. The Large-Scale Network-Savvy Content Provider model.

These players still purchase transit (generally from the Tier 1 ISPs), but they supplement it by peering openly with anyone and everyone.

Examples of these Large-Scale Content Players are Yahoo!, Google, Microsoft, Amazon, Walmart.com, Apple, Electronic Arts, and Sony Online.

This evolution represents a significant change to the Peering Ecosystem because:

• The volume of traffic being diverted to peering was huge, and it represented traffic that previously was sent through and available only from the Tier 1 ISPs.

• The LSNSCPs had an Open Peering Policy, and even actively promoted peering with them so this large amount of traffic would be freely available to peers.

• These leading players paved the way for other Large Scale Content Companies and Enterprise companies to examine Peering as a cost reducing / performance improving strategy.

We can see a graphic depiction of this evolution below in Figure 10-4.

Figure 10-4. The Large-Scale Network-Savvy Content Providers peer openly.

Evolution #3 – Cable Companies Peer Directly with Large-Scale Network-Savvy Content Providers

The peering of the Network Savvy Large Scale Content Companies with the Cable Companies put the most popular content on the Internet directly onto the same network as the broadband eyeballs. The end result was a major disruption in the Internet Ecosystem, resulting in network performance improvements and significant cost savings from peering!

Notes from the field.

We Expect Customers to Peer When Rational

I spent some time speaking with the Tier 1 ISPs at the time, asking about the massive migration of their customer traffic. Their response was that they expected it would happen. “We expect our customers will do that which is rational.”

And the middle of the Internet Peering Ecosystem got a little bit fatter as shown in Figure 10-5.

Figure 10-5. Content peers with eyeballs. The Tier 2 ISPs and the cable companies peer with the content companies that entered the peering ecosystem.

This represents a significant dynamic shift in the peering ecosystem since the Tier 1 ISPs are being cut out of a high volume traffic exchange loop. The hierarchy of the Basic Internet Peering Ecosystem has evolved into a flatter mesh, with the Tier 1 transit providers required only as the route of last resort. Traffic destined to locations too far away or too expensive to reach will ultimately need the services of an international transit provider. At this stage, the Tier 1 transit providers have lost their grip on the U.S. Internet Peering Ecosystem.

Evolution #4 – CDNs Dominate Traffic Volume

We have learned that Internet traffic volume is a key determinant as to whether peering makes sense financially. Video represents a proportionately larger amount of traffic when compared against e-mail, web browsing, and other traditional request-response messages. Amplifying the importance of video traffic is the fact that not only is it a larger amount of traffic, but it is also an increasingly popular type of traffic across the Internet. As it turns out, Content Delivery Networks (CDNs) are really good at distributing video to the edge for offloading to the last mile consumers.

Definition: A Content Delivery Network (CDN) is an entity that distributes web objects and network services as close as possible to the access network customers.

The CDN player (Figure 10-6) is similar to the Tier 2 ISP but instead of handling a steady stream of content, they distribute any “web objects” to caches at the edge. Today the service is priced the same as Internet Transit, so for modeling purposes we will make the simplifying assumption that the CDN is essentially providing transit - just a much better performing one due to the caching at their edge.

Figure 10-6. The Content Delivery Network model.

Notes from the field.

CDNs Pushing Massive Traffic volumes

Around 2006, YouTube was starting to gain popularity and started using CDN services for the most popular most viral content. The community started noticing the CDN guys were deploying massive peering gear. At the time, most were peering at 1Gbps, and perhaps an occasional 10Gbps connection. I was traveling the peering speaking circuit with the CDN guys who were deploying 10G and occasionally multiple 10G connections to the peering infrastructure. To some CDNs, videos are just another web object to be distributed at the edge of their network. These objects however are very large, so the wakes set off by movement by the CDNs were starting to be felt in a big way. The CDNs, as very large volume open peers, quickly became significant players in the Internet Peering Ecosystem.

Evolution #5 – The Video Internet Is Activated

It may have been YouTube that activated the video revolution around 2006, but there was an entire underlying Video Internet Ecosystem that conspired to enable it starting back as far back as 2002.

Consider one slice of Internet Video category is User-Generated Content. To enable content creation side, small easy-to-use video cameras came onto the market. Home video editing software (like iMovie) was inexpensive or bundled with the sale of new computers. Next in the supply chain, video distribution became free and easy thanks to YouTube, a service that leveraged the inexpensive transit and CDN services. These services leveraged volume pricing with ISPs that interconnected with last mile providers that upgraded the last mile infrastructure (to DOCSIS-3 for example). All of this would not have mattered if it wasn’t for the high-speed wireless plug-and-play networks deployed in every home today. Innovations along each link of the supply chain were required for end-to-end video distribution to work well. Once these services matured, Internet backbones quickly became dominated by video.

Cisco estimates that by 2013, about 80% of all Internet traffic will be video, leading me to coin the phrase, the “Video Internet.” Friends in Japan tell me the Internet in Japan is already 80% video, while others in the U.S. say conservatively 40%-50% of all U.S. traffic was video in 2011. While the term “Video Internet” is still gaining acceptance, I believe it more accurately reflects the dominant traffic type across the Internet today.

It was the innovations across the entire supply chain that activated the Video Internet, a system that is now positioned to service the video distribution needs of the planet.

Evolution #6 – Access Power Peering

The Internet Peering Ecosystem in the U.S. appears to be morphing again, this time positioning the access networks into a much stronger power position.

Definition: Access Networks (also known as “eyeball networks”) are Internet Service Providers that sell Internet access to end-users.

Access Networks include cable companies, telephone companies and wireless Internet providers. Since Internet users primarily download content, Access Network traffic is generally in-bound (towards the end-user).

[Note: We are naming Comcast explicitly when illustrating this evolution. There are other examples of this maneuver, but there is far more data available about the Comcast-Level 3 dispute. By walking through this story we can more succinctly and concretely demonstrate Access Power Peering.]

Video Dominates the Internet. As stated above, in 2011, at least 40-50% of all U.S. Internet traffic was video, according to conversations with ISPs. Separately, the movie industry people emphasized that video is unlike other applications. They said that disruptions (pixilation, freeze frames, audio garbling, etc. called “artifacts” in industry jargon) in a movie video stream cause a break in the “suspension of disbelief.” This effectively destroys a video watching experience for the end user. Video traffic dominating the Internet and the requirement for flawless video, taken together, highlight the importance of an optimized Internet Peering Ecosystem. Content distributers need to be able to deliver increasing volumes of video traffic with very high quality.

At the same time, significant change occurred in the U.S. Internet Peering Ecosystem.

Comcast Peering Policy. In the mid-2000s, Comcast put in place a Peering Policy stating that they require (among other things) that traffic volumes exchanged be roughly balanced. Traffic in the inbound direction (to Comcast customers) had to be (roughly) balanced with the outbound direction (from Comcast customers). Some in the industry have said that the specific not-to-exceed-ratio is around 2:1. This peering ratio prerequisite and the ratio value is somewhat common in the industry.

However, consider that

• Internet video is or soon will be the dominant Internet application, and

• Internet video tends to be massively asymmetric (as high as 30:1), and

• Comcast customers consist of tens of millions of eyeballs primarily pulling down content from the Internet.

Very few peering candidates have or will ever have a balanced ratio to offer to an access network. Video content delivery companies such as CDNs and content-heavy ISPs certainly won’t have balanced peering ratios with access networks. These companies are in the business of getting video objects to the eyeballs and have very little traffic to pull from Comcast. Since a large portion of Internet traffic is destined to be video traffic asymmetrically delivered to the access networks, such a Peering Policy effectively ensures that this traffic can not be freely peered.

Comcast Paid Peering

Comcast started offering a Paid Peering service for those that did not meet their peering prerequisites. Recall from Chapter 4 that Paid Peering is Internet Peering but with some exchange of compensation involved.

Comcast priced their metered paid peering service at about the market price for Internet Transit. Companies with content to send to the tens of millions of Comcast customers could send their traffic through their upstream ISP, or they could send it directly to Comcast for about the same price. At this price point, it is a good value proposition and several CDNs have gone down this path and purchased Paid Peering from Comcast. Let’s look at the affect on the dynamics in the ecosystem.

There is only one way to get to the Comcast customers – all traffic must directly or indirectly make its way through the Comcast network to reach the Comcast customers. Some in the industry call these customers “captive” customers since there is no alternative path to reach them. This captive customer base is the source of potentially great positional power in the Internet Peering Ecosystem as shown in Figure 10-7. With this Access Power Peering, all traffic destined for access network customers (and all corresponding Paid Peering revenue) flows directly to the access networks.

Figure 10-7. The Access Power Peering paradigm.

We will see some more Access Power Peering dynamics with the Comcast-Level 3-Netflix dispute discussed next.

Is Comcast Paid Peering Required?

In late 2010, Comcast was purchasing Internet Transit from several ISPs as shown in Figure 10-8.

Akamai was purchasing Paid Peering from Comcast and enjoying low-latency high-capacity access to Comcast customers. Limelight Networks, a competitor to Akamai had a choice to make. Should it continue to send its traffic through its upstream ISP to reach Comcast customers? By doing so, Limelight traffic will suffer higher latency and potentially greater packet loss than its competitor. Philosophically, Limelight feels that it shouldn’t have to pay Comcast to deliver the content that Comcast customers requested!

Figure 10-8. Alternative to peering is a more circuitous path.

Limelight’s hand was forced however when one of Comcast’s upstream networks experienced chronic congestion. When this occurred, Limelight content traffic destined to Comcast customers experienced performance problems (packet loss). Since Limelight is being paid by the content providers to distribute the content to the edge for a better quality end-user experience, its customers began to complain. The cause of the problem seemed to be the links between Comcast and Comcast’s upstream transit provider. Limelight’s upstream transit provider is powerless to fix this problem. As a result of this context, Limelight has to purchase Paid Peering from Comcast (Figure 10-9).

Figure 10-9. The more direct Paid Peering path to the eyeballs.

From a practical perspective, if you are paid to deliver video to Comcast customers, there is no choice but to purchase Paid Peering from Comcast. This is a potentially huge shift in power in the ecosystem. This power is demonstrated in the Netflix, Comcast and Level 3 story next.

The Netflix, Comcast and Level 3 Story

Setting the stage. Continuing this example, the relationship between Level 3 and Comcast is a bit more complex than simple transit. According to the ISPs familiar with the situation, Comcast entered into a broad business relationship with Level 3 that included as a condition of the agreement “free on-net” access. This is a well-known tactic discussed further in the ISP Peering Playbook chapter. With this “free on-net” or peering relationship, Comcast got free access to Level 3 customers (which happened to include most of the other cable companies) and Level 3 in return got free access to Comcast customers. This was shown in previous Figures 10-8 and 10-9 as “T+P.” The point is, this broader business relationship enabled Comcast to have settlement-free peering with Level 3, a Tier 1 ISP in the U.S. region.

Act I. Introducing Netflix

Netflix, a large-scale video distribution company, had gained considerable traction with its video delivery over the Internet and used CDN services from both Akamai and Limelight. Netflix was experiencing a near exponential growth in traffic. As shown in Figure 10-9, most of the ‘$’ symbols are on Comcast’s side, so as Netflix traffic grew, so did the Paid Peering fees paid to Comcast.

Act II. Level 3 Wins Netflix Contract

Also in late 2010, Level 3 bid for and won the Netflix video delivery business, undercutting Akamai. As a result, Netflix traffic moves from Akamai to the lower-priced CDN that Level 3 offers (Figure 10-10).

Figure 10-10. The shift of video traffic from a paid peer to a settlement-free peer.

Act III. Level 3 Requires More Peering Capacity

Level 3, since they have a “peering” relationship with Comcast, informs Comcast that there will need to be some additional interconnect capacity. This is a common thing that peers do (Figure 10-11).

Figure 10-11. Additional ports requested from Access Network.

At this point, who wins and who loses? Netflix wins a lower cost for video distribution. Akamai obviously loses the Netflix business, so the revenue from Netflix to Akamai decreases. Comcast also loses a corresponding amount of Paid Peering traffic and revenue. Level 3 wins the business and the additional revenue from Netflix. But Comcast loses again, because not only is it is giving up the Paid Peering revenue from Akamai, it is also being asked to spend money on additional peering ports to handle that same Netflix traffic over their free peering relationship with Level 3.

Act IV. Comcast Denies Free Peering Capacity

In response to Level 3’s request, Comcast refuses to add interconnect capacity and points out that Level 3 is “out of ratio” and needs to pay Comcast for Paid Peering. They argue that it wouldn’t be fair not to charge them since they charge the other CDN partners for Paid Peering. It would be an unfair competitive advantage if Level 3 (which offers a competing CDN service) didn’t pay as well.

This dispute was very public. Much like my airplane story in the beginning of the book, there are opinions across the spectrum as to who was in the right and who was wrong.

Dénouement. Level 3 Pays Comcast for Peering

To complete the story, Level 3 acquiesces and agrees to pay Comcast for Paid Peering as shown in Figure 10-12.

Figure 10-12. Level 3 acquiesces and becomes a Paid Peering customer.

This result further strengthens Comcast’s power peering position, since now even a Tier 1 ISP is paying for Paid Peering with Comcast to access its eyeballs. As stated in the beginning of this section, this Comcast story is merely a good demonstration of Access Power Peering evolution, and by no means the only example.

The Problem with Access Power Peering

[Note: I will step up onto the soapbox for a moment here.]

In this chapter, we have seen that the U.S. Internet Peering Ecosystem has morphed. The players and the relationships, the power positions and behaviors evolved with the context.

First, the Tier 1 ISPs were in charge and received a piece of the traffic that directly (from a customer) or indirectly (from a customer of a downstream customer) traversed the Internet region. In this case, the Tier 2 ISPs could peer their traffic between themselves to bypass their transit provider. The Tier 2 ISP could also select a different upstream ISP if they thought they could get a better price. There were choices, and both peering and buying transit in an open market led to cost and performance efficiencies. The Internet Peering Ecosystem evolved into a more efficient fat middle.

Figure 10-13. A parody likening Comcast’s control of access to its customers to the control exhibited by the Sopranos crime family.

The Access Power Peering scenario however is problematic in that it leverages the lack of alternative paths to the eyeballs.

Since 80% of Internet traffic is destined to be video, a large and massively asymmetric stream, the peering ratios clauses will prevent settlement-free peering from happening for 80% of the Internet traffic. Anyone with video content to send to the eyeballs will not qualify for free peering.

So you might say that there are alternatives to free peering: Internet Transit and Paid Peering. However, since 80% of the Internet traffic will be video, video delivery companies must get the content close to the eyeballs. The Internet Transit path is inferior and the video delivery competitors all pay for peering to get direct access to the eyeballs. If you are delivering video, from a practical perspective, there is no choice but to buy paid peering from the access network.

So then you might ask “What is the problem? Eyeballs have a choice – if they don’t like the performance of cable they can move to DSL and vice versa.” Performance would have to be very bad for all access customers to switch. During this time, the content companies will get complaints from the access customers, and the CDNs will hear those complaints as well. It is easier for the few content companies to purchase paid peering or purchase from a CDN that purchases paid peering than for the access customers to all pick up and move to a competitor. The context is such that the video delivery companies will give in first and pay the access networks. The problem is that all traffic and revenue in this model leads to the access networks, solely because they have the power position based on a captive customer base.

In the earlier example, Comcast charged market transit prices for Paid Peering. But what prevents them from charging a higher price? What is the alternative for CDNs whose business is to get the web objects as topologically close to the eyeballs as possible?

Is the next Internet Peering Ecosystem model really that everyone pays the access networks to deliver the 80% video Internet traffic?

From this we can recognize its position in the Internet Peering Ecosystem, identify its motivations and even predict its behavior. At this point you should have a good idea of the players, the positions, the corresponding motivations and behaviors observed within the Internet Region.

The Official Story

The official resolution to the Comcast-Level 3 dispute was, as expected in these sort of clashes, neutral and not very descriptive:

“Level 3 and Comcast have resolved their prior interconnect dispute on mutually satisfactory terms. Details will not be released.”

Peering Workshop Practice Questions

Here are a few practice questions from the Internet Peering Workshop:

1. What are the motivations for purchasing Paid Peering?

2. Who is in a strongest position to negotiate Paid Peering prices?

Answers to these questions are in the answer key in the back of the book.

Section IV – The Tricks of the Trade: The Playbooks

In this section we present the tactics used by some of the smartest peering coordinators in the world. They shared some tactics that they have used to obtain peering where they otherwise would not have been able to. We briefly present the foundation of the IXP, the colocation facility, and then finish by sharing some of the tricks of the trade that the most successful IXPs in the world have used to build, grow, maintain, and defend their “critical mass.”

This section contains the following chapters:

Chapter 11 – The ISP Peering Playbook

Chapter 12 – Taxonomy of Data Center Operations

Chapter 13 – The IXP Playbook

Artistic photo of the Netnod peering switch in Sweden.

Photo courtesy of Lars-Johan Liman.

Chapter 11 – The ISP Peering Playbook

This chapter presents twenty-seven “tricks of the trade” used by some of the most seasoned peering coordinators in the world. These tactics range from the practical to the clever – and even into the unethical. While these maneuvers are not necessarily recommended, they are described in order to exercise the lexicon and illustrate the types of manipulations that occur in the Internet Peering Ecosystems around the world. You should understand and be able to recognize each of these tactics.

Introduction

Sometimes peering is requested and agreed to straight away.

But what happens when a peering coordinator sends a peering request that results in a “No” answer? Or, more commonly, what happens when a peering request doesn’t get a response? Do peering coordinators just give up and go on to the next peering target?

As it turns out, the answer is no, seasoned peering coordinators don’t give up; they have their own “tricks of the trade” for obtaining peering where they otherwise might not have been able to.

This chapter enumerates these tactics. Most of these tactics have been used to obtain peering where they otherwise would not be able to. A handful of additional ones are primarily defensive in nature, intended to defend a power position in a region.

The Graphical Representation of the Plays

To describe each tactic we will apply the graphical notation employed throughout this book to highlight the relevant characteristics of the players and their actions.

The Players

Every play in the playbook has at least two players. The initiator ISP (the one who seeks peering) we will call “ISP A,” and the target of the peering we will call “ISP B.”

Throughout this chapter we will show the initiator as a cloud labeled “A” and the target ISP will always be a cloud labeled “B” (Figure 11-1).

Figure 11-1. The initiator and the target of the peering tactic.

If the play involves specific roles within the initiator or the target organization, we will show that role in subscript. For example, in Figure 11-2 we show that we are involving the Peering coordinators for ISP A and ISP B.

Figure 11-2. The peering coordinator roles are specified with this notation.

Route Announcements

Routing announcements are the mechanism for propagating reachability information between networks, and we use the color and/or the name of the network to graphically (Figure 11-3) represent these announcements.

Figure 11-3. Internet Peering, Paid Peering, Internet Transit graphical notation.

Note that the solid black marker in the routing announcement indicates a default route, the route that can be used to reach any destination in the Internet.

The ISP Customers

The customers of the initiator and the customers of the target are shown as smaller clouds attached to the ISPs as shown in Figure 11-4.

Figure 11-4. ISP Customers.

The “T” marking on the interconnection indicates that this relationship is a customer relationship, and the “$” indicates which side is getting paid. If the “$” is absent, it is assumed that the network that is higher in the diagram is the upstream ISP receiving the compensation for the transit relationship.

Interconnection Relationships

ISPs can have transit relationships with other ISPs. Figure 11-5 shows a few equivalent graphical notations for transit relationships.

Figure 11-5. Equivalent Internet Transit notations.

A “P” is used to denote an established peering relationship. The diagrams in Figure 11-6 show three notations for a peering relationship. The first two are equivalent notations of free peering relationships and the last is a Paid Peering notation.

Figure 11-6. Graphical notation of peering relationships.

Traffic Flow Notations

To demonstrate traffic traversing an interconnection, we use a directed line as shown in Figure 11-7. The directed line is colored to represent traffic destined to a particular ISP. For example, traffic destined to a purple ISP C is shown as a directed purple arrow. If packet loss is relevant to a particular tactic, it is denoted as a dashed line. If the traffic destination is not relevant, the color black is used. The thickness of the line represents the amount of traffic.

Figure 11-7. Traffic flow notations.

The Internet Service Provider Peering Playbook

To make it into the ISP Peering Playbook chapter, the tactic had to have been effective in:

• obtaining peering where the ISP would have otherwise not been able to obtain peering, or

• defending one’s peering position

This litmus test has been applied to each of these tactics.

We neither endorse nor condone any of these tactics; we are simply documenting them for the benefit of the community. If you are involved in peering on the Internet, then you should be aware of these maneuvers.

Notes from the field.

ISPs Calling Playbook Tactics By Number

I was introduced to the VP of Internet Operations for a large European ISP who shared that the ISP’s engineering teams were so well versed with these tactics that they referred to them by number. As in,

“Good morning, Geoff. We had another 19 (False NOC Outage) last night!”

Tactic 1. The Extended Direct Approach

“Success is the ability to go from one failure to another with no loss of enthusiasm.”

– Winston Churchill

The most common way to obtain peering is to simply ask for it (Figure 11-8).

Figure 11-8. Tactic 1 - The Direct Approach.

The trick here is how to request peering.

It is important to understand the internal motivations of the target ISP. The handling of peering requests is often seen as not “real network engineering” within many ISP network engineering teams. As a result, many ISPs allocate this duty as an on-duty weekly rotation. Many of the network engineers have learned that if they ignore peering requests during their week, the requestor will resend the request next week and someone else will have to deal with it. As a result, there are no shortages of excuses why a peering request could have been ignored. Don’t make it easy to drop the peering request.

The extended direct approach tactic here is to:

• Identify the right contact.

• Know the target peering prerequisites.

• Communicate the right information.

• Use the best method for contact.

• Use social engineering to make the request as easy as possible for the ISP to act upon.

The Right Contact

The following material is an excerpt from Chapter 4 – Internet Peering and is repeated here for convenience.

Top 10 ways seasoned peering coordinators contact target ISPs

1. Face-to-face at informal meetings in an “open” Internet Operations forums like NANOG, IETF, RIPE, APNIC, AFNOG, etc. (Here “open” means anyone can attend.)

2. Face-to-face at “selective” commercially-sponsored peering forums like the Global Peering Forum (Here “selective” means there are some prerequisites to attend, like being a customer of a GPF IXP sponsor.)

3. Face-to-face at “selective” IXP Member Meetings like DE-CIX, LINX, or AMS-IXP member meetings

4. Via introductions made through an IXP Chief Technical Liaison or a peer that knows the right contact

5. Via electronic mail, using the pseudo standard peering@ispdomain.net or a personal contact

6. From contacts listed on an exchange point participant list, or peeringdb registrations

7. With tech-c or admin-c from DNS or ASN registries

8. Through a search for “peering contact $ASN peering”

9. From the target ISP sales force, at trade show or as part of sales process

10. From the target ISP NOC

We will start by exploring the direct peering approach along with the back-door alternatives that seasoned peering coordinators use.

1A - E-mail Peering Requests

The direct by-the-rules approach is to send e-mail to peering@ispdomain.net as shown in Figure 11-9. Alternatively, there are many ways to find the e-mail address for an individual peering person. Dealing directly with an identifiable person as opposed to a role account may help increase the chances of a response (also shown in Figure 11-9). It is just socially harder to ignore a person-to-person e-mail.

Figure 11-9. Communicating with a person, not an alias.

The Problems with E-mail Peering Requests

E-mail is a tough communications medium for peering interactions. It is not clear how much information is needed for the interaction, or whether the target will be receptive to the overtures.

Here are the top problems with e-mailing peering requests:

• Which e-mail address? In some cases it is not at all clear which peering e-mail address to use when communicating with the target ISP. In some regions ISPs use peering@ispdomain.net as an alias for peering requests, while others use netops@ispdomain.net or peering.coordinator.name@ispdomain.net, etc.

• Not enough information in the peering request. One peering coordinator complained about constantly receiving peering requests without the basic information like AS#, amount of traffic involved, where the requester wants to peer, etc.

• Poor grammar. Others complained that some peering requests were very hard to decipher, and the peering coordinator lamented that the peering discussion that might ensue would similarly be difficult to decipher. (We examined a sample poor peering request in chapter 4.) These e-mail peering requests will ultimately drop through the cracks.

• Too much information in peering request. Still others complained that too much information made the peering request seem too burdensome to process.

• Ignorance of the peer. The notion of a “peer” includes the idea that both networks are of similar scope and scale of infrastructure and would benefit about equally from the relationship. The target peering coordinator may not know enough about the initiating ISP, so may not see the initiating ISP as a true “peer.” This ignorance may result in the target’s being unmotivated to pursue the relationship. Startup ISPs tend to be somewhat optimistic about their traffic growth futures, and since so many ISPs use intuition to determine whom to peer with, it is difficult for these unknown companies to be seen as true “peers.”

• Cultural issues. One Japanese ISP shared that it was common in Japanese ISPs to have a person with the job of translating the “rude” English peering request into a “polite” Japanese peering request. E-mail misunderstandings go in both directions.

Because of these challenges, person-to-person (ideally face-to-face) discussions help speed things along.

1B Maximize Face Time

Since face-to-face conversations improve the likelihood of obtaining peering, a popular strategy is to attend as many Internet operations conferences as possible (Figure 11-10).

The goals of attending the conferences are to:

• Collect peering intelligence.

• Maximize the chances of meeting the peering coordinator’s target ISPs.

• Be seen as a regular in the peering coordinator community.

Figure 11-10. Getting face time at Internet operations meetings.

While these conferences can be a lot of fun, you are there to seek information about your target peers. For example, “Whom do they peer with now?” If they were involved with a de-peering, what was the cause?

General information about the particular Internet Peering Ecosystem is important to discover, and it is easy in face-to-face meetings. You want to find out who the key players are and what the market prices are for transit, transport, colocation, etc. Assembling the business case for peering in the region is a critical stage for the initiator, and doing so provides credibility to the target. Familiarity with the region is important and can help maximize the likelihood of acceptance of a peering request. Most of this information is freely and readily exchanged among peering people within the community.

As a general rule, I estimate the travel budget to cost around $1000 for each domestic conference, $5000 to attend a nearby international conference, and $10,000 for far-away international conferences. A travel budget for introduction into a region might include five domestic, five international, and three far-away international conferences, for about $50,000 per year.

1C - Leverage the IXP Resources

Here we really leverage the IXP’s aligned interests to get you peering. The successful IXP has a person tasked with maximizing the value of its IXP by facilitating peering. This person performs this task with introductions, by meeting with customers and sharing contact information, by establishing a relationship with the peering coordinators to maximizing their peering, and by evangelizing peering at forums. This person is in a good position to provide introductions and promote you in the peering community (Figure 11-11).

Figure 11-11. Leverage someone who knows everyone.

Notes from the field.

Chief Technical Liaison Role

In the early days of Equinix, I would join the newbie peering coordinator to make a half dozen peering calls. I would start with people I knew would be receptive to the conversation, people who would be pleasant if not enthusiastically affirmative. After these calls the peering coordinator felt comfortable enough with the process to start making calls and meet these people face-to-face at peering forums. He would eventually became a peering consultant, helping a bunch of smaller ISPs and content companies get their peering infrastructure and sessions in place.

As Co-Founder and Chief Technical Liaison, I spent 90% of my time on the road doing research, introducing peering people to each other (hundreds of these), and running peering forums. All of these activities exposed me to peering people I could introduce to others.

If the direct approach yields no answer, the peering coordinator might reach out to an internal advocate at the target ISP, as described in the next tactic.

Tactic 2. Internet Transit with Peering Migration

The Internet Transit with Peering Migration tactic leverages an internal advocate (the target ISP salesperson) to lobby for peering on behalf of the initiating peering coordinator. The underlying assumption is that a salesperson at the target ISP is more likely to respond to e-mails than its peering committee. The initiator proposes a transit purchase agreement with an explicit migration from a transit relationship to a peering relationship should the peering prerequisites at the target ISP be satisfied (Figure 11-12).

Figure 11-12. Enlisting an internal advocate - the target ISP salesperson.

The difficulty with this tactic is that peering prerequisites evolve. Combine this reality with the fact that business interests (selling transit to the initiator) are at odds with allowing a migration to peering.

For example, Williams (an ISP at the time) executed this tactic with Sprint, and by the end of the contract term, the Sprint peering prerequisites had changed. Understandably, this situation led to heated discussions between the parties. Sprint suggested that Williams re-up for another term and try again for peering at the end of the next term. Perhaps a bit suspicious, Williams instead pursued the “end-run” tactic described next.

Tactic 3. The End-Run Maneuver

The end-run maneuver is to aggressively seek interconnections with the target ISP’s largest traffic volume customers (see Figure 11-13).

Figure 11-13. The End-Run Maneuver cherry picks the most attractive target ISP customers for interconnections.

These largest ISP B customers are then offered very-low-cost transit services by ISP A. As a fallback position, ISP A might offer Paid Peering or free peering. If these interconnect relationships are established, ISP A bypasses the target ISP, and reduces both ISP A’s cost of transit and its need to peer with the target ISP.

Once ISP B sees the customer traffic shift, ISP B will most likely decide that it would rather peer and continue to get the revenue from its customers than to lose them to ISP A. Peering is established, and, perhaps the bypass relationships discontinue.

The difficulty with this approach is that it can take a while. NetFlow traffic analysis will indicate where the traffic is flowing and can help prioritize which customers to target, but there may be a lengthy and costly sales cycle in order to obtain sufficient traffic volume to reach the desired end-run goal. Ultimately the goal is to minimize the need for peering with the target ISP. Then ironically, when you need it less, you can more easily get it. (Some people tell me it is like dating in this respect.)

Cherry picking may end up annoying the target ISP, but it has been used to obtain peering.

Tactic 4. Bundle Internet Transit with Peering

James Rice (BBC Internet at the time) shared with me the Bundle Internet Transit with Peering tactic, which is apparently more common in Europe and Japan than in the U.S. This approach uses an internal advocate to advance a transit purchase bundled with free on-net traffic exchange. The internal advocate (the target salesperson) is used here to internally advocate these interconnections, and at the end of a sales quarter, many things become possible.

This interconnect (shown pictorially in Figure 11-14) often utilizes separate routers and/or interface cards and separate transport circuits to separate the customer-to-customer traffic from the transit traffic to make the accounting easier. The initiator ISP pays for traffic exchanged on the “transit” interface card and doesn’t pay for traffic exchanged on the “peering” interface card.

Figure 11-14. Free on-net access with Internet transit sale.

The beauty of this approach is that the internal advocate handles all internal hurdles to peering, the peering can be executed immediately, and using separate interface cards makes it easy to implement.

The difficulty with this approach is that some ISPs do not have the internal mechanisms to support this dual interconnection, and might rationally prefer a simple transit relationship. Some claim that this dual relationship has been tried but is ultimately subject to being gamed, with the upstream forcing traffic along the revenue-producing path rather than the intended “free-peering” path.

Tactic 5. Buy Transit from Restrictive Tier 1 ISP

This tactic leverages the player motivations in the Internet Peering Ecosystem.

When peering requests are received, the target ISP looks to see if the proposed peer’s routes are already learned through a current peer (i.e., is the prospective peer a customer of a current peer?). If this is the case, peering is usually denied. No amount of “your customers’ performance will be better” arguments will help. They already receive your routes for free.

To maximize your chances of obtaining peering, with this tactic you purchase transit from the Tier 1 ISP with the most Restrictive Peering Policy in the ecosystem (see Figure 11-15).

Figure 11-15. Buy from the most restrictive Tier 1 ISP.

Why? By purchasing transit from the least peered Tier 1 ISP you minimize the chances that the target already learns your routes for free. Only those that were able to obtain peering from your restrictive upstream will deny peering because they already learn your routes for free.

Let’s walk through Figure 11-15 to illustrate this tactic. Let’s assume that you (ISP A) choose to purchase from the most restrictive Tier 1 ISP, ISP S. Your target, ISP B, does not already see your routes through a peer since you are not a customer of its peers. The restrictive Tier 1 doesn’t peer with many others.

If instead you bought transit from ISP L, a widely peered Tier 1 ISP, there is a greater chance that the target already peers with your upstream. ISP B will say it already learns your routes for free through its peering with ISP L.

By buying from the most restrictive Tier 1 ISP, you minimize the chances that your target ISP will already learn your routes for free.

Notes from the field.

“Once a Customer, Never a Peer”

There is an expression in the peering community: “Once a Customer, Never a Peer.” Becoming a customer may make you an unacceptable peering candidate, so relationships must be timed and managed carefully with potential peers.

One UK ISP grew larger than its upstream ISP and requested that the transit relationship be migrated to a peering relationship. Naturally, this request was denied. The ISP disconnected from upstream and dual homed to two larger ISPs and again requested peering from its former upstream ISP. This request was denied, and in fact peering wasn’t accepted for 18 months after the transit relationship ended. This scenario is not uncommon, highlighting that the transit selection strategy is important.

A related tactic is to buy only from the largest Tier 2 ISPs.

Being a customer of a Tier 1 ISP will effectively prevent peering with that Tier 1 ISP at a future date. It is often more cost-effective and strategically more effective to purchase transit from a large Tier 2 ISP. Not being a direct customer of the Tier 1 ISP means that no direct revenue is lost by peering, and therefore there is one less obstacle to overcome. There is, of course, the indirect loss of revenue if the large Tier 2 ISP is a downstream customer of the Tier 1 ISP, but this objection is an easier one to overcome than being a direct customer.

Tactic 6. Transitional Paid Peering

In this tactic (Figure 11-16), Paid Peering is positioned as a “stepping stone” toward the ultimate goal of settlement-free peering between two parties.

Figure 11-16. Purchase Paid Peering as a stepping stone to peering.

As a stepping stone, the router configurations, exchange point arrangements, and all peering interconnections logistics proceed as if it were a settlement-free peering arrangement. A settlement fee is paid every month until the peering prerequisites are met. Once the peering prerequisites are met, settlement fees are no longer required and free peering is established. It is important to note that over the years, there have been only a few completed transitions to free peering.

Tactic 7. Partial-Route Internet Transit

In this tactic, an ISP that is well-peered at an IXP sells access to the routes that it learned at the exchange.

In this scenario (Figure 11-17), ISP X agrees to sell access to all of the routes that are learned through its peering relationships at the exchange, including ISP B routes, which ISP A desires. The price for partial transit is sold at slightly above the price of transport into the IXP, so ISP X can present a compelling case to ISP A:

“For about the same price as the cost of building into the IXP, you can hear all of the routes that I hear. I have been here for a decade and have many legacy peering relationships that you will not be able to get. I will even throw in my own customer routes. You won’t have to pay the cost for IXP membership; you will have no peering port fees, no colocation costs, or costs for additional routers; and you won’t need any peering people to find peers and establish sessions. Your traffic will simply pass through my router before reaching your target peers. You get all of the benefits of peering without the headaches.”

Figure 11-17. Access peer through partial transit sale.

Notice that ISP X announces all routes through its router. This setup does not achieve a true peering relationship between ISP A and ISP B because there is a network (ISP X) in between. It is still, however, a useful tactic to quickly obtain exchange point routes. This setup is also interesting because it can be very inexpensive for ISP X to provision and sell.

Tactic 8. Play Chicken

This confrontational tactic has been employed a dozen or so times, often in a highly visible manner – very visible in the Internet operations circles anyway. These peering disputes tend to highlight a perceived asymmetry in derived value from a peering relationship.

Most of the time the tactic goes as follows (Figure 11-18): Assume that ISP A and ISP B have a peering relationship. ISP A is content-heavy, meaning that the majority of its traffic is outbound, destined to the users that requested the content. ISP B is relatively access-heavy, meaning that the majority of the traffic is inbound, presumably the content that its customers requested. When the end user requests a web page, that user’s small request results in a relatively large response. There is inherent asymmetry of traffic.

Figure 11-18. Traffic asymmetry sometimes leads to de-peering or playing chicken.

The network infrastructure required to carry the traffic between the two parties may be asymmetric as well. ISP B might have just upgraded its national backbone at great expense, and ISP A is asking for more peering interconnect bandwidth. ISP B will argue that ISP A is “dumping” its traffic onto ISP B’s network and not paying a dime towards the cost of hauling that traffic across the country.

When the Play Chicken tactic is employed, it is always ISP B that wants to re-negotiate the relationship from free peering to some sort of paid relationship. ISB B points to the recent expensive upgrade to support the ISP A-to-ISP B traffic, and that the upgraded interconnect is already saturated with ISP A traffic. ISP B reminds ISP A that it gets no revenue for this traffic and that ISP A is making a ton of money from its content providers. This situation turns into a heated discussion, and ISP B indicates that it will have no choice but to disconnect the peering session. ISP A may stretch out these negotiations for as long as possible.

Both sides believe that the other side needs the free peering relationship more. The preconditions for the Play Chicken tactic are set.

Sometimes, the two parties negotiate a more equitable business arrangement and maintain their interconnection agreement without any operation disruption.

Sometimes ISP B disconnects the peering session and both ISP A and ISP B customers experience an operational disruption. The traffic may be delivered via another path, as shown in Figure 11-19.

Figure 11-19. During the crisis peered traffic traverses a more circuitous route.

In at least one case the Play Chicken tactic resulted in an Internet segmentation where customers of one ISP could not reach customers of another ISP.

In most cases peering is resumed after they agree to spread the traffic across more interconnection points. This solution reestablishes connectivity and reduces the distance that ISP A traffic has to be carried across ISP B’s infrastructure (Figure 11-20). In some cases financial considerations were involved as well.

Figure 11-20. Re-peering addressed with more points of interconnect.

The Play Chicken tactic is employed to abruptly change the terms of the peering relationship, but it threatens operational impact if neither side acquiesces. It really tests the assertion that both sides are receiving “about equal value” from the relationship.

It is worth pointing out that the aggressor of the play-chicken tactic rarely increases revenue from this tactic. The disruption is usually so significant – and the destruction of the inter-organizational relationship so severe – that the “loser” generally does not subsequently choose the aggressor as a supplier of transit or Paid Peering services.

Tactic 9. Traffic Manipulation

One of the most clever, one of the most devious, and absolutely one of the most fun tactics to discuss is the Traffic Manipulation tactic.

This tactic leverages the fact that web traffic is asymmetric (Figure 11-21) and the content companies get to decide which of potentially many paths this relatively large proportion of traffic will flow.

Figure 11-21. The fact that small requests generate large responses can be leveraged.

In the Traffic Manipulation tactic, the content-heavy initiator (ISP A) forces its traffic to the target’s (ISP B’s) upstream transit provider, thus maximizing the financial cost to the target ISP to receive the initiator’s traffic.

To illustrate, consider Figure 11-22, where ISP A wants to peer with ISP B. Assume that the initiator ISP’s large responses normally travel through the target ISP’s peer “L,” as shown in the figure. Thus, ISP B receives ISP A traffic for free.

Figure 11-22. Initial context for Traffic Manipulation tactic.

If ISP A asks ISP B to peer, the answer will probably be “No, we already get your traffic for free through our peering arrangement with ISP L.” Rationally, ISP A does not attempt to peer with ISP B at this point.

ISP A instead forces its traffic to ISP B to go through ISP M, which is ISP B’s transit provider, causing ISP B’s transit bill to increase (Figure 11-23). I have heard stories of how this approach was amplified by using a traffic generator to replay traffic from a previous month!

Figure 11-23. Traffic shifts during the Traffic Manipulation Tactic.

After some time elapses and lots of traffic is sent through ISP B’s transit provider ISP M, ISP A opens a dialog with ISP B. ISP B reviews the traffic analysis data and is surprised that ISP A has not appeared on the radar screen before as a potential peer (Figure 11-24). Perhaps it was a spot event, a bug in the peering analysis software. In any case, it is the peering coordinator’s job to deal with these issues rapidly and keep the transit expenses low. Seeing the great transit expense that is paid for access to this traffic, the peering decision is easy for ISP B: ISP A is clearly a large traffic peer that is expensive to access over a transit link. Peering is established with the target.

Figure 11-24. The peering request occurs after some time.

Any artificial traffic amplification stops a month or two after peering is established. Since only a small percentage of ISPs perform the traffic analysis necessary to detect this maneuver, this tactic often goes undetected.

The Traffic Manipulation tactic is most effectively deployed by Large-Scale Network-Savvy Content Providers. Since web traffic is asymmetric, the producer of the responses (the content player) has the greater ability to force a larger amount of traffic along one path or another.

Along the same lines, Avi Freedman shared anecdotes of ISPs using large web spider deployments to pull content across peering and transit sessions in order to generate inbound traffic to meet peering ratios. Similarly, transit providers often offer special deals for access-heavy ISPs to help them balance their traffic ratios. Sometimes, ISPs send traffic along circuitous paths in order to maintain peering ratios.

All of these tactics require spare capacity to handle the manipulated traffic. In addition, if the tactic is detected, the peering coordinator community is small enough that everyone in the community hears about it.

This tactic is seen in the community as “evil, clever, and antisocial,” all at the same time. Conversely, Traffic Manipulation is seen by some ISPs in the community as a perfectly valid and appropriate means to manage traffic ratio requirements for peering.

Tactic 10. Bluff Issues

“All warfare is based on deception.”

– Sun Tzu, The Art of War

This tactic refers to the game of poker where players over signal their strength: they bluff. Peering prerequisites often include traffic volume minimums in multiple geographically distributed locations. Since many peering coordinators do not perform the required traffic analysis to disprove traffic volume assertions, and the cost of being wrong could be an expensive transit bill, the initiator ISP assertions may be blindly accepted as truth (Figure 11-25).

Figure 11-25. Bluffing performance issues to obtain peering.

Two forms of the bluff are seen in the field:

1. Bluffing traffic load futures. The peering coordinator in this maneuver typically asserts that the required traffic volumes for peering:

a. are met today, or

b. will be met at some point in the near future based upon some spectacular spot event, or

c. a new secret customer coming on-line soon will cause them to be met

The last form of the bluff, for some reason, often seems to involve Microsoft. For example,

“Don’t tell anyone, but Microsoft is using us for all of its update traffic, and there will be a monster launch of YYY in the next two months. We’d better peer now while we have time to plan for the impending loads!”

This approach is effective for several reasons:

• It is difficult for the target ISP to determine how much traffic will be exchanged if a peering session is established.

• It is difficult to determine if the initiating ISP is bluffing. The target has no way of validating whether the customers are planning to use the initiators network.

• It is difficult to project the resulting traffic volume if there is no bluff.

As with poker, once the bluff is called and found to be false, future peering negotiations and claims may be met with suspicion.

2. Bluffing performance problems. Eric Anderson (BT Ignite) mentioned a slightly different but related bluff. An ISP peering coordinator can also be motivated to peer if the other party can demonstrate a significant but perhaps intermittent performance problem that can be solved by peering. Since both ISPs have customers that are suffering from packet loss, for example, both parties are motivated to fix the problem.

Since few ISPs actively perform the active network performance measurements, they can be persuaded that there are network performance problems where none actually exist. In one case, the evidence of poor performance was a series of traceroutes that demonstrated the packet loss and latency between two end points. After performing some significant analysis, these traceroutes were determined to be a farce and peering was not established.

Tactic 11. Be Open. Loudly.

One tactic to obtain peering is to publicly and actively promote an Open Peering Policy (Figure 11-26).

Figure 11-26. Peering evangelism makes it easier to be found.

Peering coordinators face rejection every day. Phone calls and e-mail messages go unanswered as a part of their job. Finding an ISP that has an Open Peering Policy typically means that there will be little resistance in getting another peering session established. Since the number of peering sessions and the amount of traffic exchanged in peering relationships is used to gauge the effectiveness of a peering coordinator, obtaining peering with “Be Open. Loudly.” ISPs is an easy way for a peering coordinator to make him/herself look good.

Tactic 12. Be Everywhere

Along the same lines but separable from the “Be Open. Loudly.” tactic is the tactic of building a large number of POPs, in a large number of exchange points, in a large number of countries. This strategy may be part of the deployment strategy anyway, but doing so also maximizes the likelihood of meeting the geographic distribution peering prerequisite with the largest set of targets (Figure 11-27).

Figure 11-27. Wide-scale deployment maximizes the chances of meeting prerequisites.

“Meet you in three time zones? No Problem. Meet you in person at DE-CIX member meeting? Absolutely!” Broad participation provides many avenues for dialog with peering folks and many more opportunities for interconnection.

Tactic 13. Get Traffic

The Get Traffic tactic involves acquiring a massive amount of customer traffic that an ISP can then turn around and offer to peers via free peering arrangements (Figure 11-28). Since the alternative way to access this traffic is through expensive transit relationship(s), the target ISP is more motivated to peer with those with lots of traffic and routes. Entering a peering ecosystem with only a little local traffic to peer does not provide a compelling case for peering.

Figure 11-28. Build a healthy traffic profile prior to requesting peering.

To obtain massive amounts of customer traffic, the ISP offers something substantially different to the market, such as ultra-low-cost transit (Cogent used this tactic globally, and Level 3 started with this tactic as well). In some cases ISPs offer a strategic cooperative relationship to sell VPNs, Intranets, high-bandwidth video, or traditional carrier services over existing Internet infrastructure. The goal here is to grow into something worth peering with, not necessarily to make money.

This tactic is applicable only to Tier 2 ISPs. It is also a way for the Tier 2 ISP to ultimately meet the traffic volume prerequisites for peering with the larger players in other ecosystems. Care must be taken with this tactic in order to balance peering traffic ratios at the same time as picking up massive amounts of traffic.

Tactic 14. Friendship-Based Peering

It is often said that peering is a game of relationships, and this tactic leverages that point. Peering relationships have been established between networks of unequal size based solely upon friendships between peering coordinators. Hiring peering coordinators with extensive contacts in the peering coordinator community leverages these previous relationships to speed up peering deployments (Figure 11-29).

Figure 11-29. Peer with friends and former colleagues.

Notes from the field.

Peering With Former Colleagues

One peering coordinator said that his boss left the company for a competitor and turned around to negotiate peering with his former colleagues. Knowing the ropes aided in obtaining peering with this very selective peer.

In a similar way, it is helpful to have friends that have colocation space they can offer for free to a friend, perhaps without obtaining approval in advance. They call it “Bro-lo,” as in do you have any “bro-lo” in Dallas for me buddy?

When I worked at Equinix, part of my role involved facilitating the interactions between peering coordinators that didn’t know each other. These relationships became friendships. From these friendships, I had a good idea who would be willing to peer with new peering entrants into the ecosystem, and I used these friendships to help smooth the way to peering. This tactic has been especially effective for international ISPs that may not know any local ecosystem peers.

Friendship-based peering is not effective for obtaining peering with Tier 1 ISPs. To avoid regulation, most of the Tier 1 ISPs have a stringent peering request process that would not allow establishment of a casual peering session. These ISPs have to be able to demonstrate that their decision process is consistent across all peering requests, and they must maintain a document trail to prove that consistency.

Tactic 15. Spam Peering Requests

Everyone hates spam, but one apparently effective approach to establishing peering is to send e-mail to all participants at various exchange points.

(If you use this tactic, please make sure that “Reply to Self” and not “Reply to List” is selected.)

This approach led to dozens of peering sessions being set up very quickly. Since many companies on the Public Peering fabric are “open” peers, this approach effectively yields peering with a potentially large number of ISPs and content companies (Figure 11-30).

Figure 11-30. Spam peering requests to exchange point discussion lists.

The problems with this approach include that:

• The result might be a large number of high-maintenance low-volume peers.

• The activity signals that you are a less-desirable peer since you resorted to blanket peering requests instead of selecting the ones that mattered most and interacting with them one-by-one.

The ISP that reported this tactic said he was surprised at which ISPs responded to this approach, and said that he might not have even tried to seek peering with some of them.

Tactic 16. The Honey Approach – Be Sweet

Named for the adage, “You can get more bees with honey than with vinegar,” Yahoo! and Microsoft both adopted the Honey Approach by promoting the desirability of their content as a key reason to peer with them.

As one of the largest portals in the world, Yahoo! described the thousands of live webcast events, the hundreds of thousands of concurrent streams of content delivered, the gigabits-per-second of raw traffic flows, etc. as reasons why an ISP should want to receive that traffic over a peering relationship instead of through its transit relationship. Microsoft promotes its portals and the massive volume of traffic that can be delivered for free over peering sessions that would otherwise be delivered through transit providers at a substantial cost. Microsoft would deliver the content in-country for free or via expensive transit services from somewhere overseas. Since many ISPs see their network performance as a key differentiator, Yahoo! leverages this performance lure well (Figure 11-31).

Figure 11-31. Sell peering with you.

This tactic is also demonstrated when Yahoo! and others stand up during “peering personals” and discuss why people should want to peer with them. Peering personals is a mainstay of the peering forums where peering coordinators stand up and introduce themselves and their company to the room. After this self-introduction the peers can put a face to a name and have their peering discussions.

Peering personals is a place to execute the Honey Approach tactic. You want to stand up and participate because it is far easier to have others approach you than to have to seek out others to start the peering discussions.

Tactic 17. Purchase a Well-Peered ISP

This tactic is to simply purchase an ISP that has the desired peering you seek. This tactic was more popular in the 1990s than it is today, but it is worth noting (see Figure 11-32).

Notes from the field.

Acquiring Tier 1 ISP Status

As discussed earlier, Level 3 was unable to obtain peering with the Tier 1 ISPs in the early days, so it acquired networks that had already established peering with a couple of Tier 1 networks. Under the assumption that this peering would transfer to the larger aggregate company, Level 3 acquired the ISP in order to leverage the pre-established peering arrangement and build upon it. In Australia, the ACCC pressured Telstra to peer with a few of the largest ISPs in Australia. Within a few years of doing so, each of these ISPs was acquired by foreign ISPs who instantly became Tier 1 ISPs in Australia. Optus was acquired by SingTel (the Singaporean Tier 1 ISP), Connect.com was acquired by Telekom New Zealand (a Tier 1 ISP in New Zealand,) and Oze-mail was acquired by Verizon (a Tier 1 ISP in the U.S.).

Figure 11-32. Identify who has valuable peering relationships.

Some would say that this tactic is the easiest one to execute, but some peering coordinators advise caution here. Many peering contracts (Bi-Lateral Peering Agreements) have 30-day termination clauses, so the peering may not last long beyond the acquisition date.

In the 1990s, business people started trying to quantify the value of a peering relationship. For example, those peering relationships with a written contract may be used on the balance sheet as an asset to be used when valuing the company for acquisition.

Tactic 18. Bait-and-Switch

“Hold out baits to entice the enemy. Feign disorder, and crush him.”

– Sun Tzu, The Art of War

A large parent company may be able to obtain peering where a smaller subsidiary may not. The Bait-and-Switch tactic leverages this fact by negotiating peering as a large traffic source or sink, and then announcing a different and much smaller traffic source or sink when setting up peering (Figure 11-33).

Figure 11-33. The peering negotiated isn’t always the peering acquired.

To illustrate, let’s assume that enterprise spin-off a’ seeks peering with ISP B. However, a’ is too small to be interesting from a peering perspective. It therefore leverages its relationship with the much larger and recognizable parent, enterprise A. Enterprise A negotiates peering with the ISP. When ISP B finds out that it is peering with the smaller a’ subsidiary, and receives only a small amount of customer traffic, the ISP may realize it was a victim of a bait and switch but keep the peering session anyway.

Tactic 19. False Peering Outage

This tactic made this playbook famous (in the very narrow micro niche segment of Internet peering operations people).

With this tactic, both the initiator ISP A and the target ISP B are attached to an IXP Public Peering fabric such as an ethernet LAN (Figure 11-34). The target ISP NOC is urgently contacted to “repair” a down peering interconnection. This call is placed in the middle of the night, during a maintenance window when the NOC is manned with a junior on-call engineer. The junior engineer is manipulated into editing the router configuration and establishing peering where peering was never intended to be established.

Figure 11-34. The Fake NOC Outage.

The script goes something like this:

NOC B: Hello, ISP B NOC.

NOC A: Hi – our peering session with you is down at IXP North. I am calling to get a status update.

NOC B: I’m not seeing any alerts.

NOC A: I’m telling you it is down. Again. I know you do config runs at this time.

NOC B: Yes we are doing a config run, but I don’t see a peering session configured with you.

NOC A: Well I don’t know what you guys have done, but this is the third time this has happened this year. Don’t make me call Hans-Werner Braun. He will chew you guys out. Again.

NOC B: Umm. I did see something weird and I did hack the config for this bizarre script. I bet it was that last edit that trashed it. Give me a few minutes to fix the config and push it out and I will figure out how to make sure it doesn’t happen again.

When told about this scenario, the most common response from peering coordinators was,

“I wonder what our NOC would do?”

The VP of Engineering for a large European ISP shared with us that its NOC refers to this tactic by number, as in

“Hey, we had another 19 last night.”

Tactic 20. Leverage Broader Business Arrangement

This tactic makes use of a broader business relationship to make peering an easy “Yes.”

One large ISP met the peering prerequisites of a Tier 1 ISP but was refused peering because peering would significantly reduce transit revenue. The objection was overcome by expanding the broader business relationships to include fiber, wavelengths, colocation, etc. bundled with free on-net peering. The resulting combined arrangements left the target Tier 1 financially better off by peering (Figure 11-35).

Figure 11-35. Internet Peering is sometimes bundled with other services such as fiber, colocation space, wavelengths, etc.

There are many examples of this tactic. AOL obtained Tier 1 status by leveraging its customer relationship with the large Tier 1 ISPs. Comcast established a broad business relationship with Level 3, as discussed earlier in this book. Peering was key to obtaining incremental revenue.

Tactic 21. Overseas Power Play

In this maneuver, ISP A wants to pressure the incumbent Tier 1 ISP B to peer by forcing traffic along a more circuitous and expensive international path. The Tier 1 ISP refuses to peer, so ISP A turns off the in-country transit connection, and builds into a well-populated foreign IXP so that it can send its traffic through an international upstream to reach the incumbent Tier 1 ISP.

ISP A has to pay for the international circuit. The incumbent Tier 1 ISP also has to pay for its international circuit into the overseas IXP where it peers. The customers of both ISP A and the Tier 1 incumbent ISP both experience poor performance. The hope is that the Tier 1 ISP will come to the table and say, “We should peer. It is costing us both a lot of money to peer in this other region.”

MWEB played this tactic with Telekom South Africa in 2011:

“We have made a conscious decision that starting next month [November] we will not pay anybody for transit traffic anymore. So if you don’t want to peer with us, that is it! We will not pay you one single cent anymore.”

– MWEB CEO Rudi Jansen

Just like the Traffic Manipulation tactic, this play counts on the pricing and perhaps the performance problems to pressure the target to peer locally to improve performance and decrease its international transit costs. In this example not only will Telekom South Africa customers receive their MWEB video traffic from London at great expense, but the latency-sensitive gaming traffic will also experience really poor performance (Figure 11-36).

Figure 11-36. Peer overseas.

Several ISPs have used this tactic to get cheap transit, to force traffic along a circuitous path to pressure the local incumbent to peer, and to decrease the overall cost of traffic by taking care of its own international transit.

Notes from the field.

Pressure in South Africa

In 2010, MWEB, a Tier 2 ISP, offered a unique uncapped Internet Transit service to the South African market. The market responded enthusiastically, and by 2011 MWEB had acquired massive market share. Even with this tactic, MWEB was initially unable to obtain peering with the incumbent Tier 1 ISP. It did, however, experience massive growth and better performance from the overseas transit and peering deployment for in- and out-of-country traffic. The amount of traffic between MWEB and the incumbent became a relatively small percentage of MWEB’s total traffic. Ultimately, in September 2011, MWEB did obtain peering with Telekom South Africa, the incumbent Tier 1 ISP. This is according to some private and public sources (including http://mybroadband.co.za/news/broadband/33234-telkom-and-mweb-peering-at-last.html). As usual, the specifics of the arrangement are officially protected behind an NDA.

Tactic 22. Construct Peering from Transit

Peering can be constructed from transit by filtering out all noncustomer routes as shown in Figure 11-37.

Figure 11-37. Filtering noncustomer routes appears to be peering in the routing table.

Engineers can look at routing tables and see where there are adjacencies. A peering relationship may be inferred in a situation where only customer routes appear.

Why would you do this? So you appear more attractive. Target peers may believe that you are peering with the big players when they look at the routing table. If the big players find you worthy of peering, then perhaps the target ISP will believe that it should as well.

Just like Paid Peering, constructing peering from transit is like paying for a date to the prom.

In this tactic, only the two parties know there is a cash payment at the end of the evening. Nondisclosure agreements are generally applied to Paid Peering, and neither party is motivated to disclose its nature. The purchaser wants to appear to be a true peer, and the provider doesn’t disclose its nature because it wants to continue getting the revenue for it.

The real challenge is that the target peer that you want doesn’t have a Paid Peering product, but it would love to sell you transit. So you can construct peering out of transit by filtering all noncustomer routes.

Defensive Peering Tactics

Now we move on to the tactics used to NOT peer.

These next few tactics are different – they are executed by the larger peers (typically the Tier 1 ISPs) to avoid peering with anyone else in their Internet Region.

Tactic 23. Defensive Tactic – Do Not Peer in Your Home Market

Korea Telecom (KT) protects its home market Tier 1 advantage by not peering in its home country. In fact, to peer with Korea Telecom, peers must meet the company in a different country (Figure 11-38).

Figure 11-38. Peering only outside the home market is a common defensive maneuver.

Notes from the field.

Protecting Home Turf

As just described, peering with KT was generally allowed only outside of Korea. As John Milburn puts it, “The desire to protect one’s home market dwarfs any benefits that peering might provide.”

Tactic 24. Defensive Tactic – Protect Peering Policy

Some Tier 1 ISPs do not publish their peering prerequisites or peering policies. They say it does no good to publish them as they already know whom they want to peer with; they don’t need to advertise. Publishing the Peering Policy would only cause more work for them since the resulting peering requests will undoubtedly be denied.

So they protect these documents under nondisclosure agreements (NDAs) to:

• avoid peering requests from unqualified candidates

• avoid the public scrutiny and the potential court of public opinion that might result from public release

If the documents are protected and come under public scrutiny, then an NDA has clearly been violated, providing more evidence that the potential peer that received the policy under NDA cannot be trusted as a peer.

Tactic 25. Defensive Tactic – Change & Escalate Peering Prerequisites

In this tactic, the ISP adjusts the peering prerequisites as other ISPs approach meeting them. Once the peering requirements have been met, the ISP simply adjusts them upwards (Figure 11-39).

Figure 11-39. Adjust peering requirements over time.

In the community, this tactic is recognized as simply an implementation of a restrictive Peering Inclination, a general inclination not to peer with anyone else.

Tactic 26. MILD Defensive Tactic

The following story was told earlier, in Chapter 9, to highlight the way an ISP’s Peering Inclination may differ from its stated Peering Policy. It is repeated here as a Defensive Tactic:

At an APRICOT conference, James Spenceley (Comindico) shared a tactic that Telstra executed in Australia. Comindico met with Telstra to demonstrate that it had met the very difficult-to-meet peering prerequisites. Spenceley characterized these requirements as “bringing enough capacity into the territories for many Mbps for every man, woman, child, and kangaroo.” Once Comindico had built its transcontinental backbone, the network maps were exchanged as part of the peering discussion. Ultimately, the peering requirements were changed such that Comindico only slightly failed to meet them.

Comindico went bankrupt shortly thereafter.

It is possible that the adjustments of the peering requirements were merely a coincidence (they do change), and the story came to me second-hand, so we must allow for the possibility that there is another side to the story.

However, we have heard stories about this tactic before. Several other large Tier 2 ISPs in different parts of the world have identified this maneuver being perpetrated by the Tier 1 ISPs in the field.

The MILD tactic stands for Make It Long and Difficult. James Spenceley perhaps coined the phrase for this tactic (Figure 11-40). The approach is to prolong the peering discussions so that while you appear to be open to peering, the process never ends, and effectively peering will never happen.

Figure 11-40. The MILD defensive tactic is intended to neither provide nor deny Internet Peering requests.

To play out the example in slightly more detail, once the peering requirements are met, the peering candidate notifies the Tier 1 that it would like to meet to discuss peering. The response is that the request is being considered. Several months later a follow-up request results in an in-person meeting – scheduled for several months later. At that meeting additional documentation is requested and a follow-up meeting is scheduled for several months later. These meetings are rescheduled a few times. When the parties reconvene, there is a new team that has to be brought up-to-speed on the peering discussions, further delaying the execution of peering. Additional questions are extended to the potential peer.

During this stretched-out discussion in Australia, Comindico was paying for an overbuilt network that exceeded its revenue. As stated before, it appears that ultimately this situation led to Comindico’s bankruptcy.

Tactic 27. Congest Transit Pipes

In this tactic the ISP makes peering appear more attractive than the transit alternative.

To illustrate, consider an anonymous claim posted to the NANOG mailing list in 2010.

Note: The following material is copied from the posting at:

http://mailman.nanog.org/pipermail/nanog/2010-December/029251.html

The posting reads as follows:

Ever wonder what Comcast’s connections to the Internet look like? In the tradition of WikiLeaks, someone stumbled upon these graphs of their TATA links. For reference, TATA is the only other IP transit provider to Comcast after Level (3). Comcast is a customer of TATA and pays them to provide them with access to the Internet.

1 day graphs:

Notice how those graphs flat-line at the top? That’s because they’re completely full for most of the day. If you were a Comcast customer attempting to stream Netflix via this connection, the movie would be completely unwatchable. This is how Comcast operates: They intentionally run their IP transit links so full that Content Providers have no other choice but to pay them (Comcast) for access. If you don’t pay Comcast, your bits wont make it to their destination. Though they wont openly say that to anyone, the content providers who attempt to push bits towards their customers know it. Comcast customers however have no idea that they’re being held hostage in order to extort money from content.

Another thing to notice is the ratio of inbound versus outbound. Since Comcast is primarily a broadband access network provider, they’re going to have millions of eyeballs (users) downloading content. Comcast claims that a good network maintains a 1:1 with them, but that’s simply not possible unless you had Comcast and another broadband access network talking to each other. In the attached graphs you can see the ratio is more along the lines of 5:1, which Comcast was complaining about with Level (3). The reality is that the ratio argument is bogus. Broadband access networks are naturally pull-heavy and it’s being used as an excuse to call foul of Level (3) and other content heavy networks. But this shouldn’t surprise anyone, the ratio argument has been used for over a decade by many of the large telephone companies as an excuse to deny peering requests. Guess where most of Comcasts senior network executive people came from? Sprint and AT&T. Welcome to the new monopoly of the 21st century.

If you think the above graph is just a bad day or maybe a one off? Let us look at a 30 day graph...

Comcast needs to be truthful with its customers, regulators and the public in general. The Level (3) incident only highlights the fact that Comcast is pinching content and backbone providers to force them to pay for uncongested access to Comcast customers. Otherwise, there’s no way to send traffic to Comcast customers via the other paths on the Internet without hitting congested links.

Remember that this is not TATA’s fault, Comcast is a CUSTOMER of TATA. TATA cannot force Comcast to upgrade its links, Comcast elects to simply not purchase enough capacity and lets them run full. When Comcast demanded that Level (3) pay them, the only choice Level (3) had was to give in or have its traffic (such as Netflix) routed via the congested TATA links. If Level (3) didn’t agree to pay, that means Netflix and large portions of the Internet to browse would be simply unusable for the majority of the day for Comcast subscribers.

Love,

Backdoor Santa

These types of anonymous public disclosures are somewhat common in this community, and, therefore, should be taken with some skepticism. However, it does demonstrate a maneuver that is discussed periodically in the industry – congesting links to achieve an end.

Similarly, back in the early NSFNET transition days, a Tier 1 ISP’s peering link into MAE-East was purposely kept congested. Requests for upgrading that circuit were met with sales responses like “ ...to get good access to our customers, you should purchase our transit product...”

This tactic is well-known. Congesting a link to achieve specific business goals has proven to be an effective tactic and has led to peering, Paid Peering, or transit sales.

Peering Tactics That Don’t Work

Conversations with peering coordinators revealed several approaches that were not effective:

• Hold desirable content hostage. The Western Australian Internet Exchange (WAIX) was not successful in getting Telstra to peer by allowing access to popular content only to those that peered at the WAIX. A notice was delivered to Telstra customers that the content wasn’t available to them since Telstra didn’t peer at the WAIX. The complaints fell on deaf ears.

• Exert market dominance in a foreign market. Some foreign ISPs have tried to push as if they were the Tier 1 in the new market. This strategy proved to be ineffective and it alienated the ISPs.

• Threaten litigation. Threatening litigation and government intervention often shuts down the conversation between peering coordinators.

• Public name calling and badgering in public forums brings personality conflicts into play and often results in doors being closed that should be open.

• Blind requests for peering with the largest target peers first often fails. Make sure you research the peering requirements and build your people network well before initiating contact. Many peers, for example, insist upon geographic diversity for peering. Having some knowledge of the common colocation environments and a sense of these peering requirements signals that the peering process and ongoing operations will not be an undue burden.

• Demonstrating lack of knowledge regarding backbone operations often stops the peering discussion. Interestingly, demonstrating too much knowledge was cited in instances when arrogance led to personality conflicts. “In the end it’s knowledge and attitude.”

Peering Workshop Practice Questions

Here are a few practice questions from the Internet Peering Workshop:

1. Which maneuvers involve social engineering, the manipulation of the human elements involved in a target peer’s operation?

2. Which maneuvers would most people consider unethical?

Answers to these questions are in the answer key in the back of the book.

Cool Chill Technology. Mineral oil is a clear non-conductive liquid that provides 1200 times more heat capacity by volume than air. Here (above) we see the Green Revolution Cooling’s CarnotJet® system cooling four racks using circulated liquid cooling. Below we see a server partially submerged in liquid cooling.

Photos courtesy of Green Revolution Cooling.

Chapter 12 – Taxonomy of Data Centers

In this chapter we will build a model of an Internet Exchange based on the top of a model of an Internet Data Center. All Internet Data Centers (IDCs) are made up of similar data center components. All data centers built to house telecommunications gear have some form of:

• real estate to house the gear

• security infrastructure to prevent unauthorized access to the equipment

• reliable power infrastructure to make sure the gear continues to operate even when utility power fails

• Heating, Venting, and Air Conditioning (HVAC) systems to keep the equipment within its operating range

• data center operations staff to monitor and manage the operations environment in the data center

We present the key differentiators across the data center and colocation business models (Figure 12-1), and finish with a discussion of the differences between European- and U.S.-based IXPs.

Figure 12-1. The Internet Data Center Model.

Gallery 12-1. Inside a modern Internet Data Center. Photo courtesy of Equinix.

The Basic Data Center Model

Every data center has at least the following four components: real estate, some form of security system, reliable power infrastructure, and air conditioning systems as modeled in Figure 12-2.

Figure 12-2. All data centers have the same basic building blocks.

Now we will discuss each of these components.

Real Estate

At the core of every data center is real estate, which is, as the expression goes, all about location.

Gallery 12-2. One Wilshire is a hot bed for peering in Los Angeles, particularly with peers coming in from across the Pacific Ocean. Photo courtesy of CoreSite.

However, for real estate destined to be a networked data center, proximity to fiber is critical, as fiber is what the tenants will ride in on (Figure 12-3). If the colocation center is far from fiber, then the cost of building in is higher. The most desirable real estate is located near fiber nexus points.

Gallery 12-3. Colocation facilities are being built to support the nationwide fiber plants being incrementally built across the U.S.

Figure 12-3. Real Estate for a data center.

Notes from the field.

Hunt for Real Estate incognito

One telco exec shared that when traveling to a real estate site inspection his group travelled as stealthily as possible to decrease the chances of competitors knowing the site was available and bidding the price up. They registered for rooms under assumed names, did not disclose their trips on their calendars, etc. Many times the real estate owner is not aware of the inherent value of fiber proximity or the value potential of building infrastructure on the site. Since fiber often runs along preexisting rights-of-way along railroad tracks, real estate that opens up near these locations often comes up as key IDC locations.

Some Real Estate Selection Criteria

Selection of location is absolutely critical for Internet Data Centers since it has a direct impact on the cost of the facility and the ultimate cost of participation. Here are a few of the selection criteria that colocation providers shared as key to their location search:

1. Fiber. Ideal sites will be near or on fiber paths, or within a couple miles of dark fiber and telephone company Points of Presences (POPs).

Fiber entrance facilities run the gamut from tying fiber to the wall of an alley, as shown in Figure 12-4, to cement-encased fiber entrances, as shown in Figures 12-5 and 12-6.

Figure 12-4. A bad fiber entrance system.

Figure 12-5. Cement-encased fiber entrance. The Terremark Miami Facility.

Figure 12-6. Fiber extended in through the core of the building.

2. Size, floor loading, configuration, expandability. Sites must be large enough to support the intended operations, with floor loading sufficient for the equipment expected.

Notes from the field.

Cermack Printing Facility Becomes Core of Internet in Chicago

With respect to floor loading, one large multi-tenant colocation center in Chicago used to house a heavy newspaper printing press operation, with floor loading capable of supporting many tons of paper rolls and processing equipment. This type of venue is more than adequate for any data center needs.

Gallery 12-4. A 1999 tour of the Chicago Cermack Printing Facility during the data center conversion process. These floors used to hold large paper spools and printing presses so they can easily support the weight of racks of computer and networking equipment.

Shared Internet Data Center operators must prepare for random customer equipment, while a single customer (enterprise) data center may know in advance what types of equipment will be placed in the space.

Gallery 12-5. Massive freight elevators that used carry heavy printing loads, now help load racks of telecom equipment at the Chicago Cermack data center.

There are many other issues regarding fitness for purpose, including weather (earthquakes, hurricanes, floods, etc.), proximity to hospitals, etc. Why proximity to hospitals? If a national disaster occurs, the hospital power grids will be repaired most quickly, so being near a hospital means your data center utility power feed also comes back up from a national disaster more quickly.

Gallery 12-6. Racks of networking equipment in locked cabinets. Photo courtesy of AMS-IX.

Large data centers are often constructed outside of major metropolitan areas. For example, the Docklands in the U.K. has several large-scale data centers (Figure 12-7) that house some of the largest network providers on the continent.

Figure 12-7. Telehouse, London, U.K.

3. Utility power availability and price. Dual grid feeds are desirable, but the requirement for 3,000 amps at 480 volts per 50,000 are typically required for operations. Around 2008, power availability and unit price became critical selection criteria.

4. Facilities yard. There is often a requirement for space at ground level or on the roof for fuel tank(s), generators, chillers, etc. Space required varies, but generally at least 5200 square feet for the first 50,000 square feet of facility is needed. Even when some wholesale data center providers provide reliable power (such as Telehouse shown in Figure 12-7), some customers prefer to own and maintain their own Uninterruptable Power Supply (UPS) in their own private cage.

Gallery 12-7. Next to the building is a facilities yard for backup generators and other equipment that is best operated outside.

5. Local government support. Local, state, and federal governments often provide incentives for data center construction in distressed areas. Examples include tax abatements, ease of permitting, expedited access to rights-of-way, power, and support for fiber access that might otherwise require more extensive government, corporate, and community interactions. In some cases, discussion of the number of parking spaces, off-ramp development, signage, etc. are uncovered only at a very late stage of real estate selection.

A perfect example is Terremark’s Technology of the Americas Building in Miami, Florida. The company constructed a six-floor building in a “Federal Empowerment Zone” in downtown Miami, and along with other government programs, received several million dollars in support.

Equinix received similar support for the original Ashburn, Virginia, facility when it was still a corn field. Today Ashburn, Virginia, is known as a telecommunications hub for the U.S.

Gallery 12-8. This photo shows the first fibers run into the Equinix Ashburn data center (circa 1999).

6. Location of customers and support staff. Since these data centers will ultimately be used by customers and support engineers, it is important that local talent be available. Building a data center in the middle of nowhere would make it difficult to attract and retain people to support the equipment and infrastructure in such a building.

7. Cost of real estate. The location of the IDC to a large degree dictates the cost of the IDC.

Sean Donelan made an important point about the cost of a bad choice of real estate:

“You can save a lot of money choosing the right site, but you can spend a lot of money on the site trying to upgrade it to the appropriate standards. Many of the most expensive problems to fix at the end of the build are due to the selection of Real Estate at the beginning of the build.”

Notes from the field.

I Don’t Want My Equipment In New Jersey!

When comparing IDCs in New York and New Jersey, some customers insisted on having their equipment housed in New York. They wanted their gear to have a New York address, and they also said that they didn’t want to travel into New Jersey. The latency is negligible, and the bits don’t care, but to some of these guys the address was more important than the cost of the IDC location.

There is a demographic of so-called “Server-Huggers” that care where their equipment is located for convenience. These people anticipate visiting the location fairly often and perhaps consider each location to be their personal workspace.

For a while, the area surrounding a popular carrier hotel at 165 Halsey Street in New Jersey was considered a dangerous part of town. The running joke in the peering community was that guns are not allowed in the facility, but are handed out by the security guards so you can make your way safely back to your car.

Many criteria must be evaluated with respect to suitability of real estate. For example, all others things being equal, if there is sufficient lead time, many prefer a “greenfield” (brand new) building over an existing building. Time-to-market issues may dictate retrofitting existing facilities (to meet seismic standards, appropriate bracing for floor loading, drainage systems, etc.). Sometimes, demolition and retrofitting can save valuable time to market.

Hidden charges and unexpected delays affect deployment. For example, some telco hotels require interconnection of its tenants via risers to “meet-me” rooms. Some fiber providers have complained that every splice of fiber causes signal loss and meet-me rooms lead to another splice. Further, the expense of running a cross-connect within a building and the time to delivery are not always known.

When considering the buy decision it is important to ensure sufficient space is available for expansion and sufficient time remains on the IDC lease.

Among the core systems, real estate cost is the most variable and may have the largest overall impact on the cost of the facility.

8. Large data centers. According to Roy Earle, for shared colocation facilities there is a trend away from the smaller traditional telco hotel sizes (typically 20,000 sq ft) towards the warehouse-sized facilities (on the order of 300,000 sq ft). While this model allows scaling of infrastructure, it also dramatically reduces the choice for sites. The data center build choice doesn’t suffer from this ailment since smaller amounts of space are easier to find. This effect is compounded by the fact that large warehouses suitable for data centers are typically outside of the major metropolitan areas and therefore likely far away from telco and fiber provider Points of Presence (POPs).

Fire suppression is included in the real estate model as many buildings require sprinkler systems and/or other fire-suppression technologies (dual-trigger activated sprinklers, FM-200, etc.) regardless of the application of the building.

Some large IDC operators interviewed have real estate teams that rate possible locations by hundreds of metrics; they consider these selection-criteria documents to be highly proprietary – their real estate team “secret sauce.”

Security Systems

Security includes all systems protecting the physical assets in the Internet Data Center: perimeter and in-building access control and monitoring systems as well as security guards, escorts, and physical add-ons like bulletproof glass and Kevlar perimeter walls.

If the data center will service a single tenant, then the security systems ring can be as simple as a badge-secured exterior and a sufficiently secured employee badge reader (Figure 12-8). For multi-tenant data centers, however, the security ring gets more complicated.

Figure 12-8. The security layer of the data center model.

Shared Internet Data Center security systems must handle the added complexity of access of multiple customers. Solutions range from escorted access and supervised use to free (zoned) access. This complexity may not be required if the data center is built solely for use by a single company. In both cases, the goal is to keep unauthorized staff away from the colocated equipment.

Gallery 12-9. Data Centers follow strict physical security and data protection practices. Photo courtesy of Equinix.

The cost and scaling of the security depends more upon the application than on the scale of the data center. For example, a small section of a building that is essential to the operation (like the cashier office in a casino) may be equipped with armed guards, many surveillance cameras, etc. Likewise a shared facility may require more participant cages and authentication than a single-user facility.

Gallery 12-10. Biometrics-based security might include authentication via fingerprint reader. Photo courtesy of CoreSite.

Gallery 12-11. Surveillance cameras ensure security control within the data center. Photo courtesy of Equinix.

Gallery 12-12. Biometric and hand geometry readers protect access to customer cages within the data center. Photo courtesy of Equinix.

Having said all of that, economies of scale do exist. The incremental cost is small. Additionally, the larger facilities can allocate the indirect costs of guard access gates (Figure 12-9), circulating guards, bulletproof glass, and Kevlar perimeter protection, etc. across a larger set of equipment, helping make the actual unit cost for security of large data centers lower on a per-rack basis.

Figure 12-9. Perimeter access to an IDC.

It was pointed out that customers tend to demand more security from a shared facility than they would from their own facility because many believe that collectively they can afford more than they can individually. For example, integrated environmental (temperature, humidity, electric draw) and security (surveillance cameras on both sides of racks, cage access log) information feeds may be “required” since the cost per rack is relatively low. Shared environments generally require more complex security systems.

One practical implication of security in a shared data center is physical access. When equipment must be repaired during off hours, some Internet Data Centers (particularly telephone company facilities) do not make it easy to gain access to the failed equipment in the middle of the night. Others require escorts at all times.

Gallery 12-13. Teraco (Johannesburg, South Africa) perimeter security includes electrified fence.

After the real estate is fitted with a security system, it is secured and inherently more valuable than plain real estate. Next we will increase the value by adding conditioned power and air to the model.

Power Systems

With the addition of the Power Systems ring in the data center, the real estate is more valuable (Figure 12-10). It isn’t just square feet with fire suppression and location – it also has conditioned power, and can independently provide power during disruptions of the electrical feeds into the building (brownouts, blackouts, power spikes, etc.).

Figure 12-10. The reliable Power Systems ring of the data center model.

Gallery 12-14. Backup power generators power data centers in the event of a utility outage. Photo courtesy of Equinix.

Gallery 12-15. This San Francisco UPS system handles the full data center load until the generators can get started. Photo courtesy Jonathan Lassoff.

Gallery 12-16. CyrusOne’s Cincinnati location features 6 utility power feeds - Photo courtesy of CyrusOne.

Gallery 12-17. Generators are sometimes operated indoors. They are very loud. Photo courtesy of Equinix.

Gallery 12-18. This CyrusOne generator yard could provide enough power for a small city. These gray enclosures help contain the generator noise. Photo courtesy of CyrusOne.

Gallery 12-19. This oversized UPS room in CyrusOne’s Lewisville facility provides for expansion over time. Photo Courtesy of CyrusOne.

Gallery 12-20. Uninterruptible Power Supplies (UPS), here configured as strings of batteries, support the power load in a data center while the generators come online. Photo courtesy of Equinix.

Notes from the Field.

Nigeria and Backup Power Systems

When running a peering workshop in Nigeria I noticed that the power outages occurred several times a day and that many office buildings seemed to be running their own generators constantly. I asked how often the generators kicked on and I was told that the generators are always on. They used the generators for every day use and used the city power grid only for backup, exactly opposite from every other region I have studied.

I asked why the power systems were not upgraded, since surely there would be a massive group of Nigerians that would lobby for a reliable power grid. I was told that there was an even more powerful lobby of people who sold generators, repaired generators, and provided refueling services.

At the same time, Nigeria is undergoing a massive rebuilding process complete with modern skyscrapers. No doubt it is likely only a matter of time before these power systems are upgraded as well, but I still found this story and this dynamic fascinating.

For some customers, the availability and price of power were key selection criteria when selecting between colocation companies.

HVAC Systems

Next we add to the powered data center a Heating, Venting, and Air Conditioning (HVAC) system. Now the facility has fully conditioned space and power suitable for reliably hosting telecom equipment (Figure 12-11). This property now is much more valuable than plain real estate.

Figure 12-11. Heating, Venting, and Air Conditioning ring in the data center model.

HVAC systems are critical for maintaining an operations environment for networking and computing equipment. According to IBM, computing machinery operates best between 18° C (65° F) and 29° C (85° F) with humidity between 20% and 62% relative humidity. Too much humidity can cause system failure, while too little humidity generates static electricity. IBM recommends an optimal temperature of 24° C (76° F) and 45% relative humidity.

HVAC infrastructure provides this conditioning. HVAC systems also benefit from economies of scale. Large-scale IDCs can utilize high-efficiency (and high-cost) HVAC systems and share the costs across a potentially large number of customers. These larger systems have the added benefits of requiring fewer units to perform the same task, resulting in higher Mean Time Between Failures and lower Mean Time To Repair.

Gallery 12-21. Looped cold water system provides ample heat rejection for the entire datacenter while maintaining concurrent maintainability. Photo courtesy DuPont Fabros Technology.

Gallery 12-22. CyrusOne’s Houston facility’s chilled water system - Photo courtesy CyrusOne.

Gallery 12-23. Water pumps. Photo courtesy Equinix.

Gallery 12-24. These evaporative cooling towers are used to cool the water system while a thermal energy storage tank provides backup cooling and make up water in case of emergency. Photo courtesy DuPont Fabros Technology.

Gallery 12-25. Centrifugal water chillers remove heat from the datacenter floor. Photo courtesy of DuPont Fabros.

Data Center Operations

This infrastructure requires data center operations staff.

Gallery 12-26. Data Center Operations Center. Photo courtesy of CoreSite.

Gallery 12-27. Operations staff manage power and network cables between racks using wiring trays. Photo courtesy of Equinix.

Gallery 12-28. In a raised floor environment, the under-floor cooling space is often also used for power and networking cabling.

Gallery 12-29. An overhead wiring tray of fiber cross connects provides monthly recurring revenue for colocation operators and almost unlimited bandwidth between interconnection partners. Photo courtesy of Equinix.

Gallery 12-30. The author pointing to the first fibers entering the first Equinix Ashburn data center (circa 1999).

Notes from the field.

Even Small Data Centers Require Effort

A foreign telco built into and established its own POP in Los Angeles. At the time of my visit, the telco country manager was also the POP facilities manager and was dealing with a weekend UPS failure. The manufacturer service technician was scrolling through the alarms to troubleshoot the failure, and the detailed description of the root cause was amazingly cryptic. The root cause came from only a single piece of equipment in only one of the rings. This experience highlighted some the specialized knowledge required at each ring of a professionally managed data center. Having a support team that can monitor and repair, and coordinate and maintain support will lead to a shorter outage that will not likely happen again – at least not in the same way.

Perhaps most importantly, when things break, and they will, how will the IDC handle the situation and how will it communicate with its customers?

Notes from the field.

The 111 8th Street Story

The 111 8th Street building outage illustrates the three rules of IXP outages:

1. Things break.

2. Everyone recognizes rule #1.

3. How the IDC keeps IDC participants informed is more important to the participants than the outage itself.

This 111 8th Street story focuses on the extraordinary chain of events that led to a 24-hour+ outage. I share this story with the hope that we learn from it the lesson that notification and continual updates matter a lot to this community. George Santayana, who, in his “Reason in Common Sense, The Life of Reason, Vol.1,” wrote, “Those who cannot remember the past are condemned to repeat it.” And, in fact, the Internet operations community has repeated the failure of notification on a continual basis. Here are the broad strokes of the 111 8th Street notification story shared with DrPeering anonymously...

The 111 8th Street Outage

During the New York City blackout in August 2003, the 111 8th Street building (housing many major telecommunications company facilities) lost power. As designed, the UPS took over the building load and the generators on the roof kicked on. A crude picture of the building is shown in Figure 12-12.

Figure 12-12. During the NYC blackout, the backup power systems failed during the extended 111 8th Street outage. The cascading failures were forgiven, but the notification failures were not.

On the roof of 111 8th Street are a handful of generators fueled by a relatively small (maybe 500 gallons) diesel tank, refilled by a powerful below-ground fuel pump attached to a couple of 50,000 gallon diesel tanks safely stowed under the building. The system was designed so that when the automatic transfer switch kicked the building from the utility power feed over to the building power system, the fuel pump would also kick on to continually top off the rooftop diesel tank. And here is where the first failure occurred.

When installed, the below-ground fuel pump was tested, and sure enough, it turned on and seemed to start pumping fuel. The problem was that the polarity was reversed on the pump, so instead of pushing fuel from the underground tanks to the roof tank, it was accidentally rigged to pull fuel from the roof down to the underground tanks! When the city power was cut, the underground fuel pump indeed powered on and made noise, but did exactly the opposite of what it was supposed to do. It started draining the 500 gallon rooftop fuel tank.

During the blackout, all seemed fine at the IXP (which we will call X from here on) until the first generator on the roof stopped running, and then the second, and then the third. The tenants of the building who didn’t have their own private power systems were told that there was a problem with the building power system. IXP Operator X notified its participants at this point that there was a power problem and the building owner was checking it out.

Another generator kicked off. Only two generators were now left operating on the roof.

IXP Operator X sent a note to its customers suggesting that the building was down to its last generators and that customers should shut down all unnecessary gear at the facility.

So far, so good.

However, it turned out to be the last communication most of the customers received from IXP X. The last of the generators kicked off shortly thereafter and most of the tenants of 111 8th Street were left completely without power.

The building electricians figured out that the roof tank was empty and the underground fuel pump was indeed running but pulling fuel down from the roof. They tried the polarity reversal test and the roof tank started filling up.

The roof tank filled up. But the generators were not starting up. Why?

During the hours that all of this activity was taking place, the generator starter motors were continually trying to restart the generators. The starter motors burned out! It took many hours to get the replacement starters delivered through the partying streets of New York City.

Once the new starters were installed, the generators cranked, but the generators still would not start. Why?

At the bottom of the 500-gallon roof tank was diesel sludge that had clogged the fuel pumps that fed the generators. It took more hours to get the tank cleaned and replacement fuel filters delivered through the partying streets of New York City.

Once the new fuel pumps were installed (and the sludge cleaned out of the fuel tank and fuel lines), the generators finally started up.

This process took many hours to unfold. And no notifications were sent to customers after IXP X lost power – radio silence lasted for over 24 hours of downtime.

During the outage, the building folks were probably scrambling to diagnose and fix the problem and perhaps not updating the tenants who depended on the building for power. Perhaps the building folks updated IXP Operator X periodically. We do not know. But what we do know is that IXP Operator X did not keep its customers updated during the outage, so the customers did not know if their equipment would be powering back up in a few minutes, hours, or days, and they could not prepare. ISPs told DrPeering that they can understand that failures occur, but the fact that there were no updates for an outage that ultimately lasted over 24 hours was maddening.

Lessons we should learn:

This scenario could have happened to any exchange point operator where power is out of its control. Indeed, unanticipated failure scenarios have happened to every data center company. What we should learn is that what seems to matter most to the participants is how the company responds to these events.

The part that was under IXP Operator X’s control was its communication strategy, and in this scenario it failed completely. The ISPs interviewed were very understanding about the outage itself, but were very upset about the notification failure.

The way the system failed also demonstrates cascading failures. It was the sequence of the power outage, followed by the fuel pump polarity issue that led to the starter motor failure and the fuel pump clogging issues. If the fuel pump hadn’t emptied the roof tank, the other two failures may not have occurred, at least not at that time. It was the cascading effect of these failures that made this outage an extended physical plant outage.

The N+1 electrical and mechanical system redundancy did not help.

Some would argue the root cause here was testing. Others would argue the root cause was lack of emergency preparedness by the IXP.

As valuable as a data center is at this stage of development, with no way in or out, it is an island, so we will next add network to the data center.

The Networked Data Center

Data Centers can be operated by several different types of players, including carriers, ISPs, providers of colocation services, and completely neutral entities. This section will help you understand some of the differences between them, particularly with respect to the U.S. vs. European IXP models.

Carrier Data Center

As stated before, without a network, a data center is an island. If a carrier owns the data center, we call it either a:

• Carrier Point-of-Presence (POP) or

• Carrier colocation space

The carrier ring in the model (Figure 12-13) demonstrates the ability for multiple competing carriers to offer connectivity within a building.

Figure 12-13. The carrier ring of the data center model. Multiple carriers selling services, or just one?

Definition: Carriers are layer 2 transport providers.

When we say carriers, we are referring to the folks that bring in layer 1 and layer 2 – dark fiber, lit fiber services, and circuits for the tenants to use to get bits into and out of the data center.

Most carrier data centers do not allow competitive providers to sell transport to their customers in their space. Some companies have referred to carrier data centers as “telco jails” since they can’t get their content out of the building unless the carrier data center owner sells them transport services.

The Carrier Point-of-Presence (POP)

Definition: A Carrier POP is space within a carrier data center that is operated solely for the use of the carrier itself.

A Carrier POP is a relatively small footprint in a data center. This space is a data center space with only one customer – the carrier itself.

The Carrier Hosting Facility

Definition: A Carrier Hosting Facility is space within a carrier data center that is purpose-built for the use of hosting the carrier and its customers.

Carriers may also lease data center space from others and use it for a POP or hosting space.

Carrier-Neutral Data Center

Definition: A Carrier-Neutral Data Center is a data center that is neither owned nor operated by a company that offers carrier services.

A Carrier-Neutral Data Center does not compete with its carriers for transport (circuits) and, therefore, carriers generally do not have a disincentive to build in and support this business model. The Carrier-Neutral Data Center brings in multiple carriers for the customers to choose from, creating an open market for transport services in the building.

Notes from the field.

Specific Carrier Required

Some customers insist on being in a building with multiple carriers for redundancy and resiliency reasons. Some customers have dollar volume terms with certain carriers so they can more easily build into a building that has a certain carrier present.

Notes from the field.

Denver Rivals Won’t POP Each Others Facilities

Level 3 and Qwest are both based in Denver, Colorado, and are huge rivals. To quote a high-level Qwest executive, “there is no way we will build Qwest fiber into a Level 3 Gateway [Level 3’s name for its data center], and there is no way we will allow Level 3 fiber into our Qwest CyberCenters [Qwest’s branded data centers].” Thus, the Carrier-Neutral Data Centers serve a market that insists upon carrier neutrality and diversity.

What happens when a carrier acquires a Carrier-Neutral Colocation company?

Notes from the field.

When Carrier-Neutral Colo is Acquired By Carrier

Terremark hosted a NANOG meeting in Miami shortly after its acquisition by Verizon was announced. A representative made a few statements during the opening of the meeting and during the peering track that Terremark was to continue operating as “business as usual.” He suggested that there are many new Terremark IXPs to be built out with the same business model as before.

Will this scenario work? ISPs and carriers have purchased carrier-neutral colocation facilities before, so there are some historical precedents to consider.

In 1999, AboveNet acquired the Palo Alto Internet Exchange (PAIX) of Palo Alto, California, and the community raised great concern of the form:

“The PAIX is no longer neutral.”

The rumblings continued that ISPs would move out and that AboveNet would leverage the ownership of the PAIX to its advantage. People suggested that a new neutral exchange point would arise and take over as the peering hub for the Bay Area. Others suggested that the ISPs would not expand their presence there.

However, none of these things actually happened.

The PAIX has changed hands several times since then without significant disruptions to the business. The PAIX continued to be operated as it always did, and continued to provide a quantifiable peering value to its customers.

All IXPs go through the network externality “S” curve as shown in Figure 12-14, and the PAIX in Palo Alto was already well past critical mass – the value of peering there greatly exceeded the cost of building in and operating peering infrastructure at the PAIX. The carrier neutrality “principle” was not a strong enough force to make people move out and give up that derived peering value (where peeringValue=valueDerived-costOfPeering).

Figure 12-14. The value of an Internet Exchange.

So if history is any indication, Terremark’s facility in Miami will probably continue to contain a strong ecosystem moving forward.

However, the new Terremark buildings present a different situation altogether. Even if the Terremark buildings are operated as a completely separate Verizon subsidiary, building in ultimately supports the parent company, Verizon. Since the new buildings have to go through the exact same “start-up hump” that all colocation facilities and IXPs have to go through (where the value of participation is so much lower than the cost of participation), they will have a harder time attracting the critical mass of carriers and ISPs.

The Terremark neutrality message will also no longer ring true, so all of the heavy lifting needed to attract ISPs and build the value proposition will be much more difficult. After all, might Verizon get to pitch connectivity first to the new Terremark colocation prospects? Might Verizon get to peek through the tour and visitor logs to find prospects for bandwidth? Wouldn’t competitors prefer to see Verizon resources drain as a result of an over-built and unpopulated colocation facility? These questions are examples of concerns raised when carriers compared building into ISP-owned data centers against building into an open market Carrier-Neutral Data Center.

History has a way of repeating itself.

So most likely, Terremark colocation facilities built from now on will not be perceived as carrier or ISP neutral and will fail as colocation or IXP locations. They will probably be very-well-built data centers, though. I just don’t see the success in Miami being duplicated elsewhere with this parent.

Internet Data Center

Definition: An Internet Data Center is a data center with Internet access.

ISP Data Center

An ISP Data Center is an Internet Data Center that is owned by an ISP. It has one or more carriers providing transport into the building, but being owned by an ISP, generally allows its customers to access only its own Internet Transit services. There is no open market for Internet Transit services here (Figure 12-15).

Figure 12-15. The litmus tests differentiate data center business models.

Internet Service Provider POP

Definition: An ISP POP is space within a data center operated by an ISP that is solely for the use of the ISP itself.

An Internet Service Provider POP is where you will find routers, out-of-band access equipment, tributary routers, etc. that support the operation of the ISP network. An ISP POP supports exactly one customer – the ISP itself. If it supports more than itself, it would be categorized as an ISP Hosting Facility.

An ISP Hosting Facility

Definition: An ISP Hosting Facility is space within a data center that is operated by an ISP providing conditioned space and power along with Internet Transit services for its customers.

An ISP Hosting Facility is where you will find space for colocated customers (shown as lines across the outer ring in the model shown in Figure 12-15). There is typically only the ISP providing Internet Transit services to the population, so we show that as a solid, single ring.

An ISP and Carrier-Neutral Internet Data Center

Definition: An ISP and Carrier-Neutral Internet Data Center is a data center that is operated by an entity that neither competes with nor offers transport or Internet Transit services that compete with the carriers or ISPs.

As a result of this neutrality, the ISP and Carrier-Neutral Internet Data Center is modeled as shown in Figure 12-16, with multiple carriers (shown as the ring with lines separating the various carriers) and multiple ISPs (shown as the outermost ring with lines separating the various ISPs) providing an open market for transport and Internet Transit services.

Figure 12-16. The ISP ring in the data center model.

Notes from the field.

The Turkey Story

I learned firsthand from an Internet Service Provider in Turkey why neutrality is an important characteristic of Internet Data Centers marketed as colocation centers. He shared with me the following story.

There were really only three Internet Service Providers in Turkey, and the Internet traffic exchanged between them traversed the Atlantic Ocean (twice!) before reaching each other’s customers! This scenario was a perfect place to apply Internet Peering, a direct business relationship between competitors whereby they each freely exchange access to each other’s customers.

But there was no IXP in Turkey, so there was no Public Peering ethernet switch, and no separate commercial or association-based entity to oversee an IXP.

“No problem.” One of the ISPs “volunteered” to buy the switch and host it at his data center. He even agreed to provide free rack space for the other competitors in his data center for the purposes of Internet Peering, a free and reciprocal exchange of access to each other’s customers. The other two ISPs agreed, and they built into his data center, and the three ISPs peered with each other. Turkish traffic stayed in Turkey, so all was well in the Turkish Internet Region.

The traffic volume grew as a side effect of lower latency and lower packet loss. The Internet Region blossomed, and the volume of traffic continued to grow at a healthy pace. The importance of the Turkish Internet Exchange, owned and operated by the ISP that volunteered, was recognized by everyone (regulators, ISPs, customers, content providers, etc.). Everyone celebrated the success of the Turkish Internet Region.

Then, the volunteer ISP started touting his ISP as being the “center of the Turkish Internet,” pointing to the amount of traffic and the prestige it had gained from operating the IXP. As a side effect, it started getting increased market share, reinforcing its market positioning that it was the center of the Turkish Internet.

The other two ISPs did not like this situation at all but could not pull out from the Internet Exchange and go back to having the traffic traverse the Atlantic Ocean twice again.

This tale is a cautionary tale of why an IXP should be run by a neutral third party, one that does not compete against the participants.

The inherent value of this facility is proportional to the number, diversity, and desirability of the wide range of transport and transit services, along with the capability to peer with each other.

Another key value of the carrier-neutral IDC is the open market for Internet Transit services. The ISPs can participate in the marketplace for transit services without any market distortions. Here again, an ISP-owned data center is unlikely to have competitors in its building selling transit to its customers, but a colocation center will tend to have a variety of ISPs forming the open marketplace for transit services. Some in the community have estimated that prices for transit could be 30% less in well-populated colocation centers. The robust marketplace for transit and peering are the key value propositions that will bring additional customers in.

IDCs for Hosting Content

The outermost ring of the data center model is for content companies that take advantage of the network. Here we have the hosting companies, portals, CDNs, Content Providers, etc. These participants tend to prefer a richly networked IDC with an open marketplace of ISP services, providing the required flexibility, robustness, and ever-decreasing transit prices.

Gallery 12-31. Content providers such as Google are increasingly building and operating their own custom data centers. Here we see a Google server room in Council Bluffs, Iowa. Photo by Connie Zhou, courtesy of Google.

Colocation vs. Wholesale IDC

Some data centers are carrier-neutral and ISP-neutral, but are built with the primary intent of reselling or leasing large chunks of that space to others on a wholesale basis. This model is different from the colocation companies that focus on building the “right” population that maximizes the interaction value to its customers.

Gallery 12-32. Wholesale floor space provides a completely customizable and highly redundant platform. Photo courtesy DuPont Fabros Technology.

Gallery 12-33. Raised floor data center space can be customized for wholesale or retail consumption. Photo courtesy of CyrusOne.

One way to differentiate these two categories is with the litmus test – does the operator actively evangelize interconnection among a large population of participants? If the answer is yes, then we have a colocation company. If the answer is no, then we have a wholesale data center (Figure 12-17).

Figure 12-17. The litmus tests differentiate neutrality.

The key differentiator between providers of real estate and providers of colocation space is the managed ecosystem. Colocation is more than a real estate play – successful colocation providers understand their customers’ ecosystem, and build the right mix of players based on how customers interact with and derive value from each other.

A common question follows:

“Given enough money, can a competitor reproduce what a well-populated colocation provider has?”

It is possible, but very difficult because the value of the IXP is proportional to the number and diversity of the population there. Since these players are already in the building and presumably getting value from the presence, it will be difficult to get them to move or build in to obtain the same value. The population is a long-term sustainable differentiator for well-populated colocation centers. It is interesting that the main value of the colocation center has nothing to do with the physical data center (which can be reproduced) but has everything to do with the population.

Colocation Providers

Notes from the field.

Carrier and ISP and Colo Neutrality

When I first introduced Equinix to the European community at a RIPE meeting, they said, “But you aren’t neutral.”

“What do you mean? We are a carrier and ISP-neutral colocation center.”

“You aren’t “colo” neutral since you operate both the peering fabric and the colo. Your customers do not have a choice in colo providers.”

There are generally two models for colocation providers – those that also operate a peering service (also known as the “U.S. Peering Model”) and those that house a peering service that is operated by someone else (also known as the “European Peering Model”). This litmus test is shown in the taxonomy drawing in Figure 12-18. I also like to call the European Peering Model the “LINX” model because this model was developed by the LINX and mimicked across Europe from there.

Figure 12-18. The U.S. vs. European IXP models.

The European IXP Model

The European model IXP is modeled in Figure 12-19.

Figure 12-19. The European IXP Model.

The European IXP Model has the follow characteristics:

• The European peering community points with pride at the European IXPs as being both carrier and colocation neutral, meaning that participants in Europe can select which of the potential colocation operator providers offers the appropriate combination of services and cost points that they need. They can connect to the IXP from whichever IXP colocation site they like.

• European IXPs don’t usually have customers, they have members. They are often formed as not-for-profit associations (44%) founded by members that are also usually the first ISPs to connect. The other common form of European IXP organization is that of a service offered by an academic or regional development organization (42%). In either case, the focus is on providing mutual value, sharing the costs, and having some degree of community oversight over what they see as their organization. Even when they are not formal associations (e.g., NetNod), they tend to operate as if they were. Many European IXPs also have activities outside the mere operation of the IXP, including political, legal, and community outreach and research activities.

Gallery 12-34. The European model IXPs are often distributed across a handful of colocation providers using Wave Division Multiplexing technology shown here. Photo courtesy of AMS-IX.

Gallery 12-35. The ethernet public peering switching fabrics are similar across Europe. Photo courtesy of DE-CIX.

Notes from the field.

European Model IXP Blessing and Curse

This association structure for IXPs has proven to be both a blessing and a curse. It helps get the IXP launched and builds a population; since the founding association members share in the benefits and the costs equally, they all can exert some pressure to get additional ISPs into their IXP. However, the “same pricing for all structure” model also limits the use of discriminatory pricing that could otherwise be used to lure in strategically important peers. This ability to drop the costs of peering for the right key peers is especially important during the early stages before an IXP reaches critical mass, and the value of peering there is less than the cost of peering there.

• European IXPs have led the charge on Public Peering traffic volume. These guys are on the bleeding edge pushing over 1 Tbps of peering traffic.

• European colocation providers don’t get recurring revenue for cross-connects. In the early days, ISPs could run their own intra-building cross-connects. Later, a small one-time fee was extracted. Cross-connects in 2011 were primarily inexpensive and one-time expenses in Europe.

• Pricing in Europe for IXP services tends to approximate a cost-based pricing model. This approximation is reflective of the not-for-profit form of the business; if a not-for-profit makes too much profit, its not-for-profit status is in jeopardy. The cost of peering pricing declines, making it easier for others to justify joining the IXP, thereby increasing the economies of scale and allowing the operator to decrease the prices further.

• European IXPs are distinct from the U.S. counterparts also in that they cooperate. There is a long history of European IXPs sharing operations notes, growth trends, additional service offerings, pricing information, traffic statistics, and customer status updates at public Internet operations forums like RIPE, APRICOT, and NANOG. Now let’s compare the U.S. IXPs across the same dimensions.

The U.S. IXP Model

IXPs in the U.S. are largely for-profit commercial operations run by the colocation companies that house them (Figure 12-20).

Figure 12-20. The U.S. IXP model.

The U.S. IXP model has several characteristics.

• Commercial IXP companies generally have the ability to apply discriminatory pricing to lure in the right players, meaning a U.S. IXP can offer free colocation space, free switch ports, discounted cross-connect, etc. for ISPs that will help lure in other ISPs into the building.

For example, when Equinix Ashburn was launched and there were no tenants, public debt filings show that WorldCom was incentivized to come in with stock warrants. In Los Angeles the Any2 IXP was free for tenants in 1 Wilshire, and in Miami the NOTA IXP offered free peering ports to all tenants. The ability to incentivize the right ISPs into the IXP is a tool that the U.S. IXPs have and use, whereas the European IXPs have the same posted prices for all.

• The pricing in the U.S. for IXP services is the price the market will bear. This model is reflective of the commercial for-profit form of the U.S. IXP business. A for-profit company seeks to maximize shareholder value. As a result, U.S. exchange points are priced quite different from the European exchange points counterparts, whose prices are more closely tied to the cost of operations.

• The focus within U.S. IXPs tends to be more towards revenue-generating activities. While there is much peering outreach done here, it is motivated by revenue generation.

• The U.S. IXPs have customers, not members. Customers pay, they don’t vote. Decisions are made by the commercial IXP operator, not by the population. Next, a soft point. You tend to see that U.S. IXPs are more likely to compete with one another than cooperate. The exception to this general rule is when the customer base insists that the different IXPs work together. This situation occurred, for example, when each U.S. IXP started running its own peering forums. The customer base, faced with constant travel to so many peering events, insisted that the IXPs pool resources to run a single group event, which became the Global Peering Forum. It has turned into a singular must-attend event for U.S. peering coordinators who were otherwise splintered across many different events, so everyone won with this evolution.

• The U.S. IXP Public Peering traffic volume is much lower. In the U.S. there is much more Private Peering than Public Peering. The U.S. IXPs are primarily venture capital-backed, purpose-built, secure 100,000-sq.-ft. meet-me rooms facilitating thousands of Private Peering connections.

Gallery 12-36. Public peering fabrics can be extended across a campus and support a variety of technologies and speeds. Photo courtesy of Equinix.

Gallery 12-37. Private peering using cross connects within colocation facilities remains the dominant form of interconnection in the U.S. Photo courtesy of Equinix.

The differences that people point to the most often are shown in Table 12-1.

Table 12-1. The differences between U.S. and European model IXPs.

Notes from the field.

European Operations Meetings are Dainty

I found it interesting that even the way the Internet Operations meetings around the world “feel” is different.

The RIPE meetings, for example, feel more community-based. They focus a lot on sharing information and comparing notes in public. A few “working groups” focus on common problems, and they publish them to share the wisdom beyond the meetings. The U.S. counterpart (NANOG) feels more commercial and conflict-filled, with a greater focus on sponsorship money and the politics of the organization than on sharing experiences.

The coffee breaks feel a bit formal in European meetings. Coffee and tea service has a “queue.” The wait staff ask people in turn if they want coffee or tea. They may add cream and sugar for you. The coffee and hot water for tea are served from an elegant urn.

In the U.S. the coffee breaks involve wheeling in a few tables of 20-gallon self-serve cafeteria-style urns. A plastic container houses the sugar packets and used plastic stirrers – the meeting has a different feel to it.

The taxonomy diagram in Figure 12-21 will help you identify these different models in the wild.

Figure 12-21. The Taxonomy of Data Centers.

A Couple of Notes about Real Estate for the IXP Operator

Location is critically important as it directly affects the business case for peering, and therefore the likelihood of success as an IXP. At the highest level, IXP operators said that they first decide on the Internet Regions of interest. In the U.S., one of the first thoughts is to build in the “NFL” cities. The thinking is that if this city can support a major football team, then there is likely enough traffic to support an IXP.

Another critically important location issue is the existence of other competing IXPs. I surveyed several dozen ISPs and asked them how many IXPs they would like to have per region.

Half of them answered that they wanted to see only one IXP per region. That way, they could pick up all of the traffic they could there, and wouldn’t have to pay to reach the others that went into a different IXP. Keep the costs of peering down was their attitude. Some also pointed out that they preferred to handle the redundancy themselves through more interconnect points, diverse routing, etc.

The other half of the group said that they wanted to see exactly two IXPs per region for redundancy reasons. They wanted to see two different IXP operators, each hiring a different security guard company, a different hardware vendor for its public switch, a different fuel delivery company that would travel on different paths to refuel the diesel generators, etc.

And nobody wanted to see any more than two IXPs per region. That would “splinter the population” they said, “increasing the total cost of peering for the region.” If there were three or more IXPs in a region, each would probably have some distinct participants. To pick up all regional traffic in such a region, one would need to build into all IXPs at potentially great expense.

The point is, the competitive landscape for IXP operators is a critical factor for real estate selection.

High-Margin Cross-Connects

Another point must be made: To the colocation provider, cross-connects are wonderful high-value and high-margin products.

U.S. colocation providers charge customers for (typically fiber) cross-connects between two parties within their U.S. data centers. Within a cage, customers can of course run their own wires. It is when the wires cross the customer boundary that the colocation provider owns the interconnect and manages the interconnection process. Interconnects have a monthly recurring fee in the U.S.

These cross-connects are estimated to cost about $40 to run. This amount includes the cost of the fiber, the cost of the labor to run the fiber, and even a small cost allocation for the indirect costs of a database in which to store the customer information. There are lots of assumptions here, of course.

But the cost of the cross-connect is immaterial compared to the $300 per month that the U.S. colocation providers charge for cross-connects. The important question is: What is the value of the cross-connect to the participant? How high could the price get?

Let’s compare free peering using a cross-connect against the next-best alternative, purchasing Internet Transit. If one could freely peer 5Gbps over a cross-connect, and the alternative was to purchase 5Gbps of transit at $1/Mbps, colocation providers could charge as much as $5000/month for that cross-connect!

Based on this analysis, the $300/month cross-connect is quite a bargain.

This analysis has a lot of assumptions, but these assumptions are the broad strokes. In both cases (transit and peering) we assume peers and transit providers participate at the colocation center, no increase in equipment costs is incurred, and no incremental labor costs of peering, etc. are incurred. The point is that the value derived from the cross-connect is calculably higher than the price of the cross-connect.

The other comparison that is often made is the cost of a cross-connect compared to the cost of a circuit. Again, the cross-connect is a cost-effective alternative.

The profit on cross-connects is nearly 100%. A colocation provider with 5000 cross-connects generating $250/month yields $1.25M every month. Not a bad business.

It is interesting to observe that in Europe, cross-connects within data centers have always been run either by the tenants of the colocation center or by the colocation providers for a one-time fee of about $200. So when the European ISPs came to the U.S., they were surprised that there were monthly recurring charges for a piece of fiber run between participants.

Some of the European colocation operators I spoke with see Europe adopting the U.S. monthly recurring cost cross-connect model. They thought their customers might revolt. Some European colocation centers have already started owning the cross-connect process, so they seem to be inching along this path towards the U.S. cross-connect model.

Gallery 12-38. Cross-connects are high margin products for colocation companies. Photo courtesy of Equinix.

Notes from the field.

Bypassing U.S. Monthly Recurring Cross Connect Fees

Since there is a monthly recurring cost for a cross-connect in the U.S., ISPs are incentivized to “cheat” and bypass the system. How do they bypass the system? We learned several methods in the field.

In one example, an ISP realized that its peer was on the floor directly below it. During a midnight installation, the ISP drilled a hole through the concrete floor to run the cable to its peer.

In another example, an ISP paid the IXP technician several hundred dollars under the table to run the fiber without reporting it to the IXP operator.

In a final example, the ISPs leveraged an operations loophole at a colocation facility. In this particular colocation facility, when a cross-connect is ordered it is connected from patch panel to patch panel within the customer cages. When a disconnect order is placed, the cross-connect is simply disconnected from the two patch panels. This scenario allowed customers to order cross-connects, place the disconnect order, and then both parties would plug the two ends back to the patch panel during their next visit to their cage. In some cases, the colocation provider disconnected the fiber and pulled the cross-connect up into the overhead wiring trays. The ISPs would both then fish down the fibers and reconnect them for free cross-connect services.

Peering Workshop Practice Questions

Here are a few practice questions from the Internet Peering Workshop:

1. Between the U.S. IXP model and the European IXP model, which would you guess is more expensive and why?

2. Why can’t the European model IXPs price based on who the prospective customer is?

3. Which model (European or U.S.) probably has a higher operations cost?

4. Zaid Telekom, a large telco, opens a data center for its customers. They claim they will sell to anyone. They encourage their competitors to come and be customers there. This data center is best described as a

A. carrier-neutral IXP

B. U.S.-model IXP

C. telco hotel

D. carrier hosting facility

5. Snowhorn Real Estate Holdings converts a very large building into a multi-floor data center. They do not operate a network of any kind. They sell only floors and suites. This data center is best described as a

A. U.S.-model IXP

B. ISP POP

C. telco hotel

D. European-model IXP

6. Summa Corp. builds data centers across Africa and does not own a carrier or ISP network. They encourage carriers and ISPs to build in, but do not buy any services from them. This data center is best described as a

A. European-model IXP

B. carrier-neutral data center

C. U.S.-model IXP

D. ISP data center

7. Orlowski Internet Facilities builds data centers around the world and purchases bulk Internet Transit for resale inside its data center to its tenants. This data center is best described as a

A. hosting company

B. carrier-neutral colocation center

C. U.S.-model IXP

D. carrier POP

8. Jumoke Centre builds a colocation data center complete with a peering fabric in Nigeria. It sells rack space to anyone but markets itself to the ISP community as a peering point. It does not operate a network and does not buy or resell any network. This data center is best described as a

A. European model IXP

B. carrier-neutral colocation center

C. U.S.-model IXP

D. carrier POP

9. Witteman Int’l operates a shared peering fabric across multiple colocation buildings. Other than this LAN, it does not operate a network. This data center is best described as a

A. European-model IXP

B. carrier-neutral colocation center

C. U.S.-model IXP

D. ISP data center

Answers to these questions are in the answer key in the back of the book.

Chapter 13 – The IXP Playbook

“In theory there is no difference between theory and practice. In practice there is.”

– Yogi Berra

Two questions always come up in the Internet Peering Workshop:

1. How does an IXP build critical mass? Another form of this question is, how does an IXP attract the first ISPs to peer when there is no one to peer with?

2. Is it hopeless to compete against or attempt to displace a well-established IXP at critical mass?

In this chapter we will introduce the theoretical framework for the value of an IXP, and break the value equation into its contributory characteristics. The rest of the chapter will then present a set of the tactics that some of the smartest IXP and colocation operators in the world shared for establishing, growing, attacking, and defending their critical mass.

The Value of an IXP

Building an IXP is relatively easy. Building a successful IXP is hard.

What is a successful IXP?

Definition: A Successful IXP is an IXP where the value derived from peering exceeds the cost of peering.

To understand the value side of an IXP, we need to talk about Network Externality.

Definition: Network Externality is the economist term for the characteristic of a product or service whose value is proportional to the others using the product or service.

The classic example of Network Externality is a fax machine. If only a single fax machine existed, it would have no inherent value – there would be no one to receive the first fax! When the second fax machine was deployed, the first machine could exchange images with it, so the value for both machines was greater. Deployment of the third fax machine created the potential to send faxes to three parties, so the fax machine was inherently more valuable still. To generalize, as the number of fax machines grew, the value of owning a fax machine grew.

IXPs exhibit the Network Externality effect as well, as shown in Figure 13-1.

Figure 13-1. The IXP network externality effect.

Figure 13-1 shows a plot of the value of the IXP as a function of the number of participants. As more participants connect to the exchange point, more participants can peer with each other. From the point of view of the participants, the value of the exchange point increases with each potential peer. The tough part is getting those first participants in, a phase called the “Startup Hump.”

The IXP “Startup Hump”

An IXP must somehow attract those first participants when there is no one else to peer with at the IXP. The “IXP Startup Hump” is where the real heavy lifting occurs, because it can sell only the concept of a successful IXP, without the immediate demonstrable value. Every IXP has to make its way through these first sales gauntlets.

When at first two ISPs at the IXP peer with each other, they both derive some value from the IXP. Their traffic is exchanged directly and perhaps redundantly, and it is typically settlement-free. This traffic may bypass a more expensive transit alternative. Perhaps the traffic is between two Tier 1 ISPs that find it more cost-effective to interconnect using cross-connects in a building instead of point-to-point circuits. In any case, when these two parties peer with each other, they both derive value from their participation at the IXP.

When the third ISP joins the IXP, three ISPs can potentially peer with each other. As a result, the value of the IXP to the participants increases. To generalize, the more populated the IXP, the more traffic that can potentially be exchanged, and therefore, the more valuable the IXP.

The rest of this chapter deals with navigating through this initial phase.

Notes from the field.

The Equinix Start-Up Hump

To demonstrate the Startup Hump, consider Equinix, whose first customer was an electronic greeting card company called “Blue Mountain Arts.” At the time, our first building in Ashburn, Virginia, had just finished construction and had no carriers, no ISPs, and no content companies. It was as valuable as an empty house on an uninhabited, uncharted island.

The primary contact at Blue Mountain Arts had seen my presentation at a peering event and asked if we had any more space left at the facility. It was still the boom years, and the economy was humming. Blue Mountain Arts was on a tight deployment schedule and wanted to get the paperwork started as early as possible. With this first customer we had something tangible to talk to carriers and other ISPs about – this customer represented circuit and transit sales potential. This first sale took months, but then there was a real story to tell, some credibility, and a reason to build in. Ashburn was no longer a data center island.

We have discussed the value of the IXP, but from a participant perspective, the cost of peering must be considered. An ISP may need to purchase a circuit into the IXP, lease colocation space and a port on a Public Peering fabric, purchase equipment, etc. The ISP must be able to justify these costs with the value it believes they will derive. The cost of peering is shown as a horizontal line in Figure 13-1.

Where the value of the IXP and the cost of participation intersect is a key point called the “IXP Critical Mass Point.”

Definition: The IXP Critical Mass Point is where the derived value of the IXP exactly equals the cost of participation.

Beyond the Critical Mass Point, the IXP has a provable value proposition since it provides more value than it costs. As a result, the IXP often experiences exponential growth in the number of participants.

The IXP Exponential Growth

Why does the IXP value curve show an exponential growth? The growth is exponential because there are n peers that can peer with (n-1) other peers, leading to the IXP facilitating n(n-1) connections with peers. Up to n(n-1)/2 interconnections are possible if you don’t include both sides. Since the value is so much higher than the cost of participation, it gets easier to sell and the value grows for everyone as more people join.

Life is good for the IXP when in this situation. Salespeople become order takers, the sales cycle is compressed, customers want the service, the IXP can increase prices, and ISPs will not move out. This cycle continues until the IXP runs out of capacity.

Not surprisingly, all new IXPs want to get to and past the Critical Mass Point.

The IXP Is Full

The IXP may reach a point at which it is out of capacity and can’t expand – or chooses not to expand. The number of peers in this scenario stays constant. Over time more traffic may be peered, but the inability to add additional peers results in a sustained but flat value proposition.

A Simple Calculation of the Financial Value of an IXP

We can estimate the aggregate value of an IXP to its population by calculating the amount of traffic that is freely peered instead of traversing transit. For the sake of simplicity, we assume that all traffic peered at the IXP is exchanged, as the next-best alternative, across a commodity Internet Transit at market prices.

valueOfIXP = mbpsExchanged * avgTransitPrice - CostOfPeering

Example 1. The DE-CIX passed about 1Tbps (1,000,000 Mbps) of traffic in 2010. Assume that it charges $3500 per month uniformly across its 350 participants, and that the market price for transit is $2/Mbps.

• What is the value of the DE-CIX?

• What is the per-user value derived from peering?

• Is the DE-CIX past the Critical Mass Point?

Answers

a) To calculate the value derived by this population, we need to multiply that peered traffic volume by the market price of transit, its next-best alternative.

valueOfDE-CIX = mbpsExchanged * avgTransitPrice - CostOfPeering

valueOfDE-CIX = 1,000,000 Mbps * $2/Mbps - 350 members * $3500/month

valueOfDE-CIX = $2,000,000 - $1,225,000 = $775,000/month

From this calculation we can see that the DE-CIX is saving its peering population about $775,000 per month. The value of this IXP is $775,000 per month to its peering population.

b) To calculate the per-user value of the IXP, we simply divide the value of the IXP across the number of participants. The DE-CIX per-user value, therefore, is $775,000/350 = $2214/month

From this calculation we can say that on average, the DE-CIX participant saves $2214 per month from participation.

c) Since the per-user value of the IXP is greater than $0/month, this IXP must be past the Critical Mass Point.

Example 2. As of 2010, the Kenya IXP (KIXP) had only 28 members, each paying $300 per month for a port on the switch. In this Internet Region the price of transit is high ($400/Mbps) and the amount of traffic exchanged at the KIXP is low (only a few Mbps). This scenario may lead one to believe the KIXP is not as valuable as the large IXPs in Europe where the amount of traffic peered is measured in Tbps.

a) If the market price for Internet Transit in Kenya is $400/Mbps and the value of the traffic peered at the KIXP is $40,000 per month as shown in Figure 13-2, how much traffic is peered at the KIXP? What is the Peering Break-Even point at the KIXP?

Answer

a) The value of the peered traffic is $40,000 per month, and with the price of transit at $400/Mbps we can see that the members of this IXP peer ($40,000/$400/Mbps) 100 Mbps per month.

The price of transit is so high that it takes only 0.75Mbps ($300/$400/Mbps) of free peering traffic for peering to make financial sense at the KIXP!

Figure 13-2. Value of the Kenya IXP (KIXP).

To the population that peers there, the KIXP provides ($40,000 per month/28 members) $1128 per month in surplus value on average, as shown in Figure 13-2.

Example 3. The Belgium Neutral Internet Exchange (BNIX) is located in the outskirts of Brussels. In 2010 it had 44 participants that collectively peered about 30Gbps (30,000Mbps) in a market where the price of transit is about 5 euros per Mbps. The cost of peering, ignoring the transport, colocation, and equipment costs, is 1000 euros/month.

a) What is the value of the BNIX to its population?

Answer

a) The value of the BNIX is

valueOfBNIX = mbpsExchanged * avgTransitPrice - CostOfPeering

valueOfBNIX =30,000Mbps * 5euros/Mbps - 44*1000euros/month = 106K euros

European examples were chosen here because the European IXPs freely share their public switch traffic statistics on the Internet. Why not? It helps to demonstrate their value.

Consider Table 13-1, which compares several large IXPs.

Table 13-1. Several large IXPs with the calculated (estimated) per-user derived value (Source: Wikipedia, July 2012).

We added the column “Per-Member Transit Saved@$2/Mbps” column to show that each of these IXPs is beyond the Critical Mass Point.

Criticisms of the Simple Calculation

Some people consider the IXP traffic volume calculation to be flawed in a variety of ways.

The strongest criticism is that IXP value rankings such as this one ignore the traffic that traverses Private Peering sessions. A few years ago, the LINX started migrating some of its larger traffic exchangers from public to Private Peering. This migration extended the useful life of the LINX public switch technology, but in this ranking makes them appear less valuable.

This criticism is a valid one. IXPs typically have no visibility into Private Peering sessions and, therefore, can make no claims about how much traffic is exchanged across them. They can make estimates only about the amount of traffic that is peered. The financial value of the IXP model understates the value of these IXPs.

The second batch of criticisms focuses on the underlying assumptions in the simple model. In particular, the model assumes that

a) The traffic across the switch is all peering traffic. However in reality there could be transit traffic mixed in as well.

b) The traffic is the type of peering traffic that is bypassing the transit service, but it could be Tier 1 ISP-to-Tier 1 ISP peering.

c) The next-best alternative is transit where often the next-best alternative is peering elsewhere.

d) The cost of transport, equipment, and colocation is ignored when it could be a significant component of the cost of peering.

e) The uniqueness of the routes is ignored in the model, while an ISP would not build in to receive the same routes it received elsewhere.

f) The focus is on the “average” ISP, where all ISPs are different and will derive different value.

All of these criticisms of the simple financial value of the IXP model are valid.

This discussion leads us to a slightly more complex value model, but one that is not intended to lead us to an absolute value. The next model focuses on the parameters that make up the value of the IXP. In this model we enumerate the components of the value proposition and use them to classify the manipulations that help grow the value of the IXP.

The IXP Playbook Tactics

All tactics in the rest of this chapter focus on one of the parameters of the value of the IXP model.

The value of the IXP is proportional to the number of participants (the how many), the routes available at the IXP (the who there), and the volume of traffic peered (the how much), as shown mathematically as follows:

Value of the IXP = ValueDerived - CostToParticipate

We express this more succinctly as the function (f) below:

Value of the IXP = f(p, r, v, m) - c

where on the value side:

p = the number of participants at the IXP – the “How many?”

r = the number and uniqueness of routes available at the IXP – the “Who there?”

v = the volume of traffic that is peered at the IXP – the “How much?”

m = the match of the service to the market needs and stickiness – the “Why are they there?”

and on the cost side:

c = the cost of participation at the IXP (colocation, switch ports, equipment, cross-connect, membership fees, etc.)

All of the tactics in this chapter manipulate one or more of these parameters. We will group them based on whether they increase the value side of the equation or the cost side of the equation.

Tactics That Manipulate the Value of the IXP

These tactics all try to increase the value of the IXP

Value of the IXP = ValueDerived - CostToParticipate

Value of the IXP = f(p, r, v, m) - c

by manipulating one of the four value parameters:

p: The population

r: The routes available

v: The volume of traffic exchanged

m: The market perception of the IXP

Each tactic will highlight the targeted parameter.

Tactic 1. Bluff the Size of the Population

Value of the IXP = f(p, r, v, m) - c

“Appear strong when you are weak.”

– Sun Tzu, “The Art of War”

Observation: The IXP business has a range of deceit.

Bluff carriers. Frustrated IXPs dealing with the Startup Hump often employ tactics to signal greater value to peering prospects. One of the founders of the PAIX spoke at an APRICOT conference in the late 1990s describing their startup phase; the company needed to persuade the large carriers to build in to the PAIX facility in downtown Palo Alto. The PAIX founder contacted Pacific Bell and said that WorldCom was generating significant revenue from the PAIX population and that the customer base would like a choice in carriers. Not wanting to give up business to WorldCom, Pacific Bell built into the PAIX. WorldCom, of course, was not in the building at the time, but when Pacific Bell showed up and started selling capacity, WorldCom was contacted to come in and compete. Once these two carriers were in the building, the population of carriers and ISPs was able to grow.

Bluff two potential peers. Some ISPs have used these same tactics to attract customers; they identify potential peers that a prospect would like to peer with and signal that these peers are there or “pretty close” to installing. The prospect’s desired potential peers are then also targeted with the same type of signaling in the hope that neither ISP will realize that the other isn’t already installed at the IXP. This tactic often fails, as the peering population is a tight-knit group of professionals who talk to each other for a living. These IXPs lose credibility, and they begin to experience more difficulty building sufficient trust to build momentum out of the Startup Hump.

Another form of this tactic is the signaling of broader peering participation than actually exists. When the NAP of the Americas (NOTA) first launched, its marketing literature included a “Representative Members” list that, to some, implied that this group would be among the peering population. The NOTA was developed by a consortium of ISPs and carriers, and the consortium list was used in the early days to signal support for the NOTA – but their participation was not guaranteed. As a result, some ISPs believed there were more participants in the NOTA than there actually were.

These misleading tactics are common in the IXP sector, particularly during the Startup Hump period. If there is no value to participation, then one has to sell a vision, paint a compelling view of the future of the IXP. Scraps of interest are sometimes crafted into “marketing speak” in the form of “we are in talks with XXX right now,” or “we should have XXX in by the end of the quarter,” or other such noncommittal phrases. The problem is that the “well-chosen marketing speak” is often repeated by newly hired sales people, but without the same precision, leading to charges of misrepresentation.

Tactic 2. Build a Network Umbilical for Later IXP Migration

Value of the IXP = f(p, r, v, m) - c

Many new data centers have entered the market competing against the premium network-dense colocation centers. These new entrants are unable to charge the premium prices ($2000/rack) because they do not provide the value or sufficient differentiation. The highly competitive commodity data center market has been described as “trench warfare” competing for $400–$500/rack deals. At these prices, some customers are splitting their network deployments between the premium colocation and the commodity data center, as shown in Figure 13-3.

Figure 13-3. Network Umbilical Pitch.

In this tactic, an upstart colocation provider builds its critical mass by starting as a data center, but positioning itself as a network umbilical (colocation extension) site. This data center company pitches its data center as a cheaper place to host the larger server deployments and edge routers. The customer keeps its core routers at the premium network-dense colocation site, but deploys the rest to the discount network umbilical site. The fiber between the two data centers is paid for by the cost savings of hosting the large number of servers at the cheap site.

Consider an example where the dark fiber between the two sites costs $15,000/month. Since the difference between the premium and the commodity colocation centers is $1500/rack per month, the customer has to move only 10 racks to the commodity colocation center for the split deployment to pay for itself. Since many customers have hundreds of racks at the premium sites, the cost savings can be significant.

Once the network umbilical site has enough participants, it has a roster of impressive peers that can help it make the transition to being a peering point. While some companies may not have their core routers at the extension site, some will. In any case, the roster of participants is impressive, helping build the credibility to establish a new IXP. Over time, features in the extension sites such a greater power density and a growing peering population may help the commodity data center to raise prices and emerge as an attractive player in the colocation market.

Tactic 3. The Group Buy-In

Value of the IXP = f(p, r, v, m) - c

When @Home went bankrupt it left in its wake a dozen or so cable companies that were left to find Internet service for their customers in thirty days! Very quickly they learned that transit was very expensive, particularly as the file sharing (Kazaa, morpheus, Napster) consumed 40% of all available transit bandwidth. I spent time getting acquainted with this community to find a clear and compelling aligned interest – they each could save millions of dollars per year if they passed this traffic between each other in peering relationships at an IXP.

As a group, the North American Cable companies converged on an IXP, providing that IXP with a huge gravitational pull. The cable companies were “Open Peers,” so generally speaking, they were willing to offload as much traffic as possible with anyone at the IXP. This group buy-in tactic helped build a strong critical mass and increase the value for the IXP populations in which they participated. This tactic, shown graphically in Figure 13-4, can be a very effective startup tactic.

`

Figure 13-4. Group Buy-In.

Also, most European IXPs were started by a group of ISPs that recognized the shared benefits of IXPs and organized them as formal not-for-profit associations. This ownership and stewardship model is an effective way to build an IXP to critical mass. The value of the IXP is immediately proportional to the value of the routes available from these founding members.

Tactic 4. Buy-In Key Players

Value of the IXP = f(p, r, v, m) - c

If the promise of peering value is not sufficient to attract key peers (those with name recognition, unique routes, “Open” Peering Inclinations, large volumes of traffic available for peering, etc.), added incentives can be used to bring these select players in (Figure 13-5).

For example, in 1999 Equinix issued warrants for stock to WorldCom to help offset the cost of and expedite the building in transport and transit capabilities (see public debt filings for details). The rationale was that WorldCom could then participate in the upside as the stock appreciated as a result (in part) of WorldCom’s participation.

Figure 13-5. Buy In Key Players.

When considering this option, one needs to calculate the cost to “buy in” participants and estimate the value of their participation. Will they provide credibility? Will they peer openly? Will they allow their name to be used to draw a peering crowd? Also, one must manage the perception of their peers – the peers that may otherwise believe that they too should partake in the upside from their participation. Finally, one needs to consider how long the IXP can wait to pull these players in by natural organic forces.

Note that this tactic is not generally seen in Europe or other places the European IXP model is used. In that model, the ISPs “own” the IXP and are unlikely to enthusiastically approve providing the asymmetric benefits to their competitors, regardless of the value they bring to the IXP.

Tactic 5. Divide and Conquer

Value of the IXP = f(p, r, v, m) - c

When IXP participants are sparsely peered, they may not receive the maximum IXP value, and therefore they may be vulnerable to moving to a competitor IXP. Also, when the IXP participants are unhappy with the quality of the peering service – or if they have suffered from outages, poor customer service, or price increases from the IXP – they are vulnerable to moving to a competitor. This tactic requires that the upstart IXP identify these vulnerable participants, their pain points, and their current and desired peers (Figure 13-6).

Figure 13-6. Divide and Conquer.

Next, the upstart IXP finds out how much peering traffic is exchanged and calculates how much value is derived from participation at the old IXP. Those with few peering sessions, those with small amounts of traffic exchanged, and those deriving the least value are more willing to provide details about whom they would need to peer with to make a move worthwhile. Their existing and desired peers can then be approached and brought through the same process. The information collected is invaluable as it is used to identify the cracks in the old IXP service that need to be fixed in the new offering.

The key challenge to Divide and Conquer migrations is what I call the Operations Gear Tenet:

Operational gear rarely gets disconnected.

An ISP needs to have a reason to move. At the same time, ISPs not receiving the value from the IXP that they expect can be powerful advocates for other ISPs potentially interested in moving.

In any case, this process can provide a great wealth of market intelligence on the colocation competitive landscape.

Tactic 6. Beachhead Verticals and Niche Markets

Value of the IXP = f(p, r, v, m) - c

When the dominant IXP peering population is strong, a frontal assault may be ineffective. The startup IXP would instead attack the dominant IXP where it is weak. There needs to be a selection of reasons for ISPs to select an alternative IXP. Since the dominant IXP is strong in peering population, this tactic focuses on the dominant IXP attributes that are weak. The weak points might include:

1. Weak or no service-level agreements (SLAs). Many IXPs in Europe are run as not-for-profit associates providing best-effort peering services without a budget or inclination for SLA reimbursements. Some corporations want to buy from a commercial company that provides these guarantees.

2. Constrained Private Peering. Some IXPs do not allow Private Peering or require IXP parent approval. The JPIX, for example, refused to allow a cross-connect between participants that sought to bypass the JPIX parent (KDDI) transit services. For some large ISPs, requiring the use of a public switching fabric presents a significant downside: it introduces possibilities for bandwidth stealing by pointing default (directing all traffic to the peer instead of just sending the traffic destined to the peers customers). Further, it represents a decrease in control over their infrastructure since a third party is involved.

3. Insufficient terms and conditions. European-style IXPs are driven by their participants who are less sympathetic to the unique terms and conditions required to get the large competitors into the IXP. Commercial IXPs, on the other hand, are able to negotiate the price, the duration, and most aspects of a commercial agreement.

4. Content Providers not allowed. Some IXPs (used to) not allow Content Providers to peer at their IXP. (Note that this restriction is less common nowadays.)

5. Insufficient operations support. Security issues, access issues, and time to add racks and services may also be differentiators.

6. No transit sales allowed. Some IXPs do not allow transit traffic to traverse their infrastructure. With no revenue-generation potential at the dominant IXP, an upstart IXP can build critical mass by serving this underserved market. (Note that this restriction is less common nowadays.)

7. Underserved peering market niche. An IXP can build a differentiated service along different dimensions, including the population served. For example, Japan has three large dominant exchange points, which tend to appeal to three distinct domestic peering markets.

This tactic involves identifying and addressing underserved niche markets. For example, international ISPs looking to establish regional interconnections may be best served at an international IXP. This startup IXP might offer liaison, language, and local services in the native currencies in tune with the overseas customer.

The financial services market is another example of a market with unique needs and motivations, and a distinct critical mass from the ISP peering population. Hiring a liaison from this sector to facilitate building this market segment is a way to build a beachhead.

The Beachhead Verticals and Niche Markets can best be diagramed in Figure 13-7.

Figure 13-7. Beachhead Verticals and Niche Markets.

Other niche markets might include the emerging video ecosystems markets (ad placement, settlement, etc.), new Content Delivery markets (shared caching systems, CDN cooperatives, etc.), small ISPs that can’t afford the premium colocation services, etc. Where there is a mismatch between offered services at the dominant IXP, or where the emerging market isn’t big enough to warrant attention from the big IXPs, there is an opportunity to build a beachhead.

Once the upstart IXP population is built up, the IXP understands the market and can evolve the appropriate service bundle to uniquely service this demographic. Then there is value to the IXP and others can be attracted into the IXP.

Tactic 7. Extend the Dominant IXP

Value of the IXP = f(p, r, v, m) - c

In this tactic, the upstart IXP asks to be a reseller for the dominant IXP by extending the peering fabric into its data center space. In this way, the population of the startup IXP will immediately include all of the dominant IXP participants.

In the U.S. this tactic rarely works because of the asymmetric benefits of the extension. The dominant IXP is unlikely to be willing to support an IXP that could eventually compete with it. When approached, the dominant IXP will likely make the negotiation process long and difficult.

The IXP extension may seem similar to the European IXP model as described earlier, where the colocation site is a separate company from the IXP. The difference though is the ability for the extension colocation center to offer its own peering services as well. Ultimately, the extension site wants to have the stickiness of a network dense IXP and be able to charge the premium prices for its network-dense colocation environment.

If the extension is allowed, the startup IXP can start to build its own critical mass of peers that peer privately on site while still having access to the dominant IXP customers. Once this dual-peered population grows, the upstart has a potentially superior offering to the market and can lure peers over from the dominant IXP.

Notes from the field.

Interconnections Between Tenants of the Same Carrier Hotel.

When IXPs are collocated in the same building, such as in the Infomart in Texas, interconnections between the customers of each IXP have proven problematic. One or both colocation companies may make the interconnection process slow and expensive. Both IXPs stand to lose their stickiness by removing the necessity for the other IXP customer to build in.

Tactic 8. Prevent Rogue IXPs

Value of the IXP = f(p, r, v, m) - c

This defensive tactic prevents the erosion of unique routes from the IXP by preventing an alternative Public Peering fabric from splintering the population. Here are a couple of examples:

1. European IXP colo shield. Job Witteman (AMS-IX) pointed out that the AMS-IX colocation providers are not allowed to operate their own switching fabrics. This stipulation is part of the agreement to receive the AMS-IX switches. However, a colocation provider is allowed to run cross-connects and thus support Private Peering within its colocation environment.

2. Rogue IXP within colo. Some ISPs have complained about the cost of peering within certain IXPs and have threatened to implement their own peering fabric intended to displace the “overpriced” switching fabrics operated by the IXP.

These rogue IXP efforts have failed to attract critical mass for at least two reasons. First, the colocation operator is still responsible for running the cross-connects between the providers and the rogue IXP. They are in the position to disconnect the rogue IXP. This threat is enough to give some ISPs pause. Secondly, the colocation or dominant IXP operator has access to a flow of prospective customers as a matter of course, while the rogue IXP is typically operated as an ad hoc group of volunteers. While the low price is attractive, the volunteer staff does not communicate credibility or sustainability to the prospective customer.

Tactic 9. Run With the In Crowd

Value of the IXP = f(p, r, v, m) - c

The IXP peering business is a strange one in that the value proposition is out of the control of the IXP operator. Whether an ISP will peer – and if so how much traffic will be exchanged – is out of the hands of the IXP operator. Whether that peering will last for the duration of the IXP contract is out of the hands of the IXP operator. In fact, the amount of traffic that an ISP peers, and whom they would like to peer with, is sometimes even outside of the visibility of the IXP operator. Many ISPs consider peering relationships, traffic volumes, and even the list of peering targets to be proprietary and confidential, making building the value for all participants very difficult. Since all IXPs have to work around this information blindness issue, the data collected from the field becomes even more valuable.

Observation: Only half of a peering relationship needs to be exposed to determine a peering relationship.

Observation: Ninety percent of the desired market intelligence is available only through informal contacts. No reputable marketing firms can deliver this kind of information, and there will always be information that you would like to have but will be unable to obtain.

Valuable information about the ISPs in the peering ecosystem includes things like:

• contract terms and duration for colocation, transit, transport, etc.

• ISP peering locations

• peering relationships (current, planned, projected)

• traffic volumes between players (current, projected)

• financial stability

• desirable target peers

• transit prices

• ISP POP locations

• transport prices, both current and future

• ISP future deployment plans

• peering challenges and pain points

Market intelligence gathering requires constant reconnaissance missions, including travel to networking conferences around the world, and in particular to forums that attract target peering coordinators.

Of particular interest to the IXP operator are the customer perceptions of the IXP in the peering ecosystem:

• What are the customers and prospects saying about your IXP?

• What are competitors saying about your IXP?

• What are the trends in the IXP and colocation markets?

• What are the market price points?

• How are the competitors positioned?

Engaging the customer early and often offers several benefits. First, one can collect information and be an early problem solver. The purveyor of this information in the ecosystem is positioned to help ISPs entering the market. Collecting this information highlights opportunities and discloses vulnerabilities – and presents an excuse for a personal relationship and further contacts. The appropriate meeting at the appropriate time armed with the appropriate information can make an IXP. These behind-the-scenes tactics are sometimes referred to as “Guerilla Marketing.” An IXP that fails to collect this information is flying blind, will learn of opportunities late or never, and is vulnerable to being blind-sided by a more agile competitor.

Tactic 10. Bundling

Value of the IXP = f(p, r, v, m) - c

Several forms of bundling have proven successful in the IXP sector.

IXPs with multiple exchanges in different areas can leverage a well-populated IXP to build critical mass in another location. Customer relationships can be leveraged, extensions to existing agreements are easier than new agreements with a new vendor, and operations procedures can be made consistent across all deployments with the same vendor.

Another example of this tactic was seen when the MFN-owned PAIX bundled a free PAIX port and rack space with large fiber deals.

In the UK the LINX offers connectivity to its redundant switching port as part of the sale.

And finally, both CoreSite and NOTA launched their peering services without fee when the customer purchased colocation space.

All of these methods to build the initial peering population are common ways to get the first peers into an IXP.

Tactic 11. Build and Maintain Population Stickiness

Value of the IXP = f(p, r, v, m) - c

In the IXP sector, the peering population, the volume of traffic exchanged, and the IXP stickiness are key measures of success.

Definition: IXP Stickiness refers to the forces keeping a peer at the IXP.

Interconnections are the glue that makes it difficult for peers to leave an IXP. Essentially three forces are at play here: Value-Based Stickiness, Time-Skew-Based Stickiness, and Interdependence-Based Stickiness. After getting over the Startup Hump, building and maintaining population stickiness is the most important IXP activity. We will walk through each type of IXP stickiness in turn.

Value-Based Stickiness

As described in the section “A Simple Calculation of the Financial Value of an IXP,” ISPs are deriving more value and therefore are less inclined to leave when the value of participation is greater than the cost of participation. The value could be proportional to the volume of traffic peered, the unique routes available, or the location relative to its customers. Value-Based-Stickiness refers to the desire for a peer to continue to derive the value at the IXP.

Time-Skew–Based Stickiness

When an ISP builds into an IXP, several contracts must be negotiated. Sometimes they can all start and terminate at the same times. Sometimes this scenario is impossible. There is no good time to leave when the service termination dates are disjoint, as shown in Figure 13-8.

Figure 13-8. Time-Skew-Based Stickiness.

If all contracts and peering relationships terminated on the same date, the peer would still experience the operational burden of picking up and moving operational gear elsewhere. Additionally, in the peering industry, ISPs always like to talk about growth, as that signals success. Conversely, ISPs generally don’t talk about pulling out of an IXP since that might be interpreted as lack of commitment to the region. Moving out of a colocation center is disruptive, and for this reason operational gear rarely gets disconnected.

The exception to this rule is for Large-Scale Network-Savvy Content Providers. These peers refresh their equipment every eighteen months as a matter of practice, and they merely deploy to a new site prior to decommissioning an old colocation deployment. Moving their servers is a common activity.

Interdependence-Based Stickiness

Perhaps the most important form of stickiness is interdependence-based stickiness. The more interconnections the IXP member has the more interdependence there is among the participants. This interdependence is the glue that keeps participants from leaving the IXP; to leave, they would need to turn off their interconnections and coordinate with the other parties to pick that traffic up elsewhere.

Here are several methods for increasing interdependence-based stickiness within an IXP population:

• Personal peering introductions have proven very effective at peering forums conferences, at operations meetings, and at any place where peering coordinators congregate. Not only do these introductions increase interconnections, they also increase the value of the IXP and increase information flow between the IXP operators and the ISPs.

• Conference call peering introductions. Second only to personal introductions, peering conference calls can help increase the confidence of new peering coordinators. After 4 or 5 peering introductions, peering coordinators can effectively go through the process themselves, leading to dozens of peering sessions.

• E-mail peering introductions have proven somewhat effective. Peering e-mail introductions raise the awareness of a new potential peer and provide an exchange of the relevant contact information for both parties.

• Informal socialization. Chris Quesada (formerly with Switch & Data) hosted “Peering Lunches” to facilitate the interaction among peering coordinators, both current and prospective. These lunches proved very successful and cost-effective. I have launched similar lunches and dinners at industry events and for local peering engineers with similar results.

The IXP and the peering population have aligned interests here. Both parties wish to see an interconnection mesh as broad as possible. The IXP wants stickiness to maximize revenue and reduce customer turnover. The ISPs want their peers to stay peers at the IXP, so ISPs want to see all peers derive great value from the IXPs where they participate.

Notes from the field.

Peering Introductions Appreciated

I have run dozens of peering forums and performed hundreds of peering introductions, face-to-face at conferences and via e-mail and phone calls. I have never had an unpleasant experience. Of the hundreds, maybe a couple were uncomfortable because I didn’t realize that the two parties were already in a customer-transit provider relationship, or that one of them was not receptive to any peering requests. The point is, almost all of the experiences were positive, many were successful, and all of them helped to build and maintain my relationships with the peering population.

Tactic 12. Strengthen Peering Population

Value of the IXP = f(p, r, v, m) - c

In Europe, a defense system is automatically in place; if a rival IXP approaches a founding member of a dominant European IXP, for example, the result is effectively intelligence gathering for the dominant IXP. The founding members are unlikely to move from “their” IXP, although they may be persuaded to add a presence elsewhere if given a compelling reason. The Europeans have proven to be very protective of their IXPs.

For commercial IXPs in the U.S., however, it is critical to work very closely with the peering community to build and maintain a strong alliance with the peering population.

This tactic goes beyond understanding the customers’ businesses. It is critical to understand their inclinations. This tactic is about systematically tracking the interconnections and effectively managing introductions to maximize the degree of interconnection. The litmus test of success is the degree of the mesh.

The peering map (Figure 13-9) tracks peering sessions between classes of peers. At least one side needs to share its peering session lists in order to fill in the appropriate square with a peering mark. Let’s look at the peering map.

Figure 13-9. Strengthen Peering Population.

From the peering map in Figure 13-9 we can see that a maximally peered IXP would have all Open peers peering with all other Open peers.

Those ISPs with Selective Peering Inclinations require certain thresholds to be met (traffic volumes, interconnection points, ratios, etc.) and if met, an IXP can facilitate introductions leading to peering. Sometimes Open and Selective peers will peer with each other. Selective peers may peer with other selective peers, but their policies need to be examined before the introductions are made. Some Selective peers prefer to manage their own peering and are not receptive to peering introductions, but in my experience, very few will object to IXPs providing introductions and facilitating peering. The arrows in Figure 13-9 highlight opportunities for the IXP to facilitate peering that might otherwise not happen.

Introductions to Restrictive peers are not worth doing since Restrictive peers generally do not want additional peers. Making them aware of large Selective peers as they join may be a good idea and may indirectly facilitate peering between Restrictive and Selective peers.

With the peering map filled in and maintained, the peering policies can be systematically tracked and understood, and an IXP peering manager can facilitate the introduction of prospective customers to the current peering population, helping to ensure the value growth of the IXP. The absence of such peering initiatives means managing the IXP development based on hope and luck.

To illustrate IXP peer tracking, consider a snapshot of a peering population at a well-established IXP in Figure 13-10. The nature of the participants’ interconnections are plotted across the X and Y axes, with the intersection highlighting the peering inclination and peering state.

Figure 13-10. Track IXP Peering.

Peering Map Legend

Green indicates that both parties have an Open Peering Policy or have explicitly indicated a willingness to peer with the other party.

Yellow indicates one or the other party has not indicated affirmatively or negatively, either by policy or explicitly, an inclination to peer with the other party.

Purple indicates that there is a known Public Peering relationship.

Blue indicates that there is a known Private Peering relationship.

Black diamonds indicate that a peering introduction has been made.

White means that we have no information about whether these two parties would peer.

By systematically tracking and managing the customer base and interconnections (to the extent allowed by the population), an IXP can maximize its interdependence stickiness.

Tactic 13. Demonstrate Leadership

Value of the IXP = f(p, r, v, m) - c

One way to bring an emerging IXP to the foreground is through industry leadership. This leadership can take the form of speaking at conferences, running peering forums, chairing peering events, etc. The peering community is a fairly active and social group. Being viewed as a leader increases the number of interactions and the extent of the relationships.

IXP leadership goes beyond Tactic 9 - Swimming With The Fishes and being in touch with the customers that are or will become customers. It is beyond simply observing the market landscape and how competitors behave. The best IXPs in the world have evangelists that circle the globe to promote the benefits of Internet Peering. They write white papers, document the trends and motivations, and freely disseminate the market research in the name of growing the market for peering. Following are a few concrete steps to achieve leadership, and I recommend all of them.

• Publish white papers. Research and document interesting and undocumented aspects of peering. For example, as next-generation switching technologies are deployed, writing white papers and contributing articles regarding the deployment and operational experience would send the dual message that you have an IXP, you are operating leading-edge gear, and you are contributing what you learn back to the community. This scenario engenders support in the community and provides an “in” into otherwise congested avenues into the community.

• Promote peering with your customers by your side using any communication mechanism available (newsletters, speaking at conferences together, etc). If the information is valuable to your target customers, your message will reach prospective customers and provide another reason to interact with you.

• Develop peering discussion lists. These are used not just for peering announcements but also for events, outages, etc. These lists constitute a forum to build the community and obtain valuable feedback for growing the IXP business.

• Lead peering forums. In the Equinix days I chaired the first bunch of Gigabit Peering Forums that eventually morphed into what is now known as the Global Peering Forum. By leading the forum, I had an excuse to engage the peering customers and prospects, not to sell them but to ask them to contribute their expertise to the cause. This process provided not only the collegial relationship necessary for building a community but also insights into the key pain points and issues that need to be addressed. The point is, by leading these forums, you have access to critical information that your competitors who do not run forums do not.

• Get key staff onto program committees and steering committees for operations forums. Similar to leading peering forums, joining committees provide insight into the community mindset.

• Speak at conferences and trade shows. Develop your technical staff into effective public speakers. This type of visibility, not to sell but to contribute and participate, aids in the acceptance of new IXP offerings.

• Continually build and maintain a marketing information base for peering. When you are in the field and learning about the competitive landscape, it does no good unless you capture and store that information. I used to think I could keep it all in my head, but after many mind-numbing presentations and after many conversations, the details can get very foggy. Develop a system, even if on paper, for capturing what you learn during these interactions. In this process you develop leadership on the inside of the company when you give the company actionable information.

• Communicate key milestones, and track and communicate progress to the field. The ISPs are keenly interested in learning about happenings in the industry, so come to conferences ready to lead discussions in the hallways about the key objectives and where the company is heading. Note that we are not talking about “pitching” the company but rather, sharing with the community specific numbers and information about the metrics the community tends to care about – the number of customers, amount of traffic exchanged, new sites, outage experiences and lessons learned, operations procedures instantiated, ISO 9000 certification, etc.

The leadership tactics have many benefits:

• Greater stickiness

• Better market information

• Follow-up opportunities

• Customer loyalty

• Attendance at the next event

• Salesperson calls returned

• More public speaking opportunities

In general, it is easier when others find you than for you to have to chase down others, because no one knows who you are.

Tactic 14. Purchase Data Center During Downturns

Value of the IXP = f(p, r, v, m) - c

Getting the cost of the data center as low as possible is one way to provide a lower-priced peering solution and lure customers in. One way to decrease costs is by purchasing a data center in a down market, as described in the next notes from the field.

Notes from the field.

Ray the Hoster

(This story is repeated from an earlier chapter but with a different emphasis; here the story focuses on the acquisition of data centers during an economic downturn. )

In 2001 the economy collapsed. Ray had been waiting in the wings. Rather than fretting, he seized the opportunity to buy a moth-balled data center for literally pennies on the dollar. The generators were still shrink-wrapped, and servers were half unboxed and still there. Qwest had lit the building before the building was shut down.

Ray was network-savvy enough to identify which networking hardware he could get – also for pennies on the dollar – on Ebay to get the building connected and start his hosting company. He found the right combination of gear to get things going, and contacted Qwest at the end of its quarter to negotiate a great deal on bandwidth. He is the only one I have ever heard of who was able to buy an OC-48 “leased line” and pay for how much he used it, simply telling Qwest how much he would pay each month based on his own usage measurements! What a dealer.

Over the next few years, he filled the data center with paying customers as the economy slowly recovered. He then sold the hosting company for something in the neighborhood of $40M.

I serve on a few boards, and I have found that this story helps put things into perspective, especially for those who worry about the economy and making optimistic investments during downturns.

Ray is my favorite example of being network-savvy, opportunistic, and smart (frugal).

Tactic 15. Drop Peering Cost

Value of the IXP = f(p, r, v, m) - c

When there is not enough value in the IXP to attract ISPs, an IXP operator must find a way for it to make sense for ISPs to join. One approach is to reduce the cost of peering for early-adopter ISPs to entice them into participating. This tactic is shown graphically in Figure 13-11. With this approach it will take less traffic for the ISPs to justify building into the IXP.

Figure 13-11. Drop Peering Cost.

Note some costs of peering are within the control of the IXP operator, and some are not. An IXP operator can certainly make a free peering port available, waive install fees, and even provide free colocation, but the ISPs will still have the cost of equipment, transport into the IXP, staff time to configure the equipment, legal, etc. These costs need to be justified, and even then there is a leap of faith that the IXP will grow into something worthy of participation.

Lowering the cost of peering has the following benefits:

• Easy to do

• Results in a flow of ISP names promoting the success of the IXP

• Terms are renegotiable at the end of the contract term

• Makes it easier for customers to justify participation

Lowering the cost of peering has the following drawbacks:

• Provides little or no revenue

• Sets the expectation of a perpetual free service

• Sets a low value perception because the service is free

Tactic 16. Price on Value

Value of the IXP = f(p, r, v, m) - c

An emerging IXP can take advantage of the natural tendency for ISPs to buy only if the value exceeds the costs. An emerging IXP can’t charge the same fees as a dominant IXP without providing a corresponding value. All things considered equal, the ISPs will simply choose the IXP with the higher surplus peering value.

The value-based pricing tactic is to charge a monthly fee that is proportional to the benefit derived from participation at the time the contract is signed. This tactic commonly takes one of two forms.

For the low end of the market, the IXP can sell partial-capacity peering ports at a lower cost, helping the low end of the peering market enter the IXP. As the number of connected ISPs increases, the credibility of the IXP increases and the population incrementally derives greater value. As the value of the IXP increases, the price of the peering ports can increase, as shown in Figure 13-12.

Figure 13-12. Price on Value.

A second approach is to price the ports on the switch based on the number of ISPs attached. The first ten ports would be heavily discounted (say 60% off retail price), the next ten less discounted (maybe 40% off retail price), and the next ten priced at just below market rates (say 20%). Once the “Critical Mass Point” is reached (where the cost of participation is justified), contracts can be adjusted upwards to market rates.

Ultimately these methods set the price proportionally to the value derived by the customers and help speed the growth of the peering population. Note that these price points may initially be negative, similar to Tactic 4, buying in those first few players as shown in Figure 13-12.

What Has Not Worked

The most common failure mode for an IXP is the “Field of Dreams” approach – “build it and they will come.” Eastern Africa for example is filled with ethernet switches intended to be the base IXPs for the countries. They have few or no members and no one is promoting the IXP. This path is guaranteed to fail. The “Field of Dreams” approach is almost as bad as having an IXP operated by the incumbent ISP or required by the telecommunications regulator. As shown earlier, there is a long path to build the population up to critical mass where it is valuable to the peering population.

Summary

The operator of an IXP can focus on the goal of maximizing the value of its Internet Exchange by tracking costs of peering, the price of transit, and the volume of traffic peered across its service. Figure 13-13 illustrates that value.

Figure 13-13. All tactics seek to maximize the value of an IXP.

It is also important to track the other parameters of the value equation, including the unique routes available only at this IXP, the marketing position of the IXP, and the other derivative values derived from participation at the IXP.

Peering Workshop Practice Questions

Here are a few practice questions from the Internet Peering Workshop:

1. An IXP has 20 members exchanging 600Gbps in a market where the price of transit is around $1/Mbps. The cost of peering is $24,000 per year. Is the IXP past critical mass?

2. The argument was made that Private Peering has not been counted in the value of the IXP. Why would anyone else care that more traffic is exchanged bilaterally between two peers? Why would this traffic be relevant to the broader peering population there?

3. What is the optimal value of a private-peering-only IXP with a homogeneous population of 200 ISPs, each exchanging 100Mbps with every other peer? The price of transit is $2/Mbps. Assume the average cost of building in to the IXP is $5000 per month, and the price of a cross-connect is $300/month.

4. Go visit:

http://en.wikipedia.org/wiki/List_of_Internet_exchange_points_by_size

and determine the value of the top three IXPs (by traffic volume) and the value of the bottom three. What would the price of Internet Transit have to be in order for each of these six IXPs to be at critical mass?

Answers to these questions are in the answer key in the back of the book.

Section V – Advanced Peering Topics

The next logical progression for peered networks is to extend the network across an Internet Region and into other Internet Peering Ecosystems.

In this final section we present Remote Peering, an emerging peering optimization trend that can make peering from afar more cost effective. Then we finish up the book with a high level exploration of International Peering, applying a simplified model and diagramming technique that helps peering teams develop international peering strategies.

This section addresses the following frequently asked interconnection questions:

What is Remote Peering, how does it work, and when is it a rational option?

Is there a business case for extending into a lightly populated peering point?

Should we adopt an open, selective, or restrictive peering policy in emerging Internet Peering Ecosystems?

How should we price transit?

How can we defend this ecosystem from new foreign entrants?

This section consists of the following chapters:

Chapter 14 – Remote Peering

Chapter 15 – International Peering

Chapter 16 – The Future

Chapter 14 – Remote Peering

As of June 2013, according to Job Witteman (AMS-IX) 130 of the 630 AMS-IX customers are connecting to the peering fabric via Remote Peering. During this same year (2013), Andreas Sturm (DE-CIX) said that 50% of all new attachments to the DE-CIX peering fabric were made via Remote Peering. Remote peering is no longer a fringe interconnection technique – it is an interconnection method that brings peering within the reach of smaller and new entrant network operators. Remote Peering is breathing new life into the Internet Peering Ecosystem.

In this chapter we will describe the Remote Peering service model, how it is implemented by the Remote Peering Provider, and how and why it is being adopted across the Internet Peering Ecosystem. Since Remote Peering is a hotly debated topic today, we finish up by highlighting the strongest arguments for and against using Remote Peering.

[ Full disclosure: At the time of this publication, the author is materially employed by the International Internet Exchange (IIX), an IXP operator that offers among other things Remote Peering services. ]

The Traditional Internet Peering Model

Peering usually involves selecting Internet Exchange Points and extending one’s network into associated colocation facilities using point-to-point circuits. The network operator deploys a router into each colocation facility that hosts a popular Internet Exchange Point (IXP). The network topology and associated deployment characteristics are shown in the Figure 14-1.

Figure 14-1. The Traditional Peering Model.

Here we see ISP A extending a router presence into three colocation facilities to peer at three IXPs with a variety of networks labeled “B” to “L.”

Deployment in this traditional peering model requires overcoming a few hurdles.

1) Peering deployment is time and labor intensive. The provision of transport and the deployment of equipment into dispersed colocation facilities can take months. An ISP shared that their 2013 IXP network deployment took over 6 months to implement!

The administrative process is onerous. Deploying peering to three sites for example requires contracts with up to three colocation vendors and three transport providers, requires the purchase and deployment of three customized network kits along with the associated support contracts. These activities require substantial time and engineering resources to complete.

Notes from the Field

Do you see the Light?

A Peering Coordinator for a large-scale e-commerce portal was complaining about the workflow surrounding his job. This particular late night he was trying to bring up his network kit in Hong Kong so he was on long conference calls throughout the night with the carrier, the colocation company, the exchange point operator, and the technician flown there to install equipment and check connectivity. All of this work was to achieve the goal of “seeing light” (signal) coming in from the router on the other side of the long haul fiber optic cables. He shared:

“It sounds sexy to build out into Asia, but the reality of the situation is that this is a time-intensive brain-numbing troubleshooting job. Coordinating conference calls with people in different time zones, with different primary languages and varying technical skills, gets frustrating. Every one of these deployments is different and involves different colocation companies, with different contracts, different service levels, different exchange point models and processes, and different escalation paths. With all of this off-hour operations shift work, I have totally lost my personal life.”

2) Peering deployments require capital expenses. Each deployment in the traditional peering model requires a potentially large one-time expense of a networking kit ¬— typically a router and perhaps associated equipment such as load balancers, peering switches, out-of-band access, etc.

3) Peering deployments require ongoing operating expenses. Each deployment traditionally requires the recurring monthly transport expense, colocation expense, equipment maintenance agreement, and associated internal operating overhead.

These hurdles have made it difficult to make the business case for peering. Rather than backing away from peering, some segments of the peering population have embraced the “Remote Peering Model.”

The Remote Peering Model

Remote Peering allows network operators to peer without any additional network equipment. They simply “Remotely Peer” from their existing locations using the services of a “Remote Peering Provider.”

Definition: Remote Peering (aka ‘tethering’)is peering without one’s own physical presence at the peering point.

Definition: A Remote Peering Provider is an entity that sells access to exchange points across their transport infrastructure.

Let’s consider a Remote Peering example that mirrors the traditional peering model we just described. Figure 14-2 shows ISP A purchasing Remote Peering from a Remote Peering Provider that already has established a presence at the desired colocation facilities. Through this provider, ISP A can “virtually” extend its network and peer with networks at each of the IXPs, without deploying any incremental network equipment.

Figure 14-2. The Remote Peering Model.

The Remote Peering Provider charges ISP A a fixed monthly price for each Virtual Local Area Network (VLAN) extension from the Internet Exchange Point.

Where allowed, the Remote Peering Provider bundles the IXP peering ports and cross connects to the peering fabric, further reducing the workload and number of contracts ISP A has to sign. Most European IXPs allow port resale today, and the U.S. IXPs are increasingly enabling port resale. If the IXP does not allow peering port resale, the network operator signs a contract with the IXP for the peering port and orders a cross connect from the Remote Peering Provider equipment to the peering fabric.

Note that we have described a one-to-many Remote Peering deployment as shown in the close-up in Figure 14-3. In this form, a single 10 Gbps interface can deliver potentially many 1 Gbps VLANs extended from potentially many IXPs.

Figure 14-3. The Remote Peering service interface (one-to-many model).

Remote peering also takes the form of a one-to-one Remote Peering deployment where a single 10 Gbps peering fabric is extended to a single 10 Gbps customer router port.

The Remote Peering Provider

The Remote Peering Provider deploys networking equipment to each colocation center it intends to service. There are many different technologies that may be deployed that enable the Ethernet peering fabric to be extended across the wide area network but the most common is Multi-Protocol Label Switching (MPLS).

Beyond the networking technology, the Remote Peering Provider needs to manage the variability across colocation centers, transport providers and Internet Exchange Points. This includes handling the paperwork required to connect to or resell connections to each Internet Exchange Point, provisioning and managing a mix of transport vendor services, and facilitating the relationship between the customers and the IXP and potentially even assist with peering strategies.

The Remote Peering Provider is best positioned as a partner that handles everything except the bilateral peering negotiations. The Remote Peering Provider must be neutral in the sense that it doesn’t sell Internet Transit or compete against the customer in a material way; companies generally avoid paying their competitors or reinforcing their competitors’ position in the market.

Customer Benefits of the Remote Peering Model

Simplicity. There is no additional equipment to purchase and deploy, and there are no contracts with colocation companies to house networking equipment. Instead, a single contract with the Remote Peering Provider replaces potentially many contracts for dedicated point-to-point peering circuits and potentially many colocation contracts. In some cases the Remote Peering Provider can also handle the paperwork for the remote IXP as well.

Speed. Since the process of peering is simplified, the peering deployment process takes weeks instead of months. Remote Peering allows a network operator to start peering remotely prior to a full colocation-based POP deployment. Once the network operator determines that there is enough traffic and performance improvement to justify a physical presence, he can deploy networking gear into the colocation center and transition to a physical server presence.

These benefits are in addition to the benefits of peering in general:

Performance. Content companies go to great lengths to improve performance. Amazon is on record as tying 100 ms of latency to as much as 1% of revenue loss. PayPal believes that up to 70% of payments are at risk of being aborted if they take more than 3 seconds. Google’s real-time bidding for ads model means that content companies risk losing their ad revenue if their latency is too high (50-60 for the end-to-end interaction).

If traffic is not peered, it is handled by the commodity Internet Transit service, a service model that intermingles all Internet traffic together into the same shared network. The commodity Internet Transit service works well until times of congestion as shown in Figure 14-4.

Figure 14-4. Peered traffic is unaffected by problems in transit networks.

Security. During times of network congestion, whether from spot events, denial of service attacks, or any other reason, the commodity Internet traffic is vulnerable to packet loss and latency along impacted paths. But traffic that is peered directly between content and eyeballs is effectively segregated from commodity Internet Transit traffic, and therefore is unaffected by the congestion side effects of the commodity Internet networks.

As Andreas Sturm (DE-CIX) put it:

“Important Traffic is Peered.”

This leads to the observation that Internet Peering in general provides a degree of control and enhanced security as well.

Notes from the Field.

Peering for Internet Security

The commodity Internet traffic may also be more vulnerable to security issues than traffic that is peered. Transit traffic is passed across potentially many transit networks before reaching its destination, and each additional network in the path provides another set of machines vulnerable to packet capture, attack or manipulation. As a matter of security, some companies (and governments) require that their network traffic not traverse country borders. For them, Internet Peering provides the customer with the controls over that subset of traffic that is peered. Further, if there is a concern, the customer can directly discuss matters with the peer’s network team, an escalation avenue that may not be available if one simply sends traffic through an upstream provider.

Traditional Peering vs. Remote Peering

The most common question about Remote Peering is:

“When does it make sense to establish a network (router) presence at an IXP versus Remote Peer at an IXP?”

The answer to this question depends on many factors based on the business and network application.

If the network application requires the lowest latency possible and capital costs are not an issue, then deploying both the network and servers topologically close to the eyeballs provides the lowest latency solution. If control over and visibility into the transport layers is important to the business, or if operating all layers of the networks is strategic, then traditional peering makes sense. If there is a requirement to also buy transit on-site or privately peer or with more than a few peers, then a router presence is generally preferred.

If the goal is to peer as widely as possible, then Remote Peering is the most cost effective way to do so. If it is not easy to obtain capital, if the network engineering resources are limited, or if there is no need to have the content physically deployed and served in the region, then Remote Peering cost effectively meets the application requirements. If a regional network presence is required quickly to improve the performance for a customer base, then Remote Peering is the fastest way to improve the end-user experience in a region.

A Case Study for Remote Peering: LinkedIn

A leading social networking site, LinkedIn, compared traditional peering against Remote Peering for a European peering deployment. They already had established a presence in Dublin and so they were comparing traditional and Remote Peering models to get into four well-populated peering points in Frankfurt, London, Amsterdam, and Paris.

With the traditional peering model, they purchase, assemble and deploy their network equipment at a cost of $275K per deployment, for a total of $1.1M for all four sites. They lease colocation space and power in each city at a total cost of $16K per month so they can attach to the exchange point peering fabrics contained within. The transport cost to extend their Dublin presence into all four locations worked out to be $6K per month. These one-time and recurring costs are summarized in Table 14-1.

Table 14-1. Traditional Peering vs. Remote Peering from an expensive Internet Region.

With Remote Peering, LinkedIn was able to remove the cost of equipment, the cost of colocation and the cost of the point-to-point circuits from the equation. The Remote Peering Provider was able to extend VLANs from each of these exchange points to Dublin for a total cost of $6K/month.

Another large content provider, GoDaddy, followed the same peering analysis and verified that their conclusion was similar. GoDaddy removed over $1M from their network costs by remote peering, and found that the peering break-even point for Remote Peering was 3 Gbps as opposed to 8 Gbps with traditional peering. Remote Peering enabled GoDaddy to improve performance and substantially reduce their costs for Internet traffic exchange.

One will observe however, that in our LinkedIn example, the unit cost for peering works out to be $6/Mbps ($6K/month divided by 1000 Mbps), a price higher than the market price for transit across Europe today ($1-$4/Mbps). Zaid Ali Kahn (Head of Network Architecture and Engineering, LinkedIn) confirmed that their blend of inexpensive commodity transit and remotely peering away strategic traffic has proven to be a sound performance and revenue-enhancement strategy.

These cases illustrate the general peering perspective of content providers:

“The #1 priority is to improve the end-user experience.”

Other content providers shared that they view their Internet traffic packets as “gold-plated” in the sense that packet drops result in lost revenue far beyond the cost of traffic exchange. This sentiment is reflective of a common thread of comments from the medium to large-scale content providers.

Remote Peering Use Cases

Let’s continue to apply the Remote Peering model by demonstrating how Remote Peering is used in practice across a variety of different Internet companies.

Next Generation ISPs. Most ISPs and content companies connect to the Internet by simply purchasing Internet Transit. If network connectivity is a strategic part of the business, these networks will blend Internet Transit from different regional providers into a multi-homed interconnection solution as shown in Figure 14-5.

Figure 14-5. Remote Peering for next generations networks.

When important regional customer bases are identified, extending the regional network presence with Remote Peering enables regional network optimization. To these companies, control over routing is strategic, so they may distribute nodes into Chicago, Hong Kong and London and utilize Remote Peering to extend their directly reachable networks strategically in each region.

Remote Peering is a way to strategically offload regional traffic without the cost or time investment required for a traditional peering deployment.

Content Providers. Content Providers leverage interconnection blends to improve the end-user experience and to improve their website conversion rates. They often blend Internet Transit with Content Delivery Network (CDN) services to get their static content deployed as close to the consumers as possible. They may also blend in Internet Peering and Remote Peering to distribute their real-time dynamically-generated content directly to the access network customers.

Figure 14-6. Content Providers utilize a “blend” of interconnection options.

The choice between Remote Peering and traditional peering depends on the nature of the backend content generation. These content providers focus their attention on optimizing the traffic delivery across their transit, peering and CDN interconnections. Viral videos for example might best live on the CDN while long-tailed less popular content may be delivered via the remote peering and transit paths.

Rapid Turn-up with Transition to Full-POP Deployment. The lowest latency solution for Content-Heavy network operators is a full Point-of-Presence (POP) deployment, where the network and servers are extended into a region and the content is delivered from that regional POP. Many content providers strategically deploy a full POP system into Internet Regions to service a strong and growing customer base. Full-POP deployments however can be time-intensive and cost-prohibitive. As a result, some content-heavy network operators employ Remote Peering to service these markets until they transition to a full-POP deployment as shown in Figure 14-7.

Figure 14-7. Remote Peering enables rapid turn up and transition to Full POP.

To illustrate and example of this is, consider a content provider with a growing user base in India as well as across Asia. One solution to this challenge is to Remote Peer this content to India from a Hong Kong, Singapore or Tokyo POP until a local India POP deployment can be justified. This justification is usually based on the loss and latency performance characteristics of the current delivery system (delivering Indian traffic from Hong Kong) against the cost of bringing the content into India. When it makes sense, the Remote Peering Provider can migrate this connectivity to a POP deployment in India.

International ISPs. Some international ISPs extend a Remote Peering presence to peer traffic away for free in a foreign market, and perhaps purchase inexpensive transit (see Figure 14-8). Across Africa for example, Internet Transit in 2013 is priced around $300/Mbps, and 98% of their traffic is coming from the U.S. and Europe.

Figure 14-8. International Remote Peering use case.

An African ISP may therefore decide to deploy Remote Peering into London to pick up cheap transit ($2/Mbps) and peer away a potentially large proportion of its traffic. Given the right traffic volume and transport price points, this may be a clever financial maneuver.

Cable Companies. A common blend for cable companies in North America includes extending Remote Peering into Los Angeles to pick up Asian Internet routes and into Miami to pick up South American routes and traffic as shown in Figure 14-9.

Figure 14-9. Cable Company peers at nearby IXPs and remote peers at gateway IXPs.

There may not be enough traffic to justify a full peering deployment but enough for a Remote Peering extension into IXPs with strong regional routes that match their subscriber base.

New Entrant Internet Service Providers. For new ISPs, Remote Peering provides an easy way to build modern national or international networks. They may employ peering primarily to save money, but their motivation is also to differentiate through customer service, massive scope and scale, high availability, and performance improvements. Here remote peering plays an invaluable role. Since the cost of extending into peering points via Remote Peering is less costly, an ISP may deploy Remote Peering into smaller regional IXPs as shown in Figure 14-10.

Figure 14-10. Remote Peering allows new entrant ISPs to build into many small IXPs to localize traffic.

We have examined a number of use cases in this section. The optimal blend varies based on the network flows, the traffic volumes, and the network requirements of the applications. The job of the network architect and engineer is to determine how to best blend the interconnection methods to the needs of the network operator.

Protected or Unprotected Transport Options

One implementation decision that Remote Peering customers face is whether they require protected transport or whether they prefer unprotected transport services. There tend to be two camps on this issue.

Protected transport is implemented by deploying across two diverse paths, typically with equipment to automate recovery in the event of an outage. Those that insist on protected transport want their peering infrastructure to be as reliable and robust as their Internet Transit service, and they are willing to pay the premium for it (about double the price of unprotected transport).

Others prefer the lower cost unprotected transport option for their Remote Peering. They view their peering as a local optimization, and if the underlying transport breaks, routing will automatically failover to their Internet Transit service. Many of the network operators in this camp prefer as few active electronics in between them and their peering point. They prefer to know when the underlying transport fails and deal with these temporary failures themselves.

Implications of Remote Peering to the Internet Peering Ecosystem

How does Remote Peering impact the Internet Peering Ecosystem?

1) Remote Peering extends the life of Internet Peering. Many network operators have not attempted Internet peering because the cost, complexity, and time to implement are formidable hurdles. Remote Peering enables these players to ease into the Internet Peering Ecosystem with fewer logistical impediments and a lower Peering Break-Even Point as shown in Figure 14-11.

Figure 14-11. Remote Peering removes the transport, colocation and equipment costs from the total cost of peering.

2) Remote Peering increases the value of the Internet exchange point. As more remote peering participants connect to exchange points, those that are already peering there benefit from the additional traffic and routes available. Without Remote Peering, the IXP reaches its maximum value when there is no more space and power in the designated colocation facilities as shown in Figure 14-12.

Figure 14-12. Peering increases the value of an Internet Exchange Point.

3) Remote Peering reduces the complexity of Internet Peering. Remote Peering allows network operators to deal directly with one provider to establish a peering presence at potentially many exchange points (see Figure 14-13). Fewer contracts to sign, fewer circuits to purchase, provision and debug, and fewer colocation agreements all simplify network interconnections.

Figure 14-13. The Remote Peering service model simplifies peering by delivering all peering on one port.

4) Remote Peering speeds up deployment of peering and enables trials without deploying equipment. If the customer is present at a colocation with Remote Peering Providers, the turn up could be done in a matter of days.

5) Remote Peering provides a single neck to wring but also introduces a single point of failure. If the Remote Peering fails, there is a single point of contact to fix things: the Remote Peering Provider. These eases operations for the customer but it also means that a single Remote Peering Provider failure could take down all associated peering.

6) Remote Peering enables access into small IXPs where the cost is otherwise not justified. Remote Peering lowers the cost of peering and therefore the risk of not recouping the investment by peering away enough traffic for free. Thus, network operators may expand into IXPs that otherwise were not viable.

Collectively these implications demonstrate that Remote Peering represents not only an emerging trend, but also a major shift in the Internet Peering Ecosystem.

Breathing New Life into the Internet Peering Ecosystem

In 2013, commodity Internet Transit prices fluctuate around the $1-$4/Mbps range in well-connected Internet Regions. As always, network operators ask the question:

“Does peering makes sense anymore?”

To answer this question we typically look at the Peering versus Transit graph (see Figure 14-14). The monthly costs of the traditional peering model (colocation, equipment deployment, transport) are collectively not dropping as fast as the price of Internet Transit. This means that there is a narrow range, or “runway” where peering is financially justifiable.

Figure 14-14. Peering vs. Transit as Transit Prices Decline.

Remote Peering eliminates many of the peering costs, and therefore makes blending peering with transit a more financially attractive interconnection strategy for a larger market of potential peers. As a result, more networks attach to peering points, more traffic is peered away for free, and the value of IXPs and peering in general continues to grow. Without Remote Peering, peering points would continue to diminish in their appeal for all but the largest networks.

The Great Remote Peering Debate

It couldn’t be called a “Great Debate” without ferreting out some dissenting opinions about remote peering, and as usual, it turns out there are strongly held, almost religiously held, beliefs in the peering community about peering methodologies. Over a dinners at recent Internet operations meetings I teased out a few of these contrary views of remote peering.

Here are some of the criticisms expressed about remote peering:

1. Remote Peering is outsourcing (and hiding) some important complexity.

a. From the customer perspective, the MPLS cloud that brings the VLANs to the IXPs is not as easy to monitor and debug as a point-to-point circuit. When things break, there is a lot of active equipment and potentially many parties between the customer and its peers.

Counter-argument: Zaid Ali Kahn (LinkedIn) shared these initial concerns but after deploying remote peering, he is a big Remote Peering proponent:

“Our engineering team initially expressed concerns about the reliability of a Layer 2 Virtual Private Network (L2VPN) service, but we have found that Remote Peering solutions using our vendor’s MPLS network are very stable. Remote Peering allows us to be in control of our own destiny and get closer to our eyeballs; with transit we don’t have control of the routes.”

b. Some network operators do not want to peer with those doing Remote Peering. The added latency and complexity of the remote peering infrastructure tips the scale to make the peer not want to peer with remote peers.

2. Remote Peering can lead to routing inefficiencies. We used to say “peering keeps local traffic local,” but remote peering changes this notion. You could pick up routes in Amsterdam or Frankfurt and find out that some of this peered traffic is traversing oceans twice over layer 2 where that traffic could be peered at a colocation center in common in Ashburn, Virginia!

As a result of Remote Peering:

a. Some implicit latency benefits of peering at a local IXP may get worse.

b. If everyone remote peers, then there could be quite a tangled mesh of routing anomalies that spans layer 2 and layer 3.

The counter arguments include that there has long been a diversity of local and international peers at the popular IXPs. For example, roughly 75% of all AMS-IX traffic is from outside of Holland, 45% of all LINX traffic is from outside of the U.K. and 55% of all DE-CIX traffic is from outside of Germany. It is up to engineers to mange their own routing efficiently whether at layer 2 or layer 3.

3. BGP Timers will keep Remote Peering sessions alive long after the remote transport has disappeared. Further, when the Remote Peering transport provider network goes down, it takes ALL REMOTE PEERING WITH IT! Individual circuits don’t often all go down all at the same time, and one can engineer across different circuit providers to mitigate individual problems. With traditional peering, the router is informed immediately when the underlying transport disappears. With this more complicated layer 2 VLAN system, all peering traffic could be black holed for minutes before the BGP timers shut down the BGP session.

The counter argument is that engineers can adjust the BGP Keep Alive thresholds that turn down a BGP session. In the event of a L2VPN failure, peers can turn down the session within 30-60 seconds if they so configure.

Another counter argument is that traditional peering across an Ethernet switch also suffers from this same systemic failure – customer routers are not informed when their peer’s router gets disconnected from the fabric. The customer finds out when the BGP timers expire as well.

4. Remote Peering signals that the peer is not committed to a real physical presence. If they are not committed to a physical peering in the region, then there is some doubt whether peering with them leads to a long-term peering benefit.

A counter argument, a pivot actually, is that peering is considered a local optimization. If the remote peer pulls out, then that just means the optimization goes away. One is no worse off than before the peering was established.

5. Remote Peering with route servers may deliver suboptimal routes. Remote peering may make sense, or it may not, but history has taught us that not everyone does the careful analysis of the routing impact before turning up a peering session. Some peers, for example, will peer with Route Servers (a multi-lateral peering technology) and indiscriminately pick up routes. Remote peers that accept all routes from the Route Servers may accept and install these suboptimal paths by default.

The counter argument is that poor routing can be obtained equally at layer 2 or layer 3. It is a network engineering decision as to how to implement and blend interconnection strategies.

Summary

Remote Peering can be thought of as making peering frictionless from afar. With no incremental equipment to deploy and therefore no deployment or colocation expense, the Remote Peering Provider service model makes it relatively easy to justify and to connect to peering fabrics.

Of the 630 AMS-IX attachments, 130 are embracing remote peering, and half of the new DE-CIX attachments are embracing this remote peering trend as well. Whether you blend remote peering into your interconnection strategy or not, it is important to understand what it is and how it works.

Chapter 15 - International Peering

At the end of every Internet Peering Workshop we end up at the same discussion: How should we think about our international peering strategy? This chapter presents a simple framework that has proven helpful for creating an international peering strategy.

Simplifying the International Peering Strategy

International peering strategies rest upon a complex web of strategic intentions, the foreign business contexts, the market for Internet services, and a variety of underlying assumptions. Network operators expand their network for a variety of reasons, but primarily among these are servicing current and future customer needs. Building into a new Internet Region involves dealing with the local operations environment and understanding and adhering to the regulatory regime. Add in a few more variables associated with peering politics and the international peering discussions quickly become overly cloudy and complex.

To reduce this complexity we have found it helpful to start discussions with a simple international peering model based on explicit assumptions that can be refined or removed in later stage discussions. These discussions often lead to the broad strokes of an international peering strategy.

An International Peering Model

We will build our model by making some simplifying assumptions.

First, we will continue our naming convention and call our network operator “ISP A.” We assume that ISP A is interested in interconnecting in three Internet Regions, with one of these regions being a “gateway” Internet Region where Internet Transit is inexpensive. The other Internet Regions will be called “Internet Region X” and “Internet Region Y.” Let’s assume that ISP A has a single router (shown as ‘A’ in the figure) in each region providing interconnections for that region. These routers are interconnected across transport circuits ‘t’ as shown in Figure 15-1.

Figure 15-1. An International Interconnect Model.

Interconnections in Each Internet Region

In each Internet Region we assume that ISP A establishes a Point-of-Presence (POP) to interconnect with others in

1) free and reciprocal peering relationships (shown as ‘P’ in the figure), or in

2) transit relationships (shown as ‘T’ with a ‘$” highlighting which side gets paid).

Transit customers are numbered and the peers are lettered so we can discuss traffic flows and corresponding revenue flows. We will refer to this diagram throughout the rest of the chapter.

Simplifying Assumptions

In the context of our discussions, we will hold five variables constant.

1) Customers. All customers are assumed to be homogeneous in the sense that they all offer the same traffic load, and they are all entirely content-heavy or access-heavy. Therefore, all traffic travels in the dominant direction (Content Providers send traffic to Access Networks). These assumptions simplify the revenue arithmetic and allow us to ignore peering ratios issues for example.

2) Revenue. We are making the simplifying assumption that all customers of ISP A pay the same price ($T/Mbps) for Internet Transit. As is standard practice, Internet Transit is metered and customers pay only for traffic in the dominant direction. We further simplify the math by assuming that all customers are already colocated with ISP A, so customers and peers alike incur no transport costs when they buy access or peer with ISP A.

3) Inter-Regional Transport Cost. Each of ISP A’s POPs are interconnected using international transport, shown as ‘t’ in Figure 15-1. We will assume that all inter-regional transport has an average cost of $t/Mbps. (Note that ISPs that own their own cable will see this as a ‘sunk’ cost and therefore $t = $0/Mbps.)

4) Peering Costs. We assume that all of ISP A peers are located at ISP A’s colocation center and there are no cross-connect fees, peering port fees, and colocation fees are assumed already included in their existing presence. For the same reason, we are assuming no incremental cost for peering equipment.

5) Global Internet Transit. We will assume that ISP A can purchase access to the rest of the Internet inexpensively ($GIT/Mbps) in the “Gateway” Internet Region.

Why These Simplifying Assumptions?

These simplifying assumptions help us create a base model for discussion and ultimately help us articulate an international peering strategy. It is a valuable exercise then to challenge each assumption and evolve the model to take these refinements into account.

Discussions based upon this base model go a long way towards articulating the key drivers behind a coherent and compelling international peering strategy.

Now that we have introduced the international peering model, let’s talk about traffic flows and calculate the corresponding costs and revenue flows.

Traffic Flows

There are only three categories of traffic to be considered when looking at ISP A’s interconnect strategy:

1. Customer-to-customer traffic (also called “customer routes”), and

2. Customer-to-peer’s-customer-traffic (also called “peer routes”),

3. Customer-to-the-rest-of-the-Internet-traffic (also called “default route”).

Let’s consider each category of traffic flow.

1) Customer-to-Customer traffic

Traffic between ISP A customers and the other ISP A customers will be exchanged within the Internet Region or between Internet Regions. For example, customer #9 may send content to customer #8, or customer #6 may send traffic to customer #7 as shown in Figure 15-2.

Figure 15-2. Model of traffic flow between ISP A’s customers and other ISP A customers.

Notes from the Field.

Level 3 Acquires Eyeball and Content Customers

It is worth noting that acquiring both content customers and eyeball customers is a common interconnection strategy. The ISP “Level 3” for example courted cable companies while simultaneously courting the largest content customers that sought to get their content as topologically close to the eyeballs as possible. By satisfying both sets of customers entirely across the Level 3 network, they collect money from both sides. Eyeball networks pay for the traffic the users download and the content providers pay for the traffic that they push to the eyeballs. This strategy increases the value of the network that handles both, and therefore makes it easier to peer and acquire new customers. Selling to both traffic sources and sinks demonstrates what economists call exploiting a “two-sided market.”

Revenue. For this customer-to-customer traffic, ISP A receives revenue from both customers, or 2 * $T/Mbps. If traffic has to go between Internet Regions, there may be a cost of $t/Mbps. As a result, for this type of traffic we see that ISP A obtains revenue from customer-to-customer traffic flow of

C-C Revenue = 2*$T/Mbps - $t/Mbps

If both customers are within the same Internet Region or if the network operator owns the physical transport, the value of $t is zero.

2) Customer-to-Peer’s-Customer traffic

The second type of traffic is customer traffic that is destined to or coming from a peer’s customer. In this case, as shown in Figure 15-3, traffic between ISP A’s customers and its peers’ customers will either be satisfied within the country (such as Peer J sending content to customer #7) or satisfied by sending traffic between Internet Regions (such as Peer B sending traffic to customer #6).

Figure 15-3 - Model of Internet traffic flow between ISP A’s customers and its peer’s customers.

Revenue. ISP A collects revenue of $T/Mbps from its customers. Customers #6 and #7 pay ISP A for the traffic that is ultimately going to or coming from ISP A’s peers. If traffic has to go between Internet Regions, ISP A may bear an additional transport cost of $t/Mbps.

As a result, for this type of traffic we see that ISP A obtains revenue from customer-to-peer traffic flow of

C-P Revenue = $T/Mbps - $t/Mbps

If the customer and peer’s customer are within the same Internet Region or if the network operator owns the physical transport, the value of $t is zero.

3) Customer-to-Internet traffic

Traffic that can’t be satisfied between ISP A customers, and can’t be satisfied with ISP A’s peers, is ultimately delivered via ISP A’s upstream transit provider in the Gateway Internet Region as shown in Figure 15-4.

Figure 15-4 - Model of traffic flow between ISP A’s customers and the rest of the Internet.

Revenue. For this type of traffic, ISP A receives revenue of $T/Mbps but pays transit fees of $GIT/Mbps. If traffic has to go between Internet Regions, there may be a cost of $t/Mbps.

As a result, for this type of traffic we see that ISP A obtains revenue from customer-to-Internet traffic flow of

C-I Revenue = $T/Mbps - $GIT/Mbps - $t/Mbps

If the customer happens to interconnect with ISP A in the Gateway Region or if the network operator owns the physical transport, the value of $t is zero.

Traffic Flow and Revenue Summary

The key point is, with all three types of traffic, customer traffic generates revenue for ISP A. As long as transit revenue exceeds traffic exchange costs, ISP A will derive incremental revenue from customer traffic as shown in Table 15-1.

Table 15-1. Customer traffic generates incremental revenue.

Now that we have discussed the three traffic types and corresponding revenues and costs, lets’ take a look at a few interconnection strategies.

International Interconnection Strategies

The interconnection strategies usually consist of two parts: an Internet Transit strategy and an Internet Peering strategy.

Internet Transit Strategies

Transit strategies tend to be based on the company’s strategic intent and the market context.

Compete in the Wholesale Transit Market. If ISP A wants to be the leader in the wholesale Internet Transit across Internet Regions, it might deploy a node and a small sales office in each region. In this case, ISP A might not care so much about the Internet Region itself, nor about the Internet Peering Ecosystem dynamics in this Internet Region since it only wants to sell wholesale access for the Internet routes it picks up in the Gateway Region for example.

Lead the Internet Transit Market. On the other hand, if the strategic intent is to “own” or “lead” the market for certain Internet Regions, then they may want to accelerate the growth of this particular Internet Peering Ecosystem and take a more active role in the Internet Region. They might establish a larger local team and focus on sales and growing their own local hybrid peering and transit with an aggressive peering and customer attainment strategy. They may create an informal alliance with the local Internet Exchange Point, try to pull together the Tier 2 ISP population for peering, and perhaps establish a strategic alliance with an ISP that owns some local fiber.

Perform Transit Arbitrage. Another common strategy is to rapidly build a customer base in several Internet Regions by arbitraging the transit markets between the inexpensive gateway country and the expensive home market. For example, ISP A will drive the price for Internet Transit down within each region until the price of transit approaches the price of transit in the Gateway Internet Region.

The degree of competitive aggressiveness is a strategic decision.

Notes from the Field.

Arbitrage Africa

The price of Internet transit in the U.S. began at $1200/Mbps in 1998 and was driven down to about $12/Mbps over ten years as shown in Figure 15-5. The price for Internet Transit in South Africa was hovering $1200/Mbps in 2008, and hovered around $380/Mbps in 2011, so one might expect the price to follow a similar trajectory and reach $12/Mbps by 2018. If so, how should ISPs ride this price curve down?

Figure 15-5 - History of Internet Transit pricing in the U.S.

We have seen a couple of pricing strategies as ISPs perform arbitrage in emerging markets.

Profit Maximization. One pricing strategy is to offer Internet Transit at just below the market price, low enough to skim price-conscious customers from the competition at the highest possible price point.

Market Acquisition. Another more aggressive pricing strategy is to undercut the market by the percentages shown in the graph (20%-50% per year) regardless of the market price.

Notes from the Field.

Market Aggression

Both Level 3 and then Cogent employed the market acquisition strategy during their European expansion in the early 2000 timeframe. They presold network capacity to customers and vowed to liberate the market from the incumbents. Just after the network was ready to deliver services, the European competitors reacted by lowering their prices to just below the point at which these new market entrants could make a profit. This was an aggressive competitive response to an aggressive pricing strategy.

Both strategies make sense in certain contexts and tend to include market messaging about improving the Internet for the region, and the benefits of a lower cost or “open” Internet. In the minds of the market participants, this goodwill message helps paint the market entrant as a helper rather than as a foreign invader.

Next we turn our attention to Internet Peering in emerging Internet Peering Ecosystems, and how ISP A determines its peering inclination, its articulated peering policy, and how ISP A may evolve as these Internet Peering Ecosystems evolve.

Peering in Emerging Internet Peering Ecosystems

To discuss peering strategies, we first must understand the context of the Internet Region. For example, we need to understand:

1. How well developed is the Internet Peering Ecosystem?

2. Who are the Tier 1 ISPs in the Internet Peering Ecosystem?

3. Who are the Tier 2 ISPs and are they peering?

4. Where are the Internet Exchange Points (IXPs), who runs them, who is there, and what are the costs for participating (local loop, colocation, peering ports, membership fees, etc.)

5. What is the market price for Internet Transit?

6. What is the cost for transport capacity into the exchange?

7. What are the dominant traffic patterns?

8. How much traffic stays within the Internet Region? What are the popular destinations? Who are the leading access providers in the region?

In the Peering Workshops we go through this data collection in each region under consideration. We then create the business case for peering in each region.

While this in depth analysis of the Internet Region is important, it is beyond the scope of this book. To give you a flavor of the discussion however, we will walk through a sample peering strategy case study based the model and discussion so far.

Case Study – African ISP A Expands into London

Strategic Intent. Let’s assume that ISP A is an African ISP that is interested in expanding its network into nearby countries and more importantly, into London to purchase Internet Transit there for $2/Mbps instead of the $300/Mbps that is available in their home market. As for strategic intent, ISP A wants to provide better Internet Transit service for the home market while also being the price leader.

Let’s enumerate some relevant context generalized for the continent of Africa.

1) 98% of all African traffic is coming from the US and Europe. A trivially small percentage of African traffic is locally produced and consumed content.

2) With such a small percentage of traffic satisfied locally, it is not surprising that there is not much peering traffic exchanged within the Internet Regions across Africa.

Let’s enumerate some additional assumptions for the case study.

1) The Internet Gateway location of choice is London

2) The cost of transport up to London is $275/Mbps.

3) Relevant content providers are in London and willing to peer openly.

ISP A’s Peering Policy

Since 98% of all African traffic comes from the U.S. and Europe, ISP A should peer openly in London. Given the original assumptions that the peers are all in the ISP A colocation center in London, that Traffic will either come from the Global Internet Transit interconnect or via the free peer. Free peering is better than paid transit for this readily available traffic.

One could refine the analysis by performing a Peering vs. Transit calculation or limit peering to those that meet certain criteria, or not bother with peering at all given the low price of transit. However, by peering in the same building, ISP A will generally improve performance while simultaneously reducing costs. Peering in London will also provide a good sales and marketing tool for ISP A since it can then promote itself as having direct peering with the most popular content sites in the world up in London. Peering openly in London is aligned with the explicit strategic intent.

ISP A should therefore identify (or hire) a peering coordinator to build these peering relationships. This will require some travel budget, staff time, and a learning curve, but ultimately peering in Europe will be great preparation for peering in the African Internet Regions.

Assertion: ISP A should take a community leadership role in each Internet Region

Many Internet Regions across Africa are just now starting to grow. There is little value derived from peering in these regions today however, so an argument could be made to not bother with peering there. But these markets will not always be small. As the Internet Peering Ecosystem grows, ISP A’s market for international transit grows. It is therefore in ISP A’s best interest to promote and help grow these ecosystems. Over time,

1. Broadband access (perhaps mobile) will become more widely and inexpensively available, and

2. Lower transit prices will lead to more local Internet attachments, and

3. More Tier 2 ISPs will therefore launch services for their local markets, and

4. The local Internet Peering Ecosystem will grow enough that local eyeballs will seek local content, and

5. The business case for peering will make sense.

If ISP A takes the leadership role now, promotes local peering and the local exchange points, it will not only help establish goodwill, but it will also grow a broader base of customer prospects that may ultimately prefer to purchase transit from a richly peered ISP A.

Notes from the Field.

MWEB Goodwill

In South Africa, MWEB was hailed as a hero and a leader in the Internet Region by offering unlimited Internet access when others only offered metered service. They messaged and established a reputation for building the South African Internet. They then launched a public relations campaign to get the incumbent Tier 1 ISP to peer with them. The public relations campaign rallied the population to the cause. When the peering was denied again, MWEB publicly announced that the traffic to Telekom South Africa was going to traverse London, and that people should complain to Telekom South Africa. Ultimately, through pressure, MWEB was able to maneuver Telekom South Africa to peer and established itself as a successful liberator of the Internet from the grips of Telekom South Africa.

The point is that this community leadership is a very powerful tool, and helped position South Africa as a Internet Regional focal point. ISP A should do the same for each Internet Region of interest.

Assertion: ISP A should adopt an Open Peering in each Internet Region of interest across Africa.

At this point during the peering workshops people usually ask, “Why will customer prospects buy from ISP A when they could just peer with ISP A for free?”

The answer is that Internet Transit and Internet Peering are not perfect substitutes.

ISP A’s peers only get access to ISP A’s downstream Internet Transit customers, which, as we have already established, is a very small percentage of the ecosystems Internet traffic (less than 5% of African traffic is local sourced and sunk). So initially there is very little value to peering with ISP A. This peering discussion however can lead to this next sales discussion with prospects:

Internet Transit on the other hand provides access to all routes:

1. Routes learned from other ISP A customers, and

2. Routes from ISP A’s peers’ customers, and

3. Routes from ISP A’s upstream Internet Transit provider in London.

Ultimately, the transit service provided in each Internet Region should evaluate and compare favorably to the value of peering.

The Value of an Internet Transit Network

When one of the local ISPs purchases Internet Transit from ISP A today, it is primarily buying access to the global Internet delivered out of London. Most of the value derived by purchasing Internet Transit from ISP A today is the quality of its network capacity up to London and the quality of the global routes it picks up there. The primary question customers should ask today is “How well does ISP A deliver traffic to me from the U.S. and Europe?”

Today, the secondary question is, “How well does ISP A meet my long-term local traffic needs?” Let’s discuss the long-term effect of this open peering policy in the emerging local markets has on the value of the transit network (shown in Figure 15-6).

Figure 15-6 - The value of an Internet Transit network to its prospective customers.

Adding Peers make ISP A a more Attractive Transit Provider

The ISP A network is a more attractive network to its transit customers when it peers its traffic locally. Why? Peering provides ISP A’s customers with direct access to ISP A’s peers’ customers. This means that customers get better performance when their traffic doesn’t have to go all the way up to London to be served! As the ecosystem develops, more local ISPs and more local content comes on-line, so the value of ISP A’s Internet Transit network increases since it peers this traffic better locally as shown in Figure 15-6. From a marketing perspective, the performance benefits become an important differentiator, and from a public relations perspective, peering positions ISP A as promoting nationalistic interests. This will lead to more customers that want not only access to the global Internet but also want great performance to this emerging local content.

More Customers make ISP A a more Attractive Transit Provider

The ISP A network is a more attractive network to its transit customers as it obtains more customers. Why? First, the performance is even better when all customer traffic can be satisfied entirely on the same network. Second, these routes may be unique; if the customers are singly homed to ISP A, the next best alternative to reach these customers might require sending traffic through London. Therefore, a strong argument can be made that purchasing transit from ISP A will provide the best performing path to these local destinations.

A Strong Local Network of Customers and Peers will Fend Off Foreign Competitors

Over time, and as the Internet Peering Ecosystems across Africa develop, ISP A will be able to peer more of its traffic locally and therefore be able to demonstrate that much of this local traffic can be satisfied without leaving the country. These peering sessions will help it differentiate itself and help it compete aggressively against the foreign transit providers that build into this now more established Internet Peering Ecosystem. These foreign Internet Transit providers can initially only offer global routes from London, not the more desirable local routes that ISP A has cultivated in this emerging ecosystem. The competitor’s traffic will have to traverse London to get to ISP A’s customer base, and it may have to traverse London to get to ISP A’s peers’ customers as well. ISP A is in a more defensible market position from this strong local peering and customer acquisition strategy.

Over time, by driving down the price of Internet Transit, and perhaps by migrating to a selective peering policy, foreign competitors will see these markets as less lucrative arbitrage opportunities.

Open Peers Emerge as Tier 1 ISPs in Emerging Markets

At some point, a “Tier 1 ISP” club will probably emerge in these ecosystems similar to what we see in the rest of the world. These “Tier 1 ISPs” will peer only with each other and be able to reach all in-country routes through these free and reciprocal peering arrangements. They will adopt a restrictive peering policy in the country and de-peer all of those networks that are customers of the other Tier 1 ISPs. They will deny requests for peering with anybody else so competitors may have to send traffic through London to reach their customers. Every other Internet Region has gone through this transition and ISP A may, with this strategy, be well positioned to emerge as a Tier 1 ISP in various regions. The uniqueness of the routes and the greater volume of traffic may even allow ISP A to negotiate peering from a stronger power position in London.

International Peering for Content Providers

We have been talking about ISP A and its interconnection strategy as a provider of Internet Transit, but we have not yet discussed how the model applies to content providers. Every developing Internet Region sees cost and performance issues when content is not available locally. As a result, the large-scale network savvy content providers are constantly invited to extend their presences into other Internet Regions.

Content providers generally consider three things when evaluating these requests:

1) “How do they currently reach customers in this region, and what is the current price and performance for this traffic?” and,

2) “How might they better reach customers in this region and what would be the price and performance improvement?” and finally,

3) “What are the implications of expanding into the Internet Region?”

For content providers and content-heavy ISPs and CDNs, it is all about the end-user experience. They might even pay more for a substantial improvement in performance for an important user base. For example, companies like LinkedIn have a substantial, important, and growing user base in India and parts of South America. They might consider expanding into these markets directly rather than delivering their content from Japan or from the U.S.

So when Content Providers apply the international peering model, they don’t look at the ability to sell transit. They look at the current latency, packet loss, perhaps jitter and other metrics between customers and their servers via a transit provider from an adjacent but centrally located Internet Region. The next best alternative might be to expand into a new Internet Region and serve up the traffic locally. If they build in, how much of their traffic can be peered away for free, how much might Internet Transit cost there, and how much will it cost to build in? They will apply the business case for peering but instead of looking at the peering break-even point, these network savvy content providers primarily consider the number of milliseconds from the round trip time to display their web pages.

Generally content providers and content-heavy ISPs and CDNs will adopt an open peering policy globally. In some cases, the content provider or CDN may even require their hosting company in the region to pay for their bandwidth and hosting expenses.

Content Providers and CDNs may require that the hosting company’s competitors be allowed to interconnect with the content company or CDN directly, something the hosting company may be disinclined to allow otherwise. No “walled gardens are allowed” has been Google’s approach as they deploy caches into emerging Internet Regions to help a local Internet Peering Ecosystem grow.

Another set of issues that the content providers look at are around the business context in the foreign Internet Region. If they build into Europe for example, they may have to honor the strict European privacy standards. If they build into China they may need to honor the governmental censorship orders and deal with a strong-armed and invasive government controlled Internet Region. Some countries have large import duties and some require majority local ownership before a company is granted a license to offer telecommunications services there.

These are the sorts of discussions that come up during the peering workshops, and the framework serves as the skeleton for dissecting and discussing some of these types of topics.

Summary

This chapter provided a simple model for exploring international peering strategies that have come up in almost every Internet Peering Workshop. The group discussions surrounding the model exercise the definitions and helps us sketch out broad stroke strategies about where we want the “ISP A” in question to be in 5-10 years as the ecosystems develop. The simplifying assumptions have proven necessary to avoid getting caught up in the complexity and unique characteristics of each market.

This resulting model is a good first pass for exploring an international peering deployment. Refinements are often made to consider different price points, different peering policies, and different market penetration strategies for each region.

In this analysis we used for an example deploying across Internet Regions in Africa but the same logic applies to many international deployments. In the next few years much of the African Internet will emerge as a set of first-class independent Internet Regions. It may take some time, and there is risk here, but in a world where so much of the ecosystem is hardened, Africa looks to be a fertile ground for the next generation Internet consisting of a billion users consuming new local Internet content.

Chapter 16 – The Future

“The more things change, the more they stay the same.”

– A proverb loosely translated from

“plus ça change, plus c’est la même chose”

When asked what he thought the stock market would do next, J.P. Morgan answered,

“It will fluctuate.”

The global Internet Peering Ecosystem continues to evolve and morph to meet the demands of its users and their applications. Over the past few decades we have seen the peering model evolve from network-based metro-area peering to carrier-neutral IXP-based peering. In 2014 we will see more Remote Peering interconnection blended with Internet Transit, Traditional Peering, CDN, and full-POP deployments.

Internet Transit in 2014

The Tier 1 ISPs in mature Internet Peering Ecosystems will try to hold their ground on price in 2014. As in 2013, we will continue to see this as a bifurcated Tier 1 pricing model across commit levels. For example, Content Providers with less than 10Gbps will continue to have a hard time getting below $2/Mbps from Tier 1 ISPs such as Level 3. Only when buying more than 10G will customers see pricing towards the $1/Mbps price range from the Tier 1’s.

Of course, at the extreme end of the scale, those buying on the scale of Apple, Facebook, Microsoft, Netflix, etc. in the 100+ Gbps range, will see $0.40/Mbps price points throughout 2014 from the Tier 1 ISPs. These same content providers will have no problem though getting a 1G service for less than a dollar from the bottom feeders. Taken together, these trends are described in Figure 16-1.

Figure 16-1 – Bifurcated Internet Transit Pricing in 2014.

Across the board in 2014, Internet Transit prices will be very inexpensive, but there will be markets of exception. For example, I gave a talk for the US Telecom folks in Lake Tahoe a few years back when a rural Texan ISP approached me and said “We don’t see prices anywhere that low - we pay $75/Mbps because there is only AT&T there.” Less mature peering ecosystems like these will see outlier prices as high as hundreds of dollars per Mbps. The good news for these folks is,

“the bottom feeders are coming to town in 2014.”

Internet Peering in 2014

Peering between ISPs in the U.S. and across Western Europe is pretty much done, in the sense that most ISPs that need to peer with each other are already doing so. Major Internet Peering Ecosystems across Asia, such as Japan, Hong Kong, etc. and parts of Eastern Europe and Russia, are also mature peering ecosystems, as evidenced low Internet Transit prices.

There are many aspects to the value proposition provided by Internet Peering. Peering makes sense financially if enough traffic can be exchanged to cover the costs of peering. Peering makes sense logistically when the value derived from peering, including the performance benefits, the security benefits, along with the operations benefits collectively exceed the cost of peering.

Let’s look at some of the things that will drive these peering values into 2014.

1) Internet Traffic Always Grows so the Value of Peering Always Grows

Internet traffic has always grown, and will continue to grow through 2014. As the amount of Internet traffic grows, the amount of traffic one can peer away for free grows, and so the benefits of peering will continue to grow as well.

Even more, the value of peering tends to get compounded since peering coordinators continually add peers over time. For example, most network operator in the first few years will grow their 10% peering to 25% peering as shown in Figure 16-2. The benefits of peering will compound since the peered traffic grows at a faster rate over time as more peers are added. This compounding makes the value of the major market IXPs grow exponentially.

Figure 16-2 – Peering Value Proposition Compounding over time.

But with transit prices so low, won’t rational peers pull out of IXPs?

It depends.

• Some will still be able to peer away enough traffic to completely cover the cost of peering.

• Some will peer no matter what, purely for performance or security reasons.

• Some will continue to peer, even when it is more expensive to do so, because they don’t want to signal to the market that they are in a decline or running out of money.

• Some will keep peering because they don’t realize that peering is more expensive than transit; they simply don’t do the math.

And some will pull out of peering and resell someone else’s network. Those that go this route better have a strong differentiator, because otherwise they will have repositioned themselves as a commodity Internet reseller and ride the price curve down towards zero.

In sum, the Internet growth curve will drive the peering market segment growth and therefore the peering value proposition.

2) Remote Peering Grows the Peering Market

Remote Peering enables large enterprises, content providers, e-commerce and portals, to start peering without having to be a network operator. Without the cost of routers, colocation space, and a large network team, these companies will be able to directly connect their content to the core of the Internet. This leads to more peers, more traffic, and therefore a greater peering value proposition for peering. Because of Remote Peering, the proportion of content providers and enterprises peering at the exchanges will grow in 2014.

Network savvy enterprises will internally justify peering for the security benefits of peering, the control over routing, and bypassing the commodity Internet for direct access to their customers and service partners. In addition, small Content Providers and Cloud Services Providers in particular will start remote peering with relevant niche trading partners to help them scale. Once peering is seen as obtainable and strategic, these new peering participants will bring new traffic and new routes to the peering points over the next few years as shown in Figure 16-3.

Figure 16-3 – New Markets for Remote Peering.

3) Emerging Ecosystem Regional Growth Drives Traffic to Mature IXPs

Peering ecosystems across Africa and elsewhere will continue to extend their network reach into mature ecosystems to obtain inexpensive transit. While this may slow the development of local content and therefore the rate of local ecosystem growth, it will drive the value of peering at the mature IXPs. These new routes and resulting increased traffic volume will result in lower prices back in the local market, with better performance. The traffic growth in these other regions will lead to a stronger peering value propositions at the mature IXPs.

Figure 16-4 – Peering Growth Driven from Far Away Places.

In 2014, expect to see many more customers connecting to the large European IXPs and the emerging regional IXPs on other continents as well.

4) Video and New Traffic Types will lead to Distributed Peering Load

Video and new types of traffic will drive Internet traffic volume. Video is already 50% of all Internet traffic, and I believe the video ecosystem will require wider distribution across many more peering points, perhaps helping to establish the viability for many more IXPs across the Tier 2 and Tier 3 cities. Within a few years, these new IXPs will exchange as much traffic as the larger U.S. IXPs in Tier 1 markets today shown in Figure 16-5.

Figure 16-5 – Video Peering Growth Drives Peering Distribution.

The distribution of IXPs into the Tier 2 and Tier 3 cities will make it less expensive for the new peers to start peering and thus drive peering growth.

5) New Power Requirements Drive Distributed Peering Growth

Compounding this distribution effect, power requirements for dense server deployments will also lead to peering growth. There will be a new Request For Proposal put out for the next generation Tier 1 ISP interconnect points, backed by the large-scale cable companies, video distributors and partners, and large-scale content companies. They will deploy dense server clusters across this much more widely distributed ecosystem of IXPs.

Between 2014 and 2017 expect to see the eight interconnection regions where the Tier 1 ISPs peer, evolve to perhaps 80 IXPs across North America, each supporting massive clusters of Netflix and YouTube videos along with whatever else the Internet consists of. Given that these two content providers represent 50%+ of the Internet traffic, Google and Netflix have the clout to drive the market towards making these 80 IXPs the must-peer locations. There are other initiatives in the workings as well relating to data centers in the last mile and massive centers near the legacy IXPs. The Tier 1 ISPs might originate this RFP, but the content companies will demonstrate the power to drive fundamental changes to the peering ecosystem.

***~~~***

Answers to Practice Questions

Chapter 2 – Internet Transit

1. (C) Max(500Mbps,800Mbps)*$5/Mbps=$4000/month

2. YES – Commit early

3) Max(500Mbps,800Mbps)*$5/Mbps=$4000/month

4) Vs.

5) 1000Mbps*$3/Mbps=$3000/month

Chapter 3 – Internet Transit Playbook

1. a) The maximum savings from the Optimal Transit tactic can be calculated to be the monthly cost of traffic when transit volume hits commit point minus the monthly cost had they executed the early commit. b) The maximum savings would be the entire transit load at the market price.

2. One can imagine many additional maneuvers including swaps for services in kind, leveraging free trials, or keeping zero commit transit services in waiting indefinitely, bundling free transit with colocation, reselling agreements to consortiums of buyers, bandwidth exchanges, etc.

3. You and your buddy can share a network, share the costs, and balance the network load and leverage the aggregation benefits of not needing the bandwidth at the same time.

Chapter 4 – Internet Peering

1. No. In these situations it is good to draw a picture like Figure 4-2 to see that peering is not a transitive relationship and therefore ISP B and ISP C will not hear each other’s routes announced by ISP A.

2. Peering is the exchange of access to each other’s routes but not access to each other’s transit service. In this light, peering is not barter. If you look at the ISP service being the exchange of routes and traffic then peering looks more like barter. If you believe that routes are a commodity, each equally valuable, then peering looks more like barter. Another interesting question is then, what is the basis for taxing? The market price of transit? The price of paid peering? The cost of the next best alternative (cheapest transit in the market)?

3. Omitted.

4. Not likely since ISP C is getting paid transit fees by ISP A.

5. Some argue that paid peering should be cheaper than Internet Transit since only a subset of the routing table is provided. Others argue that peering should be the same or more expensive than transit since it provides a higher performance path. It is an interesting debate to have.

Chapter 5 – The Business Case for Peering

1. 1500Mbps*$2/Mbps=$3K+$5K=$8K

2. No: Transit is cheaper@ $2/Mbps. Peering=$3K+$1K+$3K=$7K/2000Mbps=$3.50/Mbps

3. No, you are still on the hook for 2Gbps*$2/Mbps=$4000, adding in the $2500 for the peering port puts you at $6500/month! Cheaper to just send all 3Gbps through transit.

4. The Effective Peering Range is between 2.5 Gbps ($5000/2500Mbps=$2/Mbps, the cost of transit) and 7Gbps

Chapter 6 – Selecting an IXP

1. A carrier or ISP would be competing with their customers by launching an IXP. ISPs and carrier try to avoid paying their competitors or reinforcing their competitor’s position in the market, so it will be difficult to get critical mass. While they may try and be “neutral” by allowing competitors in to sell to the population in their data center, this may cause customer churn. History has proven that ultimately, the clash between the operation of the IXP business and the operation of the carrier or ISP business will cause the IXP to fail.

2. An ISP might prefer a more expensive IXP if the peers that they seek are in that IXP. An ISP might prefer to support a local IXP operated as a non-profit over a less expensive commercial operation for business alignment reasons. An ISP might prefer a more expensive IXP if there are transit sales opportunities or if they are already in the building so extending transport is inexpensive and logistically easier that building into the less expensive IXP.

3. I would avoid talking about the telecommunications access unless there were some providers scheduled to install. I would talk about the deployment process being easy, about the customer focus and attentiveness. I would avoid talking about the ISP Current Presences since there are no ISPs in the IXP yet. Operations support can be a strong card to play, and business alignment might be a differentiator worth mentioning. If there is flexibility on cost then I would raise that. If there is strong financial support behind the next IXP I would play that card but I would avoid to the extent possible the exchange population and the existing versus emerging exchange. If the intent is to build a strong regional IXP and the routes are underserved, I would play the regional routing strength card.

Chapter 7 – Public vs. Private Peering

1. If an ISP can max out their peering port with 7Gbps then they will realize a cost of $1400/7000Mbps=$0.20/Mbps. To get the equivalent economics with a $300 cross connect the ISP would need to peer $300/$0.20 = 1500Mbps across the $300 cross connect.

2. The larger ISPs tend to have a restrictive peering policy so peer with only a few others, and peer large amounts of traffic. These ISPs are less likely to benefit from peering with a lot of the smaller players on a public switch.

3. The smaller ISPs tend to prefer the public switching fabric to peer small amounts of traffic with as many small players as possible.

Chapter 8 – The 20th Century Internet Peering Ecosystem

1. Poaching customers is easier when competitors know the customer name, contact information and their selected transit provider.

2. Not necessarily. The NSPs have routes that they learn from peers. These routes would not be propagated to other peers, so there is value in buying transit from these players besides customer routes.

3. D

Chapter 9 – The Global Internet Peering Ecosystem

1. ISP Y will say “No, we prefer the revenue.”

2. ISP Y will say “No, we prefer the revenue that we receive indirectly from ISP B, and peering would both reduce that revenue and annoy our customer ISP B.”

3. While ISP A is likely to correctly point out that peering will not reduce revenue for ISP Y, ISP Y is likely to respond by saying that it already hears the ISP A routes for free through its peering with ISP X. Having no desire to annoy his peering relationship and gaining no significant benefit from peering with ISP A, ISP Y is likely to respond by trying to sell ISP A some transit.

Chapter 10 – The 21st Century Internet Peering Ecosystem

1. The content providers seem willing to pay more for a better end-user experience. The CDNs promise the content providers that they will get the content as topologically close as possible to the eyeballs, and since the CDN competitors all pay for paid peering, every other CDN must do the same. ISPs may decide that the cost is about the same and it performs better, so why not?

2. The access networks since there is no alternative path to reach their eyeball customers.

Chapter 11 – The ISP Playbook

1. All of these tactics are manipulative and apply social engineering to some degree. These are effectively identifying the levers (the motivations) and applying pressure (the tactic) to cause the desired effect.

2. This of course is in the eyes of the beholder. Some believe that even the fake NOC outage can be justified as a counter balance to the anti-social refusal-to-peer-openly tactics employed by the target.

Chapter 12 – Taxonomy of Data Centers

1. European model IXPs are generally cheaper. In many cases I would see European model IXPs sell peering ports 30% cheaper than the U.S. model IXPs.

2. The European model IXPs are owned by members. Since the members compete with each other they probably do not want to see their competitors pay a lower price for exchange point services than they pay.

3. The U.S. model IXPs operate a data center in addition to the interconnection services so therefore have a higher operations cost than the European model IXPs.

4. D. Even though they invite competitors into the building it is still a carrier-owned data center.

5. C. This is a classic telco hotel model.

6. B. This is a carrier-neutral data center.

7. A. Buy reselling Internet services to the customers they are a hosting company.

8. C. This is the U.S. model IXP where the company owns the data center and interconnection facilities.

9. A. This is the European model where they only operate the peering fabric.

Chapter 13 – The IXP Playbook

1. Yes, the IXP is past critical mass. On average each member is paying $2000/mo for 30Gbps=$0.06/Mbps. In a market where the price of transit is $1/Mbps, peering provably makes sense here.

2. The question goes to the IXP legitimacy. Are there ISPs there that value the exchange and therefore likely to stay? Are these ISPs going to peer with me now or in the future?

3. Omitted. Send your answer to the author: ipp133@DrPeering.net

4. Omitted. Send your answer to the author: ipp134@DrPeering.net

About the Author

William B. Norton is the Executive Director for DrPeering.net, a leading Internet Peering consultancy. He is an internationally recognized expert on Internet Peering and Internet Exchange Points, a strategy consultant, and a highly sought-after public speaker. He also serves as Chief Strategy Officer for the International Internet Exchange (IIX).

From 1998 through 2008, Bill held the position of Co-Founder and Chief Technical Liaison for Equinix, Inc., a large-scale global provider of peering and colocation services. In this role he spent 90% of his time working directly with the Internet operations community facilitating interconnections, researching, and documenting the industry’s best current practices. The resulting “peering white papers” and the follow-on DrPeering.net website are the most widely cited sources for information on Internet Peering in the industry.

In this capacity, Bill led seventeen North American Network Operators Group (NANOG) Peering Birds-of-a-Feather (BOF) sessions. These came to be a popular fixture at NANOG meetings. He established himself as a thought leader in the peering community by authoring the business models for how companies could use peering relationships to dramatically reduce their network costs. His methodology described in the Business Case for Peering is now used by virtually every company that engages in large-scale traffic engineering. When Bill left Equinix in 2008, it had a market cap of $3.6B.

Prior to Equinix, Bill was the Internet Engineering Manager responsible for, among other things, chairing NANOG, the operations forum for the North American Internet from 1995 through 1998.

Bill received his MBA from the Michigan Business School and his Computer Science degree from Potsdam College.

About DrPeering Press

The DrPeering Internet Operations Series of Books provides the community with insight into operational challenges and how network operators deal with them. These titles will load a first year network operator with the wisdom of a five-year industry veteran.

Visit Us Online

http://DrPeering.net

Save on the eBook

Owning the print copy of the book entitles you to terrific discounts on the electronic version of the book.

PDFs, ePubs, etc. are easy to carry around on your laptop, are searchable and in color, contain hyperlinks, etc. Titles are available in many eBook formats.

Contact Us

Online orders: http://DrPeering.net/books

Customer Service: support@DrPeering.net

Academic: academic@DrPeering.net

Consulting and Speaking Engagements: Phone: +1.650.315.8635

The Internet Peering Playbook is also available in eBook from http://DrPeering.net/books.

The 2014

Internet Peering

Playbook

Connecting to the Core of the Internet

About the Front Cover

The cover photo was contributed by Equinix.

The 2014

Internet Peering

Playbook

Connecting to the Core of the Internet

by

William B. Norton

Published by:

DrPeering Press

http://TheCoreOfTheInter.net

Copyright © 2014 by William B. Norton

The 2014 Internet Peering Playbook: Connecting to the Core of the Internet

Print Edition ISBN: 978-1-937451-11-0

Kindle (.mobi) Edition ISBN: 978-1-937451-17-2

iPad (.iBooks) Edition ISBN: 978-1-937451-18-9

© 2014 by William B. Norton and published by DrPeering Press

Editor: Theresa Norton

All terms in this book that are known trademarks have been appropriately capitalized. The publisher cannot attest to the accuracy of this information. Use of a term in this book should not be regarded as affecting the validity of any trademark or service mark.

This book is designed to inform and entertain. Every effort has been made to make this book as complete and accurate as possible, but no warranty is implied. There may be mistakes, both typographical and in content. This playbook should be considered only as an informational guide and not the ultimate source for building a business case for connecting to the core of the Internet.

It is not the purpose of this book to reprint all of the material that is otherwise available to those engaged in the pursuit of network engineering. This work is meant to complement and supplement other available texts. The information is presented on an “as is” basis. The author and publisher shall have neither liability nor responsibility to any person or entity with respect to any loss or damages arising from the information contained in this book.

2014 Edition

All rights reserved. Printed in the United States of America. This book is protected by copyright and no part of this book may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording or by any information storage and retrieval system, without prior written permission from the author, except for the inclusion of a brief quotation in a review. For information regarding permissions, write to:

DrPeering Press: DrPeering@DrPeering.com

The author is available for a limited number of on-site consulting, workshops, and book signing/speaking engagements each year. Send requests to speaking@DrPeering.net.

This book is continuously updated and available in electronic form in a DropBox folder share. For access to the DropBox, send $9.99 via PayPal to Bill.Norton@gmail.com.

Last updated: 1/2/14

Contents

Foreword to the 2014 Edition iv

Preface – Note to the Reader vi

Section I – Connecting to the Edge of the Internet 1

Chapter 1 – Introduction 3

Chapter 2 – Internet Transit 7

Introduction to Internet Transit 7

The Internet Transit Service Model 7

The Internet Transit Pricing Model 9

Variability in Internet Transit Service 15

Implementation Model for the Internet Transit Service 16

Observations about Internet Transit 17

Chapter 3 – The Internet Transit Playbook 19

Introduction 19

Edge “Tricks of the Trade” 19

Transit Tactic 1 – Optimal Internet Transit 21

Transit Tactic 2 – Gaming the 95th Percentile 24

Transit Tactic 3 – Multi-Homing 27

Transit Tactic 4 – Renegotiate Multi-Year Contracts 29

Transit Tactic 5 – Play the Market 30

Transit Tactic 6 – Resell Transit 31

Transit Tactic 7 – Secret Sauce Transit 32

Transit Tactic 8 – Build into Cheap Transit Internet Region 33

Transit Tactic 9 – Internet Transit Troughs 35

Transit Tactic 10 – Capture Content and Access Customers 37

Transit Tactic 11 – Short-Term Transit 39

Transit Tactic 12 – Deploy Trial Gear into Operational Environment 40

Section II – Connecting to the Core of the Internet 43

Chapter 4 – Internet Peering 45

Introduction 45

Internet Peering 46

Three Key Points about Internet Peering 49

The Top Five Motivations to Peer 50

The Internet Peering Process 52

The IXP Peering Cost Model 63

Chapter 5 – The Business Case for Peering 69

The Cost of Peering 69

Calculating the Cost of Peering 71

Generalized Analysis 72

The Peering vs. Transit Economics 72

Chapter 6 – Selecting an IXP 79

Introduction 79

1 – Telecommunications Access 80

2 – Deployment Process 82

3 – ISP Current Presences 83

4 – Operations Support 84

5 – Business Alignment 85

6 – Cost 87

7 – Credibility 87

8 – Exchange Population 88

9 – Existing vs. Emerging Exchange 89

10 – Regional Route Strength 89

One Final Note 89

Chapter 7 – Public vs. Private Peering 91

Introduction 91

The Top 3 Reasons Public Peering Is Better Than Private Peering 92

The Top 5 Reasons Private Peering Is Better Than Public Peering 94

Hybrid Approach (Public + Private Peering) 96

Section III – The Global Internet Peering Ecosystem 99

Chapter 8 – The 20th Century Internet Peering Ecosystem 101

The First Peering 101

NSFNET-Era Internet Peering, 1987-1994 102

NSFNET Transition Ecosystem, 1992-1996 103

The Early Commercial Internet Model 103

Post-NSFNET Internet in the Growing-Pains Era, 1997-1998 104

Chapter 9 – The Global Internet Peering Ecosystem 109

The Global Internet Peering Ecosystem 109

The Internet Peering Ecosystem 111

The Base Internet Peering Ecosystem 125

Chapter 10 – The 21st Century Internet Peering Ecosystem 127

Evolution #1 – The Cable Companies Peer with Each Other 127

Evolution #2 – Large-Scale Network-Savvy Content Providers Peer 131

Evolution #3 – Cable Companies Peer Directly with Large-Scale Network-Savvy Content Providers 133

Evolution #4 – CDNs Dominate Traffic Volume 134

Evolution #5 – The Video Internet Is Activated 136

Evolution #6 – Access Power Peering 137

Comcast Paid Peering 138

Is Comcast Paid Peering Required? 139

The Netflix, Comcast and Level 3 Story 142

Act I. Introducing Netflix 142

Act II. Level 3 Wins Netflix Contract 142

Act III. Level 3 Requires More Peering Capacity 143

Act IV. Comcast Denies Free Peering Capacity 144

Dénouement. Level 3 Pays Comcast for Peering 145

The Problem with Access Power Peering 146

The Official Story 147

Section IV – The Tricks of the Trade: The Playbooks 149

Chapter 11 – The ISP Peering Playbook 151

Introduction 151

The Graphical Representation of the Plays 152

The Internet Service Provider Peering Playbook 158

Tactic 1. The Extended Direct Approach 159

Tactic 2. Internet Transit with Peering Migration 165

Tactic 3. The End-Run Maneuver 166

Tactic 4. Bundle Internet Transit with Peering 168

Tactic 5. Buy Transit from Restrictive Tier 1 ISP 169

Tactic 6. Transitional Paid Peering 171

Tactic 7. Partial-Route Internet Transit 172

Tactic 8. Play Chicken 173

Tactic 9. Traffic Manipulation 176

Tactic 11. Be Open. Loudly. 182

Tactic 12. Be Everywhere 183

Tactic 13. Get Traffic 184

Tactic 14. Friendship-Based Peering 185

Tactic 15. Spam Peering Requests 187

Tactic 16. The Honey Approach – Be Sweet 189

Tactic 17. Purchase a Well-Peered ISP 190

Tactic 18. Bait-and-Switch 192

Tactic 19. False Peering Outage 193

Tactic 20. Leverage Broader Business Arrangement 195

Tactic 21. Overseas Power Play 196

Tactic 22. Construct Peering from Transit 199

Defensive Peering Tactics 200

Tactic 23. Defensive Tactic – Do Not Peer in Your Home Market 200

Tactic 24. Defensive Tactic – Protect Peering Policy 201

Tactic 25. Defensive Tactic – Change & Escalate Peering Prerequisites 202

Tactic 26. MILD Defensive Tactic 203

Tactic 27. Congest Transit Pipes 206

Peering Tactics That Don’t Work 208

Chapter 12 – Taxonomy of Data Centers 211

The Basic Data Center Model 212

Real Estate 213

Security Systems 224

Power Systems 230

HVAC Systems 235

Data Center Operations 238

The Networked Data Center 245

Carrier-Neutral Data Center 246

Internet Data Center 249

An ISP and Carrier-Neutral Internet Data Center 251

Colocation vs. Wholesale IDC 254

Colocation Providers 257

The European IXP Model 258

The U.S. IXP Model 262

Chapter 13 – The IXP Playbook 273

The Value of an IXP 273

The IXP Playbook Tactics 281

Tactic 1. Bluff the Size of the Population 282

Tactic 2. Build a Network Umbilical for Later IXP Migration 284

Tactic 3. The Group Buy-In 286

Tactic 4. Buy-In Key Players 287

Tactic 5. Divide and Conquer 289

Tactic 6. Beachhead Verticals and Niche Markets 291

Tactic 7. Extend the Dominant IXP 293

Tactic 8. Prevent Rogue IXPs 294

Tactic 9. Run With the In Crowd 295

Tactic 10. Bundling 297

Tactic 11. Build and Maintain Population Stickiness 298

Tactic 12. Strengthen Peering Population 301

Tactic 13. Demonstrate Leadership 304

Tactic 14. Purchase Data Center During Downturns 306

Tactic 15. Drop Peering Cost 307

Tactic 16. Price on Value 309

What Has Not Worked 310

Section V – Advanced Peering Topics 313

Chapter 14 – Remote Peering 315

The Traditional Internet Peering Model 315

The Remote Peering Model 318

Traditional Peering vs. Remote Peering 322

Remote Peering Use Cases 324

Protected or Unprotected Transport Options 331

Implications of Remote Peering to the Internet Peering Ecosystem 331

The Great Remote Peering Debate 336

Chapter 15 - International Peering 339

Simplifying the International Peering Strategy 339

An International Peering Model 339

Traffic Flows 342

International Interconnection Strategies 347

Peering in Emerging Internet Peering Ecosystems 349

Case Study – African ISP A Expands into London 350

International Peering for Content Providers 354

Chapter 16 – The Future 357

Internet Transit in 2014 357

Internet Peering in 2014 358

Answers to Practice Questions 365

About DrPeering Press 371

About the Front Cover 373

At a Glance 383

Acknowledgements 384

At a Glance

Section I Connecting to the Edge of the Internet

Section II Connecting to the Core of the Internet

Section III The Global Internet Peering Ecosystem

Section IV The Tricks of the Trade: The Playbooks

Section V Advanced Peering Topics: Remote and International Peering

Massive interconnection is what separates commodity data centers from colocation centers. In these photos we see overhead wiring trays carrying data (fiber and copper) as well as power (A/C and D/C).

Photos courtesy of Equinix.

Acknowledgements

“You can observe a lot by watching.”

– Yogi Berra

Everything I know about Internet Peering is the result of thousands of conversations with some very smart people in the peering coordinator community. All of these folks were exceedingly generous with their time and kind enough to share their wisdom with me over the past fifteen years. Of these people, I want to single out Avi Freedman, my first peering tour guide. He has always been encouraging and consistently very generous with his time and knowledge.

I will take a few pages here to provide credit where credit is due, to those people who have contributed to the creation of this book, to the drpeering.net website, and to the original research.

The Book. Thanks to the reviewers of the new Remote Peering and International Peering chapters: Bill Flanagan (Flanagan Consulting), Steve Brownworth (Earthlink), Steve Ginsburg (Pandora). Thanks to CyrusOne, Equinix, CoreSite, DE-CIX, AMS-IX, NetNod, Teraco, Jonathan Lassoff, DuPont Fabros Technology, Connie Zhou and Google, and Matt Peterson for providing data center photos for this edition of the book. Thanks to reviewers Sergey Gorinsky (IMDEA), Ignacio de Castro Arribas (IMDEA), Juan Camilo Cardona (IMDEA), Nurani Nimpuno and Emelie Carlsson (NetNod), Christian Koch (Twitter), Zaid Ali Kahn (LinkedIn), Steve Ginsberg (Pandora), Blake Gillman and Tim Parker (GoDaddy), Al Burgio and Morgan Snyder (IIX) and Stephen Wilcox (IXReach) for sharing their experiences and insights about remote peering. Thanks to Nigel Holland of the Right On Band for his never-ending encouragement. Thanks to Will Scott, Scott Landman, Mike Hughes, David Freedman, Tim Pozar, Cat Hoffman, Michael Barrow, and Jeff Turner for their thoughtful and in some cases, very detailed reviews. Thanks to Kyle Van Beveren for his playbook metaphors as a football quarterback and to Alan Hannan for his early feedback as well. A lot of folks have contributed ideas, including Mike Hughes (formerly LINX), Cat Hoffman (Equinix), Dave Meyer (Cisco), Craig Pierantozzi (Level 3), Jonathan Heiliger (Facebook), Raphael Ho (Equinix), Job Witteman (AMS-IX), Frank Orlowski (DE-CIX), Christian Nielsen (Microsoft), Andre Retief (MWEB), Edward Punt (KPN), Sven Engelhardt (TiNet), Frédéric Libotte (BNIX), Pierre Bruyère (BELNET), Jon Terreele (BELNET), James Blessing (LimeLight Networks), and Peter Harrison. Thanks especially to Bonnie E. Hupton and then Theresa Norton for editing this book. They turned my manuscript into a much better book.

The DrPeering.net Site. I also want to thank those who provided ideas, suggestions, and feedback to the ever-evolving DrPeering.net web site. Frank Orlowski (DE-CIX) and I had our first conversations about writing an “Ask DrPeering” column at a GPF in the Dominican Republic. I wanted a web site for hosting the peering white papers. Combining these two, and the Dominican Republic Peering (from which the name is “DrPeering” is derived) idea was hatched. Thanks also to Chris Park (EasyNet), Randy Epstein, Richard Steenbergen (nLayer), Ren Provo (Comcast), and Joe Provo (ITA) for their suggestions about the site over the last few years.

The Original Peering White Papers. Most of all, thanks to all of the generous folks who have contributed their data points and their insights, citable or with anonymous attributions, during the fifteen years I spent in the peering world. All of the research in this book and in the white papers is based on these conversations in the field. All of this material has been rewritten and remains freely available on the http://DrPeering.net web site, a site sponsored by DE-CIX since the beginning.

Many contributors to the white paper research are listed in the following paragraphs. I would like to point out that not only have these people contributed to my research, but they are the ones who consistently volunteered their content, time, and promotion of the peering events that I ran at NANOGs, APRICOTs, Gigabit Peering Forums, etc. Without these contributions this book would not exist.

(The companies in the parenthesis represent the affiliation of the individuals at the time of the white paper walkthrough – many of these people have changed jobs, and in some cases have changed names as well.)

Internet Service Providers and Peering: Ren Nowlin (Onyx networks), Joe Payne (IXC), David Diaz (Netrail), Jake Khuon and Alan Hannan (Global Crossing), Dan Gisi and Jeff Rizzo (Equinix), Patricia Taylor-Dolan (Level 3), Sean Donelan (AT&T Labs), Avi Freedman (AboveNet), Patrick Gilmore (Onyx), Geoff Huston (Telstra), Steve Meuse (GTE), Aaron Dudek (Sprint), Raza Rizvi (REDNET), Wouter van Hulten (InterXion), Michael Palmaffy (Equinix), Beau Sackett (Telus), and Martin Levy.

The Business Case for Peering: Jeffrey Papen (Yahoo!), Mitchell Rose (Digital Island), Dennis L. Shreve (FACET Networks), Jon Castle (Comdisco), Jeff Swinton (Conxion), Jalil Sanad Halim (9 Telecom), Glen Larwill, Nick Monjo (PSINet Australia), Tim Griffin (AT&T Research), Rene Hatem (CA*net3), Todd Braning (Level 3), Joe Klein (Cogent), Ren Nowlin (Carrier1), Joe Provo (RCN), Vince Fuller (MFN), Mike Leber (Hurricane Electric), Brokaw Price (Yahoo! Broadcast), Waqar Khan (Qwest), John Osman (rigozsaurus.com), Mike Hughes (LINX), Ted Hardie (Equinix), Lane Patterson (Equinix), Jason Starr (Equinix), Nic Lewis (AduroNet), Wolfgang Tremmel (via Net.Works), John Zweibel (Procket Networks), James Rice (BBC Internet Services), Beau Sackett (Telus), and Martin Levy.

The Great Public vs. Private Peering Debate: Peering at 10G: Ren Provo (SBC), Richard Steenbergen (nLayer), James Rice (LoNAP), Todd Underwood (Renesys), Stephen Wilcox (TeleComplete), Vanessa Evans (LINX), Niels Bakker (AMS-IX), Chris Malayter (TDS Telecom), Patrick Gilmore (Akamai), Frank Orlowski (T-Systems), Vijay Gill (AOL), Vish Yelsangikar (NetFlix), Nathan Hickson (eBay), Steve Feldman (CNet), Lane Patterson (Equinix), Joy Fender, Falk Bornstaedt (T-Systems), Remco Donker (MCI), Danny McPhearson (Arbor Networks), Josh Snowhorn (NOTA), Vince Fuller (Cisco), Philip Smith (Cisco), Andrew Odlyzko (UMN), and Martin Levy.

The Video Internet: The Next Wave of Massive Disruption to the U.S. Peering Ecosystem: Vish Yelsangikar (NetFlix), Peter Harrison (NetFlix), Aaron Weintraub (Cogent), Jon Nistor (TorIX), Barrett Lyon (BitGravity), Dave Knight (ISC), Aaron Hughes (Caridien), David Filo (Yahoo!), Jim Goetz (Sequoia Capital), Jason Holloway (DoveTail), Matt Peterson, Richard Steenbergen (nLayer), Lane Patterson (Equinix), Eric Schwartz (Equinix), Pete Ferris (Equinix), David Cheriton (Sun), Andy Bechtolsheim (Sun), Jeffrey Papen (Peak Web Consulting), KC Broberg (Rackable), Henk Goosen (Sun), Geoffrey Noer (Rackable), Jeff Turner (InterStream/nuMetra), Vab Goel (NorWest Venture Partners), Ken Hubbard (Quad), Matt Wood (Equinix), Gary Baldus (Limelight Networks), Sylvie Laperriere (VSNL), Ted Seely (Sprint), Bryan Sutterfield (Cox), Patrick Gilmore (Akamai), Phil Thomas (Quad), Doug Wilson (Microsoft), Christian Nielsen (Microsoft), Harald Willison (Covad), Davin Overland, Douglas Galbi (FCC), Ren Provo (AT&T), Susan Martens (AT&T), Richard Clarke (AT&T), the Google Network Engineering and Network Architecture team, James Blessing (Entanet), Sean Donelan (Akamai), Allan Leinwand (Panarama Capital), and Patrick McCullough (Prism VentureWorks).

The Evolution of the US Peering Ecosystem: Ren Nowlin (SBC), Christina Chu (NTT/Verio), Joe Zhu (China Telecom USA), Joe Provo (RCN), Terry Manderson (APNIC–formerly from Optus), Dave McGaugh (PCCW/BTN), Philip Smith (Cisco), Keith Mitchell (ExchangePoint), Fumio Terashima (Japan Telecom America), Geoff Huston (Telstra), Patrick Gilmore (Akamai), Jonathan Chou (Equinix), Phil Grasso (NTT), James Spenceley (Comindico), Chris Chaudry (Comindico), Gavin Tweedie (WAIX), Lloyd Taylor (Keynote Systems), Tatsuo Sato (KDDI America), Sam Habash (SuperNews), Joe McGuckin (Via.Net), Mike Leber (Hurricane Electric), Morgan Snyder (Equinix), Eric Bell (Time Warner Telecom), Peter Cohen (Telia), Dan Golding (AOL), Alexander Tudor and Lance Tatman (Agilent Labs), Eric Aupperle (President Emeritus, Merit Network), Dave Meyer (Sprint, Cisco), Michel Py, Keith Young, Nenad Trifunovic (MCI), Sean Butler, Marcellus Nixon (XO), Mark Seery, and Martin Levy.

The Asia Pacific Peering Guidebook: Lionel Taylor (Reach), Ren Nowlin (SBC), Brokaw Price (Yahoo!), Christian Nielsen (Microsoft), James Spenceley (Comindico), Geoff Huston (Telstra), John Milbourne (Newton Technology Partners), Tom Vest (formerly of AOL Asia), Zwickey Cheung (Asia Netcom), Nigel Titley (FLAG Telecom), Joe Abley (ISC), Norman Hoy (MCI), Gavin Tweedie (WAIX), Ryan Sher (FLAG Telecom), Marcellus Nixon (XO Communications), Stuart Deane (AT&T Asia), Sylvie Laperriere (TeleGlobe), Peter Cohen (Telia), Bill Woodcock (PCH), Phil Koen (Equinix), Jonathan Chou (Equinix Asia), Bob Berger (MetroFi), Toshikazu Saito (POWEREDCOM), Lam Hon Kit (Asia Netcom), Akinori Maemura (France Telecom), Fumio Terashima (Japan Telecom), Dorian Kim (NTT/Verio), Christina Chu (NTT/Verio), James Seng (Singapore IDA), Steve Baxter (PIPE Networks), Marty Gauvin (Hostworks), Phil Tsakaros (Pacific Internet), Sean Carlin (Microsoft), Randy Whitney (MCI), Joe Zhu (China Telecom USA), Joe Provo (RCN), Terry Manderson (APNIC, formerly Optus), Dave McGaugh (PCCW/BTN), Philip Smith (Cisco), Keith Mitchell (XchangePoint), Patrick Gilmore (Akamai), Phil Grasso (NTT), Chris Chaudry (Comindico), Lloyd Taylor (Keynote Systems), Tatsuo Sato (KDDI America), Sam Habash (SuperNews), Joe McGuckin (via.net), Mike Leber (Hurricane Electric), Peter Juffernholz (T-Systems, formerly TeleGlobe), Jeremy Deutsch (Equinix), Sameer Muhammed (Qatar Telecom), Rob Seastrom (ClueTrust), Mike Hatfield, Guy Tal (Global Crossing), Jerry Murray (Global Voice Network), and Jamie Baddeley (fx.net.nz).

The Art of Peering: The Peering Playbook: Joe Klein (Adelphia), Ren Nowlin (SBC Internet), Mitchell Rose (Digital Island/Cable & Wireless), Peter Cohen (Telia), Stephen Stuart (MFN), Jeffrey Papen (Yahoo!), John Harkin (ATG), Ingrid Erkman (ICG), Jeb Linton (EarthLink), Paul Nguyen (Google), Paul Vixie (PAIX), Peter Juffernholz (TeleGlobe), Michael Winslow (Williams), Blake Williams (Williams), Scott J. Ellentuch (TTSG Internet Services), Chris Cousins (Carrier1), Michel Py, William F. Maton, Raven Alder (Intermedia), Kevin Epperson (Level 3/University of Colorado), Geoff Huston (Telstra), Joe St Sauver, Eric Aupperle (formerly President of Merit), James Rice (BBC Internet Services), Mike Hughes (LINX), Josh Horn (Terremark NOTA), Andrew Partan, Eric Anderson (BT Ignite), Chris Sandford (INS), Avi Freedman (Akamai), and several folks who wished to remain anonymous including Martin Levy.

The Folly of Peering Ratios: Peter Cohen (Telia Sonera), Richard Steenbergen (nLayer), Patrick Gilmore (Akamai), Vikas Mehta (AOL), Bryan Garrett (BellSouth), Brian Sutterfield (Cox), Kevin Mulvahill (Equinix formerly from @Home), Steve Wilcox (Telecomplete), Louie Lee (Equinix), Brokaw Price (Yahoo!), Joe Provo (RCN), John Curran, Hamish MacEwan (CityLink), Brandon Galbraith, Maureen Carroll (T-Systems), Vince Fuller (Cisco), Joshua Sahala (TWTC), and Ren Provo (SBC).

Do ATM Exchange Points Make Sense Anymore? Dorian Kim (NTT/Verio), Ingrid Erkman (ICG), Dave McGaugh (ELI), Eric T. Bell (Time Warner Telecom), Chris Parker (StarNet), Lane Patterson, Jay Adelson, Morgan Snyder, and John Hardie (Equinix), David Diaz (BellSouth), Joe Wood (Accretive Networks), Robert Seastrom (inter.net), Kevin Epperson (Level 3), Petri Helenius (FICIX), Scott Sheppard (BellSouth), Ralph Doncaster (iStop.com), Leo Bicknell (ufp.org), Paul Vixie (vix.com), Ian Somerton and Dave Wodelet (Shaw/BigPipe), Tony Hain (Cisco), Jeff S. Wheeler (five-elements.com), Cliff Hafen, Dory Liefer, Shannon Lake (Omnivergent), Nenad Trifunovic (WorldCom), Andre Gironda (eBay), Jeb Linton (EarthLink), Daniel Golding (SockEye), Peter Moyer (Juniper), Louis Lee (Equinix), Waqar Khan (Qwest), Andrew Odlyzko (UMN), and Gary Blankenship (Foundry).

The Art of Peering: The IX Playbook: Chris Quesada (PAIX/Switch and Data), Josh Snowhorn (NAP of the Americas/Terremark), Job Witteman (AMS-IX), Frank Orlowski (DE-CIX), Keith Mitchell (formerlyLINX) and Randy Whitney (UUNet).

I would also like to thank Elise Gerich, who, during my days at Merit Network, had faith that I could write the NANOG business plan and assume the role of chairing NANOG. I grew into that role, which after three years positioned me to take on the startup adventure with the title of Co-Founder and Chief Technical Liaison for Equinix. I spent 15 years researching peering with the NANOG community, so the release of this book can be traced back to that one step Elise took in 1995.

A very important thanks to Jay Adelson, Al Avery, Equinix, and more recently, Al Burgio and the International Internet Exchange (IIX) team. They allowed me and continue to allow me the freedom to travel around the world to research, document, and evangelize peering. They understood that the information I gathered for the peering community is made freely available to customers and competitors alike. The whole industry benefited from this collection and dissemination of information on how peering works, and it requires vision backed by a hefty bank account to allow me to do this research. Thanks for allowing me to help this community.

I would also like to thank the good people at the Internet Society (ISOC) for their continued support. In particular, Karen Rose, Michuki Mwangi, Dr. Dawit Bekele, Jane Coffin, Leni Nazare, and the rest of the crew, with the support of a grant from Google, have been purchasing and employing thousands of copies of this book for promoting peering and providing peering training across Africa. They have paid for this book to be translated into French, and supported my peering evangelism activities for years. The Internet Society rocks.

And of course, the biggest thanks to my loving wife, who, in addition to providing invaluable editing services, endured my endless talk of the minutia surrounding this book project. She also gave me free legal advice, carved out the time and space necessary for me to focus on writing, and always believed in my success. A more supportive and encouraging spouse could not be found. More than anyone else, this book would not have been possible without her.