Ethernet interfaces in a high traffic growth environment

(This article was written for the IEEE Communications Magazine, but not accepted due to technical limitations (the number of pages they can allocate for 100 G subject), it migth be published later)

Introduction

For this article a part of KPN’s ISP network is explored. KPN is the incumbent operator in The Netherlands, offering fixed and mobile, voice, data, internet, TV and ICT services. The local broadband market is well developed, probably due to the presence of both cable and telephony infrastructure in almost every household. And because of the generally dense population, fiber and wireless services are also offered by new entrants. 70% Of the households in The Netherlands are using broadband Internet and from these 5 million broadband connections; KPN has a market share of approximately 40%.
Currently the highest Ethernet interface speed is10 Gbps.

Configuration

The focus is on the part of the network that connects to the public infrastructure of the Amsterdam Internet Exchange. (AMS-IX). This is generally not public information, but it is the least sensitive part of the network. It is being used for peerings with over 200 AMS-IX connected organizations. The traffic between the peers and the service platform and users takes the path over a router, DWDM equipment and the AMX-IX switch. The technology used on the IX is Ethernet, as this is the most efficient solution to handle traffic between the AMS-IX connected routers.
In the event of failure, the links function as a backup for each other. Any of the two links can handle all the traffic. As the traffic is above the highest Ethernet interface speed, bundling (a.k.a. aggregation) of multiple interfaces is being used.

Traffic history

For capacity planning purposes the weekly high of all the different connections are being noted. This offers a great opportunity to create insight in the traffic history of the AMS-IX connection.

The slight decrease during the summer periods is followed by a strong growth in the months following. When studying this figure more in-depth the annual growth shows high variations.

Annual growth
2003 39 %
2004 162 %
2005 228 %
2006 81 %
Average 127 %

The high variation is partly caused by moving peers from private connections to public IX infrastructure and back. This is being done in order to limit the investments in interfaces.
The peak traffic with the AMS-IX was 27 Gbps in June 2007.

Expected traffic growth

We believe that our future traffic growth will be lower than it has been over the past few years. This decrease is mainly due to high broadband penetration already reached. The traffic will continue to grow because of increasing access speeds and intensifying use of high bandwidth applications like video.
To explore the future traffic volumes, three scenario’s have been drawn up. The first and highest growth scenario is a continuation of the average annual growth of 127%. The second and lowest growth scenario is based on the lowest annual growth of 81%. The third scenario consists of simply doubling the traffic every year, in other words: plus 100% every year.

As can be seen, in 2009 a traffic volume of 100 Gbps will be reached and around 2012 – 2013 the traffic will hit the 1 Tbps level.

Limits of current interfaces

Currently the maximum Ethernet interface speed is 10 Gbps exist. An approach for understanding the impact of the growth, is exploring the limits of aggregation of 10 Gbps interfaces.
The traffic volumes have to be handled by interfaces. An interesting point is that traffic volumes are at a level where they cannot be handled by one interface. This was especially true at the end of the life cycle of interfaces, but when 100 Gbps interfaces become available, they will be aggregated from the start.

Interface Aggregated interfaces
100 Mbps No aggregation
1 Gbps 2
10 Gbps 3 (so far)

In the current configuration on both locations, three 10 Gbps interfaces are aggregated to facilitate the traffic. Two upgrade cycles are foreseen for the remainder of this year.

The current available maximum of aggregation is eight times 10 Gbps and sixteen times 10 Gbps is underway.
There are several reasons for letting aggregating 8 interfaces being the maximum:
Even though larger numbers of aggregates will be possible, it seems wise not to overstretch above the earlier aggregation experience. There are always issues with solving hardware / interfaces limitations by software. An other reason for having 8 interfaces as the maximum limit is the total number of interfaces. In the above configuration, having 8 AMS-IX connected 10 Gbps interfaces, would mean about 60-70 10 Gbps interfaces on the two routers. They will all become obsolete when upgrading to higher speed interfaces, and reusing the 10 Gbps Router interfaces is only possible using equipment of the same supplier.When using 16 interfaces as the maximum as reference of determining the introduction moment for higher speeds interfaces, over 130 10 Gbps interfaces will have to be replaced on the two routers. Because the routers connect to other equipment, the total number of interfaces is of course twice the number of router interfaces. This means 130 interfaces in the 8 aggregate case and 260 in the 16 aggregate case.

Based on the expected traffic growth the timeline for replacing the 10 Gbps interfaces is as follows:
8 interfaces as maximum: Q4, 2008 – Q1 2009
16 interfaces as maximum: Q3, 2009 – Q3, 2010
The differences of these timelines mean installing over 60 router interfaces in 1 to 1.5 year in the core of the network. Due to the exponential growth rate, this means up to 5 new 10 Gbps router interfaces a week when reaching the 16 aggregate levels. This is obviously an operational challenge compared to the ten interfaces of a year ago.

Transport over WAN

Simply having higher speeds interfaces available for connections between routers and switches, could be the easiest part of the solution to an access provider. That is because the traffic has to be transported to end users that are typically 100’s of km’s away. This is also the case between the AMS-IX switch and the Rotterdam router, in the case used in this paper. The current optical network delivers 10 Gbps on several interfaces. There is already an ITU standard for 40 Gbps (STM-256), which can facilitate part of the traffic growth, but it is obviously not the needed Ethernet. Deploying this standard in the optical network (OTN), would mean a reduction of the number of interfaces in the network. It will include installation of routers close to the AMS-IX switches to act as an interface translators. This can probably be deployed before higher speeds Ethernet interfaces become available.
Deployment of 100 Gbps Ethernet over the OTN would solve the number of interfaces issue. The impact on the optical network however, could be rather significant. Using 40 Gbps Ethernet over the OTN will have less impact, but could be a solution for a very limited time, because of the growth rate.
An alternative could be a dedicated fiber solution for the Internet traffic.

There are options for getting 100 Gbps of traffic over the WAN before the end of 2009. Getting the technology in place for 1 Tbps over WAN in the 2012 – 2013 timeframe, is mind puzzling.

Conclusions

The case described in this article regarding the traffic between the AMS-IX and the ISP infrastructure is not the only area of concern of the network.
In other parts of the world with higher traffic volumes or higher traffic growth because of broadband deployments, similar issues may occur at an earlier stage.
For the higher speeds interfaces, it is simply the sooner the better for maintaining operational stability.
Aggregation should be regarded as an integral part of interface standard development, because the first implementations of the higher speeds interfaces, will include aggregation.
Traffic volumes of 1 Tbps between two end-points are only five years away.

-=-
Biography:
Ad Bresser is Chair Innovation Office at KPN Service Operations and one of the founding members of the Higher Speeds User group. In the past he was responsible for building KPNs ISP network infrastructure and he has been an executive board member of the AMS-IX association for four years. Before joining KPNs subsidiary Planet Internet in 1998 as an Internet architect, Mr Bresser worked as an engineer, consultant and architect with Cap Gemini for 9 years. In this timeframe he provided over 50 customers with network and Internet solutions.
Ad Bresser holds a BSc in technical computer science and in 1988 his thesis was on a diagnostics’ tool for Ethernet.