Ethernet: Past, Present and Future, Part 3

This three-part series began with the origin of Ethernet, the foundation for most networking communications today. In Part 2, we looked at the longevity and upgradability of future-proof FPGA platforms. Now, it’s time to look at how the pluggable front port technology that is available today complements the flexibility of future-proof FPGA-based smart NICs.

The flexible properties of front port technology are:

1. Level of backward compatibility
2. Number of link types supported
3. Number of link rates supported
4. Available lane breakout functionality

For those looking to support link speeds up to 400G, options abound on the market today for IEEE/MSA-defined front port form factors. Only a reduced subset of these meet the physical outline limitations imposed by the PCIe standard. With focus on the high-end smart NIC use case, this article will explore the PCIe-compliant multilane-based subset.

Level of Backward Compatibility

What constitutes backward compatibility in this instance? A front port module form factor is considered backward compatible if it supports deployment of at least one previous generation of pluggable modules. The multilane pluggable form factors are currently dominated by two families: the QSFPx family and the CFPx family.

These two families seem to provide the same functionality, at first glance, but a more careful inspection reveals significant differences in their offering, having justified their co-existence thus far. The CFPx family is targeted at telecom use cases where performance is more important than price and power consumption. Whereas the CFPx family has no backward compatible members, the QSFPx family is fully backward compatible, enabling deployment of all previous pluggable family members, in a given form factor member. The latest addition to the QSFPx family is the QSFP-DD form factor, which brilliantly introduces a doubling of the available number of lanes on the system side without sacrificing backward compatibility.

Number of Link Types Supported

What the market wants, as well as the power envelope of the form factor, determines the number of link types supported by a given form factor. The link types supported by the QSFPx form factor family is primarily defined by datacenter and enterprise applications. The SR link type is used massively in this segment, but LR and PSM volume is ramping due to the mega-sizing going on in the datacenters. With a primary focus on the telecom use cases, the CFPx form factor family members have historically provided the space needed to implement the current relevant link types, including the power-consuming long-haul distances.

The gap in the supported link types between the CFPx and the QSFPx family typically increases for each addition of new CFPx family members. Subsequently, over time, the gap decreases as the technology allows the QSFPx form-factor to catch up. This is a phenomenon that is usually driven by commercial market demand.

Number of Link Rates Supported

Form factor backward compatibility does more than enable multi-link speed support.  Deployment of dual Ethernet rate modules allows the smart NIC to support several Ethernet link speeds without physically changing the front port pluggable module.  As an example, a QSFP28 port will enable deployment of a 100GBASE-SR4 dual-rate module in the following four modes:

1. 4x 10G ports, breakout
2. 4x 25G ports, breakout
3. 1x 40G port
4. 1x 100G port

There are several dual-rate offerings available at present, supporting the QSFPx form factor family and adds further flexibility to this form factor.

Available Lane Breakout Functionality

2010 marked a turning point. This is when the 40GBASE link-type family was introduced, and for the first time, there was a standardized multi-lane-based pluggable front port form factor: the QSFP+. This form factor deploys four 10G lanes on the system side in order to handle the 40G aggregate bandwidth on the media side. In a market environment that is constantly struggling to increase port density, it was quickly recognized that this form factor provided the highest available 10G port density at that point in time, resulting in the availability of the first breakout connectivity solutions. Figure 3 shows a breakout solution mapping directly between an SR4 port and four SR ports (MPO/LC), in this case 100G to 25G but also supporting 40G to 10G.

At first, only breakout solutions could support multimode fiber infrastructure, as the single mode link types had the four 10G lanes multiplexed one fiber on the media side. With this limitation, breakout solutions had a significant limitation with respect to native 10G ports. With the MSA standardized introduction of 40GBASE-PSM4 (Parallel Single Mode 4) pluggables, this limitation is gone, leaving breakout front ports as a viable and mostly very attractive alternative to native 10G port form-factors. .

It must be noted that the MPO <-> LC breakout cable type deployed in Figure 3 is not applicable for use cases relying on existing structured LC connected cable infrastructure. A rack mount shelf holds one or more modules converting between the existing structured LC connectivity and the breakout MPO-based connectivity.

In addition to the two options above, a third breakout solution has recently been introduced that uses the QSFP to SFP adapter type. The QSFP form-factor adapter simply maps a QSFPx port into an SFPx port, obsoleting the need for the breakout solutions shown in Figure 3. Nothing comes without a cost, and in this case, the cost for obsoletion of the breakout cabling is that three-fourths of the available lanes in the front port become inaccessible.

The Flexible, Adaptive Network

Based on the information above, it seems clear that the most flexible form currently available is the QSFP28 form factor. Ethernet had humble beginnings but has expanded to be foundational to modern networking; the internet, for instance, could not exist without it. FPGA technology has adapted over time to support FPGA-based network acceleration cards. These changes now allow the design of very flexible hardware platforms. Front port technology offers today’s and tomorrow’s networks the flexibility they need. What may come tomorrow is anyone’s guess, but Ethernet will always hold a place of prominence in the history of networking.