Fabric technologies for AdvancedTCA: No king in an agnostic world

AdvancedTCA ecosystem buildup and proliferation is in an exciting phase. Attractive choices on management modules and so on have actually resulted in transforming the AdvancedTCA community. The form factor has gone well beyond the primary focus of Telco carrier grade applications into enterprise and data center applications as well. In the fabric agnostic world of AdvancedTCA, vendors will deploy applications and value added services in a competitive yet interoperable fashion taking into consideration performance, cost, reliability, availability and features. Interconnect architectures like Serial RapidIO, PCI Express, Advanced Switching, Ethernet, InfiniBand, and StarFabric will target different price points and application requirements.

Introduction
Equipment manufacturers can leverage time-to-market, cost, and product-flexibility benefits by adopting standard off-the-shelf hardware in the form of chassis, management modules, backplane interconnects, single board computers, and the like as specified by AdvancedTCA standards. By focusing instead on high availability middleware, reliable management software, and operating systems and efficient product integration, system vendors can offer differentiated application ready platforms to end customers.

Selecting the best I/O technology appropriate for a specific application is an extremely critical link in this design chain. Ethernet, for example, has been an early leader in the AdvancedTCA world because of the significantly lower hardware cost per port as a result of its widespread adoption as a cost effective bandwidth solution. Closer examination of total cost of ownership rather than just hardware per port cost might however reveal that technologies like InfiniBand will work out to be more competitive, at least for a class of solution. Attempts to extend Ethernet to address carrier grade requirements like high availability, high reliability, and Quality of Service (QoS) capabilities may however rob the technology of the advantages of cost and design simplicity. Uncertainties associated with such technology readiness might make the cost and the risk high enough to tip the balance to existing solutions like RapidIO, which can address most of these requirements today, although it is not as widespread today as Ethernet.

We can thus infer that multiplicity of fabric choices for AdvancedTCA applications are really not a hindrance to the proliferation, as some would argue. In fact, limiting interface choices will adversely affect competing innovations needed for the AdvancedTCA solution space at this stage. Current and future I/O technologies ratified through the PCI Industrial Computer Manufacturers Group (PICMG) 3.x subsidiary specifications will continue addressing different aspects of application and systems requirements, and these are expected to coexist to enable affordable application proliferation.

Overview of PICMG 3.x subsidiary specifications
The PICMG 3.0 base specification defines the systems management, thermal management, power distribution, mechanical components, and other such detailed characteristics of AdvancedTCA cards, chassis, and backplanes as well as the protocol for the base interconnect between cards (Ethernet). The PICMG 3.x subspecifications address the various high-speed interconnect technologies mapped over AdvancedTCA backplanes. An add-in card however must notify the system manager which fabric it supports before interconnects are enabled onto the backplane.

Figure 1 captures the fabrics currently mapped onto AdvancedTCA.

So far PICMG has announced releases of the following PICMG 3.x subspecifications for high-speed interconnects over AdvancedTCA backplanes.

In addition, a new subspecification, PICMG 3.6, Packet Routing Switch (Cell Switching) architecture (PRS), is currently going through the specification cycle.

PICMG 3.1, for example, is a cospecification that adds to the core AdvancedTCA specification the capability of supporting Ethernet and Fibre Channel data link (L2) and physical layers (L1) over the PICMG 3.0 generic backplane fabric interconnect. PICMG 3.1 thus will take advantage of the core PICMG 3.0 specification for its basic mechanical, thermal, electrical, power, connectivity, and system management requirements. In specific terms though, PICMG 3.1 defines all elements necessary for interoperability between multivendor PICMG 3.1 products, including but not limited to:

Similar specifications of addressing schemes, virtual lane mappings, initialization, keying, hot swap support, and hardware management have been included in the PICMG 3.2 specification, which defines how InfiniBand transport is mapped onto the PICMG 3.0 base by specifying the link physical layers, protocols, and protocol mappings needed to implement multivendor, interoperable systems.

The PICMG 3.3 specification defines the electrical environment for StarFabric as a fabric interface for AdvancedTCA system applications.

The 3.4 specification defines switched, high-speed, low-latency point-to-point connectivity among node, fabric (switch hub) and full mesh boards for PCI Express as a fabric interface for AdvancedTCA system applications.

Finally, the PICMG 3.5 specification defines the design rules and guidelines for implementation of RapidIO-based node and fabric cards on top of the PICMG 3.0 standard.

Major ecosystem players
PMC-Sierra recently announced its RSE 160 Serial RapidIO Switch Element, a scalable 16-port switch device for wireless infrastructure equipment. This is an exciting addition to the growing RapidIO ecosystem, which also includes Serial RapidIO-enabled DSPs and processor products.

The C6455 DSP announced by Texas Instruments, based on the TMS320C64x+ DSP core is now sampling, bringing together TI's highest performing DSP architecture with Serial RapidIO support to boost performance and I/O bandwidth in high-end and multichannel applications including:

System performance is boosted 12x because Serial RapidIO eliminates I/O bottlenecks by providing a low latency, high bandwidth (10 Gbps full duplex), low pin count interconnect.

Freescale Semiconductor and Tundra Semiconductor have been building up the ecosystems with RapidIO based processors and switch fabric solutions for the community and are being joined by others making major Serial RapidIO product announcements.

Gigabit Ethernet switch vendors including Marvell, Broadcom, Agere and Vitesse have been able to leverage their existing switching solutions towards PICMG 3.1 Gigabit Ethernet switch blades. Broadcom, Fujitsu and other Ethernet switch vendors are making significant investments towards extending their solutions to support 10 Gbps bandwidths along with improved traffic engineering capabilities and greater QoS.

As for the other I/O technologies, Diversified Technology, Inc. is focusing on InfiniBand based AdvancedTCA solutions like PICMG 3.2 based fabric in their ATC5232 and ATS2148 blade systems. StarGen with their AXSys architecture as well as Intel and others are leading the push for ASI bases fabric solutions in the AdvancedTCA community.

3.2 Interconnects and applications
Many of the applications now on the horizon, however, will require higher-speed switching technologies with advanced QoS capabilities. For these more demanding applications, the AdvancedTCA specifications cover a range of high-performance switching technologies, including 10 Gigabit Ethernet, PCI Express/Advanced Switching, and RapidIO.

RapidIO supports High Availability (HA) systems with significant bandwidth capabilities and quality of service. Used in proprietary systems, RapidIO will offer a scalable backplane solution with multiple suppliers. Used in AdvancedTCA systems, RapidIO provides an interoperable fabric interface for high-performance applications, significantly enhancing the value of the standardized chassis.

Announcements from PMC Sierra, Tundra, Mercury Computers and other vendors on their next generation RapidIO devices and platforms may be indicative of a trend where RapidIO is making strides in expanding market acceptance beyond DSP based applications. Although early adoption and popularity of RapidIO for media rich DSP applications can probably be attributed to a significant boost the industry got from TI’s integration of high performance DSP architecture with serial RapidIO, at this point RapidIO appears to be creating quite some excitements in next generation carrier grade and embedded application areas as well within the ATCA universe.

As for Ethernet, although it cannot strictly be termed as a system interconnect, significant number of vendors had selected Gigabit Ethernet for initial AdvancedTCA systems. Ethernet technology's mature ecosystem and the consequent competitive pricing make it attractive for example in the server blade market. Even telecom applications may find some virtue in it since it allows line card investment protection. It also provides scalability paths through the AdvancedTCA compliant 10 Gbps Ethernet XAUI interface for most of the current and future high bandwidth applications.

Ethernet however can’t currently be termed as a carrier grade technology since this best effort mechanism does not offer the high reliability, high availability and greater QoS the telecom applications demand. Upgrading the technology to address these requirements will challenge the simplicity and the cost effectiveness it currently enjoys vis-à-vis other technologies. In addition backward compatibility and system wide changes may eventually make such upgrading effort unviable.

PCI Express (PCIe) has become the dominant IO technology adopted widely by the computer industry. Advanced Switching Interconnect (ASI) as the successor of the PCIe is however yet to find widespread support as native or integrated processor interface. This necessitates deployment of gaskets and bridge devices (PCIe to ASI) for ASI based solutions. The accompanying price and complexity increase may slow down or even inhibit market adoption for this interconnect technology in areas like switch fabric applications, which had proved to be attractive for PCI Express.

Notwithstanding all the AdvancedTCA based product announcements from Diversified Technology, Inc. (DTI) and others, InfiniBand is yet to develop significant industry support. Another fabric option, a new cell-switching interconnect technology based on the existing Applied Micro Circuits Corp PRS switch product line, is currently being considered the newly formed PICMG 3.6 subcommittee.

Comparative analysis of fabric interconnects
The design choice among available topologies and interconnect protocols can ultimately be governed by overall cost and maturity considerations as well as the requirements metrics for the application, which might include bandwidth, reliability, availability, serviceability, and QoS.

With standardized components prescribed for AdvancedTCA systems, the overall cost and risk points of such designs improve considerably as developers are able to avoid proprietary designs and leverage the availability of standard Commercial Off-the-Shelf (COTS) components. This significantly benefits not only the entry cost (capital expenditure) of the system but also the total cost of ownership, which includes the operational expenses because an open standard usually enables a mature ecosystem at lower cost.

Generally, the cost of the backplane increases with the amount of bandwidth provided. However, the ability to provide a common backplane for a range of applications while allowing interconnect technology upgrades via board-set changes rather than complete shelf-up gradation can result in significant cost, support, and downtime savings.

For a cost comparison basis Gigabit Ethernet often has the best clarity – lowest per port hardware cost among all the PICMG 3.x fabrics. What needs to be added to this relatively simplistic cost basis is really the equivalent capacity cost (how does it compare, for example, with 10G capacity of x4 PCI Express?) after adding management software and other systems cost. A more expensive hardware choice may combine with cheaper open source software solution to yield a competitive price point – of course open source software also has its hidden integration cost.

Clearly considerations like CPU utilization, offload engine, and other logic costs also need to be included for a fuller analysis of performance gained for the total system cost. Such studies can always result in a surprise winner based on target applications.

Studies have often constructed different merit indices based on additional issues like ecosystem maturity, which often affects time to market and cost components positively because it generally is predicated upon more robust availability of training, development, and support expertise. The flip side of such wider dissemination of course may well have to be competitive system level differentiations against possible trend toward commoditization.

AdvancedTCA backplane example
A multiservice switch (Figure 2), which can target a wide variety of industry applications such as enterprise networking, telecom switching, wireless infrastructure, and media gateways can be realized on an AdvancedTCA platform because the backplane fabric interconnect is not tied to any single I/O technology. Figure 2 shows Schematics of a Serial RapidIO version of a multiservice switch. Multiservice switches often support the most I/O-intensive applications.

Figure 2 (click to zoom)

Conclusion
Emergence of a single, compelling, cost-competitive, and yet highly reliable high-speed interconnect for all AdvancedTCA applications does not appear feasible, at least for now. One can even argue about the desirability of such a single interconnect in view of the competing innovations needed to grow the AdvancedTCA solution space. Current and future I/O technologies ratified through the PICMG 3.x subspecifications will continue addressing different aspects of application and systems requirements and are expected to coexist.

Recognizing that there can’t be a single fabric solution which can address competing demands of cost, proliferation, performance, reliability, availability, and other QoS parameters, we have to move towards ensuring interworking of best-of-class fabric solutions across applications such as edge and core routing, media gateways, medical imaging, SIGnals INTelligence (SIGINT) for radar and military imaging, and server farms.

Admittedly interconnect architectures have a significant impact on the performance and capabilities of any system. AdvancedTCA vendors however can thank the PICMG standards committee for having a wide range of choice on interconnect technologies based on bandwidth, QoS capabilities, component availability, capital and operational expense considerations, and ecosystem maturity.

Partha Datta Ray is vice president IC Engineering and Strategic Solutions, GDA Technologies, Inc. He is an active member of the RapidIO Trade Association.

GDA Technologies, Inc.
1010 Rincon Circle
San Jose, CA 95131
Tel: 408-467-3500
Fax: 408-432-3091
E-mail: partha@gdatech.com
Website: www.gdatech.com

If you found this article valuable, click here for your complimentary subscription to CompactPCI and AdvancedTCA Systems magazine. You can also subscribe to the CompactPCI and AdvancedTCA E-Letter, our free e-mail newsletter, by clicking here.