This article, the third in a four-part series on AdvancedMC technology, provides an overview of the advantages of PCI Express architecture as an AdvancedMC interconnect. The final article in the series will examine the potential benefits of PCI Express architecture with the Advanced Switching protocol.
The PCI Industrial Computer Manufacturers Group (PICMG) AdvancedTCA specification for carrier-grade telecom applications (PICMG 3.x) supports multiple standardized system fabric interconnect technologies, including PCI Express architecture. Related to AdvancedTCA is the Advanced Mezzanine Card, also known as the AdvancedMC, or PICMG AMC.x standard. PCI Express architecture provides a compelling interconnect technology for bandwidth-intensive AdvancedMC usage models, including applications based on I/O processors, general purpose processors, network processors, and coprocessor applications. The applications include crypto acceleration, security, Digital Signal Processing (DSP), and transcoding.
Why PCI Express architecture?
Achieving Gbits/sec-class peak bandwidths means existing mezzanine card parallel buses must use faster and wider Peripheral Component Interconnect (PCI) buses. While this approach yields low Gbits/sec peak bandwidths for very brief periods, sustained bandwidths occur in the hundreds of Mbits/sec range. In order to achieve these bandwidth gains, existing parallel interconnect technology imposes tradeoffs, including higher pin count and more complex implementation requirements. Even with a large number of extra ground pins to absorb loop current, faster PCI buses remain point-to-point interconnects. Supporting multiple devices on a mezzanine card requires the addition of a bridge chip, an approach that consumes power and valuable mezzanine card real estate. PCI Express architecture represents the evolution of the PCI chip-to-chip interconnect to high-speed serial technology. Migrating from the parallel PCI bus to PCI Express architecture opens the door to higher-bandwidth devices and allows systems to support multiple connected devices, without incurring the limitations of an onboard bridge chip. PCI Express also significantly increases pin efficiency, minimizing connector cost, size, and the mezzanine area required for the external connection. Yet another great advantage of PCI Express is its backward compatibility with PCI, including full implementation of all of PCI’s address spaces, semantics, and transactions. In addition, PCI Express adds new capabilities including:
Scalable bandwidth, from 2.5 Gbits/sec per 1X lane up to over 32 Gbits/sec for 16X lanes on AdvancedMC.
Quality of Service (QoS) support with 8 Classes of Service (CoS).
High reliability based on a protocol that enables precise error detection and recovery within the link layer.
High-system availability and hot-swap capability.
Low pin count with fewer Printed Circuit Board (PCB) layers than parallel technologies require.
PCI SIG has officially sanctioned PCI Express to succeed PCI. The PCI Express architecture enables PCI products to migrate seamlessly. Expect most if not all PCI applications to migrate to PCI Express over the short term.
Physical layer The AdvancedMC connector was designed to handle up to 10 Gbits/sec signals per 1X lane, sufficient enough to meet the requirements of the first-generation PCI Express physical layer at 2.5 Gbits/sec per 1X lane, in addition to second-generation PCI Express that will double the bandwidth to 5 Gbits/sec per 1X lane. The increased bandwidth that the second-generation PCI Express physical layer supports may be used over the long term to increase delivered bandwidth for the most demanding AdvancedMC-based applications. Over the short term, it will enable cost reduction and design simplification by halving lane use. It is important to note that PCB layout and routing guidelines for PCI Express architecture specifically address edge connectors, including the connector used in the AdvancedMC specification.
(story continues below sponsor)
PCI Express usage models Figure 1 shows the basic usage models for PCI Express with AdvancedMC. A simple PCI Express system consists of a host processor and an arbitrary number of I/O devices and coprocessors or DSP chips. The number of host processor ports determines external switch requirements. This may be a fan-out switch that all traffic flows from a peripheral to the host processor or vice versa, or it could be an Advanced Switch supporting arbitrary peer-to-peer switching.
[Figure 1]
These silicon building blocks can be allocated to AdvancedMCs in a completely arbitrary manner. The three AdvancedMCs shown in Figure 1 illustrate a number of possible variations:
AdvancedMCs can contain multiple devices, each with an independent PCI Express interface offboard. Variation 1a in Figure 1 is an example of how onboard devices can have native PCI Express interfaces.
In variation 1b of Figure 1, a bridge chip is placed on the AdvancedMC, making it possible to use any device based on a load-store protocol. This design supports the long-term migration of PCI devices to PCI Express, necessary to extend migration far beyond the introductory phase of AdvancedMCs. An onboard bridge chip also allows the use of non PCI devices with PCI Express-based mezzanines, as long as they employ a load-store protocol.
Variation 1c of Figure 1 places a switch on the AdvancedMC. This enables any number of devices to reside on the mezzanine, while only requiring one mezzanine interface.
[Figure 2]
Figure 2 shows an AdvancedMC that implements a fast data path I/O card. While the data path uses an appropriate protocol such as System Packet Interface (SPI), UTOPIA, or Advanced Switching, the devices on the AdvancedMC still need a management and provisioning control plane interface. Rather than consume a large number of pins to implement a simple load-store type interface, a 1X PCI Express Link, using only two lanes, can achieve the same result using minimum real estate while preserving full backward compatibility with PCI. This approach also allows PCI Express and data-plane devices to mix.
[Figure 3]
Figure 3 shows some of the possibilities for supporting multiprocessor mezzanines. These processor mezzanine usage models could range from a simple low-power control processor, to a micro server blade. This usage model requires nontransparent bridges, typically one for each processor.
Figure 3a places the nontransparent bridge on the AdvancedMC with the processor. The bridge can be a standalone chip with very low pin count, or the processor or chipset could integrate bridge functionality. A standalone chip may include multiple inside ports, each implementing a different load-store interconnect.
Figure 3b shows the nontransparent bridges integrated into PCI Express switch ports, a design approach that consumes less total AdvancedMC real estate but assumes that the processor on the mezzanine natively supports PCI Express.
These nontransparent bridges would implement silicon features to support interrupts, mailboxes, alarms, heartbeats, failover, and other interprocessor communication mechanisms.
The configurations in Figures 1 through 3 support most AdvancedMC usage models, including general-purpose processors, coprocessors, I/O processors, and memory cards. I/O processors work especially well in storage and networking applications, such as TCP offload. Note that these configurations apply equally to appliances and chassis-based equipment.
High availability and management High Availability (HA) and manageability are also important capabilities of AdvancedMC mezzanines. PCI Express supports and extends HA because it is a fundamentally reliable fabric. It implements a reliable link layer, retry on error packets, credit-based flow control, no dropping of packets, native hot add/remove, QoS support with eight classes of service, and in-band messaging for interrupts and other control information.
PCI Express can detect a wide variety of errors, some of which are capable of automatically correcting. Flagging any errors that are not capable of automatically correcting correction by software. In either event, the system is not corrupted.
One or two 32-bit Cyclical Redundancy Checks (CRCs) protects every packet. Each packet, includes a mandatory link-layer CRC and an optional transaction-layer l CRC. Packet errors detected at the link layer are automatically retransmitted, while transaction-layer errors require software intervention. The CRCs can also detect other types of errors including overflows, timeouts, training errors, and a variety of packet format and protocol errors.
The switched nature of PCI Express architecture supports hot swapping. Isolating, removing and replacing a faulty device or mezzanine card, all takes place without disturbing other devices. High resiliency enables PCI Express architecture to continue operating despite one or more lanes failing. For example, if one or more pins on the AdvancedMC connector were to fail, affected devices would renegotiate their link to find the next narrower list link capable of supporting communications. This feature enables a complete fault diagnosis.
Summary PCI Express architecture is one of the standard system fabric interconnects supported by the PICMG AdvancedTCA and AdvancedMC specifications. The scalable bandwidth of PCI Express architecture, together with its high-availability features and real estate-saving low pin count, make it the logical interconnect for AdvancedMC in a variety of design implementations. In addition to supporting a wide range of AdvancedMC bandwidth requirements, full PCI software compatibility will greatly facilitate the migration of today’s PCI-based applications to PCI Express.
In the next and final article in this series, we will take a closer look at Advanced Switching, the standard backplane technology that reuses the physical and link layers from PCI Express to provide highly scalable bandwidth and slot count while supporting multiple protocols and topologies.
John Beaton is the interconnect manager for Intel's Network Processing Group that manages the marketing programs to support the penetration of PCI Express and Advanced Switching in communications and embedded market segments. His recent work at Intel has included strategic marketing and long range planning, product definition, and market analysis. He holds a Bachelor's and Master's degree in electrical and mechanical engineering, and has 15 years experience in engineering, sales, and marketing.
Mark Summers is a senior engineer in Intel's Embedded Intel Architecture Division. Mark is currently the chairman of the PICMG AMC.0 AdvancedMC base specification. Mark has 18 years of experience in system and component electronics packaging, with a specialization in rapid prototyping and severe environment systems packaging and analysis. Before coming to Intel, Mark worked 15 years at Motorola. He currently holds 19 U.S. patents and has written several technical papers.
. . . . .
For further information, contact either John or Mark at:
John Beaton
Intel Corporation
5000 W. Chandler Blvd.
Chandler, AZ 85226
Tel: 480-554-1327
Fax: 480-554-7674
E-mail: john.beaton@intel.com
Web site: www.intel.com
Mark Summers
Intel Corporation
5000 W. Chandler Blvd.
Chandler, AZ 85226
Tel: 480-554-1258
Fax: 480-554-7674
E-mail: mark.d.summers@intel.com
Web site: www.intel.com
