Using virtual platforms to improve AdvancedTCA software development practice
| By |
|
Jakob details how using virtual models of a rack-based system greatly helps software development, test, optimization, and debug.
The value added by system OEMs in telecommunications and data communications is moving from custom hardware solutions to standard hardware running custom software solutions. Software adds most of the value, differentiation, and customer-facing features to a system. Thus, software development becomes the dominant cost and main source of risk in new systems projects, and this requires rethinking how systems are developed. Using virtual platforms is one powerful tool that has been applied by leading system OEMs to improve the software-intense system develop-ment process.
Introduction to virtual platforms
A virtual platform is a piece of software running on a regular PC that behaves like the physical target hardware. It contains a model of the complete target system, including backplane, boards, mezzanine cards, and both standard and custom processors, chips, and FPGAs. The virtual platform is thus able to boot and run the same and complete software stack as the physical hardware platform. Using a virtual platform brings many benefits to the development, test, debug, and analysis of software. Figure 1 shows the concept of virtual platforms.
|
|
|
Figure 1 (click graphic to zoom by 1.9x) |
Virtual platforms are often discussed in the context of consumer electronics and similar systems. However, it would be a mistake to consider virtual platforms to be limited to small systems. The technology scales up to very large systems containing scores of boards, hundreds of processors, and multiple rack backplanes. Thanks to recent technology advances, multicore processors in host workstations and servers can be used to accelerate the virtual system execution. Figure 2 shows a small conceptual example based on Power Architecture processors. Note that the system uses several different processor types, contains some SMP subsystems, and runs several separate software stacks.
|
|
|
Figure 2 (click graphic to zoom) |
Whether a platform is based on custom technology or standards does not matter for software designers and virtual platforms. Regardless of whether the hardware being virtualized is a cutting-edge custom ASIC or a standard board that has been in production for years, the software designers still find a virtual platform useful.
Early hardware
The classic virtual platform use case is to start software development in parallel to hard-ware development. As soon as a specification of the hardware is available, a virtual platform can be created. The virtual platform is often available months or even years in advance of the actual hardware.
In rack-based systems, this means that new and old boards will be present in the same virtual platform, making it possible to develop software and perform system integration of the new board before it becomes available in physical form. In our experience, software development can be started many months earlier than is the case when having to wait for physical prototypes to appear, and the time required to successfully integrate a new board into an existing system is drastically reduced. Most of the work is already complete when the hardware arrives, rather than starting at that point.
Building a virtual AdvancedTCA board
For the AdvancedTCA world, virtual platforms can make designs based on new processors available well in advance of physical boards. For example, Virtutech recently provided a virtual platform for the new QorIQ P4080 processor from Freescale, enabling operating system and middleware vendors to roll out their software at the time the new processor was announced, rather than having to wait for final silicon and boards to arrive. Building a virtual AdvancedTCA board for such a processor allows customers to develop software early, without having to wait for the ecosystem to complete the board development cycle.
A virtual platform is typically more available to developers than a physical hardware platform. Seen as "just software", virtual hardware can be copied and used in any number of instances at any one time. Engineers can have target systems at their desks with no need to make lab time reservations, walk to a lab, or search for a particular type of board. There is no supply limit related to the number of physical boards of a particular type available to the development organization.
Virtual platforms can be easily scaled, from a single board to an entire network node, using the same host system. Changing the scope is a matter of changing the virtual platform configuration, rather than locating and (re)configuring a particular piece of physical hardware. This makes it much more convenient for developers to test their software in complex environments, which directly improves test coverage andthe robustness of the code developed.They also compile the software for the actual target, using the target compilers and target build settings, thus testing the actual binaries that will eventually be deployed on the physical target.
It also means that developers are run-ning their code on hardware that has the same word-lengths, instruction sets, and endianness as the actual target hardware. Usually, rack-based systems contain pro-cessors of different types, encompassing Power Architecture, MIPS, ARM, x86, and other types in 16-bit, 32-bit, and 64-bit variants. It is easy to write code that works on a particular host PC but that fails on the actual hardware. With virtual platforms, the testing on the target happens sooner, reducing the risk of bugs due to bad assumptions and coding mistakes.
Checkpointing
Virtual platforms also provide for developer convenience. It is possible to bring up a system, load software on it, and then save the state as a checkpoint before starting experiments. Loading software into flash memory and other permanent storage can be achieved using back doors, saving development time. Going back to the same state is as easy as reloading the checkpoint of the virtual system, with no need to reboot, reload, and retrace steps taken.
The system checkpoint contains the com-plete hardware configuration, software configuration, and current software state in an encapsulated package. The checkpoint can be sent to colleagues anywhere in the world. It makes working with the software more like writing documents in Word in that you can save your work at any time, as well as send out a version for others to see and make comments.
Improved debugging
Software debugging is much easier because the target system is completely control-lable, and any part of its state can be inspected. With a virtual platform, you achieve synchronized system stop across all processors, boards, and racks that are part of a system under test. For example, you can single-step interrupt handlers on a particular processor without worrying about the effect on the rest of the system.
Another unique property of virtual platforms is repeatability; any system execution can be replayed in exactly the same way that it happened before, even on a system containing multiple processors and multiple boards (Figure 3). Thanks to this, we have seen intermittent software glitches that were impossible to find on physical hardware getting fixed very quickly. Once you have a fault triggering on in a software execution on the virtual hardware, you can repeat it any number of times in exactly the same way. This is incredibly powerful to help debug concurrent, distributed, and parallel software.
|
|
|
Figure 3 (click graphic to zoom by 1.9x) |
Virtual platforms reverse debugging facilities make it possible to run the system back-wards in time. With reverse debugging, the system state is not lost when the system goes down or a piece of software crashes - developers can simply look back in time and inspect the events leading up to the crash (or other issue). We have seen this cut average bug fixing times by a factor of four or more, greatly improving software quality and reducing schedule risks due to show-stopper bugs.
As virtual platforms provide back-door access to a system, it is possible to access any part of the system state without disturbing it. There is no probe effect. For example, thisfeature was used to locate a file-system corruption bug within a matter of days that had previously required several engineering months on hardware, by the simple method of periodically pausing the virtual system, performing a global synchronized consistency check across all boards, and using interval halving to pin down the exact time at which the system became corrupted.
Virtual platforms can also diagnose and debug system execution from the first instruction executed. The development team can debug the bootup of a single processor, board, or system, observing the system behavior long before any normal diagnostics functions have had time to get started.
Fault injection
The ability to control and inspect a virtual platform's state at any point in time or space makes it possible to do precise fault injection to test the behavior of the system in the presence of faults. Redundant hardware and software only help if backups actually do take over when a primary fails, and virtual platforms make it easy to virtually remove a board from a system, create intermittent and permanent network and backplane faults, and even inject bad traffic.
Communication across the development chain
Virtual platforms can improve the communication between suppliers and users in the long chain of development from hardware vendors to operating systems and other fundamental software vendors to system developers to testing and quality assurance. Testers and developers who find a bug can send a system checkpoint to the responsible parties, making reproducing the issue much easier and more reliable than trying to communicate how to set up a system and how to run it to arrive at a problem.
With a virtual platform it is also possible to trace the detailed interaction between software and hardware in a system. This makes it easy to resolve conflicts between software and hardware teams over where the root cause of a non-functioning system is to be found, reducing the typical rounds of discussion over where a problem is located.
Steps to introducing virtual platforms
Introducing virtual platforms does not require a major change in how software is developed, but the introduction can be gradual with immediate gains and longterm benefits. A virtual platform connects to the software development tools already in use and requires no special software builds or compilers to be used.
In our experience, it works very well to start by deploying a virtual platform for a single board or small set of boards to prove the concept, and then build on from that base. Since boards tend to have a long life, these initial virtual platforms will remain useful for years to come, as more and more boards are virtualized and made available in the virtual environment. There is no need to go for a big-bang approach.
Conclusion
Virtual platforms are a proven technology that has helped many major systems companies develop software-intense products faster and with better quality. The more complex the software gets, the more important it is to have good tools to test, analyze, and debug it. Using virtual platforms can improve AdvancedTCA software development practice and help manage the complexity of modern multiprocessor, multiboard, and multirack systems.
Virtual platforms have proven to be very successful for Power Architecture processors and platforms. PowerPC and Power Architecture hardware used in telecom, datacom, mil-aero, servers, and high-performance computing have been virtualized, improving system quality and reducing time to market.
Power.org, an active industry association with the mission to develop, enable, and promote Power Architecture technology, has a technical subcommittee for simulation and virtual platforms, which is looking for technology and standards to drive the wider use of virtual platform technology as the preferred open standard hardware development platform for the electronics industry and to administer qualification programs that optimize interoperability and accelerate innovation for a positive user experience.
