Балашов eucass 2013 (Лекции 2014-2015), страница 2

To test these critical constraints on theRTA system operation (including the data exchange), the testing tools must support precise timing of the RTAsystem’s responses.1For certain types of interfaces, e.g. MIL STD-1553B, the same channel may operate as both input and output for theRTA device.3V.V. Balashov, M.V. Chistolinov, R.L. Smeliansky4. Support for testing of dependability features of the RTA system. Dependability testing requires simulation of faultsof the devices connected to the RTA device or subsystem under test, including data exchange faults.5.

Support for multiple-computer testbench configurations. The number of onboard channels in an RTA system canreach several dozens or even hundreds. Multiple instrumental computers with onboard interface adapters must beinvolved to provide full-scale testing of the RTA system. The testing tools must support such multiple-computerconfigurations, in particular provide time-synchronized sending of data into channels from different instrumentalcomputers, as well as synchronized checking of the RTA system’s responses.6.

Support for sharing the onboard interface adapters with monitoring tools. This feature allows to avoid installationof extra adapters to simultaneously perform testing of the RTA system and monitoring of data exchange throughchannels in the system under test. It is important both for large testbenches with dozens of adapters and for mobiletesting and monitoring workstations, with a limited number of adapters (no more than two in an industrial laptop).7. Support for different testing modes: automatic testing (without user’s intervention) – for checking the responses received from the RTA systemthrough onboard channels; interactive testing (the user replies to requests from the tests) – for checking the data indicated on thedisplay devices of the RTA system.8.

Support for online visualization of the testing process. This feature allows the user to track the current state oftesting and to react to possible deviations. Following information is subject to online visualization: values of test data and the data received from the system under test; testing logs.9. Support for requirements traceability and generation of reports on the testing results. Functional testing is anintegral part of the RTA system lifecycle and is aimed at verification of requirements to the RTA system.Consequently, the testing tools must support: binding the requirements to the test cases; generation of the requirements traceability matrix; generation of reports on the testing results and on fulfilment of requirements.10.

Integration with RTA system software development tools, such as: version control tools (for keeping the testing scripts for a particular RTA system’s software version togetherwith the software source code for this version); requirements management tools.11. Integration with onboard interfaces database. This feature allows the testing tools to generate the interface partof the testing scripts, in particular the specification of the set of sent/received messages and structure of their datafields, based on the information from the database. The onboard interfaces database contains the description of dataexchange protocols for the RTA system.12. Support for operation on a mobile platform. To create a mobile testing workstation based on the testing tools it isrequired that both the user interface and the test execution environment run on a common hardware/softwareplatform.4.

Applicability of general-purpose functional testing tools to testing of RTAsystemsIn this section we consider two existing functional testing toolsets and estimate their applicability to the task oftesting the RTA system software on the target hardware without target instrumentation.

The toolsets, namelyRational Test RealTime and VectorCAST, are not “purely” general purpose. They are designed to support testing ofdistributed real-time systems, however there are several common issues that substantially complicate theirapplication to the above mentioned task.Rational Test RealTime (RTRT) [1] from IBM is a cross-platform solution for testing real-time applications,including networked and embedded software. RTRT provides facilities for automated target-based software testing,memory leaks detection and code coverage analysis.

RTRT is integrated with revision control and configurationmanagement tools from IBM (ClearCase, ClearQuest).RTRT was originally developed for unit testing. System testing support for distributed applications was implementedlater. Target-based testing requires loading auxiliary modules (system testing agents) to every device (computer) ofthe target system which runs software to be tested. Agents are responsible for supplying test data to the softwareunder test, reading the resulting data and communicating with the testing control computer.RTRT supports two test description languages: Component Testing Script Language for unit tests and SystemTesting Script Language for system tests.Conventional (recommended by vendor) technology of RTRT application to target-based testing of avionics softwarerequires loading auxiliary software modules to the target devices. This approach is not suitable for acceptance and4A FAMILY OF TESTBENCHES TO SUPPORT TESTING OF REAL-TIME AVIONICS SYSTEMSfield testing.

It is also questionable whether the auxiliary modules significantly affect real-time characteristics oftarget software operation.RTRT was applied in Sukhoi company for testing of Sukhoi Superjet civilian aircraft avionics. This applicationinvolved custom “adaptation layer” software running on dedicated computers and translating test data from RTRT toonboard channel messages sent to the (non-instrumented) target system. Development of such intermediate softwareis a complex task which requires access to internal details of RTRT implementation.Testing solution based on RTRT with adaptation layer provides communication with the target system on“parameter” level, where parameter is a variable taken by the target application software as an input or generated asan output.

This level is not suitable for testing devices that take or generate “binary” data (e.g. video frames, digitalmaps etc). It is also hardly suitable for dependability testing which requires low-level stimulation of the target system(e.g. injection of communication faults into channels or simulation of power failures).VectorCAST [2] from Vector Software is an integrated toolset for unit and functional testing. It supports testgeneration, execution, code coverage analysis and regression testing.

VectorCAST can be integrated with third-partyrequirement management tools through VectorCAST / Requirement Gateway module.VectorCAST provides special features for avionics software testing. In particular, it implements recommendations ofDO-178B standard which describes avionics software development processes. VectorCAST was applied todevelopment of avionics software for several aircraft including JSF, A380, Boeing 777, A400.VectorCAST supports unit and integration testing of avionics software on target platform.

Similarly to RTRT,VectorCAST requires loading auxiliary software modules (VectorCAST/RSP, Runtime Support Package) to targetdevices. This raises the same issues as mentioned above for RTRT.There is no open information available on existence of “adaptation layer”-like software for VectorCAST which couldallow application of VectorCAST to testing of non-instrumented RTA systems through onboard channels.Development of such software layer is problematic as VectorCAST is a closed commercial product, and even if suchlayer existed, the resulting solution would have same limitations as noted above for the RTRT-based solution.Both testing toolsets reviewed in this section provide a rich feature set, but have similarly limited applicability to thetask of testing of non-instrumented RTA systems.Table 2 provides a comparison of RTRT, VectorCast and the FT toolset according to the criteria (requirements)introduced in Section 3.Table 2: Functional testing toolsets’ features according to the requirements from Section 3RequirementFT toolsetRational TestRealTimeVectorCASTSupport for functional testing ofsoftware on the target devicewithout instrumentation of thedevice+(testing is performedthrough onboardinterface channels)-/+(development ofmodules notincluded in thedelivery package ofthe tool is required)-/+(development ofmodules notincluded in thedelivery package ofthe tool is required)Support for onboard interfacestandards used in the RTA system+(support for allonboard interfacestandards used inRTA systems of theaircrafts to which thetoolset is applied:Su-35, T-50)-/+(development ofmodules notincluded in thedelivery package ofthe tool is required)-/+(development ofmodules notincluded in thedelivery package ofthe tool is required)Support for testing of real-timeconstraints on the RTA systemoperation+-+/(loading of auxiliarysoftware moduleson the target systemis required)Support for testing of dependabilityfeatures of the RTA system+-/+-/+5V.V.

Balashov, M.V. Chistolinov, R.L. SmelianskySupport for multiple-computertestbench configurations+--Support for sharing the onboardinterface adapters with monitoringtools+(cooperatively withtools from the“Channel analyzer”family [4])-/+(adaptation ispossible, but is notavailable out-of-thebox)-/+(adaptation ispossible, but is notavailable out-of-thebox)Support for different testing modes+++Support for online visualization ofthe testing process+(online visualizationfor: testing logs; testdata and responsesfrom the testedsystem, in table andgraphical forms)-/+(use of externaltools is required)-/+(use of externaltools is required)Support for requirementstraceability and generation ofreports on the testing results+/(binding ofrequirements to thetest cases;traceability matrixgeneration;generation of reportsfrom testing results)-/+(traceability – viaexternal tools,detailed reports –only for unit testing)-/+(traceability – viaexternal tools,detailed reports –only for unit testing)Integration with RTA systemsoftware development tools+/-+/--/+Integration with onboard interfacesdatabase+/(generation of theinterface part oftesting scripts fromthe DB)--Support for operation on a mobileplatform+-/+-/+5.

The toolset for functional testing of RTA systemsIn this section we describe the toolset for functional testing of RTA systems (FT toolset) utilized by Sukhoi DesignBureau for testing of modern aircraft RTA systems. This toolset is developed in the Computer Systems Laboratory(CS Lab) of Computational Mathematics and Cybernetics Department of Lomonosov Moscow State University.5.1 Toolset overviewIn contrast to the Rational Test RealTime and VectorCast toolsets considered above, the FT toolset is aimed attesting of RTA system through onboard interface channels without loading any auxiliary software to the RTAdevices.