Балашов eucass 2013 (Лекции 2014-2015)

5TH EUROPEAN CONFERENCE FOR AERONAUTICS AND SPACE SCIENCES (EUCASS)A Family of Testbenches to Support Testing of Real-TimeAvionics SystemsV.V. Balashov*, M.V. Chistolinov** and R.L. Smeliansky **** Department of Computational Mathematics and Cybernetics, Lomonosov Moscow State UniversityLeninskie Gory, MSU, 1, Bldg. 52, Room 764, Moscow, Russia, hbd@cs.msu.su** Department of Computational Mathematics and Cybernetics, Lomonosov Moscow State UniversityLeninskie Gory, MSU, 1, Bldg. 52, Room 764, Moscow, Russia, mike@cs.msu.su*** Department of Computational Mathematics and Cybernetics, Lomonosov Moscow State UniversityLeninskie Gory, MSU, 1, Bldg.

52, Room 764, Moscow, Russia, smel@cs.msu.suAbstractSystematic testing is performed on different phases of real-time avionics (RTA) system developmentto ensure fulfilment of requirements to the RTA system. In this paper we focus on the problems oftesting of an RTA system as a hardware/software system, involving the software running on the targethardware. A testing toolset and a family of testbenches to support solving of these problems arepresented, along with methodological aspects of their application on appropriate RTA systemdevelopment phases.1. IntroductionReal-time avionics (RTA) systems are subject to strict requirements to functionality, dependability and real-timeoperation. To ensure fulfillment of these requirements, systematic testing of the RTA system is performed ondifferent phases of RTA system development.Figure 1 shows a typical V-model of the RTA system’s software lifecycle.

Testing activities are performed on thephases shown on the right side of the “V” shape. While unit testing of RTA system software can be performed oninstrumental computers, integration and acceptance testing, as well as testing of series-produced RTA systems,necessarily involve the real target hardware.Production, operationand maintenanceProjects andrequirements planningAnalysis of product requirementsand specificationsAcceptance testingTop-level architectureproject designIntegrationDetailed designUnit testingCodingFigure 1: The RTA system’s software lifecycle V-modelCopyright  2013 by V.V. Balashov, M.V.

Chistolinov and R.L. Smeliansky. Published by the EUCASS associationwith permission.V.V. Balashov, M.V. Chistolinov, R.L. SmelianskyIn particular, the target hardware is required to test the requirements to real-time operation, to data exchange throughthe onboard channels, and to interaction of different software layers (OS, service software, application software).General-purpose functional testing tools, such as Rational Test RealTime [1] or VectorCAST [2], requireinstrumentation of the RTA system’s software to support testing the application software on the target hardware.Instrumentation includes loading of auxiliary software modules to the target hardware, which is incompatible withintegration testing and acceptance testing. These tools do not support analysis of data exchange through onboardchannels by the testing scripts. Consequently, application of Rational Test RealTime, VectorCAST or similar tools toRTA systems testing requires significant customization of the tools, which is hindered by the fact that their sourcecode and internal interfaces are closed.In this paper we present a toolset for functional testing of RTA systems (farther referred to as the FT toolset).

The FTtoolset is aimed at testing without target instrumentation. This toolset is developed in the Computer SystemsLaboratory (CS Lab) of Computational Mathematics and Cybernetics Department of Lomonosov Moscow StateUniversity. Testbenches based on the FT toolset are utilized by Sukhoi Design Bureau for testing of modern aircraftRTA systems. These systems consist of multiple devices (computational nodes, sensors, actuators, indicators)connected by dozens of data transfer channels.The rest of the paper is structured as follows. Section 2 highlights the main RTA system testing problems occurringon different phases of the RTA system’s lifecycle.

In Section 3, according to these problems, the requirements toRTA systems functional testing tools are stated. Section 4 analyzes applicability of two general purpose functionaltesting toolsets to the task of target hardware-based testing of RTA systems and emphasizes significant issuesundermining this applicability. Section 5 presents the FT toolset, describes its architecture and features. Section 6describes the typical architecture of a testbench based on this toolset. Section 7 describes the family of testbenchesbased on the presented toolset, each testbench aimed at a particular activity (or a group of activities) of the RTAsystem lifecycle. Section 8 presents a scheme for joint operation of these testbenches during RTA systemdevelopment and maintenance.

In the last section, future directions for development of the described technology areproposed.2. Problems of RTA system testing on different lifecycle phasesLet us consider the main RTA system testing problems, specific for different phases of RTA system’s lifecycle.Table 1 shows these problems for each phase and activity of the RTA system lifecycle on which the target hardwarebased testing is performed.Table 1: RTA system testing problems on different lifecycle phasesActivityPhaseRTA system testing problems1. Software integration with thetarget hardware, debugging of thesoftware on the target platform(typically, an individual device ofthe RTA system)IntegrationSelective execution of tests according to thesoftware functionality being debugged.2.

Integration of subsystems of theRTA system. Integration of thewhole RTA system.IntegrationEnsuring the correctness of integrated RTAdevices’ communication through theonboard interfaces.3. Acceptance testing of the RTAsystem’s software.AcceptancetestingAutomated execution of the full set of RTAsystem’s functional tests.Manual testing control to allow choosing thetests execution order and repetitiveexecution of tests.Checking the correctness of inter-devicecommunication for all devices of the RTAsystem.2A FAMILY OF TESTBENCHES TO SUPPORT TESTING OF REAL-TIME AVIONICS SYSTEMS4.

Acceptance testing of seriesproduced RTA devices andcomplete RTA system instances.Acceptancetesting,ProductionAutomated execution of the full set of RTAsystem’s functional tests.Easy replacement of RTA devices during thechange of the tested RTA system instance.5. Diagnostics of RTA devicessubject to reclamations.MaintenanceConstruction of a “reference” environmentfor the faulty device.Automated execution of test sets specific fora particular device.6. Diagnostics of RTA devicesonboard the aircraft.MaintenanceNeed for a mobile and compact hardwaresolution for testing.Activities 1-6 represent a virtually complete set of testing activities on the integration, acceptance testing andproduction/operation/maintenance phases.

For each of these activities, testing is performed in dedicatedhardware/software environments known as testbenches. The testing technology, including instrumental software andtestbench architecture, should be unified to reduce the complexity of testing and to provide reusability of test scriptsthrough different phases of the RTA system lifecycle.In this paper we do not consider unit testing of the RTA system’s application software, as it is typical to perform unittesting on software developers’ instrumental PCs without involving the target hardware.3.

Requirements to the tools for functional testing of RTA systemsThe main type of software testing performed during the activities 1-6 (see Section 2) is functional testing of softwarerunning on the target RTA devices. Each target device is considered a “black box”, inputs of which are the inputonboard interfaces and outputs are the both output1 onboard interfaces and visual indication facilities (if present). Formore details on functional testing and black-box testing, see [3].Sending of the test data to the RTA devices and obtaining the resulting output data is performed through onboardchannels. During the workout of inter-device communication through the channels, it is necessary to verify thecorrectness of data exchange. So the testing scripts must have access to the data exchange monitoring results.As the correctness of data indication on the onboard display devices is visually estimated, the testing tools mustsupport interactive mode of operation in which the user is asked for confirmation of testing success.It is reasonable to use both testing tools and data exchange monitoring tools on the testbench.

The monitoring toolscan be used to analyze “boundary” issues in which it is unclear whether a test has failed due to an error in the RTAdevice operation or in the test itself. An overview of several monitoring tools is provided in [4].According to the present testing experience of Sukhoi design bureau, the following requirements to the tools forfunctional testing of RTA systems can be stated.1. Support for functional testing of software on the target device without instrumentation of the device. The testingtools must support testing of software on the target device without loading of any auxiliary software modules to thedevice.2. Support for onboard interface standards used in the RTA system.

To perform full-scale testing of the RTA systemas a hardware/software system, the testing tools must support all the onboard interface standards used in the system.Following main features are necessary: sending of the test data through the onboard interfaces and obtaining the responses for subsequent analysis; monitoring of data exchange through the onboard interface channels and providing access to monitoringresults to the testing scripts; composition and analysis of low-level contents of the messages transferred through the channels, in order totest the devices which operate “raw” binary data without decomposition into individual parameters.3. Support for testing of real-time constraints on the RTA system operation.