A. Wood - Softare Reliability. Growth Models (798489), страница 2
Текст из файла (страница 2)
In theory all failures should be reported using TPRs. However, whencoding and unit testing the software, it might take a developer longer to submit a TPR thanto fix the problem and retest. Therefore, developers do not usually submit TPRs for theirown products until after RQA. After a product has reached the RQA point, all failures ofthat product are required to be reported via TPRs. Therefore, we have good defect datareporting after RQA. During development or test, a product group may experience a failureof another product.
These failures are also reported using TPRs, so TPRs may come fromsources other than the product test group.TPRs are reported against the release that is being tested. It is possible that a defect was putinto the software in a previous release and was not found until a later release, but it is verydifficult to determine the exact time when an error was made.
It is also possible that theoriginal code was not defective until other parts of the code changed. Reporting defectsagainst the release in which they are found may over count defects that should have beenattributed to a previous release, but it should similarly under count defects from the currentrelease that are not found until later releases. It also replicates the way customers view therelease.2There are different types of Tandem software releases. Some are large modifications tomany products, some are minor modifications to some products, and some are defect repairfor only one or a few products. The major software releases follow the product life cycleprocess and have large, coordinated QA efforts from most QA groups.
We appliedsoftware reliability modeling to these major releases. Interim releases and defect repair donot always follow a complete life cycle and QA process. The processes are tailored to fit theamount of change in the release. There is considerable variability in these types of releases,and we did not attempt to model them.There are many different software reliability growth models, and many different ways torepresent the data that is used to create those models. The current software reliabilityliterature is inconclusive as to which models and techniques are best, and some researchersbelieve that each organization needs to try several approaches to determine what works bestfor them.
This effort is an ongoing experimental effort that attempts to determine the bestapproach for Tandem. For the models to be useful, they must add value to the corporatedecision making process, e.g., they are used to help determine when to ship a product orsize the sustaining effort after the product is shipped. In order for the results to be used forthese types of corporate decisions, the results must be stable and must predict the fieldfailure rate with reasonable accuracy (see Section 2.5).TerminologyAs defined by IEEE, an error is a human action that results in afauIt. Encountering a faultduring system operation can cause afai/ure. A bug is synonymous with fault, and a defectis very similar. This paper follows those defmitions. The words defect and bug are used tomean code that does not satisfy the user requirements, either because a requirement isincorrectly designed or implemented (the vast majority) or was not implemented.
A failureis what a customer or tester encountered that caused them to report the defect.1.2Approach and OrganizationSoftware reliability growth models have been applied to portions of several releases overthe past few years. Since this document is accessible by non-Tandem employees, thespecific products and releases are not mentioned and the data has been appropriatelytransformed. The models have been applied to the QA test period - from RQA to QAOK.The purpose of this report is to describe Tandem's experience with reliability growthmodels over the past few years. The report contains the data we gathered and thetechniques used to analyze the data.
Section 2 presents software reliability growth modeltheory while Section 3 describes Tandem's experience with various models and techniques.Section 2.1 describes the data necessary for software reliability growth models, andSection 2.2 presents several model types. In Section 2.3, different statistical techniques formodel parameter evaluation are presented. Section 2.4 introduces some ideas for evaluatingmodel utility. Section 3.1 presents the raw data used for the models.
Section 3.2 presentsthe basic results we achieved and our evaluation of the model utility. Sections 3.3-3.8present the results obtained by varying model parameters.There are many variables to be considered in developing a software reliability growthmodel methodology. These variables are listed in Table 1-1. We have experimented withmany combinations of these variables over the last few years, and the results of thoseexperiments are also shown in Table 1-1. The sections of the report that describe theparameters and contain the results of varying them are also shown in the table.3OptionsExecution (CPU) timeCalendar timeNumber of test casesDefect DataDefectsTPRsGrouped Data Defect occurrence timeWeekly (grouped) summaryMany (Table 2-1)GrowthModel TypeStatisticalMaximum likelihoodClassical least squaresTechniqueAlternative least squaresParameterAmount ofTestingResult SummaryExecution time is theonly option that worksReport Section2.1.1, 3.3TPRs are a goodsurrogate for defectsGrouped data works fineand is easier to collectSimple exponential (G0) model works bestAlternative least squaresprovides the best pointestimates.
Maximum2.1.2, 3.42.1.3, 3.72.2, 3.52.3, 3.6l~elihoodispreferredfor confidence intervals.Table 1-1. Model Parameter Options2.0Software Reliability Growth ModelsReliability is usually defined as the probability that a system will operate without failure fora specified time period under specified operating conditions. Reliability is concerned withthe time between failures or its reciprocal, the failure rate.
In this report we are consideringdata from a test environment, so we report defect detection rate rather than failure rate. Adefect detection is usually a failure during a test, but test software may also detect a defecteven though the test continues to operate. Defects can also be detected during designreviews or code inspections, but we do not consider those sorts of activities in this report.Time in a test environment is a synonym for amount of testing, which can be measured inseveral ways.
Defect detection data consists of a time for each defect or group of defectsand can be plotted as shown in Figure 2-1. We can derive defect detection rates from thisdata.NumberofDefectsTest TimeFigure 2-1. Example Defect Detection Data4A cumulative plot of defects vs amount of testing such as Figure 2-1 should show that thedefect discovery rate decreases as the amount of testing increases. The theory is that eachdefect is fixed as it is discovered. This decreases the number of defects in the code, so thedefect discovery rate should decrease (the length of time between defect discoveries shouldincrease).
When the defect discovery rate reaches an acceptably low value, the software isdeemed suitable to ship. However, it is difficult to extrapolate from defect discovery rate ina test environment to failure rate during system operation, primarily because it is hard toextrapolate from test time to system operation time.
Instead, we look at the expectedquantity of remaining defects in the code. These residual defects provide an upper limit onthe number of unique failures our customers could encounter in field use.Software reliability growth models are a statistical interpolation of defect detection data bymathematical functions. The functions are used to predict future failure rates or the numberof residual defects in the code. There are different ways to represent defect detection data asdiscussed in Section 2.1.
There are many types of software reliability growth models asdescribed in Section 2.2, and there are different ways to statistically correlate the data to themodels as discussed in Section 2.3. Current software reliability literature is inconclusive asto which data representation, software reliability growth model, and statistical correlationtechnique works best. The advice in the literature seems to be to try a number of thedifferent techniques and see which works best in your environment.
In Section 3, wedescribe the application of the techniques in the Tandem environment.2. 1Software Reliability Growth Model DataThere are two relevant types of data for software reliability growth models. The first is thetime at which the defect was discovered, discussed in Section 2.1.1, and the second is thenumber of defects discovered, discussed in Section 2.1.2. Aggregated or grouped data isdescribed in Section 2.1.3.2.1.1 Test Time DataFor a software reliability growth model developed during QA test, the appropriate measureof time must relate to the testing effort.
There are three possible candidates for measuringtest time:- calendar time- number of tests run- execution (CPU) time.Test time can simply be calendar time, particularly if there is a dedicated test group thatcontinuously runs the test machines. However, the test effort is often asynchronous, sonumber of tests run or execution time is normally used in place of calendar time. Numberof tests run would be a good measure if all tests had a similar probability of detecting adefect, but often that is not the case.
We have some test suites that execute 100 tests in anhour and other more sophisticated tests that take 24 hours to execute. The longer test casesusually stress the software more and thus have a higher probability of finding a defect pertest case run. We have developed software reliability growth models using calendar time,number of tests, and execution (CPU) time as a measure of time. The results, showing thatexecution time is the best measure of test time, are described in Section 3.3.2 .
![](https://s.studizba.com/z.php?f=/templates/si/i/noavatar.png&w=100&h=100&t=1"){kind=avatar}
