A. Wood - Softare Reliability. Growth Models

~TANDEMSoftware ReliabilityGrowth ModelsAlan WoodTechnical Report 96.1September 1996Part Number: 130056Software Reliability Growth ModelsAlan WoodTandem Computers10300 N Tantau Ave. LOC55-52Cupertino, CA 95014e-mail: wood_alan@tandem.comSummarySoftware reliability is a critical component of computer system availability, so it isimportant that Tandem's customers experience a small number of software failures in theirproduction environments.

Software reliability growth models can be used as an indicationof the number of failures that may be encountered after the software has shipped and thusas an indication of whether the software is ready to ship. These models use system test datato predict the number of defects remaining in the software.

Software reliability growthmodels have been applied to portions of several releases at Tandem over the past few years.This experimental research has provided some insights into these models and their utility.The utility of a software reliability growth model is related to its stability and predictiveability. Stability means that the model parameters should not significantly change as newdata is added. Predictive ability means that the number of remaining defects predicted bythe model should be close to the number found in field use.

The major results from theresearch are:• While still in the experimental stage, software reliability growth models can be used atTandem to provide reasonable predictions of the number of defects remaining in the field.The model results are shown below and appear to be extremely good. However, no singlemethodology has been consistently used to make the predictions.

We have had to modifythe data or model technique for each different release, e.g., developing a special 2-stagemodel for release 2 and 3, so no single methodology seems capable of capturing all thevariability of different releases.Defects in First YearReleasePredicted Residual Defects34331282/3334109• Simple models perform as well or better than complex models. We evaluated 9 differentsoftware reliability growth models that appear in the literature, and the simple exponentialmodel outperformed the other models in terms of both stability and predictive ability.• Execution (CPU) time is the best measure of the amount of testing.

Using calendar timeor number of test cases to measure the amount of testing did not provide credible results.• Problem reports were a good surrogate for defects. This enhances our ability to makereal-time decisions during the test phase because we do not have to wait until problems canbe analyzed to determine if they are new defects or rediscoveries of known defects.• Grouped (weekly) data was sufficient for the models. There is no need to have daily logsof defects and execution time.Tandem Technical Report 96.1Part Number 130056© Tandem Computers, 1996Table of ContentsSectionNumberSection Title1.01.11.2IntroductionBackgroundApproach and Organization2.02.12.1.12.1.22.1.32.22.32.3.12.3.22.3.32.3.42.3.52.4Software Reliability Growth ModelsSoftware Reliability Growth Model DataDefect DataTest Time DataGrouped DataSoftware Reliability Growth Model TypesParameter EstimationMaximum LikelihoodClassical Least SquaresAlternative Least SquaresSolution Techniques and HintsTheoretical Comparison of TechniquesDefinition of a Useful Model...453.0Model ApplicationsTest DataResults From the Standard ModelResults for Different Representations of Test TimeResults From Modeling Problem ReportsResults for Different ModelsDifferent Correlation TechniquesGrouped Data StabilityRerun Test Hours15161720222324.25263.13.23.33.43.53.63.73.8PageNumberAcknowledgment12355671112121313141527References.................................................................................27Appendix 1Appendix 2Least Squares CalculationsParameter Scalingii.2829List of FiguresFigureNumber1-1.2-l.2-2.2-3.3-I.3-2.PageNumberFigure TitleResidual DefectsExample Defect Detection DataConcave and S-Shaped ModelsTwo Stage Model TransformationTest Data for All ReleasesCombined Data for Releases 2 and 32.47.101720List of TablesTableNumberTable TitlePageNumberI-I.2-I.2-2.3-I.3-2.3-3.3-4.3-5.3-6.3-7.3-8.3-9.3-10.3-1I.3-12.3-13.3-14.3-15.Model Parameter OptionsSoftware Reliability Growth Model ExamplesSoftware Reliability Model AssumptionsTest DataRelease 1 ResultsRelease 2 ResultsRelease 3 ResultsRelease 4 ResultsModel Predictions vs.

Field ExperienceRelease 4 Results for Calendar TimeRelease 3 Results for Number of Test CasesRelease 2 Results for TPRsRelease 3 Results for TPRsRelease 1 Results for Various ModelsStatistical Technique ComparisonConfidence Interval Comparison for Release 4Release 4 Results for Ungrouped Data"Release 1 Results for Discounting Rerun Test Hours.4'" ..89161818181919212122222324252626111Software Reliability Growth Models"All Models are Wrong - Some are Useful."George E. P. Box1.0IntroductionFor critical business applications, continuous availability is a requirement.

and softwarereliability is an important component of continuous application availability. Tandemcustomers expect continuous availability, and our process pair technology protects us frommost transient software defects. However, rare kinds of single software defects can cause asystem failure [Lee,93]. To avoid these failures and to decrease software support costs,Tandem needs to deliver reliable software.Developing reliable software is one of the most difficult problems facing the softwareindustry. Schedule pressure, resource limitations, and unrealistic requirements can allnegatively impact software reliability.

Developing reliable software is especially hard whenthere is interdependence among the software modules as is the case with much of existingsoftware. It is also a hard problem to know whether or not the software being delivered isreliable. Mter the software is shipped, its reliability is indicated by from customer feedback- problem reports, system outages, complaints or compliments, and so forth. However, bythen it is too late; software vendors need to know whether their products are reliable beforethey are shipped to customers.

Software reliability models attempt to provide thatinformation.There are essentially two types of software reliability models - those that attempt to predictsoftware reliability from design parameters and those that attempt to predict softwarereliability from test data. The first type of models are usually called "defect density" modelsand use code characteristics such as lines of code, nesting of loops, external references,input/outputs, and so forth to estimate the number of defects in the software. The secondtype of models are usually called "software reliability growth" models. These modelsattempt to statistically correlate defect detection data with known functions such as anexponential function. If the correlation is good, the known function can be used to predictfuture behavior.

Software reliability growth models are the focus of this report.Most software reliability growth models have a parameter that relates to the total number ofdefects contained in a set of code. If we know this parameter and the current number ofdefects discovered, we know how many defects remain in the code (see Figure 1-1).Knowing the number of residual defects helps us decide whether or not the code is ready toship and how much more testing is required if we decide the code is not ready to ship.

Itgives us an estimate of the number of failures that our customers will encounter whenoperating the software. This estimate helps us to plan the appropriate levels of support thatwill be required for defect correction after the software has shipped and determine the costof supporting the software.Software reliability growth models have been applied to portions of four software releasesat Tandem over the past 4 years. This research, while still experimental, has provided anumber of useful results and insights into software reliability growth modeling. This reportdescribes the methodology we used and the results obtained from modeling Tandemsoftware.1t--------------~-----Residual Defects - - - . ITotal DefectsDefects DiscoveredNumberofDefectsTest TimeFigure I-I. Residual Defects1. 1BackgroundSoftware quality is very important at Tandem, but being a commercial company, we don'tfollow the rigid defense and aerospace software development process, e.g., DOD2167A.We do, however, have a product life cycle, complete with product requirementsdocuments, external and internal specifications, design reviews, code inspections, and soforth.

Each product development group has a development organization and a qualityassurance (QA) organization. The development organization is responsible for developingthe software and performing unit tests. The QA organization is responsible for developingtest software and performing integration and system test.

When the developmentorganization is satisfied with the functionality and quality of the product, it delivers thesoftware to the QA organization for test. This release milestone is called release to QA(RQA). When the QA organization is satisfied with the functionality and qUality of theproduct, it approves the software for shipment, a milestone called QAOK.Failures found in the development or test process are reported using Tandem ProblemReports (TPRs).