The art of software testing. Myers (2nd edition) (2004) (The art of software testing. Myers (2nd edition) (2004).pdf), страница 27

The reason for the second clause isthat if we find 240 errors quickly, this is probably an indication that we have underestimated the total number of errorsand thus should not stop function testing early.3. System testing is complete when 111 errors are found andcorrected, or when three months of system testing have beencompleted, whichever occurs later.Table 6.2Hypothetical Estimate ofWhen the Errors Might Be FoundModule testFunction testSystem testTotalCoding and Logic-Design Errors65%30%3%98%Design Errors0%60%35%95%152The Art of Software TestingThe other obvious problem with this type of criterion is one ofoverestimation. What if, in the preceding example, there are less than240 errors remaining when function testing starts? Based on the criterion, we could never complete the function-test phase.There is a strange problem if you think about it.

Our problem isthat we do not have enough errors; the program is too good. Youcould label it a nonproblem because it is the kind of problem a lot ofpeople would love to have. If it does occur, a bit of common sensecan solve it. If we cannot find 240 errors in four months, the projectmanager can employ an outsider to analyze the test cases to judgewhether the problem is (1) inadequate test cases or (2) excellent testcases but a lack of errors to detect.The third type of completion criterion is an easy one on the surface, but it involves a lot of judgment and intuition. It requires you toplot the number of errors found per unit time during the test phase.By examining the shape of the curve, you can often determinewhether to continue the test phase or end it and begin the next testphase.Suppose a program is being function-tested and the number oferrors found per week is being plotted.

If, in the seventh week, thecurve is the top one of Figure 6.5, it would be imprudent to stop thefunction test, even if we had reached our criterion for the number oferrors to be found. Since, in the seventh week, we still seem to be inhigh gear (finding many errors), the wisest decision (rememberingthat our goal is to find errors) is to continue function testing, designing additional test cases if necessary.On the other hand, suppose the curve is the bottom one in Figure6.5.

The error-detection efficiency has dropped significantly, implying that we have perhaps picked the function-test bone clean and thatperhaps the best move is to terminate function testing and begin anew type of testing (a system test, perhaps). Of course, we must alsoconsider other factors such as whether the drop in error-detectionefficiency was due to a lack of computer time or exhaustion of theavailable test cases.Figure 6.5Estimating completion by plotting errorsdetected by unit time.6050Errors Found4030201001234W567567k6050Errors Found4030201001234Week153154The Art of Software TestingFigure 6.6Postmortem study of the testing processesof a large project.900800Errors Found per period700600500400300200100012345678910111213Two-week periodsFigure 6.6 is an illustration of what happens when you fail to plotthe number of errors being detected.

The graph represents three testing phases of an extremely large software system. An obvious conclusion is that the project should not have switched to a differenttesting phase after period 6. During period 6, the error-detectionrate was good (to a tester, the higher the rate, the better), but switching to a second phase at this point caused the error-detection rate todrop significantly.The best completion criterion is probably a combination of thethree types just discussed. For the module test, particularly becausemost projects do not formally track detected errors during this phase,the best completion criterion is probably the first.

You should requestHigher-Order Testing155that a particular set of test-case-design methodologies be used. Forthe function- and system-test phases, the completion rule might beto stop when a predefined number of errors is detected or when thescheduled time has elapsed, whichever comes later, but provided thatan analysis of the errors versus time graph indicates that the test hasbecome unproductive.The Independent Test AgencyEarlier in this chapter and in Chapter 2, we emphasized that anorganization should avoid attempting to test its own programs.

Thereasoning was that the organization responsible for developing a program has difficulty in objectively testing the same program. The testorganization should be as far removed as possible, in terms of thestructure of the company, from the development organization. Infact, it is desirable that the test organization not be part of the samecompany, for if it is, it is still influenced by the same managementpressures influencing the development organization.One way to avoid this conflict is to hire a separate company forsoftware testing. This is a good idea, whether the company thatdesigned the system and will use it developed the system or whethera third-party developer produced the system.

The advantages usuallynoted are increased motivation in the testing process, a healthy competition with the development organization, removal of the testingprocess from under the management control of the developmentorganization, and the advantages of specialized knowledge that theindependent test agency brings to bear on the problem.CHAPTER 7DebuggingIn brief, debugging is what youdo after you have executed a successful test case. Remember that asuccessful test case is one that shows that a program does not do whatit was designed to do.

Debugging is a two-step process that beginswhen you find an error as a result of a successful test case. Step 1 isthe determination of the exact nature and location of the suspectederror within the program. Step 2 consists of fixing the error.As necessary and as integral as debugging is to program testing, thisseems to be the one part of the software production process that programmers enjoy the least. These seem to be the main reasons:• Your ego may get in the way. Like it or not, debugging confirmsthat programmers are not perfect, committing errors in eitherthe design or the coding of the program.• You may run out of steam.

Of all the software developmentactivities, debugging is the most mentally taxing activity.Moreover, debugging usually is performed under a tremendousamount of organizational or self-induced pressure to fix theproblem as quickly as possible.• You may lose your way. Debugging is mentally taxing becausethe error you’ve found could occur in virtually any statementwithin the program. That is, without examining the programfirst, you can’t be absolutely sure that, for example, a numericalerror in a paycheck produced by a payroll program is notproduced in a subroutine that asks the operator to load aparticular form into the printer.

Contrast this with thedebugging of a physical system, such as an automobile. If a car157158The Art of Software Testingstalls when moving up an incline (the symptom), then you canimmediately and validly eliminate as the cause of the problemcertain parts of the system—the AM/FM radio, for example, orthe speedometer or the truck lock. The problem must be in theengine, and, based on our overall knowledge of automotiveengines, we can even rule out certain engine components suchas the water pump and the oil filter.• You may be on your own. Compared to other softwaredevelopment activities, comparatively little research, literature,and formal instruction exist on the process of debugging.Although this is a book about software testing, not debugging, thetwo processes are obviously related.

Of the two aspects of debugging,locating the error and correcting it, locating the error represents perhaps 95 percent of the problem. Hence, this chapter concentrates onthe process of finding the location of an error, given that a successfultest case has found one.Debugging by Brute ForceThe most common scheme for debugging a program is the “bruteforce” method. It is popular because it requires little thought and isthe least mentally taxing of the methods, but it is inefficient and generally unsuccessful.Brute force methods can be partitioned into at least three categories:1.