The art of software testing. Myers (2nd edition) (2004) (The art of software testing. Myers (2nd edition) (2004).pdf), страница 4

The best way to establish some confidence in this claim is to try to refute it, that is, to try to findimperfections rather than just confirm that the program works correctly for some set of input data.The Economics of TestingGiven this definition of program testing, an appropriate next step isthe determination of whether it is possible to test a program to findall of its errors. We will show you that the answer is negative, even fortrivial programs.

In general, it is impractical, often impossible, to findall the errors in a program. This fundamental problem will, in turn,have implications for the economics of testing, assumptions that thetester will have to make about the program, and the manner in whichtest cases are designed.To combat the challenges associated with testing economics, youshould establish some strategies before beginning. Two of the mostprevalent strategies include black-box testing and white-box testing,which we will explore in the next two sections.Black-Box TestingOne important testing strategy is black-box, data-driven, or input/outputdriven testing.

To use this method, view the program as a black box.Your goal is to be completely unconcerned about the internal behavior and structure of the program. Instead, concentrate on finding circumstances in which the program does not behave according to itsspecifications.In this approach, test data are derived solely from the specifications(i.e., without taking advantage of knowledge of the internal structureof the program).If you want to use this approach to find all errors in the program,the criterion is exhaustive input testing, making use of every possibleinput condition as a test case.

Why? If you tried three equilateral-10The Art of Software Testingtriangle test cases for the triangle program, that in no way guaranteesthe correct detection of all equilateral triangles. The program couldcontain a special check for values 3842,3842,3842 and denote such atriangle as a scalene triangle. Since the program is a black box, theonly way to be sure of detecting the presence of such a statement isby trying every input condition.To test the triangle program exhaustively, you would have to createtest cases for all valid triangles up to the maximum integer size of thedevelopment language. This in itself is an astronomical number of testcases, but it is in no way exhaustive; it would find errors where theprogram said that −3,4,5 is a scalene triangle and that 2,A,2 is anisosceles triangle.

To be sure of finding all such errors, you have to testusing not only all valid inputs, but all possible inputs. Hence, to test thetriangle program exhaustively, you would have to produce virtually aninfinite number of test cases, which, of course, is not possible.If this sounds difficult, exhaustive input testing of larger programsis even more of a problem. Consider attempting an exhaustive blackbox test of a C++ compiler. Not only would you have to create testcases representing all valid C++ programs (again, virtually an infinitenumber), but you would have to create test cases for all invalid C++programs (an infinite number) to ensure that the compiler detectsthem as being invalid.

That is, the compiler has to be tested to ensurethat it does not do what it is not supposed to do—for example, successfully compile a syntactically incorrect program.The problem is even worse for programs having a memory, such asoperating systems or database applications. For example, in a databaseapplication such as an airline reservation system, the execution of atransaction (such as a database query, a reservation for a plane flight)is dependent upon what happened in previous transactions.

Hence,not only would you have to try all unique valid and invalid transactions, but also all possible sequences of transactions.This discussion shows that exhaustive input testing is impossible.Two implications of this are that (1) you cannot test a program toguarantee that it is error free and (2) a fundamental consideration inprogram testing is one of economics. That is, since exhaustive testingis out of the question, the objective should be to maximize the yieldThe Psychology and Economics of Program Testing11on the testing investment by maximizing the number of errors foundby a finite number of test cases.

Doing so will involve, among otherthings, being able to peer inside the program and making certain reasonable, but not airtight, assumptions about the program (for example, if the triangle program detects 2,2,2 as an equilateral triangle, itseems reasonable that it will do the same for 3,3,3). This will formpart of the test-case-design strategy in Chapter 4.White-Box TestingAnother testing strategy, white-box or logic-driven testing, permits youto examine the internal structure of the program. This strategyderives test data from an examination of the program’s logic (andoften, unfortunately, at the neglect of the specification).The goal at this point is to establish, for this strategy, the analog toexhaustive input testing in the black-box approach.

Causing everystatement in the program to execute at least once might appear to bethe answer, but it is not difficult to show that this is highly inadequate. Without belaboring the point, since this matter is discussed inmore depth in Chapter 4, the analog is usually considered to beexhaustive path testing. That is, if you execute, via test cases, all possible paths of control flow through the program, then possibly the program has been completely tested.There are two flaws in this statement, however. One is that thenumber of unique logic paths through a program could be astronomically large.

To see this, consider the trivial program representedin Figure 2.1. The diagram is a control-flow graph. Each node or circle represents a segment of statements that execute sequentially, possibly terminating with a branching statement. Each edge or arcrepresents a transfer of control (branch) between segments. The diagram, then, depicts a 10- to 20-statement program consisting of a DOloop that iterates up to 20 times.

Within the body of the DO loop is aset of nested IF statements. Determining the number of unique logicpaths is the same as determining the total number of unique ways ofmoving from point a to point b (assuming that all decisions in theprogram are independent from one another). This number is approx-12The Art of Software Testingimately 1014, or 100 trillion. It is computed from 520 + 519 + . .

. 51,where 5 is the number of paths through the loop body. Since mostpeople have a difficult time visualizing such a number, consider it thisway: If you could write, execute, and verify a test case every five minutes, it would take approximately one billion years to try every path.If you were 300 times faster, completing a test once per second, youcould complete the job in 3.2 million years, give or take a few leapyears and centuries.Figure 2.1Control-flow graph of a small program.The Psychology and Economics of Program Testing13Of course, in actual programs every decision is not independentfrom every other decision, meaning that the number of possible execution paths would be somewhat less.

On the other hand, actual programs are much larger than the simple program depicted in Figure2.1. Hence, exhaustive path testing, like exhaustive input testing,appears to be impractical, if not impossible.The second flaw in the statement “exhaustive path testing means acomplete test” is that every path in a program could be tested, yet theprogram might still be loaded with errors. There are three explanations for this.The first is that an exhaustive path test in no way guarantees that aprogram matches its specification.

For example, if you were asked towrite an ascending-order sorting routine but mistakenly produced adescending-order sorting routine, exhaustive path testing would beof little value; the program still has one bug: It is the wrong program,as it does not meet the specification.Second, a program may be incorrect because of missing paths.Exhaustive path testing, of course, would not detect the absence ofnecessary paths.Third, an exhaustive path test might not uncover data-sensitivityerrors. There are many examples of such errors, but a simple example should suffice. Suppose that in a program you have to comparetwo numbers for convergence, that is, to see if the difference betweenthe two numbers is less than some predetermined value.

For example, you might write a Java IF statement asif (a-b < c)System.out.println("a-b < c");Of course, the statement contains an error because it should comparec to the absolute value of a-b. Detection of this error, however, isdependent upon the values used for a and b and would not necessarily be detected by just executing every path through the program.In conclusion, although exhaustive input testing is superior toexhaustive path testing, neither proves to be useful because both are14The Art of Software Testinginfeasible. Perhaps, then, there are ways of combining elements ofblack-box and white-box testing to derive a reasonable, but not airtight, testing strategy. This matter is pursued further in Chapter 4.Software Testing PrinciplesContinuing with the major premise of this chapter, that the mostimportant considerations in software testing are issues of psychology,we can identify a set of vital testing principles or guidelines. Most ofthese principles may seem obvious, yet they are all too often overlooked.

Table 2.1 summarizes these important principles, and each isdiscussed in more detail in the paragraphs that follow.Principle 1: A necessary part of a test case is a definitionof the expected output or result.This obvious principle is one of the most frequent mistakes in program testing. Again, it is something that is based on human psychology. If the expected result of a test case has not been predefined,chances are that a plausible, but erroneous, result will be interpretedas a correct result because of the phenomenon of “the eye seeingwhat it wants to see.” In other words, in spite of the proper destructive definition of testing, there is still a subconscious desire to see thecorrect result. One way of combating this is to encourage a detailedexamination of all output by precisely spelling out, in advance, theexpected output of the program.