The art of software testing. Myers (2nd edition) (2004) (The art of software testing. Myers (2nd edition) (2004).pdf), страница 5

Therefore, a test case must consistof two components:1. A description of the input data to the program.2. A precise description of the correct output of the programfor that set of input data.The Psychology and Economics of Program Testing15Table 2.1Vital Program Testing GuidelinesPrincipleNumber12345678910PrincipleA necessary part of a test case is a definition of theexpected output or result.A programmer should avoid attempting to test his or herown program.A programming organization should not test its ownprograms.Thoroughly inspect the results of each test.Test cases must be written for input conditions that areinvalid and unexpected, as well as for those that are validand expected.Examining a program to see if it does not do what it issupposed to do is only half the battle; the other half isseeing whether the program does what it is not supposedto do.Avoid throwaway test cases unless the program is truly athrowaway program.Do not plan a testing effort under the tacit assumptionthat no errors will be found.The probability of the existence of more errors in asection of a program is proportional to the number oferrors already found in that section.Testing is an extremely creative and intellectuallychallenging task.A problem may be characterized as a fact or group of facts forwhich we have no acceptable explanation, that seem unusual, or thatfail to fit in with our expectations or preconceptions.

It should beobvious that some prior beliefs are required if anything is to appearproblematic. If there are no expectations, there can be no surprises.16The Art of Software TestingPrinciple 2: A programmer should avoid attempting to testhis or her own program.Any writer knows—or should know—that it’s a bad idea toattempt to edit or proofread his or her own work. You know what thepiece is supposed to say and may not recognize when it says otherwise.And you really don’t want to find errors in your own work. The sameapplies to software authors.Another problem arises with a change in focus on a software project. After a programmer has constructively designed and coded a program, it is extremely difficult to suddenly change perspective to lookat the program with a destructive eye.As many homeowners know, removing wallpaper (a destructiveprocess) is not easy, but it is almost unbearably depressing if it wasyour hands that hung the paper in the first place.

Similarly, most programmers cannot effectively test their own programs because theycannot bring themselves to shift mental gears to attempt to exposeerrors. In addition, a programmer may subconsciously avoid findingerrors for fear of retribution from peers or from a supervisor, a client,or the owner of the program or system being developed.In addition to these psychological issues, there is a second significant problem: The program may contain errors due to the programmer’s misunderstanding of the problem statement or specification. Ifthis is the case, it is likely that the programmer will carry the samemisunderstanding into tests of his or her own program.This does not mean that it is impossible for a programmer to testhis or her own program. Rather, it implies that testing is more effective and successful if someone else does it.Note that this argument does not apply to debugging (correctingknown errors); debugging is more efficiently performed by the original programmer.Principle 3: A programming organization should not testits own programs.The Psychology and Economics of Program Testing17The argument here is similar to the previous argument.

A projector programming organization is, in many senses, a living organizationwith psychological problems similar to those of individual programmers. Furthermore, in most environments, a programming organization or a project manager is largely measured on the ability toproduce a program by a given date and for a certain cost. One reasonfor this is that it is easy to measure time and cost objectives, but it isextremely difficult to quantify the reliability of a program.

Therefore,it is difficult for a programming organization to be objective in testing its own programs, because the testing process, if approached withthe proper definition, may be viewed as decreasing the probability ofmeeting the schedule and the cost objectives.Again, this does not say that it is impossible for a programmingorganization to find some of its errors, because organizations doaccomplish this with some degree of success. Rather, it implies thatit is more economical for testing to be performed by an objective,independent party.Principle 4:Thoroughly inspect the results of each test.This is probably the most obvious principle, but again it is something that is often overlooked. We’ve seen numerous experimentsthat show many subjects failed to detect certain errors, even whensymptoms of those errors were clearly observable on the output listings.

Put another way, errors that are found on later tests are oftenmissed in the results from earlier tests.Principle 5:Test cases must be written for input conditionsthat are invalid and unexpected, as well as for thosethat are valid and expected.There is a natural tendency when testing a program to concentrateon the valid and expected input conditions, at the neglect of theinvalid and unexpected conditions.

For instance, this tendency frequently appears in the testing of the triangle program in Chapter 1.18The Art of Software TestingFew people, for instance, feed the program the numbers 1,2,5 tomake sure that the program does not erroneously interpret this as ascalene triangle instead of an invalid triangle. Also, many errors thatare suddenly discovered in production programs turn up when theprogram is used in some new or unexpected way. Therefore, testcases representing unexpected and invalid input conditions seem tohave a higher error-detection yield than do test cases for valid inputconditions.Principle 6: Examining a program to see if it does not do whatit is supposed to do is only half the battle; the other half isseeing whether the program does what it is not supposed to do.This is a corollary to the previous principle. Programs must beexamined for unwanted side effects.

For instance, a payroll programthat produces the correct paychecks is still an erroneous program if italso produces extra checks for nonexistent employees or if it overwrites the first record of the personnel file.Principle 7: Avoid throwaway test cases unless the program istruly a throwaway program.This problem is seen most often with interactive systems to testprograms. A common practice is to sit at a terminal and invent testcases on the fly, and then send these test cases through the program.The major problem is that test cases represent a valuable investmentthat, in this environment, disappears after the testing has been completed.

Whenever the program has to be tested again (for example,after correcting an error or making an improvement), the test casesmust be reinvented. More often than not, since this reinventionrequires a considerable amount of work, people tend to avoid it.Therefore, the retest of the program is rarely as rigorous as the original test, meaning that if the modification causes a previously functional part of the program to fail, this error often goes undetected.Saving test cases and running them again after changes to other components of the program is known as regression testing.The Psychology and Economics of Program Testing19Principle 8: Do not plan a testing effort under the tacitassumption that no errors will be found.This is a mistake project managers often make and is a sign of theuse of the incorrect definition of testing—that is, the assumption thattesting is the process of showing that the program functions correctly.Once again, the definition of testing is the process of executing aprogram with the intent of finding errors.Principle 9:The probability of the existence of more errorsin a section of a program is proportional to the numberof errors already found in that section.This phenomenon is illustrated in Figure 2.2.

At first glance itmakes little sense, but it is a phenomenon present in many programs.For instance, if a program consists of two modules, classes, or subroutines A and B, and five errors have been found in module A andonly one error has been found in module B, and if module A has notbeen purposely subjected to a more rigorous test, then this principletells us that the likelihood of more errors in module A is greater thanthe likelihood of more errors in module B.Another way of stating this principle is to say that errors tend tocome in clusters and that, in the typical program, some sections seemto be much more prone to errors than other sections, althoughnobody has supplied a good explanation of why this occurs. Thephenomenon is useful in that it gives us insight or feedback in thetesting process.

If a particular section of a program seems to be muchmore prone to errors than other sections, then this phenomenon tellsus that, in terms of yield on our testing investment, additional testingefforts are best focused against this error-prone section.Principle 10:Testing is an extremely creativeand intellectually challenging task.It is probably true that the creativity required in testing a large program exceeds the creativity required in designing that program. We20The Art of Software TestingFigure 2.2The Surprising Errors Remaining/Errors FoundRelationship.already have seen that it is impossible to test a program sufficiently toguarantee the absence of all errors.

Methodologies discussed later inthis book let you develop a reasonable set of test cases for a program,but these methodologies still require a significant amount of creativity.SummaryAs you proceed through this book, keep in mind these three important principles of testing:• Testing is the process of executing a program with the intent offinding errors.• A good test case is one that has a high probability of detectingan as yet undiscovered error.• A successful test case is one that detects an as yet undiscoverederror.CHAPTER 3Program Inspections,Walkthroughs,and ReviewsFor many years, most of us in theprogramming community worked under the assumptions that programs are written solely for machine execution and are not intendedfor people to read, and that the only way to test a program is to execute it on a machine.