Bishop C.M. Pattern Recognition and Machine Learning (2006) (811375), страница 15
Текст из файла (страница 15)
Indeed, the classconditional densities may contain a lot of structure that has little effect on the posterior probabilities, as illustrated in Figure 1.27. There has been much interest inexploring the relative merits of generative and discriminative approaches to machinelearning, and in finding ways to combine them (Jebara, 2004; Lasserre et al., 2006).An even simpler approach is (c) in which we use the training data to find adiscriminant function f (x) that maps each x directly onto a class label, therebycombining the inference and decision stages into a single learning problem. In theexample of Figure 1.27, this would correspond to finding the value of x shown bythe vertical green line, because this is the decision boundary giving the minimumprobability of misclassification.With option (c), however, we no longer have access to the posterior probabilitiesp(Ck |x).
There are many powerful reasons for wanting to compute the posteriorprobabilities, even if we subsequently use them to make decisions. These include:Minimizing risk. Consider a problem in which the elements of the loss matrix aresubjected to revision from time to time (such as might occur in a financial1.5. Decision Theory45application). If we know the posterior probabilities, we can trivially revise theminimum risk decision criterion by modifying (1.81) appropriately. If we haveonly a discriminant function, then any change to the loss matrix would requirethat we return to the training data and solve the classification problem afresh.Reject option.
Posterior probabilities allow us to determine a rejection criterion thatwill minimize the misclassification rate, or more generally the expected loss,for a given fraction of rejected data points.Compensating for class priors. Consider our medical X-ray problem again, andsuppose that we have collected a large number of X-ray images from the general population for use as training data in order to build an automated screeningsystem.
Because cancer is rare amongst the general population, we might findthat, say, only 1 in every 1,000 examples corresponds to the presence of cancer. If we used such a data set to train an adaptive model, we could run intosevere difficulties due to the small proportion of the cancer class.
For instance,a classifier that assigned every point to the normal class would already achieve99.9% accuracy and it would be difficult to avoid this trivial solution. Also,even a large data set will contain very few examples of X-ray images corresponding to cancer, and so the learning algorithm will not be exposed to abroad range of examples of such images and hence is not likely to generalizewell.
A balanced data set in which we have selected equal numbers of examples from each of the classes would allow us to find a more accurate model.However, we then have to compensate for the effects of our modifications tothe training data. Suppose we have used such a modified data set and foundmodels for the posterior probabilities.
From Bayes’ theorem (1.82), we see thatthe posterior probabilities are proportional to the prior probabilities, which wecan interpret as the fractions of points in each class. We can therefore simplytake the posterior probabilities obtained from our artificially balanced data setand first divide by the class fractions in that data set and then multiply by theclass fractions in the population to which we wish to apply the model. Finally,we need to normalize to ensure that the new posterior probabilities sum to one.Note that this procedure cannot be applied if we have learned a discriminantfunction directly instead of determining posterior probabilities.Combining models. For complex applications, we may wish to break the probleminto a number of smaller subproblems each of which can be tackled by a separate module. For example, in our hypothetical medical diagnosis problem,we may have information available from, say, blood tests as well as X-ray images.
Rather than combine all of this heterogeneous information into one hugeinput space, it may be more effective to build one system to interpret the Xray images and a different one to interpret the blood data. As long as each ofthe two models gives posterior probabilities for the classes, we can combinethe outputs systematically using the rules of probability. One simple way todo this is to assume that, for each class separately, the distributions of inputsfor the X-ray images, denoted by xI , and the blood data, denoted by xB , are461. INTRODUCTIONindependent, so thatp(xI , xB |Ck ) = p(xI |Ck )p(xB |Ck ).Section 8.2This is an example of conditional independence property, because the independence holds when the distribution is conditioned on the class Ck .
The posteriorprobability, given both the X-ray and blood data, is then given byp(Ck |xI , xB ) ∝ p(xI , xB |Ck )p(Ck )∝ p(xI |Ck )p(xB |Ck )p(Ck )p(Ck |xI )p(Ck |xB )∝p(Ck )Section 8.2.2(1.84)(1.85)Thus we need the class prior probabilities p(Ck ), which we can easily estimatefrom the fractions of data points in each class, and then we need to normalizethe resulting posterior probabilities so they sum to one. The particular conditional independence assumption (1.84) is an example of the naive Bayes model.Note that the joint marginal distribution p(xI , xB ) will typically not factorizeunder this model. We shall see in later chapters how to construct models forcombining data that do not require the conditional independence assumption(1.84).1.5.5 Loss functions for regressionSection 1.1So far, we have discussed decision theory in the context of classification problems.
We now turn to the case of regression problems, such as the curve fittingexample discussed earlier. The decision stage consists of choosing a specific estimate y(x) of the value of t for each input x. Suppose that in doing so, we incur aloss L(t, y(x)). The average, or expected, loss is then given byL(t, y(x))p(x, t) dx dt.(1.86)E[L] =A common choice of loss function in regression problems is the squared loss givenby L(t, y(x)) = {y(x) − t}2 . In this case, the expected loss can be written{y(x) − t}2 p(x, t) dx dt.(1.87)E[L] =Appendix DOur goal is to choose y(x) so as to minimize E[L]. If we assume a completelyflexible function y(x), we can do this formally using the calculus of variations togiveδE[L](1.88)= 2 {y(x) − t}p(x, t) dt = 0.δy(x)Solving for y(x), and using the sum and product rules of probability, we obtaintp(x, t) dt = tp(t|x) dt = Et [t|x]y(x) =(1.89)p(x)471.5.
Decision TheoryFigure 1.28The regression function y(x),which minimizes the expectedsquared loss, is given by themean of the conditional distribution p(t|x).ty(x)y(x0 )p(t|x0 )x0Exercise 1.25xwhich is the conditional average of t conditioned on x and is known as the regressionfunction. This result is illustrated in Figure 1.28. It can readily be extended to multiple target variables represented by the vector t, in which case the optimal solutionis the conditional average y(x) = Et [t|x].We can also derive this result in a slightly different way, which will also shedlight on the nature of the regression problem.
Armed with the knowledge that theoptimal solution is the conditional expectation, we can expand the square term asfollows{y(x) − t}2 = {y(x) − E[t|x] + E[t|x] − t}2= {y(x) − E[t|x]}2 + 2{y(x) − E[t|x]}{E[t|x] − t} + {E[t|x] − t}2where, to keep the notation uncluttered, we use E[t|x] to denote Et [t|x]. Substitutinginto the loss function and performing the integral over t, we see that the cross-termvanishes and we obtain an expression for the loss function in the form2(1.90)E[L] = {y(x) − E[t|x]} p(x) dx + {E[t|x] − t}2 p(x) dx.The function y(x) we seek to determine enters only in the first term, which will beminimized when y(x) is equal to E[t|x], in which case this term will vanish. Thisis simply the result that we derived previously and that shows that the optimal leastsquares predictor is given by the conditional mean. The second term is the varianceof the distribution of t, averaged over x.
It represents the intrinsic variability ofthe target data and can be regarded as noise. Because it is independent of y(x), itrepresents the irreducible minimum value of the loss function.As with the classification problem, we can either determine the appropriate probabilities and then use these to make optimal decisions, or we can build models thatmake decisions directly. Indeed, we can identify three distinct approaches to solvingregression problems given, in order of decreasing complexity, by:(a) First solve the inference problem of determining the joint density p(x, t). Thennormalize to find the conditional density p(t|x), and finally marginalize to findthe conditional mean given by (1.89).481.
INTRODUCTION(b) First solve the inference problem of determining the conditional density p(t|x),and then subsequently marginalize to find the conditional mean given by (1.89).(c) Find a regression function y(x) directly from the training data.Section 5.6Exercise 1.27The relative merits of these three approaches follow the same lines as for classification problems above.The squared loss is not the only possible choice of loss function for regression.Indeed, there are situations in which squared loss can lead to very poor results andwhere we need to develop more sophisticated approaches. An important exampleconcerns situations in which the conditional distribution p(t|x) is multimodal, asoften arises in the solution of inverse problems. Here we consider briefly one simplegeneralization of the squared loss, called the Minkowski loss, whose expectation isgiven byE[Lq ] =|y(x) − t|q p(x, t) dx dt(1.91)which reduces to the expected squared loss for q = 2.
The function |y − t|q isplotted against y − t for various values of q in Figure 1.29. The minimum of E[Lq ]is given by the conditional mean for q = 2, the conditional median for q = 1, andthe conditional mode for q → 0.1.6. Information TheoryExercise 1.28In this chapter, we have discussed a variety of concepts from probability theory anddecision theory that will form the foundations for much of the subsequent discussionin this book.
![](https://s.studizba.com/z.php?f=/templates/si/i/noavatar.png&w=100&h=100&t=1"){kind=avatar}
