The Elements of Statistical Learning. Data Mining_ Inference_ and Prediction (811377), страница 31
Текст из файла (страница 31)
Four projections onto pairs of canonical variates. Notice that asthe rank of the canonical variates increases, the centroids become less spread out.In the lower right panel they appear to be superimposed, and the classes mostconfused.1164. Linear Methods for Classification++++FIGURE 4.9. Although the line joining the centroids defines the direction ofgreatest centroid spread, the projected data overlap because of the covariance(left panel). The discriminant direction minimizes this overlap for Gaussian data(right panel).Fisher’s problem therefore amounts to maximizing the Rayleigh quotient,maxaaT Ba,aT Wa(4.15)or equivalentlymax aT Ba subject to aT Wa = 1.a(4.16)This is a generalized eigenvalue problem, with a given by the largesteigenvalue of W−1 B.
It is not hard to show (Exercise 4.1) that the optimala1 is identical to v1 defined above. Similarly one can find the next directiona2 , orthogonal in W to a1 , such that aT2 Ba2 /aT2 Wa2 is maximized; thesolution is a2 = v2 , and so on. The aℓ are referred to as discriminantcoordinates, not to be confused with discriminant functions. They are alsoreferred to as canonical variates, since an alternative derivation of theseresults is through a canonical correlation analysis of the indicator responsematrix Y on the predictor matrix X.
This line is pursued in Section 12.5.To summarize the developments so far:• Gaussian classification with common covariances leads to linear decision boundaries. Classification can be achieved by sphering the datawith respect to W, and classifying to the closest centroid (modulolog πk ) in the sphered space.• Since only the relative distances to the centroids count, one can confine the data to the subspace spanned by the centroids in the spheredspace.• This subspace can be further decomposed into successively optimalsubspaces in term of centroid separation. This decomposition is identical to the decomposition due to Fisher.4.3 Linear Discriminant Analysis1170.7•0.6•0.5•••••••••••Test DataTrain Data0.4Misclassification RateLDA and Dimension Reduction on the Vowel Data••0.3•••246••810DimensionFIGURE 4.10. Training and test error rates for the vowel data, as a functionof the dimension of the discriminant subspace.
In this case the best error rate isfor dimension 2. Figure 4.11 shows the decision boundaries in this space.The reduced subspaces have been motivated as a data reduction (forviewing) tool. Can they also be used for classification, and what is therationale? Clearly they can, as in our original derivation; we simply limitthe distance-to-centroid calculations to the chosen subspace. One can showthat this is a Gaussian classification rule with the additional restrictionthat the centroids of the Gaussians lie in a L-dimensional subspace of IRp .Fitting such a model by maximum likelihood, and then constructing theposterior probabilities using Bayes’ theorem amounts to the classificationrule described above (Exercise 4.8).Gaussian classification dictates the log πk correction factor in the distance calculation.
The reason for this correction can be seen in Figure 4.9.The misclassification rate is based on the area of overlap between the twodensities. If the πk are equal (implicit in that figure), then the optimalcut-point is midway between the projected means. If the πk are not equal,moving the cut-point toward the smaller class will improve the error rate.As mentioned earlier for two classes, one can derive the linear rule usingLDA (or any other method), and then choose the cut-point to minimizemisclassification error over the training data.As an example of the benefit of the reduced-rank restriction, we returnto the vowel data.
There are 11 classes and 10 variables, and hence 10possible dimensions for the classifier. We can compute the training andtest error in each of these hierarchical subspaces; Figure 4.10 shows theresults. Figure 4.11 shows the decision boundaries for the classifier basedon the two-dimensional LDA solution.There is a close connection between Fisher’s reduced rank discriminantanalysis and regression of an indicator response matrix. It turns out that1184. Linear Methods for ClassificationClassification in Reduced Subspaceoooooooooooooooo o ooooooo oooo ooo oo oo ooooooooooooo ooo oooo oooo • ooooooooooooo•o oooooooooooo ooooo ooo o o ooo ooo ooooo o ooooooo oo • o oooooo o o o o oooo oo oooo o o oooooo•ooooooooo o oo oo oooo oo oo oooooooooo ooooo o ooooooo oooo o oooo ooooo oooooooo•o o ooooo oo oo ooo oooooooooooooooooo• ooo oo ooooooooo oo ooo oooo oo oo o •oo oo o oo o ooooo o o oooo o oooo o oooooo ooooo oooooo ooooooooo ooooo ooo o oooo ooo oo ooooo ooooooooooooooo ooo o•ooo oo o o oooooooo• oo ooo o ooooooooo oooooooo o oooooooooooo oo o oo•ooo o oooo ooooooo ooooooooooooo oo o ooooo o o oooo oo•o ooooooo oooo ooooooo o oo o oooooCanonical Coordinate 2•••••••••••o oooooCanonical Coordinate 1FIGURE 4.11.
Decision boundaries for the vowel training data, in the two-dimensional subspace spanned by the first two canonical variates. Note that inany higher-dimensional subspace, the decision boundaries are higher-dimensionalaffine planes, and could not be represented as lines.4.4 Logistic Regression119LDA amounts to the regression followed by an eigen-decomposition ofŶT Y. In the case of two classes, there is a single discriminant variablethat is identical up to a scalar multiplication to either of the columns of Ŷ.These connections are developed in Chapter 12.
A related fact is that if onetransforms the original predictors X to Ŷ, then LDA using Ŷ is identicalto LDA in the original space (Exercise 4.3).4.4 Logistic RegressionThe logistic regression model arises from the desire to model the posteriorprobabilities of the K classes via linear functions in x, while at the sametime ensuring that they sum to one and remain in [0, 1]. The model hasthe formPr(G = 1|X = x)= β10 + β1T xPr(G = K|X = x)Pr(G = 2|X = x)= β20 + β2T xlogPr(G = K|X = x)...loglog(4.17)Pr(G = K − 1|X = x)T= β(K−1)0 + βK−1x.Pr(G = K|X = x)The model is specified in terms of K − 1 log-odds or logit transformations(reflecting the constraint that the probabilities sum to one).
Although themodel uses the last class as the denominator in the odds-ratios, the choiceof denominator is arbitrary in that the estimates are equivariant under thischoice. A simple calculation shows thatPr(G = k|X = x)=Pr(G = K|X = x)=exp(βk0 + βkT x), k = 1, . . . , K − 1,PK−11 + ℓ=1 exp(βℓ0 + βℓT x)1,(4.18)PK−11 + ℓ=1 exp(βℓ0 + βℓT x)and they clearly sum to one. To emphasize the dependence on the entire paTrameter set θ = {β10 , β1T , . . . , β(K−1)0 , βK−1}, we denote the probabilitiesPr(G = k|X = x) = pk (x; θ).When K = 2, this model is especially simple, since there is only a singlelinear function.
It is widely used in biostatistical applications where binaryresponses (two classes) occur quite frequently. For example, patients surviveor die, have heart disease or not, or a condition is present or absent.1204. Linear Methods for Classification4.4.1 Fitting Logistic Regression ModelsLogistic regression models are usually fit by maximum likelihood, using theconditional likelihood of G given X. Since Pr(G|X) completely specifies theconditional distribution, the multinomial distribution is appropriate.
Thelog-likelihood for N observations isℓ(θ) =NXlog pgi (xi ; θ),(4.19)i=1where pk (xi ; θ) = Pr(G = k|X = xi ; θ).We discuss in detail the two-class case, since the algorithms simplifyconsiderably. It is convenient to code the two-class gi via a 0/1 response yi ,where yi = 1 when gi = 1, and yi = 0 when gi = 2. Let p1 (x; θ) = p(x; θ),and p2 (x; θ) = 1 − p(x; θ). The log-likelihood can be writtenℓ(β)=N nXi=1=yi log p(xi ; β) + (1 − yi ) log(1 − p(xi ; β))N nXi=1yi β T xi − log(1 + eβTxio) .o(4.20)Here β = {β10 , β1 }, and we assume that the vector of inputs xi includesthe constant term 1 to accommodate the intercept.To maximize the log-likelihood, we set its derivatives to zero. These scoreequations areN∂ℓ(β) Xxi (yi − p(xi ; β)) = 0,(4.21)=∂βi=1which are p + 1 equations nonlinear in β.
Notice thatsince thePfirst compoPNNnent of xi is 1, the first score equation specifies that i=1 yi = i=1 p(xi ; β);the expected number of class ones matches the observed number (and hencealso class twos.)To solve the score equations (4.21), we use the Newton–Raphson algorithm, which requires the second-derivative or Hessian matrixNX∂ 2 ℓ(β)xi xi T p(xi ; β)(1 − p(xi ; β)).=−∂β∂β Ti=1(4.22)Starting with β old , a single Newton update isβ new=β old −∂ 2 ℓ(β)∂β∂β Twhere the derivatives are evaluated at β old .−1∂ℓ(β),∂β(4.23)4.4 Logistic Regression121It is convenient to write the score and Hessian in matrix notation. Lety denote the vector of yi values, X the N × (p + 1) matrix of xi values,p the vector of fitted probabilities with ith element p(xi ; β old ) and W aN × N diagonal matrix of weights with ith diagonal element p(xi ; β old )(1 −p(xi ; β old )).
Then we have∂ℓ(β)∂β2∂ ℓ(β)∂β∂β T=XT (y − p)(4.24)=−XT WX(4.25)The Newton step is thusβ new===β old + (XT WX)−1 XT (y − p)(XT WX)−1 XT W Xβ old + W−1 (y − p)(XT WX)−1 XT Wz.(4.26)In the second and third line we have re-expressed the Newton step as aweighted least squares step, with the responsez = Xβ old + W−1 (y − p),(4.27)sometimes known as the adjusted response. These equations get solved repeatedly, since at each iteration p changes, and hence so does W and z.This algorithm is referred to as iteratively reweighted least squares or IRLS,since each iteration solves the weighted least squares problem:β new ← arg min(z − Xβ)T W(z − Xβ).β(4.28)It seems that β = 0 is a good starting value for the iterative procedure,although convergence is never guaranteed.
![](https://s.studizba.com/z.php?f=/templates/si/i/noavatar.png&w=100&h=100&t=1"){kind=avatar}
