Regression models for data sciense (779323), страница 9
Текст из файла (страница 9)
. . + βp Xpi + ϵi =p∑Xik βj + ϵik=1Here X1i = 1 typically, so that an intercept is included. Least squares (and hence ML estimatesunder iid Gaussianity of the errors) minimizes:n∑i=1(Yi −p∑)2Xki βjk=1Note, the important linearity is linearity in the coefficients. Thus222Yi = β1 X1i+ β2 X2i+ . . . + βp Xpi+ ϵiis still a linear model. We’ve just squared the elements of the predictor variables.EstimationWatch this before beginning⁷⁶Recall, the LS estimate for regression through the origin is,∑∑E[Yi ] = X1i β1 , was Xi Yi / Xi2 .Let’s consider two regressors, E[Yi ] = X1i β1 + X2i β2 = µi . Least squares tries to minimize:n∑(Yi − X1i β1 − X2i β2 )2i=1We describe fitting with two regressors using residuals, since it will help us to understand howmultivariable regression adjusts an effect for another variable. The result is that the estimate for β1is:∑nee∑n i,Y |X2 2 i,X1 |X2 ,β̂1 = i=1i=1 ei,X1 |X2where, ei,Y |X2 is the residual having fit X2 on Y and ei,X1 |X2 is the residual having fit X2 on Y .
Thatis, the regression estimate for β1 is the regression through the origin estimate having regressed X2out of both the response and the predictor. Similarly, the regression estimate for β2 is the regressionthrough the origin estimate having regressed X1 out of both the response and the predictor.⁷⁶https://www.youtube.com/watch?v=BbsDGRLhluA&index=20&list=PLpl-gQkQivXjqHAJd2t-J_One_fYE55tC55Multivariable regression analysisMore generally, multivariate regression estimates are exactly those having removed the linearrelationship of the other variables from both the regressor and response. This demonstrates thesense in which multivariate regression variables adjust for the effect of the other variables.Example with two variables, simple linear regressionWe already have one of the most important examples of two variables down, linear regression.
Thelinear regression model is:Yi = β1 X1i + β2 X2iwhere X2i = 1 is an intercept term. Let’s double check our rule, since we already know what theleast squares estimates are in this case.Notice the fitted coefficient of X2i on Yi is Ȳ , the mean of the Ys. Then the residuals are ei,Y |X2 =Yi − Ȳ .Similarly, the fitted coefficient of X2i on X1i is X̄1 .
Then, the residuals are ei,X1 |X2 = X1i − X̄1 .Thus let’s work out the estimate for β1 :∑n∑n(X − X̄)(Yi − Ȳ )Sd(Y )i=1 ei,Y |X2 ei,X1 |X2∑n 2∑n i= i=1β̂1 == Cor(X, Y )2Sd(X)i=1 ei,X1 |X2i=1 (Xi − X̄)This agrees with the estimate that we came up with before.The general caseIn the general case with p regressors, least squares solutions have to minimize:n∑(Yi − X1i β1 − . . . − Xpi βp )2i=1The least squares estimate for the coefficient of a multivariate regression model is exactly regressionthrough the origin with the linear relationships with the other regressors removed from boththe regressor and outcome by taking residuals. In this sense, multivariate regression “adjusts” acoefficient for the linear impact of the other variables.Of course, we don’t fit multivariate regression models in this way ever in practice, we rely onsoftware like lm.
In fact, the programs that fit multivariate regression don’t do it this way either.However, thinking in the terms of residuals is the most conceptually useful way to think about whatmultivariate regression is accomplishing.56Multivariable regression analysisSimulation demonstrationsWatch this video demonstration.⁷⁷Let’s do some simulation exercises to convince ourselves how multivariable regression works.Linear model with three variables1234567891011121314151617> n = 100; x = rnorm(n); x2 = rnorm(n); x3 = rnorm(n)## Generate the data> y = 1 + x + x2 + x3 + rnorm(n, sd = .1)## Get the residuals having removed X2 and X3 from X1 and Y> ey = resid(lm(y ~ x2 + x3))> ex = resid(lm(x ~ x2 + x3))## Fit regression through the origin with the residuals> sum(ey * ex) / sum(ex ^ 2)[1] 1.009## Double check with lm> coef(lm(ey ~ ex - 1))ex1.009## Fit the full linear model to show that it agreescoef(lm(y ~ x + x2 + x3))(Intercept)xx2x31.02021.00900.97871.0064You can see that the estimate, 1.009, is obtained by the recipe we outlined; and it agrees with thefunction lm produces for us.Interpretation of the coefficientsLet’s go through the interpretation of the coefficients.
Consider the predicted mean for a given setof values of the regressors:E[Y |X1 = x1 , . . . , Xp = xp ] =p∑xk βk .k=1Now consider incrementing X1 (and only X1 ) by 1.⁷⁷https://www.youtube.com/watch?v=XJJxyJ6PC6I&list=PLpl-gQkQivXjqHAJd2t-J_One_fYE55tC&index=2157Multivariable regression analysisE[Y |X1 = x1 + 1, .
. . , Xp = xp ] = (x1 + 1)β1 +p∑xk βkk=2Now let’s subtract these two equations:pp∑∑E[Y |X1 = x1 +1, . . . , Xp = xp ]−E[Y |X1 = x1 , . . . , Xp = xp ] = (x1 +1)β1 +xk βk +xk βk = β1k=2k=1Thus, the interpretation of a multivariate regression coefficient is the expected change in theresponse per unit change in the regressor, holding all of the other regressors fixed.
The latter part ofthe phrase is important, by holding the other regressors constant, we are investigating an adjustedeffect, just like we described in the smoking and breath mint useage example at the beginning of thechapter.In the next chapter, we’ll do examples and go over context-specific interpretations.Fitted values, residuals and residual variationAll of our simple linear regression quantities can be extended to linear models. Here we list themout in one place. Our statistical model is:Yi =p∑Xik βk + ϵik=1where ϵi ∼ N (0, σ 2 ). Our fitted responses are:Ŷi =p∑Xik β̂k .k=1We can define our residuals exactly as in linear regression:ei = Yi − ŶiOur variance estimate is.1 ∑ 2σ̂ =en − p i=1 in258Multivariable regression analysisTo get predicted responses at new values, x1 , . .
. , xp , simply plug them into the linear model∑pk=1 xk β̂k .Coefficients have standard errors, we can label them as σ̂β̂k , andβ̂k − βkσ̂β̂kfollows a t distribution with n − p degrees of freedom. Predicted responses have standard errors andwe can calculate predicted and expected response intervals.Summary notes on linear models• Linear models are the single most important applied statistical and machine learning technique, by far.• Some amazing things that you can accomplish with linear models– Decompose a signal into its harmonics.– Flexibly fit complicated functions.– Fit factor variables as predictors.– Uncover complex multivariate relationships with the response.– Build accurate prediction models.Exercises1.
Load the dataset Seatbelts as part of the datasets package via data(Seatbelts). Useas.data.frame to convert the object to a dataframe. Fit a linear model of driver deaths withkms and PetrolPrice as predictors. Interpret your results. Watch a video Solution⁷⁸2. Predict the number of driver deaths at the average kms and petrol levels. Watch a videosolution.⁷⁹3.
Take the residual for DriversKilled having regressed out kms and an intercept Watch a videosolution.⁸⁰ and the residual for PetrolPrice having regressed out kms and an intercept. Fita regression through the origin of the two residuals and show that it is the same as yourcoefficient obtained in question 1.4. Take the residual for DriversKilled having regressed out PetrolPrice and an intercept.Take the residual for kms having regressed out PetrolPrice and an intercept. Fit a regressionthrough the origin of the two residuals and show that it is the same as your coefficient obtainedin question 1.⁷⁸https://www.youtube.com/watch?v=xcJKPyiuSMo&index=37&list=PLpl-gQkQivXji7JK1OP1qS7zalwUBPrX0⁷⁹https://www.youtube.com/watch?v=2PO8djtbDU8&index=38&list=PLpl-gQkQivXji7JK1OP1qS7zalwUBPrX0⁸⁰https://www.youtube.com/watch?v=9NS9ue8Tzm4&index=39&list=PLpl-gQkQivXji7JK1OP1qS7zalwUBPrX0Multivariable examples and tricksWatch this video before beginning.⁸¹In this chapter we cover a few examples of multivariate regression in order to get a hands on senseof the basics.Data set for discussionWe’ll start with the Swiss dataset that is part of the datasets package.
This can be loaded in R with:> require(datasets); data(swiss); ?swissStandardized fertility measure and socio-economic indicators for each of 47 French-speakingprovinces of Switzerland at about 1888.A data frame with 47 observations on 6 variables, each of which is in percent, i.e., in [0, 100].[,1][,2][,3][,4][,5][,6]Fertilitya common standardized fertility measureAgriculturepercent of males involved in agriculture as occupationExaminationpercent draftees receiving highest on army examinationEducationpercent education beyond primary school for drafteesCatholicpercent catholic (as opposed to protestant)Infant.Mortalitylive births who live less than 1 yearAll variables but Fertility give percentages of the population.⁸¹https://youtu.be/z8--IymvW4s?list=PLpl-gQkQivXjqHAJd2t-J_One_fYE55tC60Multivariable examples and tricksPlot of the Swiss data setLet’s see the result of calling lm on this data set.> summary(lm(Fertility ~ .
, data = swiss))Estimate Std. Error t value(Intercept)66.915210.706046.250Agriculture-0.17210.07030 -2.448Examination-0.25800.25388 -1.016Education-0.87090.18303 -4.758Catholic0.10410.035262.953Infant.Mortality1.07700.381722.822Pr(>|t|)1.906e-071.873e-023.155e-012.431e-055.190e-037.336e-03Agriculture is expressed in percentages (0 - 100), representing the percentage of the male populationinvolved in agriculture.The regression slope estimate for this variable is -0.1721.
![](https://s.studizba.com/z.php?f=/templates/si/i/noavatar.png&w=100&h=100&t=1"){kind=avatar}
