Regression models for data sciense (779323), страница 10
Текст из файла (страница 10)
We interpret this coefficients as follows:Our models estimates an expected 0.17 decrease in standardized fertility for every 1% increase inpercentage of males involved in agriculture in holding the remaining variables constant.Note that the the t-test for H0 : βAgri = 0 versus Ha : βAgri ̸= 0 is significant since 0.0187 isless that typical benchmarks (0.05, for exaple). Note that, by default, R is reporting the P-value forthe two sided test. If you want the one sided test, calculate it directly using the T-statistic and thedegrees of freedom. (You can figure it out from the two sided P-value, but it’s easy to get tripped upwith signs.)Interestingly, the unadjusted estimate isMultivariable examples and tricks61summary(lm(Fertility ~ Agriculture, data = swiss))$coefficientsEstimate Std.
Error t value Pr(>|t|)(Intercept) 60.30444.25126 14.185 3.216e-18Agriculture0.19420.076712.532 1.492e-02Notice that the sign of the slope estimate reversed! This is an example of so-called “Simpson’sParadox”. This purported paradox (which is actually not a paradox at all) simply points out thatunadjusted and adjusted effects can be the reverse of each other.
Or in other words, the apparentrelationship between X and Y may change if we account for Z. Let’s explore multivariate adjustmentand sign reversals with simulation.Simulation studyBelow we simulate 100 random variables with a linear relationship between X1, X2 and Y. Notably,we generate X1 as a linear function of X2. In this simulation, X1 has a negative adjusted effect onY while X2 has a positive adjusted effect (adjusted referring to the effect including both variables).However, X1 is related to X2. Notice our unadjusted effect of X1 is of the opposite sign (and way off),while the adjusted one is about right.
What’s happening? Our unadjusted model is picking up theeffect X2 as as it’s represented in X1. Play around with the generating coefficients to see how youcan make the estimated relationships very different than the generating ones. More than anything,this illustrates that multivariate modeling is hard stuff.> n = 100; x2 <- 1 : n; x1 = .01 * x2 + runif(n, -.1, .1); y = -x1 + x2 + rnorm(\n, sd = .01)> summary(lm(y ~ x1))$coefEstimate Std. Error t value Pr(>|t|)(Intercept)1.4541.0791.348 1.807e-01x196.7931.862 51.985 3.707e-73>summary(lm(y ~ x1 + x2))$coefEstimate Std.
Error t valuePr(>|t|)(Intercept) 0.001933 0.00177091.092 2.777e-01x1-1.020506 0.0163560 -62.393 4.211e-80x21.000133 0.0001643 6085.554 1.544e-272To confirm what’s going on, let’s look at some plots. In the left panel, we plot Y versus X1. Noticethe positive relationship.
However, if we look at X2 (the color) notice that it clearly goes up with Yas well. If we adjust both the X1 and Y variable by taking the residual after having regressed X2, weget the actual correct relationship between X1 and Y.62Multivariable examples and tricksPlot of the simulated dataBack to this data setIn our data set, the sign reverses itself with the inclusion of Examination and Education.
However,the percent of males in the province working in agriculture is negatively related to educationalattainment (correlation of -0.6395) and Education and Examination (correlation of 0.6984) areobviously measuring similar things.So, given now that we know including correlated variables with our variable of interest into ourregression relationship can drastically change things we have to ask: “Is the positive marginal anartifact for not having accounted for, say, Education level? (Education does have a stronger effect, bythe way.)” At the minimum, anyone claiming that provinces that are more agricultural have higherfertility rates would immediately be open to criticism that the real effect is due to Education levels.You might think then, why don’t I just always include all variables that I have into my regressionmodel to avoid incorrectly adjusted effects? Of course, it’s not this easy and there’s negativeconsequences to including variables unnecessarily.
We’ll talk more about model building and theprocess of choosing which variables to include or not in the next chapter.What if we include a completely unnecessary variable?Next chapter we’ll discuss working with a collection of correlated predictors. But you might wonder,what happens if you include a predictor that’s completely unnecessary. Let’s try some computerexperiments with our fertility data.
In the code below, z adds no new linear information, since it’s63Multivariable examples and tricksa linear combination of variables already included. R just drops terms that are linear combinationsof other terms.> z <- swiss$Agriculture + swiss$Educationlm(Fertility ~ . + z, data = swiss)\> Call:lm(formula = Fertility ~ . + z, data = swiss)Coefficients:(Intercept)Catholic66.9150.104Infant.Mortality1.077AgricultureExaminationEducation\-0.172-0.258-0.871\zNAThis is a fundamental point of multivariate regression: regression models fit the linear space of theregressors.
Therefore, any linear reorganization of the regressors will result in an equivalent fit, withdifferent covariates of course. However, the percentage of the variance explained in the response willbe constant. It is only through adding variables that are not perfectly explained by the existing onesthat one can explain more variation in the response. So, for example, models with covariates i) Xand Z, ii) X+Z and X-Z and iii) 2X and 4Z will all explain the same amount of variation in Y. A thirdvariable, W say, will only explain more variation in Y if it’s not perfectly explained by X and Z. Rlets you know when you’ve done this by putting redundant variables as having NA coefficients.Dummy variables are smartWatch this before beginning.⁸²It is interesting to note that models with factor variables as predictors are simply special cases ofregression models.
As an example, consider the linear model:Yi = β0 + Xi1 β1 + ϵiwhere each Xi1 is binary so that it is a 1 if measurement i is in a group and 0 otherwise. As anexample, consider a variable as treated versus not in a clinical trial. Or, in a more data sciencecontext, consider an A/B test comparing two ad campaigns where Y is the click through rate.Refer back to our model. For people in the group E[Yi ] = β0 + β1 and for people not in the groupE[Yi ] = β0 . The least squares fits work out to be β̂0 + β̂1 is the mean for those in the group andβ̂0 is the mean for those not in the group.
The variable β1 is interpreted as the increase or decrease⁸²https://youtu.be/fUwkLY-EDRE?list=PLpl-gQkQivXjqHAJd2t-J_One_fYE55tC64Multivariable examples and tricksin the mean comparing those in the group to those not. The T-test for that coefficient is exactly thetwo group T test with a common variance.Finally, note including a binary variable that is 1 for those not in the group would be redundant,it would create three parameters to describe two means. Moreover, we know from the last sectionthat including redundant variables will result in R just setting one of them to NA. We know thatthe intercept column is a column of ones, the group variable is one for those in the group while avariable for those not in the group would just be the subtraction of these two. Thus, it’s linearlyredundant and unnecessary.More than two levelsConsider a multilevel factor level. For didactic reasons, let’s say a three level factor.
As an exampleconsider a variable for US political party affiliation: Republican, Democrat, Independent/other. Let’suse the model:Yi = β0 + Xi1 β1 + Xi2 β2 + ϵi .Here the variable Xi1 is 1 for Republicans and 0 otherwise, the variab1e Xi2 is 1 for Democrats and0 otherwise. As before, we don’t need an Xi3 for Independent/Other, since it would be redundant.So now consider the implications of more model.
If person i is Republican then E[Yi ] = β0 + β1 . Onthe other hand, If person i is Democrat then E[Yi ] = β0 + β2 . Finally, If $i$ is Independent/OtherE[Yi ] = β0 .So, we can interpret our coefficients as follows. β1 compares the mean for Republicans to that ofIndependents/Others. β2 compares the mean for Democrats to that of Independents/Others. β1 − β2compares the mean for Republicans to that of Democrats. Notice the coefficients are all comparisonsto the category that we left out, Independents/Others. If one category is an obvious referencecategory, chose that one to leave our.
![](https://s.studizba.com/z.php?f=/templates/si/i/noavatar.png&w=100&h=100&t=1"){kind=avatar}
