Lecture 9: Linear Regression - University of Washington

1 Lecture 9: LinearRegressionGoals Linear Regression in R Estimating parameters and hypothesis testingwith Linear models Develop basic concepts of Linear Regression froma probabilistic frameworkRegression Technique used for the modeling and analysis ofnumerical data Exploits the relationship between two or morevariables so that we can gain information about one ofthem through knowing values of the other Regression can be used for prediction, estimation,hypothesis testing, and modeling causal relationshipsRegression LingoY = X1 + X2 + X3 Dependent VariableOutcome VariableResponse VariableIndependent VariablePredictor VariableExplanatory VariableWhy Linear Regression ? Suppose we want to model the dependent variable Y in termsof three predictors, X1, X2, X3Y = f(X1, X2, X3) Typically will not have enough data to try and directlyestimate f Therefore, we usually have to assume that it has somerestricted form, such as linearY = X1 + X2 + X3 Linear Regression is a Probabilistic Model Much of mathematics is devoted to studying variablesthat are deterministically related to one another!

2 Y = "0 + "1x! "0! y! x! "1=#y#x! "y! "x But we re interested in understanding the relationshipbetween variables related in a nondeterministic fashionA Linear Probabilistic Model! "0! y! x! y = "0 + "1x + # Definition: There exists parameters , , and , such that forany fixed value of the independent variable x, the dependentvariable is related to x through the model equation! "0! "1! "2 ! " is a rv assumed to be N(0, #2)! y = "0 + "1xTrue Regression Line! "1! "2! "3 Implications The expected value of Y is a Linear function of X, but for fixedx, the variable Y differs from its expected value by a randomamount Formally, let x* denote a particular value of the independentvariable x, then our Linear probabilistic model says:! E(Y| x*)= Y|x* = mean value of Y when x is x*! V(Y| x*)= "Y|x*2 = variance of Y when x is x*Graphical Interpretation!

3 Y = "0 + "1x! "0 + "1x1! "0 + "1x2! x1! x2! y! x For example, if x = height and y = weight then is the averageweight for all individuals 60 inches tall in the population! Y|x1=! Y|x2=! Y|x=60 One More ExampleSuppose the relationship between the independent variable height(x) and dependent variable weight (y) is described by a simplelinear Regression model with true Regression liney = + and Q2: If x = 20 what is the expected value of Y?! Y|x=20 = + (20) = Q3: If x = 20 what is P(Y > 22)? Q1: What is the interpretation of = ! "1 The expected change in height associated with a 1-unit increasein weight! "=3! P(Y>22|x=20)= " # $ % & ' =1()( )= Model Parameters Point estimates of and are obtained by the principle of leastsquares! " 0! " 1! f("0,"1)=yi#("0+"1xi)[]i=1n$2! "0! y! x ! " 0=y # " 1x Predicted and Residual Values Predicted, or fitted, values are values of y predicted by the least-squares Regression line obtained by plugging in x1,x2.

4 ,xn into theestimated Regression line! y 1= " 0# " 1x1! y 2= " 0# " 1x2 Residuals are the deviations of observed and predicted values! e1=y1" y 1e2=y2" y 2! y! x! e1! e2! e3! y 1! y1 Residuals Are Useful!! SSE=(eii=1n")2=(yii=1n"# y i)2 They allow us to calculate the error sum of squares (SSE): Which in turn allows us to estimate :! "2! " 2=SSEn#2 As well as an important statistic referred to as the coefficient ofdetermination:! r2=1"SSESST! SST=(yi"y )2i=1n#Multiple Linear Regression Extension of the simple Linear Regression model to two ormore independent variables! y = "0 + "1x1 + "2x2+..+"nxn+# Partial Regression Coefficients: i effect on thedependent variable when increasing the ith independentvariable by 1 unit, holding all other predictorsconstantExpression = Baseline + Age + Tissue + Sex + ErrorCategorical Independent Variables Qualitative variables are easily incorporated in regressionframework through dummy variables Simple example: sex can be coded as 0/1 What if my categorical variable contains three levels:xi =0 if AA1 if AG2 if GGCategorical Independent Variables previous coding would result in colinearity Solution is to set up a series of dummy variable.

5 In generalfor k levels you need k-1 dummy variablesx1 =1 if AA0 otherwisex2 =1 if AG0 otherwiseAAAGGGx1x2110000 Hypothesis Testing: Model Utility Test (orOmnibus Test) The first thing we want to know after fitting a model is whetherany of the independent variables (X s) are significantly related tothe dependent variable (Y):! H0 : "1="2=..="k=0HA : At least one "1#0f=R2(1$R2) kn$(k+1)! Rejection Region: F",k,n#(k+1)Equivalent ANOVA Formulation of Omnibus Test We can also frame this in our now familiar ANOVA framework- partition total variation into two components: SSE (unexplainedvariation) and SSR (variation explained by Linear model)Equivalent ANOVA Formulation of Omnibus Test We can also frame this in our now familiar ANOVA framework! Rejection Region: F",k,n#(k+1)- partition total variation into two components: SSE (unexplainedvariation) and SSR (variation explained by Linear model)n-1 Totaln-2 ErrorkRegressionFMSSum of SquaresdfSource ofVariation!

6 SSRk! SSEn"2! MSRMSE! SSR=( y i"y )2#! SSE=(yi" y i)2#! SST=(yi"y )2#F Test For Subsets of Independent Variables A powerful tool in multiple Regression analyses is the ability tocompare two models For instance say we want to compare:! Full Model: y = "0 + "1x1 + "2x2+"3x3+"4x4+#! Reduced Model: y = "0 + "1x1 + "2x2+#! f=(SSER"SSEF)/(k"l)SSEF/([n"(k+1)] Again, another example of ANOVA:SSER = error sum of squares forreduced model with predictors! lSSEF = error sum of squares forfull model with k predictorsExample of Model Comparison We have a quantitative trait and want to test the effects at twomarkers, M1 and M2.! f=(SSER"SSEF)/(3"2)SSEF/([100"(3+1)]=(SS ER"SSEF)SSEF/96 Full Model: Trait = Mean + M1 + M2 + (M1*M2) + errorReduced Model: Trait = Mean + M1 + M2 + error! Rejection Region: Fa, 1, 96 Hypothesis Tests of Individual RegressionCoefficients Hypothesis tests for each can be done by simple t-tests:!))

7 " i! H0 : " i=0HA : " i#0T= " i$"ise("i) Confidence Intervals are equally easy to obtain:! " i t#/2,n$(k$1) se( " i)! Critical value: t"/2,n#(k#1)Checking Assumptions Critically important to examine data and check assumptionsunderlying the Regression model Outliers Normality Constant variance Independence among residuals Standard diagnostic plots include: scatter plots of y versus xi (outliers) qq plot of residuals (normality) residuals versus fitted values (independence, constant variance) residuals versus xi (outliers, constant variance) We ll explore diagnostic plots in more detail in RFixed -vs- Random Effects Models In ANOVA and Regression analyses our independent variables canbe treated as Fixed or Random Fixed Effects: variables whose levels are either sampledexhaustively or are the only ones considered relevant to theexperimenter Random Effects.

8 Variables whose levels are randomly sampledfrom a large population of levelsExpression = Baseline + Population + Individual + Error Example from our recent AJHG paper.