Gaussian Linear Models - MIT OpenCourseWare

1 Gaussian Linear Models Gaussian Linear Models MIT Dr. Kempthorne Spring 2016 1 MIT Gaussian Linear Models Gaussian Linear Models Linear Regression: Overview Ordinary Least Squares (OLS) distribution Theory: normal Regression Models Maximum Likelihood Estimation Generalized M Estimation Outline 1 Gaussian Linear Models Linear Regression: Overview Ordinary Least Squares (OLS) distribution Theory: normal Regression Models Maximum Likelihood Estimation Generalized M Estimation 2 MIT Gaussian Linear Models Gaussian Linear Models Linear Regression: Overview Ordinary Least Squares (OLS) distribution Theory: normal Regression Models Maximum Likelihood Estimation Generalized M Estimation General Linear Model: For each case i, the conditional distribution [yi | xi ] is given by yi = yi + Ei where y i = 1xi,1 + 2xi,2 + + i,pxi,p = ( 1, 2,.., p)T are p regression parameters (constant over all cases) Ei Residual (error) variable (varies over all cases) Extensive breadth of possible Models Polynomial approximation (xi,j = (xi )j , explanatory variables are different powers of the same variable x = xi ) Fourier Series: (xi,j = sin(jxi ) or cos(jxi ), explanatory variables are different sin/cos terms of a Fourier series expansion) Time series regressions: time indexed by i, and explanatory variables include lagged response values.

2 Note: Linearity of yi (in regression parameters) maintained with non- Linear x. 3 MIT Gaussian Linear Models Gaussian Linear Models Linear Regression: Overview Ordinary Least Squares (OLS) distribution Theory: normal Regression Models Maximum Likelihood Estimation Generalized M Estimation Steps for Fitting a Model (1) Propose a model in terms of Response variable Y (specify the scale) Explanatory variables X1, X2,.. Xp (include different functions of explanatory variables if appropriate) Assumptions about the distribution of E over the cases (2) Specify/define a criterion for judging different estimators. (3) Characterize the best estimator and apply it to the given data. (4) Check the assumptions in (1). (5) If necessary modify model and/or assumptions and go to (1). 4 MIT Gaussian Linear Models Gaussian Linear Models Linear Regression: Overview Ordinary Least Squares (OLS) distribution Theory: normal Regression Models Maximum Likelihood Estimation Generalized M Estimation Specifying Estimator Criterion in (2) Least Squares Maximum Likelihood Robust (Contamination-resistant) Bayes (assume j are s with known prior distribution ) Accommodating incomplete/missing data Case Analyses for (4) Checking Assumptions Residual analysis Model errors Ei are unobservable Model residuals for fitted regression parameters j are: ei = yi [ 1xi,1 + 2xi,2 + + p xi,p ] Influence diagnostics (identify cases which are highly influential ?)

3 Outlier detection 5 MIT Gaussian Linear Models Gaussian Linear Models Linear Regression: Overview Ordinary Least Squares (OLS) distribution Theory: normal Regression Models Maximum Likelihood Estimation Generalized M Estimation Outline 1 Gaussian Linear Models Linear Regression: Overview Ordinary Least Squares (OLS) distribution Theory: normal Regression Models Maximum Likelihood Estimation Generalized M Estimation 6 MIT Gaussian Linear Models Gaussian Linear Models Linear Regression: Overview Ordinary Least Squares (OLS) distribution Theory: normal Regression Models Maximum Likelihood Estimation Generalized M Estimation Ordinary Least Squares Estimates Least Squares Criterion: For = ( 1, 2,.., p)T , define LNQ( ) = [yi y i ]2 i=1LN =[yi ( 1xi,1 + 2xi,2 + + i,pxi,p)]2 i=1 Ordinary Least-Squares (OLS) estimate : minimizes Q( ). Matrix Notation y1 x1,1 x1,2 x1,p 1 y = y2.

4 X = x2,1 x2,2 x2,p .. = .. pyn xn,1 xn,2 xp,n 7 MIT Gaussian Linear Models Gaussian Linear Models Linear Regression: Overview Ordinary Least Squares (OLS) distribution Theory: normal Regression Models Maximum Likelihood Estimation Generalized M Estimation Solving for OLS Estimate y 1 y 2 .. y nL = X and y = n (yi y i )2 = (y y)T (y y)i=1Q( ) = = (y X )T (y X ) Q( ) OLS solves =0, j = 1, 2,.., p j Q( ) L n =[yi (xi,1 1 + xi,2 2 + xi,p p)]2 ji=1L j n 2( xi,j )[yi (xi,1 1 + xi,2 2 + xi,p p)]i=1 = = 2(X[j])T (y X ) where X[j] is the jth column of X 8 MIT Gaussian Linear Models Gaussian Linear Models Linear Regression: Overview Ordinary Least Squares (OLS) distribution Theory: normal Regression Models Maximum Likelihood Estimation Generalized M Estimation Solving for OLS Estimate Q XT (y X ) 1 [1] Q = = 2XT (y X ) Q 2.

5 XT [2](y X ) = 2 .. Q XT (y X ) p [p]So the OLS Estimate solves the normal Equations XT (y X ) = 0 XT X = XT y = = (XT X) 1XT y For to exist (uniquely) (XT X) must be invertible X must have Full Column Rank 9 MIT Gaussian Linear Models Gaussian Linear Models Linear Regression: Overview Ordinary Least Squares (OLS) distribution Theory: normal Regression Models Maximum Likelihood Estimation Generalized M Estimation (Ordinary) Least Squares Fit OLS Estimate: 1 2 = = (XT X) 1XT y Fitted Values: .. p 1 + + x1,p p y 1 x1,1 1 + py 2 .. + x2,px2,1 y = = .. 1 + py n + xn,pxn,1 = X = X(XT X) 1XT y = Hy Where H = X(XT X) 1XT is the n n Hat Matrix 10 MIT Gaussian Linear Models Gaussian Linear Models Linear Regression: Overview Ordinary Least Squares (OLS) distribution Theory: normal Regression Models Maximum Likelihood Estimation Generalized M Estimation (Ordinary) Least Squares Fit The Hat Matrix H projects Rn onto the column-space of X Residuals: Ei = yi y i , i = 1, 2.

6 , n = E 1 E 2 .. E n = y y = (In H)y 0 XT (y X ) = XT = 0p = .. 0 normal Equations: The Least-Squares Residuals vector is orthogonal to the column space of X 11 MIT Gaussian Linear Models Gaussian Linear Models Linear Regression: Overview Ordinary Least Squares (OLS) distribution Theory: normal Regression Models Maximum Likelihood Estimation Generalized M Estimation Outline 1 Gaussian Linear Models Linear Regression: Overview Ordinary Least Squares (OLS) distribution Theory: normal Regression Models Maximum Likelihood Estimation Generalized M Estimation 12 MIT Gaussian Linear Models Gaussian Linear Models Linear Regression: Overview Ordinary Least Squares (OLS) distribution Theory: normal Regression Models Maximum Likelihood Estimation Generalized M Estimation normal Linear Regression Models distribution Theory Yi = xi,1 1 + xi,2 2 + xi,p p + Ei = i + Ei Assume {E1,E2.}

7 ,En} are N(0, 2). = [Yi | xi,1, xi,2,.., xi,p, , 2] N( i , 2), independent over i = 1, 2,.. n. Conditioning on X, , and 2 E1 E2 .. En Nn(On, 2In)Y = X + , where = 13 MIT Gaussian Linear Models Gaussian Linear Models Linear Regression: Overview Ordinary Least Squares (OLS) distribution Theory: normal Regression Models Maximum Likelihood Estimation Generalized M Estimation distribution Theory 1 = .. n = E (Y | X, , 2) = X 14 MIT Gaussian Linear Models Gaussian Linear Models Linear Regression: Overview Ordinary Least Squares (OLS) distribution Theory: normal Regression Models Maximum Likelihood Estimation Generalized M Estimation 2 0 0 0 0 2 0 0 0 0 2 0 .. 0 0 2 = Cov(Y | X, , 2) = = 2In That is, i,j = Cov(Yi , Yj | X, , 2) = 2 i,j . Apply Moment-Generating Functions (MGFs) to derive Joint distribution of Y = (Y1, Y2.)

8 , Yn)T Joint distribution of = ( 1, 2,.., p)T . 15 MIT Gaussian Linear Models Gaussian Linear Models Linear Regression: Overview Ordinary Least Squares (OLS) distribution Theory: normal Regression Models Maximum Likelihood Estimation Generalized M Estimation MGF of Y For the n-variate Y, and constant n vector t = (t1,.., tn)T , t1Y1+t2Y2+ tnYn )MY(t) = E (etT Y) = E(e = E (et1Y1 ) E (et2Y2 ) E (etnYn ) = MY1 (t1) MY2 (t2) MYn (tn) 1n ti i + t2 2 i= 2 i=1 e n 1 n 1 i=1 2 i,k=1 2 ti i + ti i,k tk tT u+ tT t = e = e = Y Nn( , ) multivariate normal with mean and covariance 16 MIT Gaussian Linear Models Gaussian Linear Models Linear Regression: Overview Ordinary Least Squares (OLS) distribution Theory: normal Regression Models Maximum Likelihood Estimation Generalized M Estimation MGF of For the p-variate , and constant p vector = ( 1.

9 , p)T , T 1 1+ 2 2+ p p )M ( ) = E (e ) = E (e Defining A = (XT X) 1XT we can express = (XT X) 1XT y = AY and T ( ) = E (e )M T AY)= E (e = E (etT Y), with t = AT = MY(t) 1tT u+2 tT t = e 17 MIT Gaussian Linear Models Gaussian Linear Models Linear Regression: Overview Ordinary Least Squares (OLS) distribution Theory: normal Regression Models Maximum Likelihood Estimation Generalized M Estimation MGF of For T M ( ) = E(e ) tT 1 u+2 tT t = e Plug in: t = AT = X(XT X) 1 = X = 2In Gives: tT = T tT t = T (XT X) 1XT [ 2In]X(XT X) 1 = T [ 2(XT X) 1] So the MGF of is 1 T + T [ 2(XT X) 1] M ( ) = e 2 Np( , 2(XT X) 1) 18 MIT Gaussian Linear Models Gaussian Linear Models Linear Regression: Overview Ordinary Least Squares (OLS) distribution Theory: normal Regression Models Maximum Likelihood Estimation Generalized M Estimation Marginal Distributions of Least Squares Estimates Because Np( , 2(XT X) 1) the marginal distribution of each j is: j N( j , 2Cj,j ) where = jth diagonal element of (XT X) 1 19 MIT Gaussian Linear Models Gaussian Linear Models Linear Regression: Overview Ordinary Least Squares (OLS) distribution Theory: normal Regression Models Maximum Likelihood Estimation Generalized M Estimation The Q-R Decomposition of X Consider expressing the (n p) matrix X of explanatory variables as X = Q R where Q is an (n p) orthonormal matrix, , QT Q = Ip.

10 R is a (p p) upper-triangular matrix. The columns of Q = [Q[1], Q[2],.., Q[p]] can be constructed by performing the Gram-Schmidt Orthonormalization procedure on the columns of X = [X[1], X[2],.., X[p]] 20 MIT Gaussian Linear Models Gaussian Linear Models Linear Regression: Overview Ordinary Least Squares (OLS) distribution Theory: normal Regression Models Maximum Likelihood Estimation Generalized M Estimation If R = r1,1 r1,2 r1,p 1 r1,p 0 r2,2 r2,p 1 r2,p .. 0 0 .. 0 0 rp 1,p 1 rp 1,p 0 0 0 rp,p , then X[1] = Q[1]r1,1 = 2 XT r= 1,1 [1]X[1] =Q[1] X[1]/r1,1 X[2] = Q[1]r1,2 + Q[2]r2,2 = QT QT = [1]X[2] Q[1]r1,2 + QT Q[2]r2,2[1][1]= 1 r1,2 + 0 r2,2 = r1,2 (known since Q[1] specfied) 21 MIT Gaussian Linear Models Gaussian Linear Models Linear Regression: Overview Ordinary Least Squares (OLS) distribution Theory: normal Regression Models Maximum Likelihood Estimation Generalized M Estimation With r1,2 and Q[1] specfied we can solve for r2,2 : = Q[2]r2,2 = X[2] Q[1]r1,2 Take squared norm of both sides: 22r= XT X[2] 2r1,2QT X[2] + r2,2 [2][1]1,2 (all terms on RHS are known) With r2,2 specified = 1 =Q[2] X[2] r1,2Q[1]r2,2 Etc.