Lecture 13: Simple Linear Regression in Matrix Format

1 11:55 Wednesday 14thOctober, 2015 See updates and corrections ~cshalizi/mreg/ Lecture 13: Simple Linear Regression in MatrixFormat36-401, Section B, Fall 201513 October 2015 Contents1 Least Squares in Matrix The Basic Matrices .. Mean Squared Error .. Minimizing the MSE ..42 Fitted Values and Residuals .. Expectations and Covariances ..73 Sampling Distribution of Estimators84 Derivatives with Respect to Second Derivatives .. Maxima and Minima .. 115 Expectations and Variances with Vectors and Matrices126 Further Reading1312So far, we have not used any notions, or notation, that goes beyond basicalgebra and calculus (and probability).

2 This has forced us to do a fair amountof book-keeping, as it were by hand. This is just about tolerable for the simplelinear model, with one predictor variable. It will get intolerable if we havemultiple predictor variables. Fortunately, a little application of Linear algebrawill let us abstract away from a lot of the book-keeping details, and makemultiple Linear Regression hardly more complicated than the Simple notes will not remind you of how Matrix algebra works. However, theywill review some results aboutcalculuswith matrices, and about expectationsand variances with vectors and , bold-faced letters will denote matrices, asaas opposed to Least Squares in Matrix FormOur data consists ofnpaired observations of the predictor variableXand theresponse variableY, , (x1,y1).

3 (xn,yn). We wish to fit the modelY= 0+ 1X+ (1)whereE[ |X=x] = 0, Var [ |X=x] = 2, and is uncorrelated across The Basic MatricesGroup all of the observations of the response into a single column (n 1) matrixy,y= (2)Similarly, we group both the coefficients into a single vector ( , a 2 1matrix) =[ 0 1](3)We d also like to group the observations of the predictor variable together,but we need something which looks a little unusual at first:x= (4)1 Historically, Linear models with multiple predictors evolved before the use of Matrix alge-bra for Regression .

4 You may imagine the resulting I need to also assume that is Gaussian, and strengthen uncorrelated to inde-pendent , I ll say :55 Wednesday 14thOctober, Mean Squared ErrorThis is ann 2 Matrix , where the first column is always 1, and the second columncontains the actual observations ofX. We have this apparently redundant firstcolumn because of what it does for us when we multiplyxby :x = 0+ 1x1 0+ 0+ 1xn (5)That is,x is then 1 Matrix which contains the point matrixxis sometimes called thedesign Mean Squared ErrorAt each data point, using the coefficients results in some error of prediction,so we havenprediction errors.

5 These form a vector:e( ) =y x (6)(You can check that this subtracts ann 1 Matrix from ann 1 Matrix .)When we derived the least squares estimator, we used the mean squarederror,MSE( ) =1nn i=1e2i( )(7)How might we express this in terms of our matrices? I claim that the correctform isMSE( ) =1neTe(8)To see this, look at what the Matrix multiplication really involves:[ ] (9)This, clearly equals ie2i, so the MSE has the claimed us expand this a little for further ( ) =1neTe(10)=1n(y x )T(y x )(11)=1n(yT TxT)(y x )(12)=1n(yTy yTx TxTy+ TxTx )(13)11:55 Wednesday 14thOctober, Minimizing the MSEN otice that (yTx )T= TxTy.

6 Further notice that this is a 1 1 Matrix , soyTx = TxTy. ThusMSE( ) =1n(yTy 2 TxTy+ TxTx )(14) Minimizing the MSEF irst, we find the gradient of the MSE with respect to : MSE( =1n( yTy 2 TxTy+ TxTx )(15)=1n(0 2xTy+ 2xTx )(16)=2n(xTx xTy)(17)We now set this to zero at the optimum, :xTx xTy= 0(18)This equation, for the two-dimensional vector , corresponds to our pair of nor-mal or estimating equations for 0and 1. Thus, it, too, is called an , = (xTx) 1xTy(19)That is, we ve got one Matrix equation which gives us both coefficient this is right, the equation we ve got above should in fact reproduce theleast-squares estimates we ve already derived, which are of course 1=cXYs2X=xy x yx2 x2(20)and 0=y 1x(21)Let s see if that s a first step, let s introduce normalizing factors of 1/ninto both the matrixproducts.)

7 = (n 1xTx) 1(n 1xTy)(22)Now let s look at the two factors in parentheses separately, from right to [ xn] (23)=1n[ iyi ixiyi](24)=[yxy](25)11:55 Wednesday 14thOctober, 20155 Similarly for the other factor:1nxTx=1n[n ixi ixi ix2i](26)=[1xxx2](27)Now we need to take the inverse:(1nxTx) 1=1x2 x2[x2 x x1](28)=1s2X[x2 x x1](29)Let s multiply together the pieces.(xTx) 1xTy=1s2X[x2 x x1][yxy](30)=1s2X[x2y xxy xy+xy](31)=1s2X[(s2X+ x2)y x(cXY+ x y)cXY](32)=1s2X[s2xy+ x2y xcXY x2 ycXY](33)=[y cXYs2 XxcXYs2X](34)which is what it should :n 1xTyis keeping track ofyandxy, andn 1xTxkeeps track ofxandx2.

8 The Matrix inversion and multiplication then handles all the book-keeping to put these pieces together to get the appropriate (sample) variances,covariance, and intercepts. We don t have to remember that any more; we canjust remember the one Matrix equation, and then trust the Linear algebra totake care of the Fitted Values and ResidualsRemember that when the coefficient vector is , the point predictions for eachdata point arex . Thus the vector of fitted values, m(x), or mfor short, is m=x (35)Using our equation for , m=x(xTx) 1xTy(36)11:55 Wednesday 14thOctober, 20156 Notice that the fitted values are Linear iny.

9 The matrixH x(xTx) 1xT(37)does not depend onyat all, but does control the fitted values: m=Hy(38)If we repeat our experiment (survey, observation.. ) many times at the samex, we get differentyevery time. ButHdoes not change. The properties ofthe fitted values are thus largely determined by the properties ofH. It thusdeserves a name; it s usually called thehat Matrix , for obvious reasons, or, ifwe want to sound more respectable, theinfluence s look at some of the properties of the hat not a function ofy, we can easily verify that mi/ yj=Hij.

10 Thus,Hijis the rate at which theithfitted value changes as we varythejthobservation, the influence that observation has on that s easy to see thatHT= square matrixais calledidempotent3whena2=a(and soak=afor any higher powerk). Again, by writing out themultiplication,H2=H, so it s , Projection, GeometryIdempotency seems like the mostobscure of these properties, but it s actually one of the more maren-dimensional vectors. If we project a vectoruon to the line in thedirection of the length-one vectorv, we getvvTu(39)(Check the dimensions:uandvare bothn 1, sovTis 1 n, andvTuis1 1.)