OLS Estimation of the Multiple (Three-Variable) Linear ...

1 ECONOMICS 351* -- NOTE 12 Abbott ECON 351* -- Note 12: OLS Estimation in the Multiple CLRM .. Page 1 of 17 pages ECON 351* -- NOTE 12 OLS Estimation of the Multiple (Three-Variable) Linear regression Model This note derives the Ordinary Least Squares (OLS) coefficient estimators for the three-variable Multiple Linear regression model. The population regression equation, or PRE, takes the form: (1) ii22i110ii22i110i2i1uXXY+ + + = where ui is an iid random error term. The OLS sample regression equation (OLS-SRE) for equation (1) can be written as (i = 1, .., N). (2) iiii22i110iu Y u X X Y+=+ + + = where the are the OLS estimators of the corresponding population regression coefficients j (j = 0, 1, 2), $ j (i = 1, .., N) i22i110iiiiX X YY Yu = = are the OLS residuals, and (i = 1.)

2 , N) i22i110iX X Y + + = are the OLS estimated (or predicted) values of Yi. The function is called the OLS sample regression function (or OLS-SRF). X X )X,X(f + + =ECONOMICS 351* -- NOTE 12 Abbott ECON 351* -- Note 12: OLS Estimation in the Multiple CLRM .. Page 2 of 17 pages 1. The OLS Estimation Criterion The OLS coefficient estimators are those formulas (or expressions) for , , and that minimize the sum of squared residuals RSS for any given sample of size N. 0 1 2 The OLS Estimation criterion is therefore: ()() == == N1i2i22i110iN1i2i210X X Yu , , RSS Minimize (3) {} $ j Interpretation of the ()210 , , RSS function: The knowns in the ()210 , , RSS function are the sample observations )i2i1 for i = 1, .., N. In other words, the N sample values of the observable variables Y, X1, X2 are taken as known (or given). (iX,X,1,Y The unknowns in the ()210 , , RSS function are therefore the coefficient estimators 21.

3 0 , , For purposes of deriving the OLS coefficient estimators, the ()210 , , RSS 0 , , function is interpreted as a function of the three unknowns 21 . ECONOMICS 351* -- NOTE 12 Abbott ECON 351* -- Note 12: OLS Estimation in the Multiple CLRM .. Page 3 of 17 pages 2. The OLS Normal Equations: Derivation of the FOCs STEP 1: Re-write the ()210 , , RSS function in (3) as follows: () ==== N1iiN1i2i210)u (fu , , RSS where 2iiu )u (f=i22i110iiX X Yu = (3) Note: The function is a function of , and is in turn a function of , , and . 2iiu )u (f=iu iu 0 1 2 STEP 2: Partially differentiate the ()210 , , RSS function in (3) with respect to , , and : 0 1 2 Using the chain rule of differentiation, each partial derivative of the ()210 , , RSS function takes the general form = = N1ijiij u u dfd RSS. (4) Using the power rule of differentiation, the derivative iu dfdis ii2iiu 2u d)u (du dfd==.

4 The partial derivatives j RSS for j = 0, 1, 2 are therefore === = = = N1ijiiN1ijiiN1ijiij u u 2 u u 2 u u dfd RSS j = 0, 1, 2. (5) ECONOMICS 351* -- NOTE 12 Abbott ECON 351* -- Note 12: OLS Estimation in the Multiple CLRM .. Page 4 of 17 pages Since the i-th residual is i22, the partial derivatives i110iiX X Yu = $$uijfor j = 0, 1, 2 are: i22ii11i0iX u ;X u ;1 u = = = . Substitute the partial derivatives $$uijfor j = 0, 1, 2 into equation (5): = = N1ijiij u u 2 RSS j = 0, 1, 2. (5) The partial derivatives RSSj$ for j = 0, 1, 2 thus take the form: === = = = N1iiN1iiN1i0ii0u 2)1(u 2 u u 2 RSS ( ) === = = = N1iii1N1ii1iN1i1ii1u X2)X(u 2 u u 2 RSS ( ) === = = = N1iii2N1ii2iN1i2ii2u X2)X(u 2 u u 2 RSS ( ) ECONOMICS 351* -- NOTE 12 Abbott ECON 351* -- Note 12: OLS Estimation in the Multiple CLRM.

5 Page 5 of 17 pages STEP 3: Obtain the first-order conditions (FOCs) for a minimum of the RSS function by setting the partial derivatives ( )-( ) equal to zero, then dividing each equation by 2, and finally setting : i22i110iiX X Yu = = = N1ii0u 2 RSS ( ) 0 RSS0= ( ) 0u 2N1ii= =0u N1ii= = ( ) (0X X YN1ii22i110i= =) = = N1iii11u X2 RSS ( ) 0 RSS1= ( ) 0u X2N1iii1= =0u XN1iii1= = ()0X X YXN1ii22i110ii1= = ( ) = = N1iii22u X2 RSS ( ) 0 RSS2= ( ) 0u X2N1iii2= =0u XN1iii2= = ()0X X YXN1ii22i110ii2= = ( ) ECONOMICS 351* -- NOTE 12 Abbott ECON 351* -- Note 12: OLS Estimation in the Multiple CLRM .. Page 6 of 17 pages STEP 4: Rearrange each of the equations ( )-( ) to put them in the conventional form of the OLS normal equations.

6 Thus, taking summations and rearranging terms, we obtain the OLS normal equations: )0 ( ) (X X YN1ii22i110i= =0X X NYN1ii22N1ii110N1ii= === === = N1iiN1ii22N1ii110YX X N (N1) ==== + + N1iiN1ii22N1ii110YX X N )0 ( ) (X X YXN1ii22i110ii1= =()0XX X X YXN1ii2i122i11i10ii1= = 0XX X X YXN1ii2i12N1i2i11N1ii10N1iii1= ==== ==== = N1iii1N1ii2i12N1i2i11N1ii10 YXXX X X (N2) =====++N1iii1N1ii2i12N1i2i11N1ii10 YXXX X X )0 ( ) (X X YXN1ii22i110ii2= =()0X XX X YXN1i2i22i1i21i20ii2= = 0X XX X YXN1i2i22N1ii1i21N1ii20N1iii2= ==== ==== = N1iii2N1i2i22N1ii1i21N1ii20 YXX XX X (N3) ===== + + N1iii2N1i2i22N1ii1i21N1ii20 YXX XX X ECONOMICS 351* -- NOTE 12 Abbott ECON 351* -- Note 12: OLS Estimation in the Multiple CLRM .. Page 7 of 17 pages RESULT: Assemble the three OLS normal equations (N1)-(N3): ==== + + N1iiN1ii22N1ii110YX X N (N1) ===== + + N1iii1N1ii2i12N1i2i11N1ii10 YXXX X X (N2) ===== + + N1iii2N1i2i22N1ii1i21N1ii20 YXX XX X (N3) The OLS normal equations (N1)-(N3) constitute three Linear equations in the three unknowns 0 , 1 , and 2.

7 Solution of the OLS normal equations (N1)-(N3) yields explicit expressions (or formulas) for 0 , 1 , and 2 ; these expressions are the OLS estimators 0 , 1 , and 2 of the partial regression coefficients 0, 1, and 2 respectively. ECONOMICS 351* -- NOTE 12 Abbott ECON 351* -- Note 12: OLS Estimation in the Multiple CLRM .. Page 8 of 17 pages 3. Expressions for the OLS Coefficient Estimators The expressions (formulas) for the OLS estimators are most conveniently written in deviation-from-means form, which uses lower case letters to denote the deviations of the sample values of each observable variable from their respective sample means. Thus, define the deviations-from-means of Yi, X1i, and X2i as: ;XXx ;XXx ;YYy2i2i21i1i1ii where YYNYN iiii== is the sample mean of the Yi values; NXNXXi1ii1i1 = =is the sample mean of the X1i values; NXNXXi2ii2i2 = =is the sample mean of the X2i values.

8 The OLS slope coefficient estimators 1 and 2 in deviation-from-means form are: ()()( )()()()()2i2i1i2i2i2i1iii2ii2i1iii1i2i2i 1xxxxyxxxyxx = ; ( ) ()()( )()()()()2i2i1i2i2i2i1iii1ii2i1iii2i2i1i 2xxxxyxxxyxx = . ( ) The OLS intercept coefficient estimator 0 is: 22110X X Y = . ( ) ECONOMICS 351* -- NOTE 12 Abbott ECON 351* -- Note 12: OLS Estimation in the Multiple CLRM .. Page 9 of 17 pages 4. The OLS Variance-Covariance Estimators An unbiased estimator of the error variance 2 For the general Multiple Linear regression model with K regression coefficients, an unbiased estimator of the error variance 2 is the degrees-of-freedom-adjusted estimator ()()KNRSSKNu 2ii2 = = where K = k + 1 = the total number of regression coefficients in the PRF. For the three-variable Multiple Linear regression model (such as regression equation (1) above) for which K = 3, the unbiased estimator of the error variance 2 is therefore ()()$$ 2233= = iiuNRSSN.

9 (10) where (N 3) is the degrees of freedom for the residual sum of squares RSS in the OLS-SRE (2). is an unbiased estimator of 2 because it can be shown that E() = E(RSS) = (N 3) 2 . $ 2 iiu$2 The error variance estimator $ 2 is used to obtain unbiased estimators of the variances and covariances of the OLS coefficient estimators 0 , 1 , and 2 . ECONOMICS 351* -- NOTE 12 Abbott ECON 351* -- Note 12: OLS Estimation in the Multiple CLRM .. Page 10 of 17 pages Formulas for the variances and covariances of the slope coefficient estimators 1 and 2 in the three-variable Multiple regression model ()()()()2i2i1i2i2i2i1i2i2i21xxxxx Var = ; ()()()()2i2i1i2i2i2i1i2i1i22xxxxx Var = ; ()()()()2i2i1i2i2i2i1ii2i1i221xxxxxx , Cov = . Unbiased estimators of the variances of the slope coefficient estimators 1 and 2 are obtained by substituting the unbiased estimator $ 2 for the unknown error variance 2 in the formulas for ) ( and ) (Var2 : Var1 ()()()()2i2i1i2i2i2i1i2i2i21xxxxx ra V = ; ( ) ()()()()2i2i1i2i2i2i1i2i1i22xxxxx ra V = ; ( ) Similarly, an unbiased estimator of the covariance between the slope coefficient estimators 1 and 2 is obtained by substituting the unbiased estimator $ 2 for the unknown error variance 2 in the formula for ) , (1 : Cov2 ()()()()2i2i1i2i2i2i1ii2i1i221xxxxxx , vo C =.

10 ( ) ECONOMICS 351* -- NOTE 12 Abbott ECON 351* -- Note 12: OLS Estimation in the Multiple CLRM .. Page 11 of 17 pages Interpretive formula for the variances of the OLS slope coefficient estimators j , j = 1, 2, .., k Consider the general Multiple Linear regression equation given by the PRE ikikjiji110iuXXXY+ ++ ++ + =LL ( ) OLS Estimation of the PRE in (11) yields the OLS SRE ikikjiji110iu X X X Y+ ++ ++ + =LL ( ) The formula for ) (for j = 1, 2, .., k can be written as Varj ()2jj2jR1 TSS) (Var = for j = 1, 2, .., k (13) where () ==N1i2jjiN1i2jijXXxTSS the total sample variation in the regressor Xj; 2jR the R2 from the OLS regression of regressor Xj on all the other K 1 regressors in ( ), including the intercept. That is, measures the 2jRproportion of the total sample variation in Xj that is explained by the other regressors in the PRE. Alternatively, measures the degree of 2jRlinear dependence between the sample values Xji of the regressor Xj and the sample values of the other regressors in regression equation ( ).