Lecture 4: Multivariate Regression Model in Matrix …

1 1 Takashi Yamano Lecture Notes on Advanced Econometrics Lecture 4: Multivariate Regression Model in Matrix Form In this Lecture , we rewrite the multiple Regression Model in the Matrix form. A general multiple Regression Model can be written as iikkiiiuxxxy+++++= ..22110 for i = 1, .. ,n. In Matrix form, we can rewrite this Model as + = n x 1 n x (k+1) (k+1) x 1 n x 1 uXY+= We want to estimate . Least Squared Residual Approach in Matrix Form (Please see Lecture Note A1 for details) The strategy in the least squared residual approach is the same as in the bivariate linear Regression Model . First, we calculate the sum of squared residuals and, second, find a set of estimators that minimize the sum.

2 Thus, the minimizing problem of the sum of the squared residuals in Matrix form is min )()( XYXYuu = 1 x n n x 1 2 Notice here that uu is a scalar or number (such as 10,000) because u is a 1 x n Matrix and u is a n x 1 Matrix and the product of these two matrices is a 1 x 1 Matrix (thus a scalar). Then, we can take the first derivative of this object function in Matrix form. First, we simplify the matrices: ))(( XYXYuu = XXXYYXYY + = XXYXYY + =2 Then, by taking the first derivative with respect to , we have: XXYXuu + = 22)( From the first order condition ( ), we have 0 22= + XXYX YXXX = Notice that I have replaced with because satisfy the , by definition.

3 Multiply the inverse Matrix of 1)( XXon the both sides, and we have: YXXX = 1)( (1) This is the least squared estimator for the Multivariate Regression linear Model in Matrix form. We call it as the Ordinary Least Squared (OLS) estimator. Note that the first order conditions (4 2) can be written in Matrix form as 3 0) (= XYX 0 .. knknkknnkkknxxxxxxyyyxxxxxx 0 .. nkknnkkkknkkknxxyxxyxxyxxxxxx (k+1) x n n x 1 This is the same as the first order conditions, k+1 conditions, we derived in the previous Lecture note (on the simple Regression Model ): 0)..(221101= = ikkiiniixbxxy 0)..(2211011= = ikkiiniiixbxxyx 0).

4 (221101= = ikkiiniiikxbxxyx Example 4-1 : A bivariate linear Regression (k=1) in Matrix form As an example, let s consider a bivariate Model in Matrix form. A bivariate Model is iiiuxy++=110 for i = 1, .., n. In Matrix form, this is uXY+= 4 + = nnnuuuxxxyyy21102121111 From (1), we have YXXX = 1)( (2) Let s consider each component in (2). = = = ===== This is a 2 x 2 square Matrix . Thus, the inverse Matrix of XX is, = == nxnxnxxnxnXXniinii1221211)( = ==nxnxnxxxnniinii1221)(1 The second term is 5 = = = === Thus the OLS estimators are: = = === niiiniiniiyxynnxnxnxxxnYXXX112211)(1)( + = ====niiiniiiniiniiyxnynxnyxxnxynxxn11122 1)(1 = ====yxnyxyxxxyxxniiiniiiniinii111221)(1 + = ====niiiniiiniiniiyxyxyxxxyxyxyxx1122122 1)()(1 = ====niiiniiiniiniiyyxxxyyxxxxyxx1121221) )(()()()(1 = ==2111)())(( xxyyxxxyniiniii =10 6 This is what you studied in the previous Lecture note.)

5 End of Example 4 1 Unbiasedness of OLS In this sub section, we show the unbiasedness of OLS under the following assumptions. Assumptions: E 1 (Linear in parameters): uXY+= E 2 (Zero conditional mean): 0)|(=XuE E 3 (No perfect collinearity): X has rank k. From (2), we know the OLS estimators are YXXX = 1)( We can replace y with the population Model (E 1), )()( 1uXXXX+ = uXXXXXXX + = 11)()( uXXX += 1)( By taking the expectation on the both sides of the equation, we have: )()() (1uXEXXE += From E2, we have 0)|(=XuE. Thus, =) (E Under the assumptions E1 E3, the OLS estimators are unbiased. The Variance of OLS Estimators 7 Next, we consider the variance of the estimators.

6 Assumption: E 4 (Homoskedasticity): 2)|( =XuVariand 0),(=jiuuCov, thus IXuVar2)|( =. Because of this assumption, we have []IuuEuuEuuEuuEuuEuuEuuEuuEuuEuuuuuuEuuE nnnnnnnn22222122212121112121000000)()()( )()()()()()()( = = = = n x 1 1 x n n x n n x n n x n Therefore, ])([) (1uXXXVarVar += ])[(1uXXXVar = ])()[(11 =XXXuuXXXE 11)()()( =XXXuuEXXX 121)()( =XXXIXXX (E4: Homoskedasticity) 12)() ( =XXVar (3) GAUSS-MARKOV Theorem: Under assumptions 1 4, is the Best Linear Unbiased Estimator (BLUE). 8 Example 4-2: Step by Step Regression Estimation by STATA In this sub section, I would like to show you how the Matrix calculations we have studied are used in econometrics packages.

7 Of course, in practices you do not create Matrix programs: econometrics packages already have built in programs. The following are Matrix calculations with STATA using data called, Here we want to estimate the following Model : iiiiiuedusqedufemaleyincome++++=3210)ln( All the variables are defined in Example 3 1. Descriptive information about the variables are here: . su; Variable | Obs Mean Std. Dev. Min Max -------------+-------------------------- ------------------------------ female | 648 .2222222 .4160609 0 1 edu | 648 -8 19 edusq | 648 0 361 -------------+-------------------------- ------------------------------ ln_income | 648 First, we need to define matrices.

8 In STATA, you can load specific variables (data) into matrices. The command is called mkmat. Here we create a Matrix , called y, containing the dependent variable, ln_nfincome, and a set of independent variables, called x, containing female, educ, educsq.. mkmat ln_nfincome, Matrix (y) . mkmat female educ educsq, Matrix (x) Then, we create some components: XX , 1)( XX, and YX : 9. Matrix xx=x'*x; . mat list xx; symmetric xx[4,4] female edu edusq const female 144 edu 878 38589 edusq 8408 407073 4889565 const 144 4197 38589 648 . Matrix ixx=syminv(xx); . mat list ixx; symmetric ixx[4,4] female edu edusq const female.

9 0090144 edu .00021374 .00053764 edusq const .00007321 .00806547 Here is YX : . Matrix xy=x'*y; . mat list xy; xy[4,1] ln_nfincome female edu edusq const Therefore the OLS estimators are YXXX 1)(: . ** Estimating b hat; . Matrix bhat=ixx*xy; 10. mat list bhat; bhat[4,1] ln_nfincome female edu .04428822 edusq .00688388 const . ** Estimating standard error for b hat; . Matrix e=y-x*bhat; . Matrix ss=(e'*e)/(648-1-3); . Matrix kk=vecdiag(ixx); . mat list ss; symmetric ss[1,1] ln_nfincome ln_nfincome . mat list kk; kk[1,4] female edu edusq const r1 .0090144.

10 00053764 .00806547 Let s verify what we have found.. reg ln_nfincome female edu edusq; Source | SS df MS Number of obs = 648 -------------+-------------------------- ---- F( 3, 644) = Model | 3 Prob > F = Residual | 644 R-squared = -------------+-------------------------- ---- Adj R-squared = Total | 647 Root MSE = ---------------------------------------- -------------------------------------- ln_nfincome | Coef. Std. Err. t P>|t| [95% Conf. Interval] 11-------------+------------------------ ---------------------------------------- female |.