What is the Error Term in a Regression Equation?

1 What is the Error Term in a Regression Equation? byDavid A FreedmanIt is often said that the Error term in a Regression equation represents the effect of the variablesthat were omitted from the equation. This is unsatisfactory, even in simple contexts, as the followingdiscussion should indicate. Suppose subjects are IID, and all variables are jointly normal withexpectation 0. Suppose the explanatory variables have variance 1. The explanatory variables maybe correlated amongst themselves, but anypof them have a non-singularp-dimensional parameters jare real. Let(1)Yi= j=1 jXijFor eachp=1,2,..,consider the Regression model(2)Yi=p j=1 jXij+ i(p)where(3) i(p)= j=p+1 jXijThe jare identifiable.

2 If theXijare independent forj=1,2,..,the standard assumptionshold, and i(p)does indeed represent the effect onYiof the omitted variables{Xij:j=p+1,..},at least in an algebraic sense. On the other hand, if theXijare dependent, the matter is we take (1 3) as written, then i(p)represents the effect onYiof the omitted variables but i(p)is correlated with the explanatory variables. The standard assumptions fail, and fitting (2) to datafori=1,..,nwill estimate the wrong parameters. If i(p)is replaced by i(p) , namely, thepart of i(p)independent ofXi1,..,Xip, we have a bona fide Regression model, but with different is no easy way out of the difficulty.

3 The conventional interpretation for Error termsneeds to be reconsidered. At a minimum, something like this would need to be said: the Error termrepresents the combined effect of the omitted variables, assuming that(i) the combined effect of the omitted variables is independent of each variable included inthe equation,(ii) the combined effect of the omitted variables is independent across subjects,(iii) the combined effect of the omitted variables has expectation is distinctly harder to swallow. Pratt and Schlaifer have a discussion in great technical detailsIf the jvanish for all but finitely manyj, there are no technical issues.

4 The inferential issueremains, provided the largestjwith j =0 is an unknown parameter. Suppose next that j =0 forinfinitely manyj. Summability and identifiability must be demonstrated. To avoid interesting butunnecessary probabilistic complications, suppose j| j|< . Fixi. Suppose also that part ofeachXij:j=1,2,..is independent of all the otherXik, and hasL2norm at least >0. Morespecifically, letX ijbeXijnet of{Xik:k=1,..,pwithk =j}. Thus, we assume X ij ,where is theL2norm. See below for definitions and some i(p) j=p+1| j|is small, so the sum on the right hand side of (1) converges inL2. Fixjandpwith 1 j p.

5 The Regression of i(p)on{Xi1,..,Xip}has a small coefficientonXij, because(i) i(p)is small,(ii) we get the coefficient by regressing i(p)onX ij, and(iii) X ij .In more formal terms, by Lemma 2 below, a Regression ofYionXi1,..,Xipin the random-variable domain gives a coefficient onXijof cov(X ij,Y)/var(X ij). This coefficient is j, with anerror that is at most(4)cov(X ij, i(p))var(X ij) X ij i(p) X ij 2 1 i(p) 1 j=p+1| j| 0asp . That proves mistake to avoidSome may conclude from the forgoing that bigger models are better. Perhaps, but (i) eventuallywe run out of data, and (ii) there is always the ugly possibility of inadvertently including anendogenous variable.

6 Also see exercise 15 on page 105 ofStatistical Modelsfor information onstandard errors in the presence of misspecification. Kitchen-sink models have their problems in the domain of random variablesChanging notation, letqbe a positive integer. LetU1,..,Uq,Vbe jointly normal randomvariables, each having expectation 0. LetCij=cov(Ui,Uj). This is a symmetricq qmatrix,assumed to be positive definite. LetDi=cov(Ui,V). TakeD=(D1,..,Dq) as aq 1 1D, which is also aq 1 vector. LetV =V (U1,..,Uq) B, a scalar 1. (i)V is normal with expectation 0, and (ii)V (U1,..,Uq)in the sense thatcov(Uj,V )=E(UjV )=0 for eachj=1.

7 ,q. In particular, (iii)V and(U1,..,Uq) the proof, assertion (i) is immediate. For (ii), we need only check thatcov(Uj,V)=cov(Uj,(U1,..,Uq) B)=q k=1cov(Uj,Uk)Bk=q i=1 CjkBk, ,D=CB. ButB=C 1 Dby construction, completing the short,(U1,..,Uq) Bis the Regression ofVonU1,..,Uq; the coefficient onUiisBi;andV is the part ofVindependent ofU1,..,Uq. This is also Vnet ofU1,..,Uq. Normalityis relevant only to convert orthogonality into independence. Without normality,(U1,..,Uq) Bis the linear projection ofVontoU1,..,Uq, , the linear combination ofU1,..,Uqclosest toVinL2 becauseV is orthogonal toU1,..,Uq. The simplest special case hasq=1.

8 Then theregression coefficient takes a form that may be more familiar, cov(U1,V)/var(U1).Lemma 2. The Regression ofVonU=(U1,..,Uq)can be computed by the followingstepwise procedure, with U=(U2,..,Uq).(i) RegressVonU2,..,Uq. Let be the(q 1) 1 vector of Regression coefficients. Let V= U andV =V V.(ii) RegressU1onU2,..,Uq. Let be the(q 1) 1 vector of Regression U1= U andU 1=U1 U1.(iii) RegressVonU 1. Let be the Regression coefficient, a 1 vector of Regression coefficients ofVonU1,..,Uqis then( )Proof. SinceV= V+V and V U 1, whether we regressVonU 1orV onU 1, thecoefficient onU 1will be the same, viz.

9 , .So =V U 1 U 1. Plainly, U2,..,Uq,because is a linear combination ofV andU 1. Thus,V= V+V = V+U 1 + (5)= U +(U1 U ) + =U1 + U( )=( )U+ with U, as required. To clarify the notation,Uis 1 qand Uis 1 (q 1); both are randomvectors; V,V , U1,U 1, are all scalar random variables. IfU1,..,Uq,Vare taken as jointlynormal, these derived quantities are jointly normal too. The quantities , , are parameters notestimates, being computed from the joint distribution not from data. Exercise 17 on page 34 ofStatistical Modelscovers Regression in the data domain using a method exactly like that in Lemma 2,although the notation is little DA (2005).

10 Statistical Models: Theory and Practice. Cambridge University J, Schlaifer R (1984). On the Nature and Discovery of of the AmericanStatistical Association79: 9 J, Schlaifer R (1988). On the Interpretation and Observation of of Econometrics39: 23 for Statistics 215 Department of StatisticsUC Berkeley, CA 94720-3860 November, 20054