Weighted Least Squares - McMaster University

1 Weighted Least Squares The standard linear model assumes that Var( i) = 2fori= 1,..,n. As we have seen, however, there are instances whereVar(Y|X=xi) = Var( i) = 2wi. Herew1,..,wnare known positive constants. Weighted Least Squares is an estimation technique whichweights the observations proportional to the reciprocal ofthe error variance for that observation and so overcomes theissue of non-constant Least Squares in Simple Regression Suppose that we have the following modelYi= 0+ 1Xi+ ii= 1,..,nwhere i N(0, 2/wi) forknownconstantsw1,..,wn. The Weighted Least Squares estimates of 0and 1minimizethe quantitySw( 0, 1) =n i=1wi(yi 0 1xi)2 Note that in this Weighted sum of Squares , the weights areinversely proportional to the corresponding variances; pointswith low variance will be given higher weights and points withhigher variance are given lower Least Squares in Simple Regression The Weighted Least Squares estimates are then given as 0=yw 1xw 1= wi(xi xw)(yi yw) wi(xi xw)2wherexwandyware the Weighted meansxw= wixi wiyw= wiyi wi.

2 Some algebra shows that the Weighted Least Squares esti-mates are still Least Squares in Simple Regression Furthermore we can find their variancesVar( 1) = 2 wi(xi xw)2 Var( 0) = 1 wi+x2w wi(xi xw)2 2 Since the estimates can be written in terms of normal randomvariables, the sampling distributions are still normal. The Weighted error mean squareSw( 0, 1)/(n 2) also givesus an unbiased estimator of 2so we can derivettests forthe parameters Weighted Least Squares Solution LetWbe a diagonal matrix with diagonal elements equal tow1,..,wn. The the Weighted Residual Sum of Squares is defined bySw( ) =n i=1wi(yi xti )2= (Y X )tW(Y X ). Weighted Least Squares finds estimates of by minimizingthe Weighted sum of Squares .

3 The general solution to this is = (XtWX) Least Squares as a Transformation Recall from the previous chapter the modelyi= 0+ 1xi+ iwhere Var( i) =x2i 2. We can transform this into a regular Least Squares problemby takingy i=yixix i=1xi i= ixi. Then the model isy i= 1+ 0x i+ iwhere Var( i) = Least Squares as a Transformation The residual sum of Squares for the transformed model isS1( 0, 1) =n i=1(y i 1 0x i)2=n i=1(yixi 1 01xi)2=n i=1(1x2i)(yi 0 1xi)2 This is the Weighted residual sum of Squares withwi= 1/x2i. Hence the Weighted Least Squares solution is the same as theregular Least Squares solution of the transformed Least Squares as a Transformation In general suppose we have the linear modelY=X + where Var( ) =W 1 2.

4 LetW1/2be a diagonal matrix with diagonal entries equalto wi. Then we have Var(W1/2 ) = Least Squares as a Transformation Hence we consider the transformationY =W1/2Y X =W1/2X =W1/2 . This gives rise to the usual Least Squares modelY =X + Using the results from regular Least Squares we then get thesolution =((X )tX ) 1(X )tY =(XtWX) 1 XtWY. Hence this is the Weighted Least Squares of Weighted Least Squares In the transformed model, the interpretation of the coeffi-cient estimates can be difficult. In Weighted Least squaresthe interpretation remains the same as before. In the transformed model, there will often not be an inter-cept which means that the F-tests and R-squared values arequite different.

5 In Weighted Least Squares we generally in-clude an intercept retaining the usual interpretation of thesequantities. Weighted Least Squares gives us an easy way to remove oneobservation from a model by setting its weight equal to 0. We can also downweight outlier or influential points to reducetheir impact on the overall Weights To apply Weighted Least Squares , we need to know the weightsw1,..,wn. There are some instances where this is true. We may have a probabilistic model for Var(Y|X=xi) inwhich case we would use this model to find thewi. For example, with Poisson data we may usewi= 1/xiif weexpect an increasing relationship between Var(Y|X=x) Weights Another common case is where each observation is not asingle measure but an average ofniactual measures and theoriginal measures each have variance 2.

6 In that case, standard results tell us thatVar( i)= Var(Yi|X=xi)= 2ni Thus we would use Weighted Least Squares with weightswi=ni. This situation often occurs in cluster Weights In many real-life situations, the weights are not known apriori. In such cases we need to estimate the weights in order touse Weighted Least Squares . One way to do this is possible when there are multiple re-peated observations at each value of the covariate vector. That is often possible in designed experiments in which anumber of replicates will be observed for each set value ofthe covariate vector. We can then estimate the variance ofYfor each fixed co-variate vector and use this to estimate the Error Suppose that we haveniobservations atx=xj,j= 1.

7 ,k. Then a fitted model could beyij= 0+ 1xj+ iji= 1,..,nj;j= 1,..,k. The (i,j)thresidual can then be written aseij=yij yij= (yij yj) + (yj yij). The first term is referred to as the pure Error Note that the pure error term does not depend on the meanmodel at all. We can use the pure error mean Squares to estimate thevariance at each value 1nj i=1(yij yj)2j= 1,..,k. Then we can use the weightswij= 1/s2j(i= 1,..,nj) asthe weights in a Weighted Least Squares regression Weights For observational studies, however, this is generally not pos-sible since we will not have repeated measures at each valueof the covariate(s). This is particularly true when there are multiple covariates.

8 Sometimes, however, we may be willing to assume that thevariance of the observations is the same within each levelof some categorical variable but possibly different betweenlevels. In that case we can estimate the weights assigned to obser-vations with a given level by an estimate of the variance forthat level of the categorical variable. This leads to a two-stage method of Estimation In the two-stage estimation procedure we first fit a regularleast Squares regression to the data. If there is some evidence of non-homogenous variance thenwe examine plots of the residuals against a categorical vari-able which we suspect is the culprit for this problem.

9 Note that we do still need to have some apriori knowledge ofa categorical variable likely to affect variance. This categorical variable may, or may not, be included in themean Estimation LetZbe the categorical variable and assume that there arenjobservations withZ=j(j= 1,..,k). If the error variability does vary with the levels of this cate-gorical variable then we can use 2j=1nj 1 i:zi=jr2ias an estimate of the variability whenZ=j. NoteYour book uses the raw residualseiinstead of thestudentized residualsribut that does not work Estimation If we now assume that 2j=cj 2we can estimate thecjby cj= 2j 2=1nj 1 i:zi=jr2i1nn i=1r2i We could then use the reciprocals of these estimates as theweights in a Weighted Least Squares regression in the secondstage.

10 Approximate inference about the parameters can then bemade using the results of the Weighted Least Squares with Two-Stage Estimation The method described above is not universally accepted anda number of criticisms have been raised. One problem with this approach is that different datasetswould result in different estimated weights and this variabilityis not properly taken into account in the inference. Indeed the authors acknowledge that the true sampling dis-tributions are unlikely to be Student-t distributions and areunknown so inference may be suspect. Another issue is that it is not clear how to proceed if nocategorical variable explaining the variance heterogeneity canbe