6 Dealing with Model Assumption Violations

1 6 Dealing with Model Assumption ViolationsIf the regression diagnostics have resulted in the removal of outliers and influential observations, butthe residual and partial residual plots still show that Model assumptions are violated, it is necessaryto make further adjustments either to the Model (including or excluding predictors), or transformingthe response and/or predictors, and/or weighting the measurements, and if this all does not helpswitching to a different Model or estimation method the inclusion or exclusion of predictors do not resolve the concerns about the violation of the modelassumptions further approaches can be on the type of violation different remedies can TransformationsTransformations can help when1. the homoscedasticity Assumption , or2. the linearity Assumption , or3. normalityis HeteroscedasticityIf the Assumption of constant variance is violated, the least squares estimators are still unbiased, butthe Gauss-Markov theorem does not hold anymore, and standardized scores do not have the assumeddistribution, and therefore test results and confidence intervals are unreliable.

2 Usually the standarderrors of the regression coefficients are too transformation of the response variable can help to resolve such a problem. Depending on thetype of violation, different transformations are helpful:Useful transformations:Relationship between the errorTransformation ofYvariance and the mean response 2 E(Y)square root 2 E(Y)2log 2 E(Y)3reciprocal square root (1/ (y)) 2 E(Y)4reciprocal 2 E(Y)(1 E(Y))if 0 Y 1, arcsin, (sin 1( (y))However, after applying the transformation the interpretation of the regression coefficients is notstraight forward anymore, and the inverse transformation is not necessarily resulting in unbiasedestimates on the original intervals and prediction intervals though can be transformed back to the original data set consists on 400 randomly sampled elementary schools from the California Departmentof Education s API in the 2000.)

3 This data file contains a measure of school academic performance(api00) as well as other attributes of the elementary schools, such as, class size, enrollment, poverty, enrollment have an effect on academic performance?The scatterplot indicates the higher the enrollment the lower the academic the modelapi00 = 0+ 1enrollment+ results in the following residual plots2 Both indicate a violation of the Assumption of homoscedasticity. To investigate the nature of therelationship of the violation plot the squared residuals against the fitted the different transformations suggested in the table above1/ api00 = 0+ 1enrollment+ results in the following residual plotsthe best of the fitted curve based on the transformed Model is3 The transformation of the Model ensures that the Model assumptions are met, and improves thefit of the Model to data and hopefully therefore to the population.

4 Tests and confidence intervalsbecome reliable statements which hold the associated error by transforming the response variable we loose the straight forward interpretation of the estimateof . Non-linear RelationshipsIf the scatterplot for response and predictor does indicate a non linear relationship transformationsof the response and/or predictor can result in a linear Model for fitting the non-linear easiest example is to permit for a polynomial relationship, Where the Model becomes:Y= 0+ 1x+ 2x2+ + kxk+ , N(0, 2)Some other non-linear relationships we can deal with by transforming the response and/or predictorvariable(s) are given in the graphs and table general a non-linear regression Model should be non-linear relationships which can be dealt with by transformations are:FunctionTransformations ofxand/oryResulting modely= 0x 1y =log(y),x =log(x)y =log( 0) + 1x y= 0e 1xy =ln(y)y =ln( 0) + 1xy= 0+ 1log(x)x =log(x)y= 0+ 1x y=x 0x 1y =1y,x =1xy = 0 1x 4 The transformation have a large influence on the error structure.

5 The Assumption is that the error5in the transformed Model is normal with mean 0 and constant variance looking closer at the second transformation:The MLRM islog(Y) =log( 0) + 1x+ which is equivalent to (take e to the power of sides):Y= 0e 1xe Since we assume that is normal, this implies that the multiplicative error in the Model forYis lognormal (it s logarithm is normal).To check Model Assumption the transformed Model has to be checked using residuals and sample analyzed consists of 50 observations of per capita expenditure on public schools and percapita income for each state and the District of Columbia in 1979. Wisconsin was removed becauseno information was available on the per capita expenditure for public scatterplot suggest a quadratic relationship. Use the data to fit the modelexpend= 0+ 1(income) + 2(income)2+ Resulting in:6 The residual plot shows that for states with larger per capita income the variation in the per capitaexpenditure is larger than for states with lower per capita income.

6 This is in violation of the the log transformation for the per capita expenditure results in the following scatterplot:Showing a linear (expend) = 0+ 1(income) + results in the following residual plot:This residual plot for the transformed Model shows a big improvement over the residual plot for theinitial Model . Tests and confidence intervals based on this Model are more reliable than for the Choosing a Box-Cox: transforming the responseAnother violation which can be coped with by transforming the data is non-normality of the and Cox(1964) developed a method for choosing the best transformation from the set of powertransformations to correct for this set of power transformations can be parameterized with the following definitionDefinition R, theny( )= y 1 y 1, 6= 0 y ln(y), = 0where y=e1nn i=1ln(yi)is the geometric mean of the geometric mean is a scaling factor, which enters the equation when finding themaximum like-lihood estimatorsfor and~ simultaneously.

7 It makes the residual sum of squares for different No formula exists which will produce the value of , which will produce the smallest residual sumof squares for a transformationy( ),SSRes( ). Therefore usuallySSRes( ) is found for a number ofdifferent values for , which are then plotted against . From the graph then the with the smallestSSRes( ) can be read. This procedure can be done iteratively, by first determining a rough estimateand then investigating the neighbourhood of the rough estimate using a finer has been decided on, sometimes after rounding so that the result is an easier to interpretmodel (use = instead of = ). The transformationy is used in the analysis, omittingthe scaling factor, which was only necessary to make the sum of squares confidence interval for Based on the theory about maximum likelihood a (1 ) 100% confidence interval for is givenby all withL( ) L( ) 12 2 ,1whereL( ) = 12nln[SSRes( )]is the log likelihood is equivalent to all withL( ) L( ) 12 2 ,1or all withSSRes( ) SSRes( )e 2 ,1/nThe bounds can be found graphically by including a horizontal line with the plot of againstSSRes( ).

8 Continue back to the blood pressure example and find the confidence interval for .9 L( ) to the graph the 95% confidence interval for falls between and , with particular 1 is not the confidence interval indicating that some transformation should be done tomake the residuals more normal, and the residual sum of squares smaller, and therefore result in abetter fit of the Model . A good choice for seems to be Model would be1Y2= 0+ 1(weight) + 2(age) + Comparing the fit of the two models give an adjustedR2for the original Model of , andfor the transformed Model of The already well fitting Model could be improved by the observed versus the fitted blood pressure for the two models illustrates the same fact,the Model based on the transformation fits better than the original Transforming a predictorIn some instances the residual plots or the initial scatter plots indicate a non linear relationshipbetween a regressor and the response, which can be linearized by transforming the the following Model (one regressor to illustrate, but also applies to MLRMs)Y= 0+ 1t(x) + , N(0, 2)Wheretis a functiont.

9 R7 and Tidwell (1962) proposed an analytic approach to choosing the (x) ={x , 6= 0ln(x), = 0then for the Model given above witht=t E(Y) = 0+ 1t (x) =f( 0, 1,t )Their iterative approach is based on the Taylor series forfabout an initial guess 0= 1, seeMPV( ).The process:1. Start with an initial guess 0= 1, and leti= Until you are convinced to have found a proper value for , repeatFind 1,ifrom fitting the modelY= 0,i+ 1,ix i+ Letw=x iln(x i)Find from fitting the modelY= 0,i+ 1,ix i+ w+ 11 The new guess for is i+1= 1,i+ 1 Leti < i+ R to find the Box Tidwell transformation for WEIGHT in the bloodpressure (car)attach( )boxTidwell(BP~WEIGHT,~AGE,verbose=TRUE) Generating the following output:iter = 1 powers = = 2 powers = = 3 powers = Statistic p-value MLE of = 3 The output give , so we could use transformations WEIGHT2or WEIGHT3.}

10 Both do notimprove the Model fit, which was expected since the Score Statistic does not indicate a again the per capita expenses on education and the mean per capitaincome in the different American states.> boxTidwell(expend~income, verbose=TRUE)iter = 1 powers = = 2 powers = = 3 powers = = 4 powers = Statistic p-value MLE of = 4 The Box Tidwell procedure suggest a polynomial of degree (lm(expend~poly(income,5)))Shows improved Model fit for the Model of expenditure being a polynomial of degree 5 in :Estimate Std. Error t value Pr(>|t|)(Intercept) < 2e-16 **poly(income, 5)1 **poly(income, 5)2 **poly(income, 5)3 *poly(income, 5)4 *poly(income, 5)5 **---Signif. codes: 0 ** ** * . 1 Residual standard error: on 44 degrees of freedomMultiple R-squared: ,Adjusted R-squared: : on 5 and 44 DF, p-value: Generalized and Weighted Least-SquaresIn order to deal with a violation of the homoscedasticity Assumption is to generalize Model , bypermitting, a less restrictive form for the covariance matrix of the General Multiple Linear Regression Model (GMLRM) is given by~Y=X~ + , N(~0, 2V)For this Model the (ordinary) least squares estimator is no longer appropriate.