Lecture 12 Linear Regression: Test and Confidence Intervals

1 Lecture 12 Linear Regression: Test and Confidence IntervalsFall 2013 Prof. Yao Xie, H. Milton Stewart School of Industrial Systems & Engineering Georgia Tech 1 Outline Properties of and as point estimators Hypothesis test on slope and intercept Confidence Intervals of slope and intercept Real example: house prices and taxes 2 1 0 Regression analysis Step 1: graphical display of data scatter plot: sales vs. advertisement cost !!!!!!! calculate correlation 3 Step 2: find the relationship or association between Sales and Advertisement Cost Regression 4 Simple Linear regressionBased on the scatter diagram, it is probably reasonable to assume that the mean of the random variable Y is related to X by the following simple Linear regression model: 5iiiXY ++=10ni,,2,1!

2 =Intercept Slope i Random error Response Regressor or Predictor i 0, 2()where the slope and intercept of the line are called regression coefficients. The case of simple Linear regression considers a single regressor or predictor x and a dependent or response variable Y. Regression coefficients 611-2 SIMPLE Linear REGRESSION407 Simplifying these two equations yields(11-6)Equations 11-6 are called the least squares normal solution to the normalequations results in the least squares estimators and ! 1.! 0! 0 ani"1 xi#! 1 ani"1 x i2"ani"1 yi xi n! 0#! 1 ani"1 xi"ani"1 yiThe least squares estimatesof the intercept and slope in the simple Linear regressionmodel are(11-7)(11-8)where y"11$n2gni"1 yi and x"11$n2gni"1 xi.! 1"ani"1yi xi%aani"1yib aani"1xibnani"1x 2i%aani"1xib2n! 0"y%! 1xLeast SquaresEstimatesThe fittedor estimated regression lineis therefore(11-9)Note that each pair of observations satisfies the relationshipwhere ei"yi%is called the residual describes the error in the fit of themodel to the ith observation yi.

3 Later in this chapter we will use the residuals to provideinformation about the adequacy of the fitted , it is occasionally convenient to give special symbols to the numerator anddenominator of Equation 11-8. Given data (x1, y1), (x2, y2), p, (xn, yn), let(11-10)and(11-11)Sx y"ani"11yi%y21xi%x2"ani"1xiyi%aani"1xib aani"1 yibnSx x"ani"1 1xi%x22"ani"1x 2i%aani"1xib2ny iyi"! 0#! 1xi#ei, i"1, 2,p, ny "! 0#! 1/15/10 4:53 PM Page 40711-2 SIMPLE Linear REGRESSION407 Simplifying these two equations yields(11-6)Equations 11-6 are called the least squares normal solution to the normalequations results in the least squares estimators and ! 1.! 0! 0 ani"1 xi#! 1 ani"1 x i2"ani"1 yi xi n! 0#! 1 ani"1 xi"ani"1 yiThe least squares estimatesof the intercept and slope in the simple Linear regressionmodel are(11-7)(11-8)where y"11$n2gni"1 yi and x"11$n2gni"1 xi.! 1"ani"1yi xi%aani"1yib aani"1xibnani"1x 2i%aani"1xib2n!

4 0"y%! 1xLeast SquaresEstimatesThe fittedor estimated regression lineis therefore(11-9)Note that each pair of observations satisfies the relationshipwhere ei"yi%is called the residual describes the error in the fit of themodel to the ith observation yi. Later in this chapter we will use the residuals to provideinformation about the adequacy of the fitted , it is occasionally convenient to give special symbols to the numerator anddenominator of Equation 11-8. Given data (x1, y1), (x2, y2), p, (xn, yn), let(11-10)and(11-11)Sx y"ani"11yi%y21xi%x2"ani"1xiyi%aani"1xib aani"1 yibnSx x"ani"1 1xi%x22"ani"1x 2i%aani"1xib2ny iyi"! 0#! 1xi#ei, i"1, 2,p, ny "! 0#! 1/15/10 4:53 PM Page 407xy10 =xxxySS=1 iixy10 +=Fitted (estimated) regression model Caveat: regression relationship are valid only for values of the regressor variable within the range the original data.

5 Be careful with of variance Using the fitted model, we can estimate value of the response variable for given predictor !! Residuals: Our model: Yi = 0 + 1Xi + i, i =1,..,n, Var( i) = 2 Unbiased estimator (MSE: Mean Square Error) ! 7iixy10 +=iiiyyr =212 = =nrMSEnii 2=Punchline the coefficients !!and both calculated from data, and they are subject to error. if the true model is , are point estimators for the true coefficients ! we can talk about the ``accuracy of 8 1 and 0y= 1x+ 0 1 and 0 1 and 0 Assessing Linear regression model Test hypothesis about true slope and intercept !

6 ! Construct Confidence Intervals !! Assume the errors are normally distributed 9 i~ 0, 2() 1=?, 0=? 1 1 a, 1+a 0 0 b, 0+b with probability 1 10 Properties of Regression Estimatorsslope parameter 1 intercept parameter 0unbiased estimator unbiased estimator414 CHAPTER 11 SIMPLE Linear REGRESSION AND CORRELATIONx ! , , , , , , , , , , , , , , , , , , , , , , , , , , ! , , , , , , , , , , , , , , , , , , , , , , , , , , (a) Find the least squares estimates of the slope and the inter-cept in the simple Linear regression model. Find an esti-mate of .(b) Estimate the mean length of dugongs at age 11.(c) Obtain the fitted values that correspond to each ob-served value yi. Plot versus yi, and comment on whatthis plot would look like if the Linear relationship betweenlength and age were perfectly deterministic (no error).

7 Does this plot indicate that age is a reasonable choice ofregressor variable in this model? the regression model developed in Ex-ercise 11-2.(a) Suppose that temperature is measured in "C rather than " the new regression model.(b) What change in expected pavement deflection is associ-ated with a 1"C change in surface temperature? the regression model developed in Exercise11-6. Suppose that engine displacement is measured in cubiccentimeters instead of cubic iy i#2(a) Write the new regression model.(b) What change in gasoline mileage is associated with a1 cm3change is engine displacement? that in a simple Linear regression modelthe point () lies exactly on the least squares regression , y() points. Use the two plots to intuitivelyexplain how the two models, Y!$0%$1x%&and, are related.(b) Find the least squares estimates of and in the model.

8 How do they relate to the leastsquares estimates and ? we wish to fit a regression model for whichthe true regression line passes through the point (0, 0). The ap-propriate model is Y!$x%&. Assume that we have npairsof data (x1, y1), (x2, y2),p, (xn, yn). (a) Find the least squares estimate of $.(b) Fit the model Y!$x%&to the chloride concentration-roadway area data in Exercise 11-10. Plot the fittedmodel on a scatter diagram of the data and comment onthe appropriateness of the model.$ 1$ 0Y!$*0%$*1z%&$*1$*0Y!$*0%$*1z%&zi!xi'x, yi11-3 PROPERTIES OF THE LEAST SQUARES ESTIMATORSThe statistical properties of the least squares estimators and may be easily that we have assumed that the error term &in the model Y!$0%$1x%&is a randomvariable with mean zero and variance #2. Since the values of xare fixed, Yis a random vari-able with mean !$0%$1xand variance #2. Therefore, the values of and dependon the observed y s ; t h u s , t h e l e a s t s q u a r e s e s t i m a t o r s o f t h e r e g r e s s i o n c o e f fi c i e n t s m a y b eviewed as random variables.

9 We will investigate the bias and variance properties of the leastsquares estimators and .Consider first . Because is a Linear combination of the observations Yi, we can useproperties of expectation to show that the expected value of is(11-15)Thus, is an unbiased estimatorof the true slope $ consider the variance of . Since we have assumed that V(&i)!#2, it follows thatV(Yi)!#2. Because is a Linear combination of the observations Yi, the results inSection 5-5 can be applied to show that(11-16)V1$ 12!#2 Sxx$ 1$ 1$ 1E1$ 12!$1$ 1$ 1$ 1$ 1$ 0$ 1$ 0(Y0x$ 1$ the simple Linear regression model Y!$0%$1x%&. Suppose that the analyst wants to use z!x'asthe regressor variable.(a) Using the data in Exercise 11-11, construct one scatterplot of the () points and then another of thexi, 1/14/10 8:02 PM Page 41411-4 HYPOTHESIS TESTS IN SIMPLE Linear REGRESSION415 For the intercept, we can show in a similar manner that(11-17)Thus, is an unbiased estimator of the intercept !)

10 0. The covariance of the random vari-ables and is not zero. It can be shown (see Exercise 11-98) that cov()"#$ estimate of $2could be used in Equations 11-16 and 11-17 to provide estimates ofthe variance of the slope and the intercept. We call the square roots of the resulting varianceestimators the estimated standard errorsof the slope and intercept, ! 0, ! 1! 1! 0! 0E1! 02"!0 and V1! 02"$2 c1n&x2 SxxdIn simple Linear regression the estimated standard error of the slopeand the estimated standard error of the interceptare respectively, where is computed from Equation 11-13.$ 2se1! 12"B$ 2 Sxx and se1! 02"B$ 2 c1n&x2 SxxdEstimatedStandard ErrorsThe Minitab computer output in Table 11-2 reports the estimated standard errors of the slopeand intercept under the column heading SEcoeff. 11-4 HYPOTHESIS TESTS IN SIMPLE Linear REGRESSIONAn important part of assessing the adequacy of a Linear regression model is testing statisticalhypotheses about the model parameters and constructing certain Confidence Intervals .