Simple Linear Regression Least Squares Estimates ... - Amherst

1 Simple Linear RegressionLeast Squares Estimates of 0and 1 Simple Linear Regression involves the model^Y= YjX= 0+ 1X:This document derives the Least Squares Estimates of 0and 1. It is simply for your owninformation. You will not be held responsible for this Least Squares Estimates of 0and 1are:^ 1= ni=1(Xi X)(Yi Y) ni=1(Xi X)2^ 0= Y ^ 1 XThe classic derivation of the Least Squares Estimates uses calculus to nd the 0and 1parameter Estimates that minimize the error sum of Squares :SSE= ni=1(Yi ^Yi)2. Thisderivation uses no calculus , only some lengthy algebra . It uses a very clever method thatmay be found in:Im, Eric Iksoon,A Note On Derivation of the Least Squares Estimator, Working Paper SeriesNo.

2 96-11, University of Hawai'i at Manoa Department of Economics, DerivationThe Least Squares Estimates are Estimates ^ 0and^ 1that minimize the error sum of squaresSSE=n i=1(Yi ^Yi)2:1We can algebraically manipulate things to getSSE=n i=1(Yi ^Yi)2=n i=1(Yi 0 1Xi)2=n i=1[(Yi+ Y Y) 0 1(Xi+ X X)]2=n i=1[( Y 0 1 X) +Yi Y 1Xi+ 1 X]2=n i=1[( Y 0 1 X) ( 1Xi 1 X Yi+ Y)]2=n i=1[( Y 0 1 X)2]+n i=1[( 1Xi 1 X Yi+ Y)]2=n( Y 0 1 X)2+n i=1[( 1Xi 1 X Yi+ Y)]2=n( Y 0 1 X)2+n i=1[ 1(Xi X) (Yi Y)]2=n( Y 0 1 X)2+n i=1[ 21(Xi X)2 2 1(Xi X)(Yi Y) + (Yi Y)2]=n( Y 0 1 X)2+ 21n i=1(Xi X)2 2 1n i=1(Xi X)(Yi Y) +n i=1(Yi Y)2=n( Y 0 1 X)2+( 21 2 1 ni=1(Xi X)(Yi Y) ni=1(Xi X)2+ ni=1(Yi Y)2 ni=1(Xi X)2)n i=1(Xi X)2=n( Y 0 1 X)2+ 21 2 1 ni=1(Xi X)(Yi Y) ni=1(Xi X)2+[ ni=1(Xi X)(Yi Y) ni=1(Xi X)2]2+ ni=1(Yi Y)2 ni=1(Xi X)

3 2 [ ni=1(Xi X)(Yi Y) ni=1(Xi X)2]2 n i=1(Xi X)2=n( Y 0 1 X)2+( 1 ni=1(Xi X)(Yi Y) ni=1(Xi X)2)2+n i=1(Yi Y)2 1 [ ni=1(Xi X)(Yi Y) ni=1(Xi X)2 ni=1(Yi Y)2]2 2We're still trying to minimize the SSE, and we've split the SSE into the sum of three that the rst two terms involve the parameters 0and 1. The rst two terms are alsosquared terms, so they can never be less than zero. The third term is only a function of thedata and not the parameter. So, we know thatSSE=n( Y 0 1 X)2+( 1 ni=1(Xi X)(Yi Y) ni=1(Xi X)2)2+n i=1(Yi Y)2 1 [ ni=1(Xi X)(Yi Y) ni=1(Xi X)2 ni=1(Yi Y)2]2 n i=1(Yi Y)2 1 [ ni=1(Xi X)(Yi Y) ni=1(Xi X)2 ni=1(Yi Y)2]2 This is the minimum possible value for the SSE.

4 We actually achieve this minimum valuewhen the rst two terms of the equation above are zero. Setting each of these two termsequal to zero gives us two equations in two unknowns, so we can solve for 0and =n( Y 0 1 X)20 =( 1 ni=1(Xi X)(Yi Y) ni=1(Xi X)2)2 From the rst equation we get0 =n( Y 0 1 X)2)0 = Y 0 1 X) 0= Y 1 XFrom the second equation we get0 =( 1 ni=1(Xi X)(Yi Y) ni=1(Xi X)2)2)0 = 1 ni=1(Xi X)(Yi Y) ni=1(Xi X)2) 1= ni=1(Xi X)(Yi Y) ni=1(Xi X)2As these areestimates, we put hats on them. We are done! We've now shown that^ 1= ni=1(Xi X)(Yi Y) ni=1(Xi X)2^ 0= Y ^ 1 X:3