Lecture 16 - Correlation and Regression

1 Lecture 16 - Correlation and RegressionStatistics 102 Colin RundelApril 1, 2013 Modeling numerical variablesModeling numerical variablesSo far we have worked with single numerical and categorical variables,and explored relationships between numerical and categorical, andtwo categorical week we will learn to quantify the relationship between twonumerical variables, as well as modeling numerical response variablesusing a numerical or categorical explanatory week we will learn to model numerical variables using manyexplanatory variables at 102 (Colin Rundel)Lec 16 April 1, 20132 / 34 Modeling numerical variablesPoverty vs. HS graduate rateThescatterplotbelow shows the relationship between HS graduate rate inall 50 US states and DC and the % of residents who live below the povertyline(income below$23,050 for a family of 4 in 2012).

2 Llllllllllllllllllllllllllllllllllllllll lllllllllll808590681012141618% HS grad% in povertyResponse?Explanatory?Relationship ?Statistics 102 (Colin Rundel)Lec 16 April 1, 20133 / 34 CorrelationQuantifying the relationshipCorrelationdescribes the strength of thelinearassociation betweentwo takes values between -1 (perfect negative) and +1 (perfectpositive).A value of 0 indicates no linear use to indicate the population Correlation coefficient, andRorrto indicate the sample Correlation 102 (Colin Rundel)Lec 16 April 1, 20134 / 34 CorrelationCorrelation ExamplesFrom 102 (Colin Rundel)Lec 16 April 1, 20135 / 34 CorrelationCovariance and CorrelationCovarianceWe have previously discussed the variance as a measure of uncertainty of arandom variable.

3 Var(X) = 2=1nn i=1(xi X)2In order to define Correlation we first need to define covariance, which is ageneralization of variance to two random variablesCov(X,Y) =1nn i=1(xi X)(yi Y)Covariance is not a measure of uncertainly but rather a measure of thedegree to whichXandYtend to be large (or small) at the same time orthe degree to which one tends to be large while the other is 102 (Colin Rundel)Lec 16 April 1, 20136 / 34 CorrelationCovariance and CorrelationCovariance, magnitude of the covariance is not very informative since it is affectedby the magnitude of bothXandY. However, the sign of the covariancetells us something useful about the relationship the following conditions.

4 Xi> Xandyi> Ythen (xi X)(yi Y) will be < Xandyi< Ythen (xi X)(yi Y) will be > Xandyi< Ythen (xi X)(yi Y) will be < Xandyi> Ythen (xi X)(yi Y) will be 102 (Colin Rundel)Lec 16 April 1, 20137 / 34 CorrelationCovariance and CorrelationProperties of CovarianceCov(X,X) =Var(X)Cov(X,Y) =Cov(Y,X)Cov(X,Y) = 0 ifXandYare independentCov(X,c) = 0 Cov(aX,bY) =ab Cov(X,Y)Cov(X+a,Y+b) =Cov(X,Y)Cov(X,Y+Z) =Cov(X,Y) +Cov(X,Z)Statistics 102 (Colin Rundel)Lec 16 April 1, 20138 / 34 CorrelationCovariance and CorrelationCorrelationSinceCov(X,Y) depends on the magnitude ofXandYwe would preferto have a measure of association that is not affected by changes in thescales of the most common measure oflinearassociation is Correlation which isdefined as (X,Y) =Cov(X,Y) X Y 1< (X,Y)<1 Where the magnitude of the Correlation measures the strength of thelinearassociation and the sign determines if it is a positive or 102 (Colin Rundel)Lec 16 April 1, 20139 / 34 CorrelationCovariance and CorrelationCorrelation and IndependenceGiven random variablesXandYXandYare independent= Cov(X,Y) = (X,Y) = 0 Cov(X,Y) = (X,Y) = 06= XandYare independentStatistics 102 (Colin Rundel)

5 Lec 16 April 1, 201310 / 34 CorrelationCovariance and CorrelationGuessing the correlationWhich of the following is the best guess for the Correlation between % inpoverty and % HS grad?lllllllllllllllllllllllllllllllllll llllllllllllllll808590681012141618% HS grad% in poverty(a) (b) (c) (d) (e) 102 (Colin Rundel)Lec 16 April 1, 201311 / 34 CorrelationCovariance and CorrelationGuessing the correlationWhich of the following is the best guess for the Correlation between % inpoverty and % single mother household?llllllllllllllllllllllllllllll lllllllllllllllllllll8101214161868101214 1618% female householder, no husband present% in poverty(a) (b) (c) (d) (e) 102 (Colin Rundel)Lec 16 April 1, 201312 / 34 CorrelationCovariance and CorrelationAssessing the correlationWhich of the following is has the strongest Correlation , correlationcoefficient closest to +1 or -1?

6 Llllllllllllllllllllllllllllllllllllllll llllllllllllllllllllllllllllllllllllllll lllllllllllllllllllll(a)llllllllllllllll llllllllllllllllllllllllllllllllllllllll llllllllllllllllllllllllllllllllllllllll lllll(b)llllllllllllllllllllllllllllllll llllllllllllllllllllllllllllllllllllllll lllllllllllllllllllllllllllll(c)llllllll llllllllllllllllllllllllllllllllllllllll llllllllllllllllllllllllllllllllllllllll lllllllllllll(d)Statistics 102 (Colin Rundel)Lec 16 April 1, 201313 / 34 Best fit line - least squares regressionEyeballing the lineEyeballing the lineWhich of the following appears to be the line that best fits the linearrelationship between % in poverty and % HS grad?lllllllllllllllllllllllllllllllllll llllllllllllllll808590681012141618% HS grad% in poverty(a)(b)(c)(d)Statistics 102 (Colin Rundel)Lec 16 April 1, 201314 / 34 Best fit line - least squares regressionResidualsQuantifying best fitlllllllllllllllllllllllllllllllllllll llllllllllllll808590681012141618% HS grad% in povertyStatistics 102 (Colin Rundel)

7 Lec 16 April 1, 201315 / 34 Best fit line - least squares regressionResidualsResidualsResidualResi dual is the difference between the observed and yillllllllllllllllllllllllllllllllllllll lllllllllllll808590681012141618% HS grad% in ^y ^DCRI% living in poverty inDC is more living in poverty in RIis less 102 (Colin Rundel)Lec 16 April 1, 201316 / 34 Best fit line - least squares regressionResidualsA measure for the best lineWe want a line that has small residuals:1 Option 1: Minimize the sum of magnitudes (absolute values) ofresiduals|e1|+|e2|+ +|en|2 Option 2: Minimize the sum of squared residuals least squarese21+e22+ +e2nWhy least squares?1 Most commonly used2 Easier to compute by hand and using software3In many applications, a residual twice as large as another is more thantwice as badStatistics 102 (Colin Rundel)Lec 16 April 1, 201317 / 34 Best fit line - least squares regressionResidualsThe least squares line y= 0+ 1x predicted y interceptAAAU slopeHHHHH jexplanatory variableNotation:Intercept:Parameter: 0 Point estimate:b0 Slope:Parameter: 1 Point estimate.

8 B1 Statistics 102 (Colin Rundel)Lec 16 April 1, 201318 / 34 Best fit line - least squares regressionThe least squares HS grad% in poverty% HS grad% in poverty(x)(y)mean x= y= 102 (Colin Rundel)Lec 16 April 1, 201319 / 34 Best fit line - least squares regressionThe least squares lineSlopeSlopeThe slope of the Regression can be calculated asb1=sysxRIn = each % point increase in HS graduate rate, we wouldexpectthe %living in poverty to decreaseon averageby 102 (Colin Rundel)Lec 16 April 1, 201320 / 34 Best fit line - least squares regressionThe least squares lineInterceptInterceptThe intercept is where the Regression line intersects they-axis.

9 The cal-culation of the intercept uses the fact the a Regression line always passesthrough ( x, y).b0= y b1 xlllllllllllllllllllllllllllllllllllllll llllllllllll020406080100010203040506070% HS grad% in povertyinterceptb0= ( ) = 102 (Colin Rundel)Lec 16 April 1, 201321 / 34 Best fit line - least squares regressionThe least squares lineInterpreting InterceptsWhich of the following is the correct interpretation of the intercept?(a) For each % point increase in HS graduate rate, % living in poverty isexpected to increase on average by (b) For each % point decrease in HS graduate rate, % living in poverty isexpected to increase on average by (c) Having no HS graduates leads to of residents living below thepoverty line.

10 (d) States with no HS graduates are expected on average to have residents living below the poverty line.(e) In states with no HS graduates % living in poverty is expected toincrease on average by 102 (Colin Rundel)Lec 16 April 1, 201322 / 34 Best fit line - least squares regressionThe least squares lineRegression line [%in poverty] = [%HS grad]lllllllllllllllllllllllllllllllllll llllllllllllllll808590681012141618% HS grad% in povertyStatistics 102 (Colin Rundel)Lec 16 April 1, 201323 / 34 Best fit line - least squares regressionThe least squares lineInterpretation of slope and interceptIntercept:Whenx= 0,yis expected to equalthe :For eachunitincrease inx,yis expected toincrease/decreaseon average bythe 102 (Colin Rundel)Lec 16 April 1, 201324 / 34 Best fit line - least squares regressionPrediction & extrapolationPredictionUsing the linear model to predict the value of the response variablefor a given value of the explanatory variable is calledprediction,simply by plugging in the value ofxin the linear model will be some uncertainty associated with the predicted value -we ll talk about this next HS grad% in povertyStatistics 102 (Colin Rundel)