Assessing Studies Based on Multiple Regression Chapter 7

1 Assessing Studies Based on Multiple RegressionChapter 7 Michael AshCPPAA ssessing Regression Studies notes Last time Multiple Regression Alternative specifications Sign, Size and, Significance Time for some Stata questions Today Fast recap of Chapter 6 Assessing Studies Based on Multiple RegressionAssessing Regression Studies s in Chapter 6?Non-linear Regression functions Curved or bent Regression functions (curved relationship betweenXandY Does the effect of STR tail off at some point? Interactions between independent variables Is the effect of distance from college different for men andwomen (already done in the homework)? Does STR matter more for classrooms with many Englishlearners?Techniques are fairly easy to implement but beyond the scope ofthiscourse. See me for more detail when you do workshop and Regression Studies and External validity Focus: causal effect of independent variable on dependentvariable, basis of policy.)

2 Assessing threats to validity for improvement and critique internal validity : statistical inferences about causal effects arevalid for the population being studied. What would happen to California elementary school testscores if every district reduced the STR by two students? External validity : statistical inferences about causal effects canbe generalized from the population and setting being studied toother populations and settings. What would happen to CaliforniaHStest scores if everydistrict reduced the STR by two students? What would happen toIowaelementary school test scores ifevery district reduced the STR by two students? What would happen toJapaneseelementary school testscores if every district reduced the STR by two students? Assessing Regression Studies validity Want to be unbiased and consistent estimator of Hypothesis tests should have the intended significance level andCI s should have the desired confidence level.

3 Depends on SE s being accurately estimated Example of a threat to internal validity : omitted variable bias Solution: include omitted variablesAssessing Regression Studies ValidityExample Laboratory animal toxicity Studies to study, predict, and regulatehuman exposure and health effectsWhat can go wrong? Differences in Populations between population studied andthepopulation of interest (geography, time [ , RAND HIE]) Differences in Settings (legal, institutional, and physicalenvironmentsTest scores and STR ES scores in the more likely to be an externally validapplication than HS scores in or ES scores in Japan. Except: High-stakes testing (for students, teachers, etc.) Assessing Regression Studies External ValidityRequires Specific knowledge of the population and setting studied andthepopulation and setting of interest.)

4 Or Studies on several populations and settings that generate Regression Studies to internal validity , and Solutions Omitted Variable Bias Misspecification of the Functional Form Imprecise Measurement of the Independent Variables( Errors-in-Variables ) Sample Selection Simultaneous CausalityEach of these is an instance of correlation between the regressor (X)and the error term (u), which violates the first least squares Regression Studies Variable Bias Already discussed the problem at length Solution when Omitted Variable is Observed Identify the key coefficient(s) of interest Consider which control variables to include Based on expertjudgment Estimate alternative specifications, keep additional variablesthat are themselves statistically significant; or affect the sign, size, or significance of the coefficients onkey variables.

5 Full disclosure of the specifications tested Solution when Omitted Variable is Unobserved Compare a unit to itself (over time or within super-unit) Experimental or quasi-experimental designAssessing Regression Studies of the Functional Form Curved and changing relationships ( Chapter 6) beyond thescope of this course. Use scatterplots to identify non-linearrelationships. Discreteoutcomevariables ( Chapter 9) Assessing Regression Studies Only a problem with imprecise measurement of theindependentvariables The imprecise measurement is not biased up or down, simplyXis true signal butwis noise (imprecise measurement ofX) isadded: X=X+w. Leads to Attenuation Bias. 1p 2X 2X+ 2w 1 1is always an underestimate of 1(estimated effect is closer tozero, smaller than true effect).

6 Imprecise measurement of the dependent variable is not aproblem: imprecise measurement is simply one of the otherfactors uthat Regression Studies , cont d Solutions Multiple independent measures ofX(even if all areimprecise) Adjust estimates for attenuation bias Based on estimatedsize of the imprecisionAssessing Regression Studies Selection Availability of the data is influenced by a selection processthat isrelated to the value of the dependent variable. In all these cases, other factors umay be correlated withX. 1936 Presidential poll limited to car and telephone owners People who apply for job-training programs likely havebarriers to employment. People with jobs may have high earning potential (controllingfor their characteristics). InnerChange program evaluation, attrition in general Solutions: various and complex; create an explicit model of theselection Regression Studies Regression Studies , or Reverse, Causality Government may hire additional teachers in low-performingdistricts (or now government may penalize low-performingdistricts).

7 Yi= 0+ 1Xi+uiXi= 0+ 1Yi+vi Induces correlation betweenuandX. Consider case whereuiis low, henceYiis low. IfYiis low, then (assuming 1positive)Xiis low. But this means thatuiandXiare low together correlated! Solutions: randomized controlled experiments ( Chapter 11) andeconometric quasi-experimental methods (beyond the scope ofthis course). Assessing Regression Studies Note that every problem discussed so far involved a violation ofOLS assumption #1: the conditional distribution ofuigivenXihasmean zero. General language for discussing problems with causal models(within and beyond econometrics) Assessing Regression Studies in OLS Standard Errors The OLS estimates of remain consistent and unbiased; but Inference (CI s, hypothesis tests) will be wrong because the SE sare wrong.

8 Heteroskedasticity: use robust standard errors Correlation of the error term across observationsYi= 0+ 1Xi+uiYj= 0+ 1Xj+ujuiandujshould not be related. Repeated sampling of the same unit over time, serialcorrelation Sampling within the same household or geographical unit Less (fewer observations) than meets the eyeAssessing Regression Studies