Transcription of October 19 - CEGA
1 ValidityCaseStudiesExternalValidityErick Gong( )UCBerkeley&CEGAO ctob er19 ValidityCaseStudiesAgendaInternalValidit yExternalValidityCaseStudiesValidityCase StudiesWhatisvalidity?Twotyp esInternalValidityExternalValidityIntern alValidity:Accurateestimateofthecausale :Accurateestimateofthecausale eenoninternalvalidityValidityCaseStudies Whatisvalidity?Twotyp esInternalValidityExternalValidityIntern alValidity:Accurateestimateofthecausale :Accurateestimateofthecausale eenoninternalvalidityValidityCaseStudies Whydowecare?ExternalvalidityProgramExpan sionorScalabilityReplicationEconomicTheo ryExample:Progressa&CCTA nswer:Wecareb ecausewewantourresearch/ ndingstoleadtomoree ectivep :Toconvincep olicymakers,NGOs,governmentstoadopte ectivep :Progressa&CCTA nswer:Wecareb ecausewewantourresearch/ ndingstoleadtomoree ectivep :Toconvincep olicymakers,NGOs,governmentstoadopte ectivep :Progressa&CCTA nswer:Wecareb ecausewewantourresearch/ ndingstoleadtomoree ectivep :Toconvincep olicymakers,NGOs,governmentstoadopte ectivep d:Randomassignmentoftreatmentandcontrola rmExamples:Dewormingmedication,monetaryi ncentivesforHIVtests,scholarshipsforstud entsWhatarewemeasuring?
2 (ATE)AverageTreatmentE ectsRandomizedEvaluationshavetwotyp esofe ects2E ects:ITT(IntenttoTreat)=Peoplemadeeligib lefortreatment/interventionTOT(Treatment ontheTreated)=Peoplewhoactuallyto okthetreatment/interventionPolicyImplica tions?ValidityCaseStudiesInternalValidit yRegressionDiscontinuityComparetwogroups oneachsideofthethresholdExamples:E ectofwinningscholarship,classsizeE ect:LATE(Lo calAverageTreatmentE ect)PolicyImplication?ValidityCaseStudie sInternalValidityRegressionDiscontinuity Comparetwogroupsoneachsideofthethreshold Examples:E ectofwinningscholarship,classsizeE ect:LATE(Lo calAverageTreatmentE ect)PolicyImplication?ValidityCaseStudie sInternalValidityUsingmetho dswecanmeasureAverageTreatmentE ect(ATE):E ectoftreatmentonrandomlyselectedp ersonIntenttoTreatE ect(ITT):E ectofb eingmadeeligiblefortreatmentTreatmentonT reatedE ect(TOT):E ectoftakingtreatmentLo calAverageTreatmentE ect(LATE):E ectoftreatmentforp opulationnearthresholdAllourcausale :ScholarshipPrograminKenya(Busia&Teso)Sc holarshipsawardedtohighestscoring15%ofgi rlsOverallscholarshipprogramhasp ositivee ectontestscoresPolicyimplication:scholar shipsnationwide?
3 Wait!Howwouldscholarshipprograma ecttheseothergroups:1)Girlsinurbanscho ols2)Girlsinhighincomeareas(Nairobisubur bs)3)Girlswhohaddropp edoutofscho ol4)Firstgradersinsampleregion(notjust6t hgraders)5)Boysin6thgrade6)GirlsinUganda Heterogeneityintreatmente :Wantinterventiontohaveane ect-OR-noreasontomotivatep :ScholarshipPrograminKenya(Busia&Teso)Sc holarshipsawardedtohighestscoring15%ofgi rlsOverallscholarshipprogramhasp ositivee ectontestscoresPolicyimplication:scholar shipsnationwide?Wait!Howwouldscholarship programa ecttheseothergroups:1)Girlsinurbanscho ols2)Girlsinhighincomeareas(Nairobisubur bs)3)Girlswhohaddropp edoutofscho ol4)Firstgradersinsampleregion(notjust6t hgraders)5)Boysin6thgrade6)GirlsinUganda Heterogeneityintreatmente :Wantinterventiontohaveane ect-OR-noreasontomotivatep :ScholarshipPrograminKenya(Busia&Teso)Sc holarshipsawardedtohighestscoring15%ofgi rlsOverallscholarshipprogramhasp ositivee ectontestscoresPolicyimplication:scholar shipsnationwide?
4 Wait!Howwouldscholarshipprograma ecttheseothergroups:1)Girlsinurbanscho ols2)Girlsinhighincomeareas(Nairobisubur bs)3)Girlswhohaddropp edoutofscho ol4)Firstgradersinsampleregion(notjust6t hgraders)5)Boysin6thgrade6)GirlsinUganda Heterogeneityintreatmente :Wantinterventiontohaveane ect-OR-noreasontomotivatep :ScholarshipPrograminKenya(Busia&Teso)Sc holarshipsawardedtohighestscoring15%ofgi rlsOverallscholarshipprogramhasp ositivee ectontestscoresPolicyimplication:scholar shipsnationwide?Wait!Howwouldscholarship programa ecttheseothergroups:1)Girlsinurbanscho ols2)Girlsinhighincomeareas(Nairobisubur bs)3)Girlswhohaddropp edoutofscho ol4)Firstgradersinsampleregion(notjust6t hgraders)5)Boysin6thgrade6)GirlsinUganda Heterogeneityintreatmente :Wantinterventiontohaveane ect-OR-noreasontomotivatep opulationofinterest?
5 OrWhodowecareab out?Example:CashrewardsinthisclassExampl e:ScholarshipPrograminKenya1 Allprimaryscho olstudents2 Female/MaleStudents3 Urban/RuralAreas4 Studentsfromdisadvantagedfamiliesorbackg roundsHelpssometimestothinkab o oroutofp opulationtostudyValidityCaseStudiesPopul ationConceptsWhatisthep opulationofinterest?orWhodowecareab out?Example:CashrewardsinthisclassExampl e:ScholarshipPrograminKenya1 Allprimaryscho olstudents2 Female/MaleStudents3 Urban/RuralAreas4 Studentsfromdisadvantagedfamiliesorbackg roundsHelpssometimestothinkab o oroutofp opulationtostudyValidityCaseStudiesPopul ationConceptsWhatisthep opulationofinterest?orWhodowecareab out?Example:CashrewardsinthisclassExampl e:ScholarshipPrograminKenya1 Allprimaryscho olstudents2 Female/MaleStudents3 Urban/RuralAreas4 Studentsfromdisadvantagedfamiliesorbackg roundsHelpssometimestothinkab o oroutofp opulationtostudyValidityCaseStudiesSampl esInanevaluation/studyweusuallyanalyzesa mplesDe nep opulationofinterest&thentakesamplePopula tion>Sample>TreatmentGroupTreatmentGroup ->SampleSample->PopulationConcern:Howwas sampleselected?
6 Example:ScholarshipprogramforyounggirlsP opulationofInterest:Scho olgirlsfromp o orfamiliesSelectionofSample:Flyersandrad ioadsaskingstudentstosignupforprogramPro blem:SelectionProblem-whenp eopleinthesamplearedi erentfromourp opulationofinterestinsomeunobservedway( )ValidityCaseStudiesSamplesInanevaluatio n/studyweusuallyanalyzesamplesDe nep opulationofinterest&thentakesamplePopula tion>Sample>TreatmentGroupTreatmentGroup ->SampleSample->PopulationConcern:Howwas sampleselected?Example:Scholarshipprogra mforyounggirlsPopulationofInterest:Scho olgirlsfromp o orfamiliesSelectionofSample:Flyersandrad ioadsaskingstudentstosignupforprogramPro blem:SelectionProblem-whenp eopleinthesamplearedi erentfromourp opulationofinterestinsomeunobservedway( )ValidityCaseStudiesSamplesInanevaluatio n/studyweusuallyanalyzesamplesDe nep opulationofinterest&thentakesamplePopula tion>Sample>TreatmentGroupTreatmentGroup ->SampleSample->PopulationConcern:Howwas sampleselected?
7 Example:ScholarshipprogramforyounggirlsP opulationofInterest:Scho olgirlsfromp o orfamiliesSelectionofSample:Flyersandrad ioadsaskingstudentstosignupforprogramPro blem:SelectionProblem-whenp eopleinthesamplearedi erentfromourp opulationofinterestinsomeunobservedway( )ValidityCaseStudiesExternalValidityTwoC oncernsab outp opulation/sampleSelectionProblem:Isthewa ythesamplewaschosenrepresentativeofthep opulationofinterest?Heterogeneity:Isthet reatmente ectgoingtob evary?Example:CCTforsafesexualb ehaviortoreduceHIVtransmissionPopulation ofInterest:SexuallyActiveYoungPeopleSele ction:Scho ols?Homes?Bars?Heterogeneityintreatment: Female/MaleValidityCaseStudiesExternalVa lidityTwoConcernsab outp opulation/sampleSelectionProblem:Isthewa ythesamplewaschosenrepresentativeofthep opulationofinterest?Heterogeneity:Isthet reatmente ectgoingtob evary?
8 Example:CCTforsafesexualb ehaviortoreduceHIVtransmissionPopulation ofInterest:SexuallyActiveYoungPeopleSele ction:Scho ols?Homes?Bars?Heterogeneityintreatment: Female/MaleValidityCaseStudiesHowdo esitwork?Inrandomizedevaluations,wehavee videnceIFinterventionhasane ect,butwedon' (vaccinations)ValidityCaseStudiesHowdo esitwork?ConditionalCashTransfers(CCT)Tr eatment:Conditionedcashpaymentsonscho olenrollmentHowdo esitwork?Isitb ecausestudentsadheringtotheconditionalit y?OR-Isthereanincomee ect?Wouldunconditionalcashpaymentsdothes ame?Mechanism/Channel:Howistheinterventi onactuallyworking?Whyisthisimp ortant?Wecanimproveprogramifweknowsp eci :CCTscanb everyexp esitwork?ConditionalCashTransfers(CCT)Tr eatment:Conditionedcashpaymentsonscho olenrollmentHowdo esitwork?Isitb ecausestudentsadheringtotheconditionalit y?
9 OR-Isthereanincomee ect?Wouldunconditionalcashpaymentsdothes ame?Mechanism/Channel:Howistheinterventi onactuallyworking?Whyisthisimp ortant?Wecanimproveprogramifweknowsp eci :CCTscanb everyexp esitwork?ConditionalCashTransfers(CCT)Tr eatment:Conditionedcashpaymentsonscho olenrollmentHowdo esitwork?Isitb ecausestudentsadheringtotheconditionalit y?OR-Isthereanincomee ect?Wouldunconditionalcashpaymentsdothes ame?Mechanism/Channel:Howistheinterventi onactuallyworking?Whyisthisimp ortant?Wecanimproveprogramifweknowsp eci :CCTscanb everyexp otheticalExample:Vo cationaltrainingprogramResearchDesign:Sa mpleinruralKenyain10villagesTreatment:Tr ainingtob ecomeaskilledlab orer( er,mechanic,electrician)E ect:Peopleinprogramearnmoremoney(+40%gai ninwages)Whatifvo cationaltrainingrolledoutnationwide?
10 Asmorep eopleb ecameskilledlab orers,wageswillgodown,leadingtodecreasei nwagegainsGeneralEquilibriumE ectsValidityCaseStudiesGeneralEquilibriu mMostevaluationsassumepartialequilibrium Hyp otheticalExample:Vo cationaltrainingprogramResearchDesign:Sa mpleinruralKenyain10villagesTreatment:Tr ainingtob ecomeaskilledlab orer( er,mechanic,electrician)E ect:Peopleinprogramearnmoremoney(+40%gai ninwages)Whatifvo cationaltrainingrolledoutnationwide?Asmo rep eopleb ecameskilledlab orers,wageswillgodown,leadingtodecreasei nwagegainsGeneralEquilibriumE ectsValidityCaseStudiesGeneralEquilibriu mMostevaluationsassumepartialequilibrium Hyp otheticalExample:Vo cationaltrainingprogramResearchDesign:Sa mpleinruralKenyain10villagesTreatment:Tr ainingtob ecomeaskilledlab orer( er,mechanic,electrician)E ect:Peopleinprogramearnmoremoney(+40%gai ninwages)Whatifvo cationaltrainingrolledoutnationwide?
