SIEF Kenya Impact Evaluation Workshop Di erence-in-Di ...

1 SIEF Kenya Impact Evaluation WorkshopDifference-in-Difference EstimationMay 6, 2015 Instructor: Pamela JakielaUniversity of Maryland, College Park, USAO verview Review: false counterfactuals Difference-in-differences: the intuition Difference-in-differences: the Stata code Checking the common trends assumption A practical exampleSIEF IE Workshop : Difference-in-Difference EstimationSlide 2 Motivation: False CounterfactualsWhat Is an Impact Evaluation ? An Impact Evaluation assesses changes in the well-being of individuals,households, communities or firms that can be attributed to a particularproject, program or policy. The central Impact Evaluation question iswhat would have happened to those receiving the intervention if they hadnot in fact received the program.

2 Since we cannot observe this groupboth with and without the intervention, the key challenge is to develop acounterfactual that is, a group which is as similar as possible (inobservable and unobservable dimensions) to those receiving theintervention. This comparison allows for the establishment of definitivecausality attributing observed changes in welfare to the program, whileremoving confounding factors. SIEF IE Workshop : Difference-in-Difference EstimationSlide 4 What Is an Impact Evaluation ?Goal: measurecausalimpacts of policy on participants We didA; as a result,Bhappened Ais a policy or intervention Bis an outcome of interest (what we hope to Impact ) Examples:IWe gave out insecticide-treated bednets, and fewer children underthe age of 5 got sick with or died from malaria as a resultIWe distributed free lunches in elementary schools, and schoolattendance and/or academic performance went up as a resultSIEF IE Workshop : Difference-in-Difference EstimationSlide 5 What Is an Impact Evaluation ?

3 Goal: measurecausalimpacts of policy on participants We didA; as a result,Bhappened Ais a policy or intervention Bis an outcome of interest (what we hope to Impact ) Examples:IWe gave out insecticide-treated bednets, and fewer children underthe age of 5 got sick with or died from malaria as a resultIWe distributed free lunches in elementary schools, and schoolattendance and/or academic performance went up as a resultSIEF IE Workshop : Difference-in-Difference EstimationSlide 5 What Is an Impact Evaluation ?Goal: measurecausalimpacts of policy on participants We didA; as a result,Bhappened Ais a policy or intervention Bis an outcome of interest (what we hope to Impact ) Examples:IWe gave out insecticide-treated bednets, and fewer children underthe age of 5 got sick with or died from malaria as a resultIWe distributed free lunches in elementary schools, and schoolattendance and/or academic performance went up as a resultSIEF IE Workshop : Difference-in-Difference EstimationSlide 5 Establishing CausalityGoal: measurecausalimpacts of policy on participants We want to be able to sayBhappened because ofAIWe need to rule out other possible causes ofB If we can say this, then we can also say.

4 If we didAagain (inanother place), we think thatBwould happen there as wellIn an ideal world (research-wise), we could clone each programparticipant and observe the impacts of our program on their IE Workshop : Difference-in-Difference EstimationSlide 6 Establishing CausalityGoal: measurecausalimpacts of policy on participants We want to be able to sayBhappened because ofAIWe need to rule out other possible causes ofB If we can say this, then we can also say: if we didAagain (inanother place), we think thatBwould happen there as wellIn an ideal world (research-wise), we could clone each programparticipant and observe the impacts of our program on their IE Workshop : Difference-in-Difference EstimationSlide 6 Establishing CausalityIn an ideal world (research-wise), we could clone each programparticipant and observe the impacts of our program on their is the Impact of giving Lisa a book on her test score?

5 Impact = Lisa s score with a book - Lisa s score without a bookIn the real world, we either observe Lisa with a book or without We never observe thecounterfactualSIEF IE Workshop : Difference-in-Difference EstimationSlide 7 Establishing CausalityIn an ideal world (research-wise), we could clone each programparticipant and observe the impacts of our program on their is the Impact of giving Lisa a book on her test score? Impact = Lisa s score with a book - Lisa s score without a bookIn the real world, we either observe Lisa with a book or without We never observe thecounterfactualSIEF IE Workshop : Difference-in-Difference EstimationSlide 7 Establishing CausalityTo measure the causal Impact of giving Lisa a book on her test score, weneed to find a comparison group that did not receive a estimate of the Impact of the book is then the difference in testscores between thetreatment groupand thecomparison group Impact = Lisa s score with a book - Bart s score without a bookAs this example illustrates, finding a good comparison group is hardSIEF IE Workshop .

6 Difference-in-Difference EstimationSlide 8 Establishing CausalityTo measure the causal Impact of giving Lisa a book on her test score, weneed to find a comparison group that did not receive a estimate of the Impact of the book is then the difference in testscores between thetreatment groupand thecomparison group Impact = Lisa s score with a book - Bart s score without a bookAs this example illustrates, finding a good comparison group is hardSIEF IE Workshop : Difference-in-Difference EstimationSlide 8 The Potential Outcomes FrameworkTwo potential outcomes for each individual, community, etc:Potential outcome ={Y0iPi= 0Y1iPi= 1 The problem: we only observe one ofY1iandY0i Each individual either participates in the program or not The causal Impact of program (P) oniis:Y1i Y0iWe observei s actual outcome:Yi=Y0i+ (Y1i Y0i) impactPiSIEF IE Workshop : Difference-in-Difference EstimationSlide 9 The Potential Outcomes FrameworkTwo potential outcomes for each individual, community, etc:Potential outcome ={Y0iPi= 0Y1iPi= 1 The problem: we only observe one ofY1iandY0i Each individual either participates in the program or not The causal Impact of program (P) oniis:Y1i Y0iWe observei s actual outcome:Yi=Y0i+ (Y1i Y0i) impactPiSIEF IE Workshop : Difference-in-Difference EstimationSlide 9 The Potential Outcomes FrameworkTwo potential outcomes for each individual, community, etc.}}

7 Potential outcome ={Y0iPi= 0Y1iPi= 1 The problem: we only observe one ofY1iandY0i Each individual either participates in the program or not The causal Impact of program (P) oniis:Y1i Y0iWe observei s actual outcome:Yi=Y0i+ (Y1i Y0i) impactPiSIEF IE Workshop : Difference-in-Difference EstimationSlide 9 Defining the CounterfactualTo estimate the Impact of a program, we need to know what would havehappened to every participantiin the absence the program We call this thecounterfactualOf course, we can t actually clone our participants and see what happensto the clones if they don t participate in the program Instead, we estimate the counterfactual using acomparison groupThe comparison group needs to: Look identical to the treatment group prior to the program Not be impacted by the program in anywayYOU CANNOT HAVE A GOOD Impact EVALUATIONWITHOUT A CREDIBLE, CONVINCING COMPARISON GROUPSIEF IE Workshop : Difference-in-Difference EstimationSlide 10 Defining the CounterfactualTo estimate the Impact of a program, we need to know what would havehappened to every participantiin the absence the program We call this thecounterfactualOf course, we can t actually clone our participants and see what happensto the clones if they don t participate in the program Instead, we estimate the counterfactual using acomparison groupThe comparison group needs to.}

8 Look identical to the treatment group prior to the program Not be impacted by the program in anywayYOU CANNOT HAVE A GOOD Impact EVALUATIONWITHOUT A CREDIBLE, CONVINCING COMPARISON GROUPSIEF IE Workshop : Difference-in-Difference EstimationSlide 10 Defining the CounterfactualTo estimate the Impact of a program, we need to know what would havehappened to every participantiin the absence the program We call this thecounterfactualOf course, we can t actually clone our participants and see what happensto the clones if they don t participate in the program Instead, we estimate the counterfactual using acomparison groupThe comparison group needs to: Look identical to the treatment group prior to the program Not be impacted by the program in anywayYOU CANNOT HAVE A GOOD Impact EVALUATIONWITHOUT A CREDIBLE, CONVINCING COMPARISON GROUPSIEF IE Workshop : Difference-in-Difference EstimationSlide 10 Defining the CounterfactualTo estimate the Impact of a program, we need to know what would havehappened to every participantiin the absence the program We call this thecounterfactualOf course, we can t actually clone our participants and see what happensto the clones if they don t participate in the program Instead, we estimate the counterfactual using acomparison groupThe comparison group needs to.

9 Look identical to the treatment group prior to the program Not be impacted by the program in anywayYOU CANNOT HAVE A GOOD Impact EVALUATIONWITHOUT A CREDIBLE, CONVINCING COMPARISON GROUPSIEF IE Workshop : Difference-in-Difference EstimationSlide 10 The Moving Parts of an Impact EvaluationA policy or program of interest (aka the treatment ) Pi= 1 if individual/communityiparticipated in the program Pi= 0 otherwise Thetreatment group: a group of people for whomPi= 1 Thecomparison group: a group of people for whomPi= 0 Theoutcome of interest: the dependent variable in our analysis Something that we care about Something that we expect to be impacted by the treatmentAn Impact Evaluation compares values of the outcome of interest in thetreatment group to values in the comparison group We attribute the difference to the Impact of treatmentSIEF IE Workshop : Difference-in-Difference EstimationSlide 11 The Moving Parts of an Impact EvaluationA policy or program of interest (aka the treatment ) Pi= 1 if individual/communityiparticipated in the program Pi= 0 otherwise Thetreatment group: a group of people for whomPi= 1 Thecomparison group: a group of people for whomPi= 0 Theoutcome of interest.

10 The dependent variable in our analysis Something that we care about Something that we expect to be impacted by the treatmentAn Impact Evaluation compares values of the outcome of interest in thetreatment group to values in the comparison group We attribute the difference to the Impact of treatmentSIEF IE Workshop : Difference-in-Difference EstimationSlide 11 The Moving Parts of an Impact EvaluationA policy or program of interest (aka the treatment ) Pi= 1 if individual/communityiparticipated in the program Pi= 0 otherwise Thetreatment group: a group of people for whomPi= 1 Thecomparison group: a group of people for whomPi= 0 Theoutcome of interest: the dependent variable in our analysis Something that we care about Something that we expect to be impacted by the treatmentAn Impact Evaluation compares values of the outcome of interest in thetreatment group to values in the comparison group We attribute the difference to the Impact of treatmentSIEF IE Workshop : Difference-in-Difference EstimationSlide 11 The Moving Parts of an Impact EvaluationA policy or program of interest (aka the treatment ) Pi= 1 if individual/communityiparticipated in the program Pi= 0 otherwise