A COUNTERFACTUAL APPROACH FOR IMPACT EVALUATION

1 INTRODUCTIONCOUNTERFACTUAL FRAMEWORKIE DESIGNS & METHODSCASE STUDIESA COUNTERFACTUAL APPROACH FORIMPACT EVALUATIOND epartment of EpidemiologyMiguel-Angel Luque-Fernandez,PhDJuly 19, 2014 MIGUEL ANGEL LUQUE-FERNANDEZA COUNTERFACTUAL APPROACH FOR IMPACT EVALUATIONINTRODUCTIONCOUNTERFACTUAL FRAMEWORKIE DESIGNS & METHODSCASE STUDIESOUTLINE1 INTRODUCTIONIE DefinitionsIE CharacteristicsDifferences between IE M&E OR2 COUNTERFACTUAL FRAMEWORKH istory, definition and justificationWhat is a causal effect?3IE DESIGNS & METHODSIE Designs: methodsIE controversies and mythsThe best design4 CASE STUDIESPre/post surveysMSMMIGUEL ANGEL LUQUE-FERNANDEZA COUNTERFACTUAL APPROACH FOR IMPACT EVALUATIONINTRODUCTIONCOUNTERFACTUAL FRAMEWORKIE DESIGNS & METHODSCASE STUDIESIMPACT EVALUATIONMIGUEL ANGEL LUQUE-FERNANDEZA COUNTERFACTUAL APPROACH FOR IMPACT EVALUATIONINTRODUCTIONCOUNTERFACTUAL FRAMEWORKIE DESIGNS & METHODSCASE STUDIESIE DefinitionsIE CharacteristicsDifferences between IE M&E ORImpact EVALUATION DefinitionsWorld Bank A systematic identification of theeffectspositive or negative, intendedor not on individual households, institutions.

2 And the environmentcausedby a given development activity such aprogram or project ANGEL LUQUE-FERNANDEZA COUNTERFACTUAL APPROACH FOR IMPACT EVALUATIONINTRODUCTIONCOUNTERFACTUAL FRAMEWORKIE DESIGNS & METHODSCASE STUDIESIE DefinitionsIE CharacteristicsDifferences between IE M&E ORImpact EVALUATION DefinitionsUS Environmental Protection Agency A form of EVALUATION that assess thenet effectof a program bycomparing program outcomeswith an estimate ofwhat would havehappened in the absence of the program ANGEL LUQUE-FERNANDEZA COUNTERFACTUAL APPROACH FOR IMPACT EVALUATIONINTRODUCTIONCOUNTERFACTUAL FRAMEWORKIE DESIGNS & METHODSCASE STUDIESIE DefinitionsIE CharacteristicsDifferences between IE M&E ORIE characteristicsImpact EvaluationImpact EVALUATION (IE)assesses changesthan can be attributed toa (CAUSAL mechanism), that is,a comparison between what actually happenedandwhat would have happenedin theabsence of the ANGEL LUQUE-FERNANDEZA COUNTERFACTUAL APPROACH FOR IMPACT EVALUATIONINTRODUCTIONCOUNTERFACTUAL FRAMEWORKIE DESIGNS & METHODSCASE STUDIESIE DefinitionsIE CharacteristicsDifferences between IE M&E ORIE characteristicsImpact EvaluationImpact EVALUATION (IE)assesses changesthan can be attributed toa (CAUSAL mechanism)

3 , that is,a comparison between what actually happenedandwhat would have happenedin theabsence of the ANGEL LUQUE-FERNANDEZA COUNTERFACTUAL APPROACH FOR IMPACT EVALUATIONINTRODUCTIONCOUNTERFACTUAL FRAMEWORKIE DESIGNS & METHODSCASE STUDIESIE DefinitionsIE CharacteristicsDifferences between IE M&E ORIE characteristicsImpact EvaluationImpact EVALUATION (IE)assesses changesthan can be attributed toa (CAUSAL mechanism), that is,a comparison between what actually happenedandwhat would have happenedin theabsence of the ANGEL LUQUE-FERNANDEZA COUNTERFACTUAL APPROACH FOR IMPACT EVALUATIONINTRODUCTIONCOUNTERFACTUAL FRAMEWORKIE DESIGNS & METHODSCASE STUDIESIE DefinitionsIE CharacteristicsDifferences between IE M&E ORIE characteristicsImpact EvaluationIEanswers the question: What works for whom in whatcircumstances?

4 Thus,IEinvolvesMixed Methods: contextual andqualitative main purpose ofIEis to improveevidence-basedpolicymaking by means of providingeffectivenessevaluations of publichealth ANGEL LUQUE-FERNANDEZA COUNTERFACTUAL APPROACH FOR IMPACT EVALUATIONINTRODUCTIONCOUNTERFACTUAL FRAMEWORKIE DESIGNS & METHODSCASE STUDIESIE DefinitionsIE CharacteristicsDifferences between IE M&E ORIE characteristicsImpact EvaluationIEanswers the question: What works for whom in whatcircumstances? Thus,IEinvolvesMixed Methods: contextual andqualitative main purpose ofIEis to improveevidence-basedpolicymaking by means of providingeffectivenessevaluations of publichealth ANGEL LUQUE-FERNANDEZA COUNTERFACTUAL APPROACH FOR IMPACT EVALUATIONINTRODUCTIONCOUNTERFACTUAL FRAMEWORKIE DESIGNS & METHODSCASE STUDIESIE DefinitionsIE CharacteristicsDifferences between IE M&E ORIE characteristicsImpact EvaluationIEanswers the question: What works for whom in whatcircumstances?

5 Thus,IEinvolvesMixed Methods: contextual andqualitative main purpose ofIEis to improveevidence-basedpolicymaking by means of providingeffectivenessevaluations of publichealth ANGEL LUQUE-FERNANDEZA COUNTERFACTUAL APPROACH FOR IMPACT EVALUATIONINTRODUCTIONCOUNTERFACTUAL FRAMEWORKIE DESIGNS & METHODSCASE STUDIESIE DefinitionsIE CharacteristicsDifferences between IE M&E ORDifferencesImpact EvaluationIEis the general framework whereMonitoring and EVALUATION (ME)andOperational Research(OR) are available from the project over time interms of goals, indicators and for tools that can enhance the quality of element ofORis that theresearch questionsare generated byidentifying the constraints and challenges encountered during theimplementation of program ANGEL LUQUE-FERNANDEZA COUNTERFACTUAL APPROACH FOR IMPACT EVALUATIONINTRODUCTIONCOUNTERFACTUAL FRAMEWORKIE DESIGNS & METHODSCASE STUDIESIE DefinitionsIE CharacteristicsDifferences between IE M&E ORDifferencesImpact EvaluationIEis the general framework whereMonitoring and EVALUATION (ME)andOperational Research(OR)

6 Are available from the project over time interms of goals, indicators and for tools that can enhance the quality of element ofORis that theresearch questionsare generated byidentifying the constraints and challenges encountered during theimplementation of program ANGEL LUQUE-FERNANDEZA COUNTERFACTUAL APPROACH FOR IMPACT EVALUATIONINTRODUCTIONCOUNTERFACTUAL FRAMEWORKIE DESIGNS & METHODSCASE STUDIESIE DefinitionsIE CharacteristicsDifferences between IE M&E ORDifferencesImpact EvaluationIEis the general framework whereMonitoring and EVALUATION (ME)andOperational Research(OR) are available from the project over time interms of goals, indicators and for tools that can enhance the quality of element ofORis that theresearch questionsare generated byidentifying the constraints and challenges encountered during theimplementation of program ANGEL LUQUE-FERNANDEZA COUNTERFACTUAL APPROACH FOR IMPACT EVALUATIONINTRODUCTIONCOUNTERFACTUAL FRAMEWORKIE DESIGNS & METHODSCASE STUDIESIE DefinitionsIE CharacteristicsDifferences between IE M&E ORDifferencesImpact EvaluationIEfocuses on theEFFECTIVENESSof the on theIMPLEMENTATION and EVOLUTION ofthe project over ANGEL LUQUE-FERNANDEZA COUNTERFACTUAL APPROACH FOR IMPACT EVALUATIONINTRODUCTIONCOUNTERFACTUAL FRAMEWORKIE DESIGNS & METHODSCASE STUDIESIE DefinitionsIE CharacteristicsDifferences between IE M&E

7 ORDifferencesImpact EvaluationIEfocuses on theEFFECTIVENESSof the on theIMPLEMENTATION and EVOLUTION ofthe project over on enhancing theQUALITYof the project ANGEL LUQUE-FERNANDEZA COUNTERFACTUAL APPROACH FOR IMPACT EVALUATIONINTRODUCTIONCOUNTERFACTUAL FRAMEWORKIE DESIGNS & METHODSCASE STUDIESIE DefinitionsIE CharacteristicsDifferences between IE M&E ORDifferencesImpact EvaluationIEfocuses on theEFFECTIVENESSof the on theIMPLEMENTATION and EVOLUTION ofthe project over on enhancing theQUALITYof the project ANGEL LUQUE-FERNANDEZA COUNTERFACTUAL APPROACH FOR IMPACT EVALUATIONINTRODUCTIONCOUNTERFACTUAL FRAMEWORKIE DESIGNS & METHODSCASE STUDIESIE DefinitionsIE CharacteristicsDifferences between IE M&E ORDifferencesImpact EvaluationIEfocuses on theEFFECTIVENESSof the on theIMPLEMENTATION and EVOLUTION ofthe project over on enhancing theQUALITYof the project ANGEL LUQUE-FERNANDEZA COUNTERFACTUAL APPROACH FOR IMPACT EVALUATIONINTRODUCTIONCOUNTERFACTUAL FRAMEWORKIE DESIGNS & METHODSCASE STUDIESH istory, definition and justificationWhat is a causal effect?

8 COUNTERFACTUAL FRAMEWORKMIGUEL ANGEL LUQUE-FERNANDEZA COUNTERFACTUAL APPROACH FOR IMPACT EVALUATIONINTRODUCTIONCOUNTERFACTUAL FRAMEWORKIE DESIGNS & METHODSCASE STUDIESH istory, definition and justificationWhat is a causal effect? COUNTERFACTUAL frameworkWhen an RCT is not possibleThis framework was developed first by statisticians (Rubin, 1983)and econometricians (Heckman, 1978) as a new APPROACH for theestimation ofcausal effectsfrom observational known as theNeyman-Rubin ANGEL LUQUE-FERNANDEZA COUNTERFACTUAL APPROACH FOR IMPACT EVALUATIONINTRODUCTIONCOUNTERFACTUAL FRAMEWORKIE DESIGNS & METHODSCASE STUDIESH istory, definition and justificationWhat is a causal effect? COUNTERFACTUAL frameworkWhen an RCT is not possibleThis framework was developed first by statisticians (Rubin, 1983)and econometricians (Heckman, 1978) as a new APPROACH for theestimation ofcausal effectsfrom observational known as theNeyman-Rubin COUNTERFACTUAL framework offers an APPROACH toIEwhen aRandom Clinical Trial(RCT) is unfeasible or ANGEL LUQUE-FERNANDEZA COUNTERFACTUAL APPROACH FOR IMPACT EVALUATIONINTRODUCTIONCOUNTERFACTUAL FRAMEWORKIE DESIGNS & METHODSCASE STUDIESH istory, definition and justificationWhat is a causal effect?

9 COUNTERFACTUAL frameworkWhen an RCT is not possibleThis framework was developed first by statisticians (Rubin, 1983)and econometricians (Heckman, 1978) as a new APPROACH for theestimation ofcausal effectsfrom observational known as theNeyman-Rubin COUNTERFACTUAL framework offers an APPROACH toIEwhen aRandom Clinical Trial(RCT) is unfeasible or ANGEL LUQUE-FERNANDEZA COUNTERFACTUAL APPROACH FOR IMPACT EVALUATIONINTRODUCTIONCOUNTERFACTUAL FRAMEWORKIE DESIGNS & METHODSCASE STUDIESH istory, definition and justificationWhat is a causal effect? COUNTERFACTUAL frameworkWhen an RCT is not possibleThis framework was developed first by statisticians (Rubin, 1983)and econometricians (Heckman, 1978) as a new APPROACH for theestimation ofcausal effectsfrom observational known as theNeyman-Rubin COUNTERFACTUAL framework offers an APPROACH toIEwhen aRandom Clinical Trial(RCT) is unfeasible or ANGEL LUQUE-FERNANDEZA COUNTERFACTUAL APPROACH FOR IMPACT EVALUATIONINTRODUCTIONCOUNTERFACTUAL FRAMEWORKIE DESIGNS & METHODSCASE STUDIESH istory, definition and justificationWhat is a causal effect?

10 JustificationWhen a RCT is not possibleThe COUNTERFACTUAL framework offers an APPROACH to IE whenresearchers need to assess treatment effects from survey data,census data, administrative data, or other types of data. Data collected through the observation of systems as theyoperate in normal practice without any interventionsimplemented by randomized assignments rules (Rubin, 1977, )MIGUEL ANGEL LUQUE-FERNANDEZA COUNTERFACTUAL APPROACH FOR IMPACT EVALUATIONINTRODUCTIONCOUNTERFACTUAL FRAMEWORKIE DESIGNS & METHODSCASE STUDIESH istory, definition and justificationWhat is a causal effect?JustificationWhen a RCT is not possibleThe COUNTERFACTUAL framework offers an APPROACH to IE whenresearchers need to assess treatment effects from survey data,census data, administrative data, or other types of data.