SRA Workshop 10 March 2008 British Library …

1 Philip Davies, PhDAmerican Institutes for ResearchOxford England and Washington, DCPhilip Davies, PhDAmerican Institutes for ResearchOxford England and Washington, DCUsing Monitoring and Evaluationto Improve Public PolicySRA Workshop 10 March 2008 British Library Conference Centre, London Context of Policy evaluation in the UK Modernising Government Better Policy Making Evidence-Based Policy and Practice Greater Accountability Performance Management Public Spending and Fiscal Control Strategic Development Why Use M&E For Public Policy? Effectiveness - ensure we do more good than harm Efficiency - use scarce public resources to maximum effect Service Orientation - meet citizen s needs/expectations Accountability - transparency of what is done and why Democracy - enhance the democratic process Trust - help ensure/restore trust in government and publicservices The Experimenting Society Donald T.

2 Campbell ..a society that would use social science methods andevaluation techniques to vigorously try out possiblesolutions to recurrent problems and would make hard-headed, multidimensional evaluations of outcomes, andwhen the evaluation of one reform showed it to have beenineffective or harmful, would move on and try otheralternatives (Campbell, 1999a:9). A family of research methods which seeks tosystematically investigate the effectiveness of ways that improve socialconditions (Rossi, Freeman and Lipsey, 1999:20)What is evaluation ?

3 Types of evaluation Impact (or summative) evaluationsDoes the policy (programme, intervention)work? How large is the likely effect size? Process (or formative) evaluation How,why, and under what conditions does thepolicy (programme, intervention) work? What is the Policy Question? Effectiveness of What? Intervention effectiveness - what works? Resource effectiveness - at what cost/benefit? Likely diversity of effectiveness across different groups what works for whom and when? Implementation effectiveness - how it works? Experiential effectiveness - public s views of policy4 Evidencefor PolicyTheoriesof ChangeHow is the policy supposed to work?

4 Logic Model Establishing the Policy Logic/Theory of ChangeProgramme TheoryProgramme EvidenceVisit to aPrison byJuvenilesFirst HandExperienceof PrisonLifeExposure toPrison Lifeand Prisonersas NegativeRole ModelsFrightens orScaresJuvenilesAway fromCrimeReducesCrime andOffendingVisit to aPrison byJuvenilesFirst HandExperienceof PrisonLifeExposure toPrison Lifeand Prisonersas PositiveRole ModelsStimulates orAttractsJuvenilesTowardsCrimeIncreases Crime andOffending Evidencefor PolicyTheoriesof ChangeHow is the policy supposed to work?HarnessExistingEvidenceWhat evidencealready exists?

5 Systematic ReviewsLogic Model Evidencefor PolicyTheoriesof ChangeHow is the policy supposed to work?What is the nature, size and dynamicsof the problem?Logic ModelHarnessExistingEvidenceWhat evidencealready exists?Systematic ReviewsStatisticsSurveysQualitative ResearchDescriptiveAnalyticalEvidence Evidencefor PolicyTheoriesof ChangeDescriptiveAnalyticalEvidenceHow is the policy supposed to work?What is the nature, size and dynamicsof the problem?Logic ModelHarnessExistingEvidenceWhat evidencealready exists?Systematic ReviewsStatisticsSurveysQualitative ResearchAttitudinalandExperientialEviden ceSurveysQualitative ResearchObservationalStudiesHow do citizensfeel about thepolicy?

6 Evidencefor PolicyTheoriesof ChangeDescriptiveAnalyticalEvidenceHow is the policy supposed to work?What is the nature, size and dynamicsof the problem?Logic ModelHarnessExistingEvidenceWhat evidencealready exists?Systematic ReviewsStatisticsSurveysQualitative ResearchAttitudinal andExperientialEvidenceSurveysQualitativ e ResearchObservationalStudiesEvidence ofEffectiveInterventions(Impact)What Works?At What Costs?With What Outcomes?Experimental andQuasi-ExperimentalStudiesHow do citizensfeel about thepolicy? Evidencefor PolicyTheoriesof ChangeDescriptiveAnalyticalEvidenceHow is the policy supposed to work?

7 What is the nature, size and dynamicsof the problem?Logic ModelHarnessExistingEvidenceWhat evidencealready exists?Systematic ReviewsStatisticsSurveysQualitative ResearchAttitudinal andExperientialEvidenceSurveysQualitativ e ResearchObservationalStudiesEvidence ofEffectiveInterventionsWhat Works?At What Costs?With What Outcomes?Experimental andQuasi-ExperimentalStudiesEconomicandE conometricEvidenceWhat is the Cost,Benefit andEffectiveness of Interventions?Cost-BenefitCost-Effective nessCust-Utility AnalysisHow do citizensfeel about thepolicy? Evidencefor PolicyTheoriesof ChangeDescriptiveAnalyticalEvidenceHow is the policy supposed to work?

8 What is the nature, size and dynamicsof the problem?Logic ModelHarnessExistingEvidenceWhat evidencealready exists?Systematic ReviewsStatisticsSurveysQualitative ResearchAttitudinal andExperientialEvidenceHow do citizens and patients feel about health, illness and health policy?SurveysQualitative ResearchObservationalStudiesEvidence ofEffectiveInterventionsWhat Works?At What Costs?With What Outcomes?Experimental andQuasi-ExperimentalStudiesEconomicandE conometricEvidenceWhat is the Cost,Benefit andEffectiveness of Interventions?Cost-BenefitCost-Effective nessCust-Utility AnalysisEthicalEvidenceWhat are the ethical implications ofthe policy?

9 Social EthicsPublic Consultation PolicyEvaluationPolicyImplementationPoli cyDevelopment evaluation Evidence in The Policy Process(Linear Model)Ideas evaluation Evidence in The Policy ProcessIdeasEvaluationPolicyDevelopmentE valuationPolicyImplementationEvaluationE vidence-BasedPolicy Impact Evaluations Evaluations of Outcome Attainment (have targets been met?) Evaluations of Net Effects (against a counterfactual) Single Group Pre- and Post- Tests Interrupted Time Series Designs Matched Comparisons Designs Difference of Differences Propensity Score Matching Regression Discontinuity Designs Randomised Controlled Trials`Increasing strength of internal validity andcausal inference yyj5055606570758085909519961997199819992 0002001200220032004200520062007200820092 010 Delivery IndicatorLow Trajectory (policy has a lagged impact)Mid trajectoryHigh Trajectory (policy has an immediate impact)

10 Policy Step ALong Term Strategic GoaMid term Delivery Contract GoalIntermediate progress indicators or milestonesHistorical performanceProject Plan StreamsProject Plan StreamsPolicy Step BPolicy Step CPolicy Delivery: trajectoriesEvaluations of Outcome Attainment(Have Targets Been Met?) Evaluations of Net Effects(Against a Counterfactual) Counterfactual: what would have happened withoutthe program Need to estimate the counterfactual find a control or comparison group Counterfactual Criteria Intervention & counterfactual groups have identicalcharacteristics on average, Only reason for the difference in outcomes is due tothe intervention Evaluations of Net Effects(Against a Counterfactual)