MONITORING AND IMPACT EVALUATION …

1 WORLD BANK INDEPENDENT EVALUATION GROUP & THE THEMATIC GROUP FOR POVERTY ANALYSIS, MONITORING AND IMPACT EVALUATION EVALUATION CAPACITY DEVELOPMENTTHE WORLD BANK1818 H Street, , 20433, : 202-477-1234 Facsimile: 202-477-6391 Telex: MCI 64145 WORLDBANKMCI 248423 WORLDBANKI nternet: EVALUATION GroupKnowledge Programs and EVALUATION Capacity Development (IEGKE)E-mail: 202-458-4497 Facsimilie: 202-522-3125 CONDUCTINGQUALITY IMPACTEVALUATIONSUNDER BUDGET,TIME AND DATACONSTRAINTSCONDUCTINGQUALITY IMPACTEVALUATIONSUNDER BUDGET,TIME AND DATACONSTRAINTSI ndependent EVALUATION Group,The World BankPoverty Analysis, MONITORING andImpact EVALUATION Thematic GroupPREM Network The World BankNAcknowledgementThis booklet was prepared by Michael Bamberger, and was jointly sponsored by the World Bank s Independent EVALUATION Group (IEG) and the PovertyAnalysis, MONITORING and IMPACT EVALUATION Thematic Group (PREM Network).

2 It draws on earlier work commissioned by IEG which was formerlyknown as the Operations EVALUATION Department and on a more detailed volume subsequently developed jointly by Michael Bamberger, Jim Rugh andLinda Mabry (2006). Valuable comments on earlier drafts of this booklet wereprovided by a number of Bank staff, including Judy Baker, Tara Bedi, ArielFiszbein, Emanuela Galasso and Howard White. The joint task managers of thebooklet were Keith Mackay, Aline Coudouel and Markus generous financial support for this paper, provided by the NorwegianAgency for Development Cooperation (Norad) in the Ministry of ForeignAffairs, is gratefully TilmesManagerKnowledge Programs & EVALUATION Capacity DevelopmentIndependent EVALUATION GroupCopyright 2006 The International Bank for Reconstruction and Development/The World Bank1818 H Street, 20433, rights in the United States of opinions expressed in the report do not necessarily represent the views of the World Bank or itsmember governments.

3 The World Bank does not guarantee the accuracy of the data included in thispublication and accepts no responsibility whatsoever for any consequence of their OFCONTENTS1 Overview ..12 Simplifying the EVALUATION design ..5 Case study 1: Reducing costs through a post-project cross-sectional design gender and time-use impacts of the Ecuador cut-flower industry ..93 Working with comparison group designs ..10 Case study 2: Using propensity scores to produce a matched comparison group the Viet Nam Rural Roads Project ..124 Working with secondary data ..13 Case study 3: Using multiple sources of secondary data the Bangladesh Integrated Nutrition Project ..155 Reconstructing baseline study 4: Reconstructing baseline data the Nicaragua Social Fund ..186 Reducing the costs of data collection ..19 Case study 5: Using PRA techniques to reduce data collection costs the Flores, Indonesia Village Water Supply and Sanitation Project ..217 Strengthening EVALUATION designs when working under budget, time and data constraints .

4 22 Table 1. Design options for reducing data collection costs ..8 Table 2. How budget, time and data constraints affect the quality of an IMPACT EVALUATION ..23 Endnotes ..27 References ..29 Additional Resources on MONITORING and EVALUATION .. and purpose of this publicationThere is a growing appreciation within the development community of the merits of conducting evaluations. EVALUATION is a powerful tool for learning about what works, what doesnot, and the reasons why. EVALUATION is also an important accountability tool. For these reasons,the World Bank requires that all of its projects be evaluated. A growing number of developingcountries also recognize the benefits from EVALUATION , and many are making efforts to institutionalize MONITORING and EVALUATION systems as part of sound this context, an increasing number of rigorous IMPACT evaluations are being conducted, and itis importantthat the EVALUATION methods, findings and recommendations are as reliable as possible.

5 At the same time, however, these evaluations can be expensive to conduct. Project andprogram managers who wish to conduct an EVALUATION are often faced with severe budget, timeor data constraints these can act as a disincentive to conduct rigorous evaluations. The purpose of this booklet is to provide advice to those planning an IMPACT EVALUATION , so that theycan select the most rigorous methods available within the constraints they face. The booklet isalso intended to clarify the nature of the trade-offs between EVALUATION rigor and the budget,time and data which are available for an is hoped that this booklet will encouragemanagers to conduct IMPACT evaluations when they might otherwise have viewed them as tooexpensive or time-consuming to be conducted to a high standard. Thus the desired outcome isan increase in the quality and quantity of rigorous IMPACT evaluations which are conducted. An extensive literature is now available on appropriate methodologies for evaluating the impactsof development projects and programs.

6 This booklet applies these methodologies to the real-world situations and constraints faced by task managers and researchers. It is intended tocomplement other recent World Bank publications including Baker (2000), Operations EVALUATION Department (2004), Ravallion (2001, 2005), White (2006), and the methodologicalguidelines and IMPACT EVALUATION case studies on the Bank s Poverty IMPACT Analysis, Monitoringand EVALUATION EVALUATION constraints Two sets of constraints shape the choice of EVALUATION methods. The first comprises budget, timeand data constraints . Budget constraintsaffect the number of interviews that can be conducted, the ability to combine quantitative and qualitative data collection and analysis, thesize and professional experience of the research team, and the analysis that can be constraintsaffect when the EVALUATION begins and ends, how long researchers can be inthe field, and time available for feedback from stakeholders.

7 When new surveys are conducted,data constraintsaffect the ability to collect information from a suitable comparison group andobtain baseline information on the project population; or to collect sensitive information andNto interview difficult-to-reach groups. When the EVALUATION is based on secondary data or whendata are obtained through studies conducted by other agencies (piggy-back or synchronizedstudies), data constraints may affect the compatibility of sample coverage and timing, orwhether the data cover the required variables and define them in the required manner. In contrast, program design and delivery mechanismsdetermine how project participants areselected (randomly, according to administrative criteria, or through self-selection) and the quality and uniformity of project implementation. These design features, which derive from theinner working of a project, produce a range of project-specific issues which determine the rangeof EVALUATION methods which can be applied.

8 These issues will not be covered here; readers interested in guidance on how to approach these questions of EVALUATION methods should seeRavallion (2001, 2005). Real-world EVALUATION scenariosWe discuss two common scenarios. Under the first, the evaluator is involved from the start ofthe project but budget, time and data constraints or program design and delivery mechanismslimit the range of available EVALUATION designs. For example, it may not be possible to include acontrol group or to conduct a comprehensive baseline study on the project population, or theremay be limits on the number of interviews that can be conducted. Under the second scenario the EVALUATION is not commissioned until the project is nearing completion or has ended. Data may be collected in one of four ways (see White, 2006): from aproject-specific survey; by piggy-backing a special module onto an ongoing survey; through asynchronized survey in which the project population is interviewed but a comparison group isobtained from another survey (national household survey, etc); or the EVALUATION is based on secondary data that include information on the project and potential comparison groups.

9 Amajor constraint facing all post-project EVALUATION designs is the absence of a baseline study, andthe options for addressing this are discussed in Section requirements for a quality IMPACT evaluationunder real-world constraintsThe challenge for the evaluator and the client is to decide whether it is possible to conduct a quality IMPACT EVALUATION under the real-world constraints , and to select the strongest possibledesign within the particular set of budget, time and data constraints . For example: at what pointdoes the sample become too small, or too limited in its coverage to permit sound statisticalanalysis? What are the criteria for assessing the adequacy of secondary data for estimating baseline conditions? And when is it possible to construct a valid counterfactual in the absenceof a baseline study? A quality IMPACT EVALUATION must: Develop a set of indicators that can meaningfully and reliably define and measure projectinputs, implementation processes, outputs, intended outcomes and impacts.

10 Develop a logically sound counterfactualpresenting a plausible argument that observed2changes in outcome indicators after the project intervention are in fact due to the projectand not to other unrelated factors, such as improvements in the local economy or programsorganized by other agencies. Determine, in accordance with accepted statistical procedures, whether a project has contributed to the intended impacts and benefited a significant proportion of the targetpopulation. In addition many evaluations are required to: Assess the distribution of benefits among different sectors of the target population. Identify factors influencing the magnitude and distribution of the impacts. Assess the sustainability of impacts over time. When resources are not a constraint, the conventional EVALUATION approach is to use a pre- andpost-intervention project and control group comparison similar to the following: Figure 1. Pre-test Post-test Control Group DesignTimeT1T2T3 Start of Project intervention End of project project (this may last a few weeks (baseline)or as long as several years)Project groupP1XP2 Randomized or C1C2non-randomized control groupThere are a number of methodological advantages from EVALUATION designs in which subjects arerandomly assigned to the project and control groups as this avoids systematic differencesbetween the two groups prior to the project.