1 Impact Evaluations Impact Evaluations and Development Nonie Guidance on Impact evaluation and Development Inputs Nonie Guidance on Impact evaluation . Activities . Outputs . Outcomes . Impacts ISBN 978-1-60244-120-0. What Is NONIE? NONIE is a Network of Networks for Impact evaluation comprised of the Organisation for Economic Co-operation and Development 's Development Assistance Committee (OECD/. DAC) evaluation Network, the United Nations evaluation Group (UNEG), the evaluation Cooperation Group (ECG), and the International Organization for Cooperation in evaluation (IOCE) a network drawn from the regional evaluation associations. NONIE was formed to promote quality Impact evaluation . NONIE fosters a program of Impact evaluation activities based on a common understanding of the meaning of Impact evaluation and approaches to conducting Impact evaluation .
2 NONIE focuses on Impact evaluation and does not attempt to address wider monitoring and evaluation issues. To this end NONIE aims to . Build an international collaborative research effort for high-quality and useful Impact evalu- ations as a means of improving Development effectiveness. Provide its members with opportunities for learning, collaboration, guidance, and support, leading to commissioning and carrying out Impact Evaluations . Develop a platform of resources to support Impact evaluation by member organizations. Impact Evaluations and Development Nonie Guidance on Impact evaluation Frans Leeuw Maastricht University Jos Vaessen Maastricht University and University of Antwerp 2009 NONIE The Network of Networks on Impact evaluation , Frans Leeuw, and Jos Vaessen c/o Independent evaluation Group 1818 H Street, NW.
3 Washington, DC 20433. Internet: All rights reserved This volume is a product of the volume's authors, Frans Leeuw and Jos Vaessen, who were commis- sioned by NONIE. The findings, interpretations, and conclusions expressed in this volume are those of the authors and do not necessarily reflect the views of NONIE, its members, or other participating agencies. NONIE does not guarantee the accuracy of the data included in this work and accepts no responsibility for any consequence of their use. Rights and Permissions The material in this publication is copyrighted. Copying and/or transmitting portions or all of this work without permission may be a violation of applicable law. NONIE encourages dissemination of its work and will normally grant permission to reproduce portions of the work promptly.
4 All queries on rights and licenses, including subsidiary rights, should be addressed to NONIE, c/o IEG, 1818 H St., NW, Washington, DC, 20433, Cover: Pakistani girl reading. Photo by Curt Carnemark, courtesy of World Bank Photo Library. ISBN-10: 1-60244-120-0. ISBN-13: 978-1-60244-120-0. Printed on recycled paper Contents vii Acknowledgments ix Executive Summary xix Introduction 1 Part I Methodological and Conceptual Issues in Impact evaluation 3 1 Identify the (type and scope of the) intervention 3 The Impact evaluation landscape and the scope of Impact evaluation 4 Impact of what? 7 Impact on what? 9 Key message 11 2 Agree on what is valued 11 Stakeholder values in Impact evaluation 12 Intended versus unintended effects 12 Short-term versus long-term effects 12 The sustainability of effects 13 Key message 15 3 Carefully articulate the theories linking interventions to outcomes 15 Seeing interventions as theories: The black box and the contribution problem 15 Articulating intervention theories on Impact 17 Testing intervention theories on Impact 19 Key message 21 4 Address the attribution problem 21 The attribution problem 23 Quantitative methods addressing the attribution problem 29 Applicability of quantitative methods for addressing the attribution problem 31 Other approaches 34 Key message 35 5 Use a mixed-methods approach.
5 The logic of the comparative advantages of methods 35 Different methodologies have comparative advantages in addressing particular concerns and needs 36 Advantages of combining different methods and sources of evidence 38 Average effect versus distribution of costs and benefits 39 Key message iii I m pa c t E va l u at i o n s a n d De v e l o p m e n t N o n ie G u i d a n c e O n I m pa c t E va l u at i o n 41 6 Build on existing knowledge relevant to the Impact of interventions 43 Key message 45 Part II Managing Impact Evaluations 47 7 Determine if an Impact evaluation is feasible and worth the cost 48 Key message 49 8 Start collecting data early 49 Timing of data collection 49 Data availability 51 Quality of the data 51 Dealing with data constraints 52 Key message 53 9 Front-end planning is important 53 Planning tools 53 Staffing and resources 54 The balance between independence and collaboration between evaluators and stakeholders 54 Ethical issues 55 Norms and standards 56 Ownership and capacity building 56 Key message 57 Appendices 59 1.
6 Examples of diversity in Impact evaluation 61 2. The General Elimination Methodology as a basis for causal analysis 63 3. Overview of quantitative techniques of Impact evaluation 65 4. Technical aspects of quantitative Impact evaluation techniques 69 5. Evaluations using quantitative Impact evaluation approaches 71 6. Decision tree for selecting quantitative evaluation designs to deal with selection bias 73 7. Hierarchical modeling and other statistical approaches 75 8. Multi-site evaluation approaches 77 9. Methodological frameworks for assessing the effects of interventions , mainly based on quantitative methods 79 10. Where to find reviews and synthesis studies on mechanisms underlying processes of change 81 11. Evaluations based on qualitative and quantitative descriptive methods 101 12.
7 Further information on review and synthesis approaches in Impact evaluation 105 13. Basic education in Ghana 109 14. Hierarchy of quasi-experimental designs 111 15. International experts who contributed to the subgroup documents 113 Endnotes 117 References Boxes 7 Unpacking the aid chain 16 Social funds and government capacity: Competing theories iv CONT E NTS. 18 Social and behavioral mechanisms as heuristics for understanding processes of change and Impact 25 Using propensity scores to select a matched comparison group . The Vietnam Rural Roads Project 33 Participatory Impact monitoring in the context of the poverty reduction strategy process 39 Brief illustration of the logic of comparative advantages 42 Narrative review and synthesis study: Targeting and Impact of community-based Development initiatives 73 Impact of the Indonesian financial crisis on the poor.
8 Partial equilibrium modeling and CGE modeling with microsimulation Figures xi Levels of intervention, programs, and policies and types of Impact xii Simple graphic of net Impact of an intervention 8 Levels of intervention, programs, and policies and types of Impact 17 Basic intervention theory of a fictitious small business support project 22 Graphic display of the net Impact of an intervention 29 Regression discontinuity analysis 66 Estimation of the effect of class size with and without the inclusion of a variable correlated with class size 87 Final Impact assessment triangulation 92 Generic representation of a project's theory of change 93 Components of Impact evaluation framework 96 Project outputs and outcomes 99 Framework to establish contribution 99 Model linking outcome to Impact Tables 5 Aspects of complication in interventions 6 Aspects of complexity in interventions 26 Double difference and other designs 52 evaluation scenarios with time, data.
9 And budget constraints 96 Project outcome 97 Change in key ecological attributes over time 97 Current threats to the global environment benefits v Acknowledgments This Guidance document could not have The Guidance document represents the views of existed without the numerous contributions the authors, who were commissioned by NONIE. of Network of Networks on Impact evaluation Given the fact that perspectives on the defini- (NONIE) members and others in terms of papers, tion, scope, and appropriate methods of Impact PowerPoint presentations, and suggestions. evaluation differ widely among practitioners and other stakeholders, the document should not be In particular, this Guidance document builds on two taken to represent the agreed positions of all of existing draft guidance documents, a document on the individual NONIE members.
10 The network experimental and quasi-experimental approaches membership and the authors recognize that to Impact evaluation (NONIE subgroup 1, May 17, there is scope to develop the arguments further 2007) and a document on qualitative approaches in several key areas. to Impact evaluation (NONIE subgroup 2, January 9, 2008). A third draft document prepared by We would like to thank all of the above people NONIE members on the Impact evaluation of for their contributions to the process of writing macroeconomic policies and new aid modalities the Guidance document. First, we thank the such as budget support is outside the scope of this authors of the subgroup documents for provid- Guidance document. The subgroup 1 document ing building blocks for this document. In was prepared mainly by Howard White and Antonie addition, we would like to thank the steering De Kemp.