Example: confidence

Impact Evaluations Development - World Bank

Impact Evaluations Impact Evaluations and Development Nonie Guidance on Impact evaluation and Development Inputs Nonie Guidance on Impact evaluation . Activities . Outputs . Outcomes . Impacts ISBN 978-1-60244-120-0. What Is NONIE? NONIE is a Network of Networks for Impact evaluation comprised of the Organisation for Economic Co-operation and Development 's Development Assistance Committee (OECD/. DAC) evaluation Network, the United Nations evaluation Group (UNEG), the evaluation Cooperation Group (ECG), and the International Organization for Cooperation in evaluation (IOCE) a network drawn from the regional evaluation associations. NONIE was formed to promote quality Impact evaluation . NONIE fosters a program of Impact evaluation activities based on a common understanding of the meaning of Impact evaluation and approaches to conducting Impact evaluation . NONIE focuses on Impact evaluation and does not attempt to address wider monitoring and evaluation issues.

Impact Evaluations and Development – No NIE Guidance on Impact Evaluation ISBN 978-1-60244-120-0 Impact Evaluations and Development NoNIE Guidance

Tags:

  Development, Evaluation, Impact, Impact evaluations, Impact evaluations development

Information

Domain:

Source:

Link to this page:

Please notify us if you found a problem with this document:

Other abuse

Transcription of Impact Evaluations Development - World Bank

1 Impact Evaluations Impact Evaluations and Development Nonie Guidance on Impact evaluation and Development Inputs Nonie Guidance on Impact evaluation . Activities . Outputs . Outcomes . Impacts ISBN 978-1-60244-120-0. What Is NONIE? NONIE is a Network of Networks for Impact evaluation comprised of the Organisation for Economic Co-operation and Development 's Development Assistance Committee (OECD/. DAC) evaluation Network, the United Nations evaluation Group (UNEG), the evaluation Cooperation Group (ECG), and the International Organization for Cooperation in evaluation (IOCE) a network drawn from the regional evaluation associations. NONIE was formed to promote quality Impact evaluation . NONIE fosters a program of Impact evaluation activities based on a common understanding of the meaning of Impact evaluation and approaches to conducting Impact evaluation . NONIE focuses on Impact evaluation and does not attempt to address wider monitoring and evaluation issues.

2 To this end NONIE aims to . Build an international collaborative research effort for high-quality and useful Impact evalu- ations as a means of improving Development effectiveness. Provide its members with opportunities for learning, collaboration, guidance, and support, leading to commissioning and carrying out Impact Evaluations . Develop a platform of resources to support Impact evaluation by member organizations. Impact Evaluations and Development Nonie Guidance on Impact evaluation Frans Leeuw Maastricht University Jos Vaessen Maastricht University and University of Antwerp 2009 NONIE The Network of Networks on Impact evaluation , Frans Leeuw, and Jos Vaessen c/o Independent evaluation Group 1818 H Street, NW. Washington, DC 20433. Internet: All rights reserved This volume is a product of the volume's authors, Frans Leeuw and Jos Vaessen, who were commis- sioned by NONIE. The findings, interpretations, and conclusions expressed in this volume are those of the authors and do not necessarily reflect the views of NONIE, its members, or other participating agencies.

3 NONIE does not guarantee the accuracy of the data included in this work and accepts no responsibility for any consequence of their use. Rights and Permissions The material in this publication is copyrighted. Copying and/or transmitting portions or all of this work without permission may be a violation of applicable law. NONIE encourages dissemination of its work and will normally grant permission to reproduce portions of the work promptly. All queries on rights and licenses, including subsidiary rights, should be addressed to NONIE, c/o IEG, 1818 H St., NW, Washington, DC, 20433, Cover: Pakistani girl reading. Photo by Curt Carnemark, courtesy of World Bank Photo Library. ISBN-10: 1-60244-120-0. ISBN-13: 978-1-60244-120-0. Printed on recycled paper Contents vii Acknowledgments ix Executive Summary xix Introduction 1 Part I Methodological and Conceptual Issues in Impact evaluation 3 1 Identify the (type and scope of the) intervention 3 The Impact evaluation landscape and the scope of Impact evaluation 4 Impact of what?

4 7 Impact on what? 9 Key message 11 2 Agree on what is valued 11 Stakeholder values in Impact evaluation 12 Intended versus unintended effects 12 Short-term versus long-term effects 12 The sustainability of effects 13 Key message 15 3 Carefully articulate the theories linking interventions to outcomes 15 Seeing interventions as theories: The black box and the contribution problem 15 Articulating intervention theories on Impact 17 Testing intervention theories on Impact 19 Key message 21 4 Address the attribution problem 21 The attribution problem 23 Quantitative methods addressing the attribution problem 29 Applicability of quantitative methods for addressing the attribution problem 31 Other approaches 34 Key message 35 5 Use a mixed-methods approach: The logic of the comparative advantages of methods 35 Different methodologies have comparative advantages in addressing particular concerns and needs 36 Advantages of combining different methods and sources of evidence 38 Average effect versus distribution of costs and benefits 39 Key message iii I m pa c t E va l u at i o n s a n d De v e l o p m e n t N o n ie G u i d a n c e O n I m pa c t E va l u at i o n 41 6 Build on existing knowledge relevant to the Impact of interventions 43 Key message 45 Part II Managing Impact Evaluations 47 7 Determine if an Impact evaluation is feasible and worth the cost 48 Key message 49 8 Start collecting data early 49 Timing of data collection 49 Data availability 51 Quality of the data 51 Dealing with data constraints 52 Key message 53 9 Front-end planning is important 53 Planning tools 53 Staffing and resources 54 The balance between independence and collaboration between evaluators and stakeholders 54

5 Ethical issues 55 Norms and standards 56 Ownership and capacity building 56 Key message 57 Appendices 59 1. Examples of diversity in Impact evaluation 61 2. The General Elimination Methodology as a basis for causal analysis 63 3. Overview of quantitative techniques of Impact evaluation 65 4. Technical aspects of quantitative Impact evaluation techniques 69 5. Evaluations using quantitative Impact evaluation approaches 71 6. Decision tree for selecting quantitative evaluation designs to deal with selection bias 73 7. Hierarchical modeling and other statistical approaches 75 8. Multi-site evaluation approaches 77 9. Methodological frameworks for assessing the effects of interventions, mainly based on quantitative methods 79 10. Where to find reviews and synthesis studies on mechanisms underlying processes of change 81 11. Evaluations based on qualitative and quantitative descriptive methods 101 12. Further information on review and synthesis approaches in Impact evaluation 105 13.

6 Basic education in Ghana 109 14. Hierarchy of quasi-experimental designs 111 15. International experts who contributed to the subgroup documents 113 Endnotes 117 References Boxes 7 Unpacking the aid chain 16 Social funds and government capacity: Competing theories iv CONT E NTS. 18 Social and behavioral mechanisms as heuristics for understanding processes of change and Impact 25 Using propensity scores to select a matched comparison group . The Vietnam Rural Roads Project 33 Participatory Impact monitoring in the context of the poverty reduction strategy process 39 Brief illustration of the logic of comparative advantages 42 Narrative review and synthesis study: Targeting and Impact of community-based Development initiatives 73 Impact of the Indonesian financial crisis on the poor: Partial equilibrium modeling and CGE modeling with microsimulation Figures xi Levels of intervention, programs, and policies and types of Impact xii Simple graphic of net Impact of an intervention 8 Levels of intervention, programs.

7 And policies and types of Impact 17 Basic intervention theory of a fictitious small business support project 22 Graphic display of the net Impact of an intervention 29 Regression discontinuity analysis 66 Estimation of the effect of class size with and without the inclusion of a variable correlated with class size 87 Final Impact assessment triangulation 92 Generic representation of a project's theory of change 93 Components of Impact evaluation framework 96 Project outputs and outcomes 99 Framework to establish contribution 99 Model linking outcome to Impact Tables 5 Aspects of complication in interventions 6 Aspects of complexity in interventions 26 Double difference and other designs 52 evaluation scenarios with time, data, and budget constraints 96 Project outcome 97 Change in key ecological attributes over time 97 Current threats to the global environment benefits v Acknowledgments This Guidance document could not have The Guidance document represents the views of existed without the numerous contributions the authors, who were commissioned by NONIE.

8 Of Network of Networks on Impact evaluation Given the fact that perspectives on the defini- (NONIE) members and others in terms of papers, tion, scope, and appropriate methods of Impact PowerPoint presentations, and suggestions. evaluation differ widely among practitioners and other stakeholders, the document should not be In particular, this Guidance document builds on two taken to represent the agreed positions of all of existing draft guidance documents, a document on the individual NONIE members. The network experimental and quasi-experimental approaches membership and the authors recognize that to Impact evaluation (NONIE subgroup 1, May 17, there is scope to develop the arguments further 2007) and a document on qualitative approaches in several key areas. to Impact evaluation (NONIE subgroup 2, January 9, 2008). A third draft document prepared by We would like to thank all of the above people NONIE members on the Impact evaluation of for their contributions to the process of writing macroeconomic policies and new aid modalities the Guidance document.

9 First, we thank the such as budget support is outside the scope of this authors of the subgroup documents for provid- Guidance document. The subgroup 1 document ing building blocks for this document. In was prepared mainly by Howard White and Antonie addition, we would like to thank the steering De Kemp. The subgroup 2 document, which was committee of this project, Andrew Warner, David somewhat broader in content than methodol- Todd, Zenda Ofir, and Henri Jorritsma, for their ogy, was coordinated by Sukai Prom-Jackson. pertinent suggestions. We also would like to The primary authors were Patricia Rogers, Zenda thank Antonie De Kemp for exchanging ideas Ofir, Sukai Prom-Jackson, and Christine Obester. on design questions. We are grateful to Patricia Case studies were prepared by Jocelyn Delarue, Rogers, the external peer reviewer, for providing Fabrizio Felloni, Divya Nair, Christine Obester, valuable input to this document. Our thanks also Lee Risby, Patricia Rogers, David Todd, and Rob go to Victoria Gunnarsson and Andrew Warner van den Berg.

10 The Development of this document from the NONIE secretariat for accompanying benefited extensively from a reference group of us throughout the whole process and providing international evaluators. excellent feedback. Nick York, Howard White, David Todd, Indran Nadoo, and John Mayne Whereas the two subgroup documents provided provided helpful insights in the final phase of the basis for the current Guidance document, the this project. We thank Arup Banerji for drafting purpose of the current document was to develop the executive summary. Comments from NONIE. a new structure that could accommodate some members were received at the Lisbon European of the diversity in perspectives on Impact evalua- evaluation Society Conference (October, 2008). tion. In addition, within this new structure, new and the Cairo Conference on Impact evaluation content was added where necessary to support key (March, 2009). Networks within NONIE, such as points. The process of developing this Guidance the International Organization for Cooperation in was supervised by a steering committee of NONIE evaluation and the European evaluation Society, members.


Related search queries