Ten Steps to a Results- Based Monitoring - OECD.org

1 THE WORLD BANKJody Zall Kusek Ray C. RistTen StepsTen Stepsto aResults-BasedMonitoringandEvaluationSys temto aResults-BasedMonitoringandEvaluationSys temA Handbook for Development PractitionersTen Stepsto aResults- Based Monitoring andEvaluationSystemA Handbook for Development PractitionersJody Zall Kusek Ray C. RistTHE WORLD BANKW ashington, Handbook for Development PractitionersTen Steps to aResults- Based MonitoringandEvaluationSystemTen Stepsto aResults- Based Monitoring andEvaluationSystem 2004 The International Bank for Reconstruction and Development / The World Bank1818 H Street, NWWashington, DC 20433 Telephone 202-473-1000 Internet rights findings, interpretations, and conclusions expressed herein are those ofthe author(s) and do not necessarily reflect the views of the Board ofExecutive Directors of the World Bank or the governments they World Bank does not guarantee the accuracy of the data included inthis work. The boundaries, colors, denominations, and other informationshown on any map in this work do not imply any judgment on the part ofthe World Bank concerning the legal status of any territory or the endorse-ment or acceptance of such and PermissionsThe material in this work is copyrighted.

2 Copying and/or transmitting portions or all of this work without permission may be a violation of applicable law. The World Bank encourages dissemination of its work andwill normally grant permission permission to photocopy or reprint any part of this work, please send arequest with complete information to the Copyright Clearance Center, Inc.,222 Rosewood Drive, Danvers, MA 01923, USA, telephone 978-750-8400,fax 978-750-4470, other queries on rights and licenses, including subsidiary rights, should be addressed to the Office of the Publisher, World Bank, 1818 HStreet NW, Washington, DC 20433, USA, fax 202-522-2422, e-mail of Congress Cataloging-in-Publication DataKusek, Jody Zall, 1952 Ten Steps to a results - Based Monitoring and evaluation system : a hand-book for development practitioners / Jody Zall Kusek and Ray C. bibliographical references and 0-8213-5823-51. Government productivity Developing countries Performance standards Developing countries evaluation .

3 3. Totalquality management in government Developing countries Public administration Developing countries evaluation . I. Rist, RayC. II. 5 dc222004045527 Preface xiAbout the Authors xivIntroductionBuilding a results - Based Monitoring and evaluation System 1 Part I New Challenges in Public Sector Management 2 International and External Initiatives and Forces for Change 3 National Poverty Reduction Strategy Approach 8 Internal Initiatives and Forces for Change 10 Part 2 Results- BasedM&E APowerfulPublicManagementTool 11 Monitoring and evaluation : What Is It All About? 12 Key Features of Traditional Implementation-Focused and results - Based M&E Systems 15 Many Applications for results - Based M&E 17 Political and Technical Challenges to Building a results - Based M&E System 20 Introducing the 10-Step Model for Building a results - Based M&E System 23 Where to Begin: Whole-of-Government, Enclave, or Mixed Approach 24 Part 3 M&E Experience in Developed and Developing Countries 27M&E Experience in Developed and OECD Countries 27 Special M&E Challenges Facing Developing Countries 32M&E Experience in Developing Countries 35 Chapter 1 Step 1: Conducting a Readiness Assessment 39 Part 1 Why Do a Readiness Assessment?

4 40 Part 2 The Readiness Assessment: Eight Key Questions 43 Part 3 Readiness Assessments in Developing Countries: Bangladesh, Egypt, and Romania 48 ContentsPart 4 Lessons Learned 49 Chapter 2 Step 2: Agreeing on Outcomes to Monitor and Evaluate 56 The Importance of Outcomes 56 Issues to Consider in Choosing Outcomes to Monitor and Evaluate 57 The Importance of Building a Participatory and Consultative Process involving Main Stakeholders 58 The Overall Process of Setting and Agreeing upon Outcomes 59 Examples and Possible Approaches 61 Chapter 3 Step 3: Selecting Key Performance Indicators to Monitor Outcomes 65 Indicators Are Required for All Levels of results - Based M&E Systems 66 Translating Outcomes into Outcome Indicators 66 The CREAM of Good Performance Indicators 68 The Use of Proxy Indicators 70 The Pros and Cons of Using Predesigned Indicators 72 Constructing Indicators 74 Setting Indicators: Experience in Developing Countries 75 Chapter 4 Step 4: Setting Baselines and Gathering Data on Indicators 80 Establishing Baseline Data on Indicators 81 Building Baseline Information 82 Identifying Data Sources for Indicators 83 Designing and Comparing Data Collection Methods 84 The Importance of Conducting Pilots 86 Data Collection: Two Developing Country Experiences 89 Chapter 5 Step 5.

5 Planning for Improvement Selecting results Targets 90 Definition of Targets 90 Factors to Consider When Selecting Performance Indicator Targets 91 Examples of Targets Related to Development Issues 93 The Overall Performance- Based Framework 94 Chapter 6 Step 6: Monitoring for results 96 Part 1 Key Types and Levels of Monitoring 98 Links between Implementation Monitoring and results Monitoring 101 Part 2 Key Principles in Building a Monitoring System 103 Achieving results through Partnership 105 Needs of Every results - Based Monitoring System 106 The Data Quality Triangle: Reliability, Validity, and Timeliness 108viContentsAnalyzing Performance Data 111 Pretesting Data Collection Instruments and Procedures 112 Chapter 7 Step 7: The "E" in M&E Using evaluation Information toSupport a results - Based Management System 113 Uses of evaluation 115 The Timing of Evaluations 118 Types of Evaluations 121 Characteristics of Quality Evaluations 126 Examples of evaluation at the Policy, Program, and Project Levels 128 Chapter 8 Step 8: Reporting the Findings 129 The Uses of Monitoring and evaluation Findings 130 Know and Target the Audience 130 Presentation of Performance Data in Clear and Understandable Form 132 What Happens If the M&E System Produces Bad Performance News?

6 136 Chapter 9 Step 9: Using the Findings 138 Uses of Performance Findings 138 Additional Benefits of Using Findings: Feedback, Knowledge, and Learning 140 Strategies for Sharing Information 146 Chapter 10 Step 10: Sustaining the M&E System within the Organization 151 Six Critical Components of Sustaining results - Based M&E Systems 152 The Importance of Incentives and Disincentives in Sustaining M&E Systems 155 Possible Problems in Sustaining results - Based M&E Systems 155 Validating and Evaluating M&E Systems and Information 160M&E: Stimulating Positive Cultural Change in Governments and Organizations 160 Last Reminders 160 Chapter 11 Making results - Based M&E Work for You and Your Organization 162 Why results - Based M&E? 162 How to Create results - Based M&E Systems 165 Summing Up 170 Contents viiAnnexes:Annex I:Assessing Performance- Based Monitoring and evaluation Capacity: An Assessment Survey for Countries, Development Institutions, and Their Partners 174 Annex II:Readiness Assessment: Toward results - Based Monitoring and evaluation in Egypt 178 Annex III:Millennium Development Goals (MDGs): List of Goals and Targets 200 Annex IV:National evaluation Policy for Sri Lanka: Sri Lanka evaluation Association (SLEva) jointly with the Ministry of Policy Development and Implementation 204 Annex V: Andhra Pradesh (India) Performance Accountability Act2003: (Draft Act) (APPAC Act of 2003) 211 Annex VI:Glossary.

7 OECD Glossary of Key Terms in evaluation and results - Based Management (2002) 223 Notes 230 References 231 Useful Web Sites 235 Additional Reading 236 Index Millennium Development Goals of Millennium Development Goal, Targets, and Indicators Transparency International The Power of Measuring results Features of Implementation Monitoring versus results Monitoring Australia s Whole-of-Government Model : Lagging Behind but Now Speeding Ahead in Governmental Reform Republic of Korea: Well on the Road to M&E : Outcome- Based Budgeting, Nation Building, and Global Competitiveness Uganda and Poverty Reduction Impetus toward M&E The Case of Bangladesh Building from the Bottom Up The Case of Egypt Slow, Systematic Moves toward M&E Case of Romania Some Opportunities to Move toward M&E Indicator Dilemmas The Africa Region s Core Welfare Indicators Sri Lanka s National evaluation Policy Albania s Three-Year Action Plan and Project Level results Indicators: An Example from the Irrigation Sector Outcome: Increased Participation of Farmers in Local Markets Albania s Strategy for Strengthening Data Collection Capacity Lebanon.

8 Joining the IMF Data System Examples of Development Targets results Monitoring in Mexico results Monitoring in Brazil Provides Information on Strategy, Operations, andLearning Ten Uses of results Findings Performance Data to Track and Reduce Crime in New YorkCity Department of Labor An Organization with a Mature, Functioning results - Based M&E System of Improving Conditions for evaluation - Based Learning inGerman Aid Agencies Obstacles to Learning for Learning, Knowledge Building, and Greater Use of Performance Findings Active and Passive Approaches to Using results Information Canadian Government Performance Reports to Parliament Citizen s Charter in the United Kingdom Government Performance and results Act of 1993 for Staff Incentives That Encourage Learning-Oriented, Participatory M&E for Staff Disincentives That Hinder Learning-Oriented, Participatory M&E evaluation Culture and Collaborative Partnerships Help BuildAgency Capacity Roles of results - Based Monitoring and evaluation Building Baseline Information Comparison of Major Data Collection Methods Outcomes Reporting Format: Actual Outcomes versus Targets Table for Reporting Descriptive Data.

9 Gender Differences inVoting evaluation Capacity Development and Institutionalization KeyIssues Addressed in Colombia, China, and Indonesia Illustrative Logic Model for One National Development Goal Ten Steps to Designing, Building, and Sustaining a Results- BasedMonitoring and evaluation System Conducting a Readiness Assessment Agreeing on Outcomes to Monitor and Evaluate Developing Outcome Statements Outcome Statements Derived from Identified Problems or Issues How NOT to Construct Outcome Statements Developing Outcomes for One Policy Area 64 Contents Selecting Key Indicators to Monitor Outcomes Developing a Set of Outcome Indicators for a Policy Area Checklist for Assessing Proposed Indicators Baseline Data on Indicators Where Are We Today? Developing Baseline Data for One Policy Area Data Collection Methods Planning for Improvement Selecting results Targets Identifying Desired Level of results Requires Selecting PerformanceTarge ts Developing Targets for One Policy Area Monitoring for results Sample Gant Chart results - Based Monitoring Examples of results Monitoring between Implementation Monitoring and results Monitoring Linking Implementation Monitoring to results Monitoring Achieving results through Partnership Monitoring System Needs Ownership, Management,Maintenance, and Credibility Key Criteria for Collecting Quality Performance Data The Data Quality Triangle: Reliability The Data Quality Triangle: Validity The Data Quality Triangle.

10 Timeliness Analyzing results Data The Role of Evaluations Using evaluation to Explain Performance Divergence evaluation to Determine the Impacts of Design and Implementation on Outcome Seven Types of Evaluations Characteristics of Quality Evaluations Examples of evaluation Reporting Findings of Graphic Excellence and Sample Charts for DisplayingInformation Using Findings Sustaining the M&E System within the Organization 151xContentsAn effective state is essential to achieving sustainable socioeconomicdevelopment. With the advent of globalization, there are growingpressures on governments and organizations around the world to bemore responsive to the demands of internal and external stakeholdersfor good governance, accountability and transparency, greater devel-opment effectiveness, and delivery of tangible results . Governments,parliaments, citizens, the private sector, nongovernmental organiza-tions (NGOs), civil society, international organizations, and donorsare among the stakeholders interested in better performance.