Guidelines for the Evaluation of Public Sector - ADB

1 Guidelines April 2016 Guidelines for the Evaluation of Public Sector Operations Independent Evaluation Department Asian Development Bank NOTE In this report, "$" refers to US dollars. Director General V. Thomas, Independent Evaluation Department (IED) Deputy Director General V. Salze-Lozac'h, IED Team leader W. Kolkma, Director, Independent Evaluation Division 1, IED Team members F. Ahmed, Lead Evaluation Specialist, IED J. Tominaga, Principal Evaluation Specialist, IED B. Nguyen, Senior Evaluation Specialist, IED J.

2 Dimayuga, Senior Evaluation Officer, IED F. De Guzman, Senior Evaluation Officer, IED V. Melo-Cabuang, Senior Evaluation Assistant, IED C. Malilay, Evaluation Assistant, IED ABBREVIATIONS ADB Asian Development Bank AER Annual Evaluation Review DMF design and monitoring framework EAL emergency assistance loan EAP Evaluation approach paper ECG Evaluation Cooperation Group EIRR economic internal rate of return FILP financial intermediary loan project FIRR financial internal rate of return IED Independent Evaluation Department MFF multitranche financing facility O&M operation and maintenance OECD Organisation for Economic Co-operation and Development OM Operations

3 Manual PAI Project Administration Instructions PBL policy-based loan PCR project (or program) completion report PPER project (or program) performance Evaluation report PPTA project preparatory technical assistance PVR project completion report validation RRP report and recommendation of the President SDP Sector development program TA technical assistance TCR technical assistance completion report TPER technical assistance performance Evaluation report WACC weighted average cost of capital CONTENTS Page I. INTRODUCTION 1 II.

4 APPROACH TO PROJECT Evaluation IN ADB 2 III. EVALUATING AND RATING PROJECT PERFORMANCE 4 A. Relevance 4 B. Effectiveness 9 C. Efficiency 14 D. Sustainability 19 IV. OTHER PERFORMANCE ASSESSMENTS 23 A. Development Impacts 23 B. ADB and Cofinancier Performance 27 C. Borrower and Executing Agency Performance 25 V. OVERALL ASSESSMENT 27 APPENDIXES: 1. Project Performance Evaluation Report 30 2. Validation of Project or Program Completion Reports 40 3. technical assistance Performance Evaluation Report 48 4. Project Performance Evaluation Report and Validation of Project Completion Report Processes 56 I.

5 INTRODUCTION 1. The se Guidelines provide the framework for evaluating completed Public Sector operations (projects) supported by the Asian Development Bank (ADB) through loans, grants, and technical They apply to both core and non-core Evaluation criteria for assessing such Assessments using the core criteria of relevance, effectiveness, efficiency, and sustainability contribute to the overall project performance assessment and the project s success rating. The non-core criteria of development impacts, ADB and cofinancier performance, and borrower and executing agency performance provide further depth to the Evaluation .

6 2. ADB s Independent Evaluation Department (IED) has formulated the se Guidelines in consultation with ADB s regional departments. 3 The Guidelines are in line with ADB s Evaluation policy4 and are designed to ensure that t he a ssessments and ratings standards used in self -evaluations undertaken by regional departments and independent evaluations by IED are consistent. The Guidelines aim to provide a basis for an update of ADB s project administration instructions (PAI) , which currently govern the preparation by regional departments of self- Evaluation project completion reports (PCRs) and technical assistance (TA) completion reports (TCRs).

7 5 Separate Guidelines govern ADB-financed private Sector operations (also called nonsovereign operations).6 3. The principles underpinning the Evaluation criteria are based on the 1991 Organisation for Economic Co-operation and Development Development assistance Committee (OECD-DAC) principles for Evaluation of development The approach also incorporates good practice standards for evaluating Public Sector operations as endorsed by t he Evaluation Cooperation Group (ECG) of the multilateral development The Guidelines are consistent with ADB s Updated Design and Monitoring Framework (DMF) Guidelines , 4.

8 Chapter II describes ADB s approach to project Evaluation and how self-evaluations and independent evaluations complement each other. Chapter III explains the Evaluation criteria, and the rating standards for each core Evaluation criterion. The Guidelines apply to all project modalities, but special guidance is provided for multitranche financing facilities (MFFs), Sector projects, Sector development programs (SDPs), policy-based loans (PBLs), 1 The Guidelines replace IED. 2006. Guidelines for Preparing Performance Evaluation Reports for Public Sector Operations.

9 January (updated March 2013). Manila: ADB; and IED. 2014. Guidelines for the Validation of Project/Program Completion Reports of the Independent Evaluation Department. Manila: ADB. 2 Projects in these Guidelines refer to Public Sector operations approved by ADB and may cover various types and modalities, including investment projects, Sector projects, program loans, Sector development programs (SDPs), multitranche financing facilities (MFFs), results-based lending, emergency assistance loan (EAL) projects, and technical assistance (TA) operations. 3 Central and West Asia Department, East Asia Department, Pacific Department, South Asia Department, and Southeast Asia Department.

10 4 ADB. 2008. Review of the Independence and Effectiveness of the Operations Evaluation Department. Policy Paper. Manila. 5 Available at: 6 IED. 2014. Guidelines for the Preparation of Project Performance Evaluation Reports on Nonsovereign Operations. Manila: ADB. 7 OECD-DAC. 1991. Principles for Evaluation of Development assistance . Paris. 8 ECG. 2011. Good Practice Standards for Public Sector Operations. Washington, DC; ECG. 2012. Big Book on Evaluation Good Practice Standards. Manila. 9 ADB. 2015. Updated Design and Monitoring Framework Guidelines .