Measuring Value for Money - OECD

1 EVALUATION REPORT EV645. November 2003. reports Measuring Value for Money ? An independent review of DFID's Value for Money (VFM) Indicator, Public Service Agreement 2003-2006. By Derek Poate and Christopher Barnett DEPARTMENT FOR INTERNATIONAL DEVELOPMENT. EVALUATION REPORT EV645. Measuring Value FOR Money ? An Independent Review of DFID's Value for Money (VFM) Indicator, Public Service Agreement 2003 2006. Derek Poate and Christopher Barnett The opinions expressed in this report are those of the authors and do not necessarily represent the views of the Department for International Development. Preface PREFACE. This study was undertaken as part of the programme of independent evaluation studies commissioned by the Evaluation Department of the Department for International Development (DFID). The purpose of these studies is to improve the quality of development activities by providing evidence of what makes for effective development. Lessons learned from evaluation can be applied to strengthen current and future policies and programmes.

2 Evaluation of development assistance provided by DFID also helps to strengthen DFID's accountability. DFID's Evaluation Department (EvD) is independent of the spending divisions in DFID, and reports to DFID's Management Board through the Director General (Corporate Performance and Knowledge Sharing). Each year, Evaluation Department commissions a number of evaluations which rigorously examine the design, implementation and results of selected DFID policies and programmes. The findings and lessons from each evaluation are published. This report presents findings and conclusions from an independent assessment of the Value for Money indicator, which constitutes an important element of DFID's Public Service Agreement. The Public Service Agreement sets out DFID's corporate objectives and targets, which define the contribution DFID aims to make towards achieving the global Millennium Development Goals. Unlike most of the PSA objectives, which focus on progress towards the Millennium Development Goals, the Value for Money indicator is directly linked to the effectiveness of activities funded by DFID.

3 It is therefore an important measure of DFID's operational performance. The study was conducted by Derek Poate and Christopher Barnett of ITAD Ltd. This was undertaken largely a desk study, supplemented by interviews and internal consultations. Further work is planned to test some of the conclusions. DFID's management response is included at Annex 6. Annex 7 presents data on scores for achievement and risks at the time of going to press. The study was managed by Joanne Asquith and edited by Caryn McLean. Colin Kirk Head, Evaluation Department November 2003. i Preface ii 7. Table of Contents CONTENTS. List of Acronyms and Abbreviations v 1. Key Findings and Conclusions 1. 2. Introduction 2. 3. The VFM Indicator 3. Data Capture by PRISM 4. Measuring Value for Money 4. The VFM Baseline 5. Frequencies of Overall Scores 6. 4. Guidance and Internal Procedures 8. Guidance on Assessment of Riskiness of an Operation 8. Guidance on Scoring Operations 8. The Monitoring and Reporting System 10.

4 5. Compliance 11. 6. Trends 13. Trends by Risk Categories 13. Regional Trends 14. Trends by Sector 18. Instruments (SWAps, DBS) 18. 7. Evidence and Practice to Support Scores 20. Documentary Evidence 20. Review Methodologies 21. Quality of Supporting Evidence 22. The Review Process 23. 8. Conclusions and Key Issues 26. What Does the VFM Measure? 26. Are the Measurements Consistent Across the Portfolio? 26. Are the Measurements Supported with Evidence? 26. What is the Quality of this Evidence? 27. Does the DFID System Effectively Support Quality and Consistency? 27. Ways Forward 27. iii Annex 1: Summary of the World Bank System 29. Annex 2: Review of OPR Documentation in Support of PRISM VFM Scores 31. Annex 3: Key Informant Interviews with DFID Country Offices 34. Annex 4: References and People Interviewed 37. Annex 5: Terms of Reference 38. Annex 6: DFID's Management Response 43. Annex 7: Data on scores for achievement and risk as at November 2003 45. iv Abbreviations and Acronyms ABBREVIATIONS AND ACRONYMS.

5 APPR Annual Plan and Performance Review ARDE Annual Review of Development Effectiveness ARPP Annual Review of Portfolio Performance CAP Country Assistance Plans CHAD Conflict and Humanitarian Affairs Department DAC Development Assistance Committee DBS Direct Budgetary Support DER Development Effectiveness Report DFID Department for International Development EAPD Eastern Asia and Pacific Department EEWH Eastern Europe and Western Hemisphere Department EMAD Europe, Middle East and Americas Division FAO Food and Agriculture Organisation of the United Nations ICR Implementation Completion Report MIS Management Information System MTR Mid-Term Review NAO National Audit Office OECD Organisation for Economic Co-operation and Development OI Office Instructions OPR Output to Purpose Reviews OVI Objectively Verifiable Indicators PAD Project Appraisal Document PCR Project Completion Reports PPCM Programme & Project Cycle Management PRISM Performance Reporting Information System for Management PSA Public Service Agreements PSR Project Status Report QAE Quality at Entry Assessment QAG Quality Assurance Group QER Quality Enhancement Review QQT Quantity, Quality and Timing QSR Quality of Supervision Review SDA Service Delivery Agreement SWAps Sector Wide Approaches UNESCO United Nations Educational, Scientific and Cultural Organisation VFM Value for Money v vi Key Findings and Conclusions 1.

6 KEY FINDINGS AND CONCLUSIONS. The VFM indicator captures a large proportion of DFID's bilateral expenditure but there are important concerns, some of which may need to be addressed for subsequent Public Service Agreements (PSAs). Key concerns include: The criteria for eligibility excludes 84% of projects1 and it is not clear how important these operations are for DFID's poverty objectives, nor the extent to which the same types of projects are excluded in each region. There are apparent regional anomalies in scoring, especially across risk categories. The current system is not particularly well suited to Measuring the performance of Sector Wide Approaches (SWAps) and Direct Budgetary Support (DBS). Supporting evidence and quality of Output to Purpose Reviews (OPR) is reasonably thorough, with innovative approaches being developed in Country Offices, but: Only a small proportion of review documents are available from the Performance Reporting Information System for Management (PRISM)2.

7 There are many gaps in the supporting narrative3. The style and content of the text varies widely. The scores in the system show a tendency to clustering and clear anomalies in the treatment of risk. This may arise from the nature of the scoring system or the limited guidance given to staff. At present, processes that are thorough in each country may yield results that are contradictory in aggregate. There is evidence that the scoring system has settled down over a relatively short time. Despite the small samples studied in this review, the issues that have emerged are important if there is to be confidence in the robustness of the indicator. Summaries of the main findings can be found in boxed text at the end of each section, and a detailed list of the key issues is provided in section 8 (Conclusions and Key Issues). 1. This is according to an analysis of PRISM data for operational projects, undertaken in January 2003. However, only 12% of operational projects are excluded by commitment Value .

8 2. Only a small proportion of the original review documents (OPRs, Project Completion Reports (PCRs), etc.) are uploaded onto PRISM. This is distinct from the summary information (scores, justifications, etc.) which is entered directly onto the system. 3. It is not presently mandatory to complete all the text fields. 1. Introduction 2. INTRODUCTION. This study examines the use of scoring by DFID to report performance against the PSA Value for Money (VFM) indicator. The indicator4 has two parts: an objective for increasing overall Value for Money and a measure of progress against it. Value for Money under the 2003 2006 PSA is achieved in DFID when: 1. The proportion of DFID's bilateral programme going to low income countries increases from 78%. to 90%; and 2. There is a sustained increase in the index of DFID's bilateral projects' evaluated success. The first part is measured from records of new commitments. The second part is based on scored assessments of performance by DFID staff.

9 This latter part is the subject of the review. Three recent studies have drawn attention to limitations in current arrangements for project and programme monitoring. A peer review by the Organisation for Economic Co-operation and Development/Development Assistance Committee (OECD/DAC) in 2001 found that monitoring and evaluation of portfolio performance had little ownership by DFID staff, resulting in low compliance The study identified a need to reconcile the targets in the PSA with DFID's longer-term development objectives. The DFID Development Effectiveness Report raised concerns about the use of self-assessment, concluding that operational staff need more support to achieve adequate coverage, consistency, timeliness and quality of A National Audit Office (NAO) review in 2002 found strengths in DFID's approach to performance management but called for a stronger focus and more direct relationship with performance management in order to influence resource allocation and choice of Most of the PSA indicators cover progress towards key targets linked to the Millennium Development Goals.

10 Owing to significant lags between current spending and development outcomes, the Value for Money indicator is the only PSA indicator that reflects current operational performance. As such, it is a critical measure for DFID. This study starts by reviewing the coverage and content of the VFM indicator. Next, we examine guidance to staff and internal procedures. Compliance is summarised, followed by a review of trends in performance scores with observations on current operational practice in reporting from a small sample of telephone interviews. The study ends with a summary of key issues and conclusions. 4. See DFID Public Service Agreement, 2003 2006; and DFID SDA, 2003 2006. 5. DAC Journal, Volume 2, No. 4. United Kingdom: Development Co-operation Review Main Findings and Recommendations 6. DFID (2002) Development Effectiveness Report. 7. NAO (2002) Department for International Development. Performance Management Helping to Reduce World Poverty. 2. The VFM Indicator 3.