Equity-focused evaluations - EvalPartners

1 How to design and manageEquity- focused evaluationsHow to design and manage Equity-focused evaluationsUNICEF evaluation Office3 United Nations PlazaNew York, NY 10017, USA 011 How to design and manageEquity- focused evaluationsThe evaluation Working Papers (EWP) are documents that present strategic evaluation findings, lessons learned and innovative approaches and methodologies. We would like to encourage proposals for relevant papers which could be published in the next EWP issues. Papers can be prepared by UN staff and by additional information and details please contact Marco Segone, Systemic strengthening, UNICEF evaluation Office, ISSUE # 1: New trends in development evaluation . Published jointly with IPEN. # 2: Bridging the gap: The role of monitoring and evaluation in evidence-based policy making.

2 Published by UNICEF in partnership with the World Bank, IDEAS, DevInfo and MICS, # 3: Country-led monitoring and evaluation systems. Better evidence, better policies, better development results. Published by UNICEF in partnership with the World Bank, IDEAS, IOCE, unece , DevInfo and MICS, # 4: Country-led monitoring and evaluation systems. Watch and listen international keynote speakers. DVD published by UNICEF in partnership with IDEAS, IOCE, WFP, OECD/DAC Network on development evaluation and DevInfo, 2009. ISSUE #5: From policies to results: Developing capacities for country monitoring and evaluation systems. Published by UNICEF in partnership with DevInfo, IDEAS, ILO, IOCE, The World Bank, UNDP, UNIFEM, WFP, :The opinions expressed are the personal thinking of the contributors and do not necessarily reflect the policies or views of UNICEF or any other organization involved or named in this publication.

3 The text has not been edited to official publication standards and UNICEF and partner organizations accept no responsibility for from this publication may be freely reproduced with due byPhoto Credits: UNICEF/ NYHQ2011-1158/Kate HoltHow to design and manageEquity- focused evaluationsMichael Bamberger and Marco SegoneHow to design and manageEquity- focused evaluationsContentsEditorial .. 1 equity and Equity-focused evaluationsSection 1. What is equity and why does it matter? .. The challenge of achieving equitable development results for children .. What is equity ? .. Why does equity matter? .. Why is equity now so urgent ? .. What are the implications for the evaluation function? ..8 Section 2. Defining Equity-focused evaluations .. What is an Equity-focused evaluation ?

4 Why are Equity-focused evaluations needed? .. Purposes of Equity-focused evaluations .. Empowering worst-off groups, including children, through Equity-focused evaluation processes ..12 Part 2 Managing Equity-focused evaluationsSection 3. Preparing for the evaluation .. Determining the evaluability of the equity dimensions of the intervention .. Identifying evaluation stakeholders, including worst-off groups .. Identifying the intended use and intended users .. Identifying potential challenges in promoting and implementing Equity-focused evaluations ..29 Section 4. Preparing the evaluation Terms of Defining the scope and purpose of the evaluation .. Framing the evaluation Selecting a technically-strong and culturally-sensitive evaluation team.

5 39 How to design and manageEquity- focused evaluationsSection 5. Designing the evaluation .. Selecting the appropriate evaluation framework ..41A. Theory-based Equity-focused evaluation ..41 Basic components of a programme theory of change ..43 Refinements to the basic logic model ..45B. The bottleneck analysis framework ..45 Use of services by worst-off groups ..46 Supply side factors ..49 Demand side factors ..49 Contextual factors .. Selecting the appropriate evaluation design ..50A. Mixed methods designs ..51 Types of mixed method designs ..52 Triangulation: a powerful tool for assessing validity and for deepening understanding ..56B. Attribution, contribution and the importance of the counterfactual ..58C. Equity-focused evaluation at the policy level.

6 61 Systems approaches to evaluation ..62 Unpacking complex policies ..66 Pipeline designs ..66 Policy gap analysis ..67 Using other countries or sectors as the comparison group ..68 Concept mapping ..69 Portfolio analysis ..69D. Equity-focused evaluations at the project and programme levels ..70 Conventional quantitative impact evaluation designs ..70 Estimating project impacts using non-experimental designs (NEDs) ..72 Potentially strong non-experimental designs ..73E. Feasibility analysis .. Collecting and analyzing data ..74A. Collecting data and analyzing contextual factors ..75B. Collecting and analyzing surveys and qualitative information to understand knowledge, attitude and practice ..78C. Collecting and analyzing information on the quality of services delivered and the satisfaction of citizens.

7 81 Citizen report Carrying out cost-effectiveness studies to compare costs and results of alternative intervention ..82 Cost-effectiveness analysis ..82 Public expenditure tracking (PETS) studies ..83 Public expenditure Benefit Incidence Analysis (BIA) ..84 Section 6. Utilizing the evaluation .. Preparing the evaluation report and alternative forms of reporting .. Disseminating the evaluation and preparing a Management Response ..87 Section 7. Conducting Equity-focused evaluations under real-world constraints .. Understanding the evaluation scenario .. Reconstructing the programme theory when it is non-existent or is very weak .. Conducting credible Equity-focused evaluations when working under budget and time constraints .. Reconstructing baseline data when the evaluation is not commissioned until late in the implementation cycle.

8 Conducting Equity-focused evaluations in countries where government is not supportive of an equity focus ..96 Section 8. Case studies of UNICEF-supported Equity-focused evaluations ..98 References ..102 Editorial & AcknowledgementsEditorial ..XAcknowledgements ..XIIE ditorial & AcknowledgementIXHow to design and manageEquity- focused evaluationsXEDITORIALThe push for a stronger focus on equity in human development is gathering momentum at the international level. Its premise is increasingly supported by United Nations reports and strategies as well as by independent analysis and donors. More and more national policies and international alliances are focusing on achiev-ing equitable development results for children.

9 While this is the right way to go, it poses important challenges and opportunities for the evaluation function. How can one strengthen the capac-ity of Governments, organizations and communities to evaluate the effect of interventions on equitable outcomes for children? What are the evaluation questions to ensure interventions are relevant and are having an impact in decreasing inequity, are achieving equi-table results, and are efficient and sustainable? What are the meth-odological implications in designing, conducting, managing and using Equity-focused evaluations ? This document represents a first attempt to address these questions. The purpose of this document is to provide guidance to UNICEF Country Offices, their partners and Governmental and Civil Society stakeholders, including communities and worst-off groups, on how to design and manage evaluations to assess the contribution of pol-icies, programmes and projects to equitable development results for children.

10 The document is complemented by a resource centre on Equity-focused evaluation available at , in which readers can access in more detail the methodological mate-rial presented in this document. Therefore, this document should be used together with that electronic resource centre. This document is divided into two parts. Part I discusses equity and Equity-focused evaluations . Section 1 defines equity , why equity matters and why equity is so urgent now. Section 2 defines Equity-focused evaluations , explaining what their purpose should be and potential challenges in their promotion and implementation. Part II explains how to manage Equity-focused evaluations , explaining the key issues to take into account when preparing for Equity-focused evaluations (section 3) and developing the Terms of Reference (section 4); designing the evaluation , including iden-tifying the appropriate evaluation framework, evaluation design and appropriate methods to collect data (section 5); and utilizing the evaluation (section 6).