MONITORING AND EVALUATION: AN OVERVIEW …

1 MONITORING AND. evaluation : AN. OVERVIEW . NODDY JINABHAI. Planning, MONITORING and evaluation for Results-Based Strategies Based on GHAP. Managing for Results Approach . By Rosal a Rodriguez-Garc a and Jody Z. Kusek, GHAP, The World Bank, March 1, 2007, with valuable contributions from GHAP and WBI colleagues at the World Bank. Module Outcomes AIM : Utilize M&E data to improve decision making and implementation performance SKILLS: By the end of the course, participants should be able to z Integrate the M&E System in the Strategic Action Plans z Understand the rationale for effective M&E based on the UNAIDS. Tool Kit and the Results Based M&E System z Explain the elements of a comprehensive UNAIDS M&E system, z Understand elements of an M&E logical framework, with key indicators, targets and data sources z Contribute to an M&E operational plan and budget, z Contribute to a tailored, functional M&E system z Stress the epidemiological and economic foundations of M&E.

2 Z Identify and disseminate proven, evidence-based HIV/AIDS. practices Benefits of MONITORING & evaluation MONITORING and evaluation (M&E) helps programme implementers to: determine the extent to which the programme/ project is on track and to make any needed corrections accordingly;. make informed decisions regarding operations management and service delivery;. ensure the most effective and efficient use of resources;. evaluate the extent to which the programme/ project is having or has had the desired impact. Difference between MONITORING and evaluation MONITORING means tracking the key elements of programme performance on a regular basis (inputs, activities, results). In contrast, evaluation is the episodic assessment of the change in targeted results that can be attributed to the programme/ project intervention, or the analysis of inputs and activities to determine their contribution to results.

3 STRATEGY RESULTS CYCLE. Formulate/Revise HIV/AIDS Strategy Analyze HIV/AIDS. Evaluate and National Changes in Response Data Outcomes Annual Action Plan, costing, funding Identify Key National Implementation Outcomes & Priorities & process MONITORING (indicators, resource Select Interventions tracking). Resource Needs, Select Funding Principal Program Strategies Planning Process based on the Strategy Results Cycle Purpose Methodology/process Key persons & stakeholders Links with other strategies Finalize Roadmap/timetable Disseminate Use plans Monitor results HIV/AIDS situation Evaluate impacts Current response Capacity . available & needed Data sources Data collection Analyze & use Expected results Strategic programs Workplan Responsibilities Resources Source: Adapted from P. Lumumba Osewe, WBI, World Bank. Presentation in St. Lucia, 2006. Global HIV/AIDS M&E Framework & Illustrative Data Assessment Inputs Activities Outputs Outcomes Impact & Planning (Resources) (Interventions, (Immediate (Intermediate (Long-term Services) Effects) Effects) Effects).

4 Situation Analysis Staff Trainings # Staff Trained Provider Behavior Social Norms Response Analysis Funds Services # Condoms Distributed Risk Behavior HIV prevalence Stakeholder Needs Materials Education # Clients Served Service Use STI Incidence Resource Analysis Facilities Treatment # Tests Conducted Clinical Outcomes AIDS Morbidity Collaboration plans Supplies Interventions Quality of Life AIDS Mortality Econ. Impact Program Population-based Biological, Program-based Data Development Behavioral & Social Data Data In addition to MONITORING programs, conduct process and outcome evaluations Source: D. Rugg, G. Peersman, G. & M. Carael in Advances in Global HIV/AIDS MONITORING and evaluation : New Directions in evaluation , UNAIDS, 2004. The Logical Approach of the Results Chain Goal Long-term, widespread improvement (Long-term in society Impacts). Big picture (country longer term Strategy strategy).

5 RESULTS PLANNING. Effects or behavior changes resulting Outcomes from a strategic program Outputs Products and services that Programming need to be delivered to achieve the expected outcomes Activities What actually was done with the available resources to produce the intended outputs Inputs Critical resources (expertise, equipment, supplies) needed to Source: Binnenen and Kusek&Rist, 2004. implement the planned activities Apply the Results Chain to HIV/AIDS Strategy Planning For a Results-based Strategy ask: 1. What is the aim of longer term improvement in HIV/AIDS? (national goal) = impact and outcomes 2. What improvements are aimed at by the end of the strategy period? = intermediate outcomes 3. How will one know success - which outcome targets need to be met? = intermediate outcomes and final outputs 4. What strategic programs and critical interventions should be the focus of the national response?

6 = outputs 5. What financial, human, material, and technical resources are needed? = inputs Note: Refer to definitions in previous slide Applying the Logic of the Results Chain to HIV/AIDS. IMPACTS Reduced HIV Incidence and Prevalence. HIV prevalence is the bedrock of surveillance, MONITORING , and evaluation (longer-term goal). gy te ra Increased Coverage. Access and use St OUTCOMES of services and behavior changes (strategy objectives and results). Products and Services. First ng level results needed to achieve the OUTPUTS. mi outcomes (program outputs). ram og Critical resources. Includes Pr money, people, equipment, supplies INPUTS and know how (program inputs). Measuring the Performance of the National HIV/AIDS Strategy Once the results and outcomes of the strategy are identified, how should performance be measured? z Performance of the HIV/AIDS strategy is measured through outcome indicators.

7 Z For each indicator identify: - a baseline (what is the value now), and - performance targets (what value should one aim to achieve) over the time period of the strategy. Information Triangle INTERNATIONAL. OVERVIEW , TRENDS, ANNUAL NATIONAL. COMPARISONS REPORT PROVINCIAL. INFORMATION MONTHLY REPORT DISTRICT /. AGGREGATED DATA WEEKLY TALLY INSTITUTION. DETAILED DAILY PATIENT RECORDS CLINIC. DATA. INFO. Surveillance & Research Surveillance comprises: biological, behavioural and social impact surveillance. WHO/UNAIDS/CDC support ensures sound antenatal biological surveillance in countries with generalized HIV/AIDS epidemics. In concentrated HIV epidemics, efforts to strengthen serosurveillance among high-risk groups are in progress. Surveillance should be complemented by essential research, including epidemiological, evaluation and social impact research. First Generation HIV Surveillance is Necessary to Assess the Evolution of the Epidemic Public Heath Types of Surveillance: surveillance helps: - HIV patient case reports - define the nature and from clinical settings extent of HIV infection, - Sentinel sero- and surveillance which uses - assess the impact that blood samples programs and services have on the HIV.

8 Problem. Surveillance is the bedrock of public health MONITORING Source: Compiled from UNAIDS 2006 Report on the Global AIDS Epidemic, pp 278-279. First Generation HIV Surveillance is Necessary but not Sufficient to Assess the Evolution of the Epidemic z It records infections that have already taken place, but does not give early warning of the potential for infection z Second Generation HIV Surveillance expands the scope to include: z At risk behaviors (unprotected sex). - Biological markers (STDs). - Knowledge (or lack of) of how HIV is transmitted - Behavioral and other studies (DHS). Source: Compiled from UNAIDS 2006 Report on the Global AIDS Epidemic, pp 278-279. Evaluate Changes in Outcomes Evaluate Changes in Outcomes RELEVANCE OF EVALUATIONS. EVALUATIONS FOR IMPROVED PERFORMANCE. The extent to which the objectives of a programme/ project are consistent with country and direct beneficiaries'.

9 Needs. To what extent were the objectives of the programme still valid? Relevance of the programme to the country's needs and target group. Are the activities of the programme consistent with the overall goal and the attainment of its objectives? Are the activities of the programme consistent with the intended impacts & effects? Evaluations to Measure Efficiency Efficiency is the extent to which the inputs (funds, expertise, time, etc.) were converted to outputs economically. Compare alternative approaches to achieving the same outputs. (1) Were activities cost-efficient? (Direct support versus outsourcing; relevance of skills required for a given activity.). (2) Were objectives achieved on time? (3) Was the programme/ project implemented in the most efficient way, compared to alternatives? Impact evaluation The positive and negative changes produced by a programme/ project directly or indirectly, intended or unintended.

10 This involves the main impacts and effects resulting from the programme on the local social, economic, environmental and other development indicators. (1) What has happened as a result of the programme/ project ? (2) What real difference has the activity made to the beneficiaries? (3) How many people have been affected? METHODOLOGY OF EVALUATIONS. Combination of qualitative (desk reviews, key informants interviews, focus group discussions, observations) and quantitative (household surveys, health facility surveys or other special surveys). methods need to be used and spelled out in this section. Measuring the Performance of the National HIV/AIDS Strategy Once the priorities and outcomes are identified, how should performance be measured? z Performance of the HIV/AIDS strategy is measured through outcome indicators. z For each indicator identify: z a baseline (what is the value now), and z performance targets (what value should one aim to achieve).