A Guidebook for Nutrition Project Managers in Developing ...

1 Monitoring and EvaluationA Guidebook for Nutrition ProjectManagers in Developing CountriesF. James LevinsonBeatrice Lorge RogersKristin M. HicksThomas SchaetzelLisa TroyCollette YoungSeptember 1999 Human Development NetworkThe World BankiiPrepared for The World Bank, Human Development Network byInternational Food and Nutrition CenterTufts University School of Nutrition Science and PolicyiiiContents1111122222 Acronyms .. viiGlossary .. ixIntroduction: How to Use this Guidebook .. xiiiAn Overview of Monitoring and Evaluation .. 1 Clarifying Project Goals, Objectives and Information Needs .. 23 Pre-Design Tasks.

2 23 Clarifying Project Goals and Objectives .. 23 Mapping the Project .. 26 Determining Information Needs .. 31 Designing a Monitoring System .. 35 Monitoring for Project Improvement with a ManagementInformation System (MIS) .. 36 Responding to Monitoring Results .. 47 Monitoring the Budgetary Health of the Project .. 51 Planning the Flow of Information .. 53 Selecting an Evaluation Design .. 59 Designs for Evaluation .. 60 Preparing for Evaluations .. 73 Planning an Evaluation .. 73 Determine the Sample Size .. 74 Identify a Control Group .. 75 Choose a Sampling Frame.

3 77 Collecting Pre- Project Information through a Baseline Survey .. 79333334444455555ivSelecting Indicators .. 81 What Is an Indicator? .. 81 Characteristics of a Good Indicator .. 81 Types of Indicators for Measuring Program Effectiveness .. 86 How to Select Indicators .. 88 Deciding Data Collection Strategies .. 91 Qualitative vs. Quantitative Data Collection Methods .. 91 Maximizing the Efficiency of Data Collection .. 99 Controlling for Bias through Data Collection Methods .. 100 Developing Data Collection Instruments .. 103 Steps in Designing M&E Instruments .. 103 Analyzing the Data.

4 115 Analyzing Quantitative Information .. 115 Analyzing Qualitative Information .. 129 Returning to the Conceptual Framework .. 131 Maximizing the Usefulness of Results .. 133 Ownership of the M&E Process .. 133 Effective Presentation of Findings and Recommendations .. 136 The Usability of Results .. 137 Annexes1: Program Constraints Assessment .. 1432: Conceptual Framework Examples from Three NutritionProjects .. 1453: General M&E Checklist .. 149666667777788888999991010101010v4: Illustrative Data Collection Forms Which Facilitate Management by Exception .. 1515: Sampling Tips.

5 1756: Instrumentation Checklist .. 1797: Illustrative Use of the Conceptual Framework Modelfor Backwards Mapping .. 181 References .. 185viviiBMIBody mass indexCNPC ommunity Nutrition promoterCNOC ommunity Nutrition organizerCNWC ommunity Nutrition workerCSCluster samplingFFWFood-for-workHAZH eight-for-age Z-scoreICDSI ntegrated child development servicesIDDI odine deficiency disorderLBWLow birth weight (< 2,500 grams)MISM anagement information systemM&EMonitoring and evaluationNGONon-governmental organizationPRAP articipatory rural appraisalRRAR apid rural appraisalSDStandard deviation(s)SRSS imple random samplingSSSystematic samplingStSStratified samplingTINPT amil Nadu Integrated Nutrition ProjectUPGK Family Nutrition Improvement Program in IndonesiaVADV itamin A deficiencyWAZW eight-for-age Z-scoreWHZW eight-for-height Z-scoreAcronymsviiiixGlossaryBenefitsBen efitsBenefitsBenefitsBenefits the broader.

6 Sustainable changes in public health or economicstatus that a program seeks to achieve but which are inevitably influ-enced by a wide range of other Assessment Constraints Assessment Constraints Assessment Constraints Assessment Constraints Assessment the systematic identification of constraintsinhibiting Project effectiveness. The constraints are then organized topermit the identification of technical, policy, research, and training meansof addressing Grol Grol Grol Grol Groupoupoupoupoup a group of individuals in an evaluative study who sharethe same characteristics as a participant Analysisy Analysisy Analysisy Analysisy Analysis study of the cost incurred to deliver a specifiedset and quantity of goods and services (outputs) to a targeted Analysisfectiveness Analysisfectiveness Analysisfectiveness Analysisfectiveness Analysis study of the cost incurred to achieve aspecific change in nutritional status (impact)

7 In a targeted GrFocus GrFocus GrFocus GrFocus Groupsoupsoupsoupsoups small group discussions led by a trained moderator whointroduces a topic and facilitates participation by all group the broad aims of the Project , the significant, longer-termchanges that planners expect to occur as a an objectively verifiable measurement which reflects the ac-tivity, assumption, or effect being InforKey InforKey InforKey InforKey Informant Intermant Intermant Intermant Intermant Interviewsviewsviewsviewsviews a face-to-face meeting between a trainedinterviewer and a person selected to represent a certain group whoseknowledge, attitudes or practices are being monitored or evaluated.

8 Or aperson likely to offer informed Indicatoroxy Indicatoroxy Indicatoroxy Indicatoroxy Indicator a measurement used as a substitute when true indica-tors are too difficult to measure changes in the condition of the target population which gener-ally reflect the primary objectives of the the materials, goods and actions necessary to carry out theprimary Project assumptionsInput assumptionsInput assumptionsInput assumptionsInput assumptions the expectations regarding the effectiveness andquality of the Project inputsEvaluationEvaluationEvaluationEval uationEvaluation a process of data collection designed to assess the effec-tiveness of the Project in attaining its originally stated objectives, and theextent to which observed changes are attributable to the DesignExperimental DesignExperimental DesignExperimental DesignExperimental Design a rigorous evaluation design which includes acontrol group.

9 Randomization, and pre-post Project InforManagement InforManagement InforManagement InforManagement Information Systemmation Systemmation Systemmation Systemmation System a tool, often computerized, which isused to compile and analyze monitoring the ongoing collection and review of information on projectimplementation, coverage and utilization of operationalized goals which specify the results and thelevel of change the intermediate effects, often behavioral, resulting directlyfrom Project outputs that may be necessary to achieve a desired the provision of Project goods and services to the target popu-lation.

10 The primary Project AssumptionsOutput AssumptionsOutput AssumptionsOutput AssumptionsOutput Assumptions expectations regarding the ways goods and ser-vices (outputs) will be used by the target subset of a population which is used to represent the AnalysisSensitivity AnalysisSensitivity AnalysisSensitivity AnalysisSensitivity Analysis a means of exploring how plausible changes inassumptions about uncertain variables affect Studies Special Studies Special Studies Special Studies Special Studies studies to investigate issues raised before or duringproject implementation which can not be addressed through ongoingproject individuals or organizations associated with or affectedby a Design Quasi-experimental Design Quasi-experimental Design Quasi-experimental