EVALUATION OF AGRICULTURAL EXTENSION AND ADVISORY …

1 EVALUATION OF AGRICULTURAL . EXTENSION AND ADVISORY SERVICES. A MEAS Training Module By Murari Suvedi September 2011. EVALUATION OF AGRICULTURAL . EXTENSION AND ADVISORY SERVICES. A MEAS Training Manual Professor Murari Suvedi Michigan State University Murari Suvedi, Michigan State University and the MEAS Project This work is licensed under a Creative Commons Attribution Non Commercial Share Alike License. This license lets others remix, tweak, and build upon this work non-commercially, as long as they credit Murari Suvedi and license their new creations under the identical terms. This training module was produced as part of the United States Agency for International Development, USAID, project Modernizing EXTENSION and ADVISORY Services , MEAS. Leader with Associates Cooperative Agreement No.

2 AID-OAA-L-10-00003. The module was made possible by the generous support of the American people through USAID. The contents are the responsibility of the authors and do not necessarily reflect the views of USAID or the United States Government. ACKNOWLEDGEMENTS. EVALUATION practitioners and international AGRICULTURAL development experts have contributed to the development of this training manual by offering guidance and/or reviewing the manuscript. Mary Andrews, Michigan State University Kirk Heinze, Michigan State University Diane Ruonavaara, Michigan State University Gail Vander Stoep, Michigan State University R. Kirby Barrick, University of Florida Shawn Morford, Program Manager, Rural Development Initiatives, Inc. Angela Hobson, Graduate Assistant in the Department of Community, Agriculture, Recreation and Resources Studies at Michigan State University as well as Andrea Bohn, MEAS Project Manager at the University of Illinois at Urbana-Champaign helped in editing and formatting the manual.

3 Resources from diverse research and EVALUATION literature have been utilized to develop this training manual. Samples of EVALUATION instruments, reports and presentation materials are drawn from author's work at Michigan State University. Specifically, this manual draws heavily from the following previous digital and print publications: Suvedi, M., & Morford, S. (2003). Conducting Program and Project Evaluations: A Primer for Natural Resource Program Managers in British Columbia. Forrex-Forest Research EXTENSION Partnership, Kamloops, Forrex Series 6. Suvedi, M., Heinze, K., & Ruonavaara, D. (2008). How to conduct EVALUATION of EXTENSION programs. In Suvedi, M., Singh, B., Vijayaragavan, K., Padaria, R., & Wason, M. (Eds.) EVALUATION capacity building in rural resource management: A manual.

4 India: Golden Printers & Publishers. i TABLE OF CONTENTS. About the EVALUATION Training Manual .. v Chapter 1. History and Development of AGRICULTURAL EXTENSION and ADVISORY Services .. 1. AGRICULTURAL EXTENSION and ADVISORY Service Models and Approaches .. 2. AGRICULTURAL EXTENSION in the 21st Century .. 5. Challenges to EVALUATION of EXTENSION Programs and Services .. 6. Chapter 8. Introduction to Program EVALUATION .. 8. Why Evaluate a Program?.. 10. What is the Role of the Evaluator? .. 11. What are Common EVALUATION Approaches and Types? .. 12. Evaluative Studies Conducted at the Planning Stage .. 14. Evaluative Studies Conducted During Program Implementation .. 15. Evaluative Studies Conducted at the End of the Program or After .. 16. Qualitative versus Quantitative Studies.

5 16. Evaluative Studies to Measure Long-term Changes .. 19. Does EVALUATION Involve Research? .. 19. Which Experimental Designs are Useful for EVALUATION ? .. 21. Ethics of EVALUATION .. 25. Popular Program EVALUATION Frameworks and Designs .. 25. Program Logic Model .. 25. Hierarchy of Evidence (Hierarchy of Program EVALUATION ) .. 27. Program EVALUATION Cycle .. 30. Results 32. Chapter 33. Step 1: Creating a Climate for EVALUATION .. 33. Chapter IV .. 36. Developing an EVALUATION Plan .. 36. ii Step 2: Identifying the Program Phase and EVALUATION Type .. 36. Step 3: Assess the Feasibility of Implementing an EVALUATION Study .. 37. Step 4: Identify and Consult Key Stakeholders .. 38. Clarify EVALUATION Issues, Questions, Indicators, and Criteria .. 39. Step 5: Identify Approaches to Data Collection.

6 41. Chapter V .. 45. Managing EVALUATION Data Collection .. 45. Step 6: Selecting Data Collection Techniques .. 45. Secondary Data (Existing Information).. 47. Data from Testing and Examination .. 47. Questionnaire Surveys and Interviews .. 49. Creating Quality Surveys by Avoiding Errors .. 55. Focus Group .. 63. Rapid Rural Appraisal .. 66. Case Study .. 68. Semi-structured Interviews with Key Informants .. 68. Participant 69. Economic Analyses for EVALUATION .. 71. Benefit/Cost Analysis .. 72. Step 7: Sampling for Program EVALUATION .. 74. Chapter VI .. 81. Step 8: Collecting and Analyzing EVALUATION data .. 81. Organizing EVALUATION Data .. 82. Analyzing and Interpreting 82. Chapter VII .. 98. Step 9: Communicate EVALUATION Findings .. 98. Reporting Negative Findings.

7 99. Step 10: Use of EVALUATION Results .. 100. Reflecting on the EVALUATION .. 100. iii Appendix A: Online Resources on Program EVALUATION .. 102. Appendix B: Sample Cover Letter to a Mail Survey .. 103. Appendix C: Sample Questionnaire .. 104. Appendix D: Sample Short Course EVALUATION 107. Appendix E: Sample Introduction to Focus Group .. 108. Appendix F: Sample Focus Group Questions .. 109. Appendix G: Sample Case Study .. 110. Appendix H: Example of Qualitative Data Generated from Open-ended Survey Questions .. 111. References .. 113. iv ABOUT THE EVALUATION TRAINING MANUAL. Planners and policy-makers in EXTENSION systems are busy professionals who may lack the skills and core competencies to guide a systematic EVALUATION of AGRICULTURAL EXTENSION initiatives and programs.

8 With exposure to EVALUATION methods and procedures, they could contribute significantly in guiding program EVALUATION and use results to improve future programs. This training module is designed for policy makers, upper level administrators, and program managers in public sector institutions who have an EXTENSION mandate (including university faculty having responsibilities in EXTENSION , rural development, and food and nutrition security programs) and administrators and program officers who fund and manage projects having EXTENSION components. The purpose of this training module is to expose national level policy makers, project managers, and funding agency personnel to various models and theories of program EVALUATION . Specifically, participants in this EVALUATION training module will be able to: understand program EVALUATION what it is and reasons for doing it.

9 Describe EVALUATION principles and frequently used models of program EVALUATION ;. identify indicators of program success for a given AGRICULTURAL EXTENSION project/policy;. select appropriate methods/techniques of data collection for conducting process and impact evaluations;. understand the use of statistics to analyze data, interpret results, write EVALUATION reports, and share findings with stakeholders; and develop EVALUATION plans to document impacts of EXTENSION programs. Workshops corresponding to this manual can be offered either as a four-day training course in-country or as a regional course, based on demand. The structure and content are: Day One: Introduce participants to the context of program EVALUATION in AGRICULTURAL EXTENSION , EVALUATION concepts and principles, and frequently used models for EVALUATION of programs and policies (Chapters 1 and 2).

10 Day Two: Introduce various methods of EVALUATION data collection qualitative and quantitative . including strengths and weaknesses of each method (Chapters 3-6). Day Three: Introduce the concept of sampling, review commonly used data analysis procedures and interpretation of results, and present strategies for sharing EVALUATION findings with diverse stakeholders (Chapters 7-10). Day Four: Participants develop an EVALUATION plan for a participant-identified EXTENSION program needing EVALUATION . The content of the training manual is brief and straightforward. Some of the text is presented as bullet points for ease of reading. Power Point presentations are available on each major topic and a list of key resources is provided for further information. For illustrative purposes, samples of EVALUATION plans, data collection instruments, and EVALUATION reports are included in the Appendix.