What is program evaluation?

1 What is program evaluation 1. What is program evaluation ? A beginners guide Last revised 3/10/2020. Gene Shackman, The Global Social Change Research Project Free Resources for Methods in program evaluation With contributions, suggestions, reviews by Michael Quinn Patton, David Fetterman Electronic copy available at: What is program evaluation 2. Table of Contents Logic Theory of Experimental Design:..16. Quasi-experimental Multiple evaluation In depth case Methods for collecting Administrative Focus Key informant Qualitative Research Methods: Additional Data Reviewing Descriptive Inferential Successful Putting it all What else does an evaluator need to Additional Electronic copy available at: What is program evaluation 3.

2 Introduction In this guide, we focus on one particular kind of evaluation : program evaluation . Basically, when beginning an evaluation , program people will often want the answer to this question: Does the program work? And how can it be improved? There are many equally important questions Is the program worthwhile? Are there alternatives that would be better? Are there unintended consequences? Are the program goals appropriate and useful? Evaluations, and those who request them, may often benefit from a consideration of all of these questions.

3 This guide, however, focuses on the first of these issues: does the program work , and how program evaluation can contribute to improving program services. Electronic copy available at: What is program evaluation 4. How does program evaluation answer questions about whether a program works, or how to improve it. Basically, program evaluations systematically collect and analyze data about program activities and outcomes. The purpose of this guide is to briefly describe the methods used in the systematic collection and use of data.

4 Additional Resources program evaluation answers questions like: To what extent does the program achieve its goals? How can it be improved? Should it continue? Are the results worth what the program costs? program evaluators gather and analyze data about what programs are doing and accomplishing to answer these kinds of questions.. American evaluation Association blog, January 2014. Norms and Standards for evaluation . United Nations evaluation Group. 2016. Definition of evaluation 1. An evaluation is an assessment, conducted as systematically and impartially as possible, of an activity, project, programme, strategy, policy, topic, theme, sector, operational area or institutional performance.

5 It analyses the level of achievement of both expected and unexpected results by examining the results chain, processes, contextual factors and causality using appropriate criteria such as relevance, effectiveness, efficiency, impact and sustainability. An evaluation should provide credible, useful evidence-based information that enables the timely incorporation of its findings, recommendations and lessons into the decision-making processes of organizations and stakeholders. 2. The purposes of evaluation are to promote accountability and learning.

6 evaluation aims to understand why and to what extent . intended and unintended results were achieved and to analyse the implications of the results. evaluation can inform planning, programming, budgeting, implementation and reporting and can contribute to evidence-based policymaking, development effectiveness and organizational effectiveness. Electronic copy available at: What is program evaluation 5. Planning Electronic copy available at: What is program evaluation 6. The first step is planning. Evaluations should follow a systematic and mutually agreed on plan.

7 Plans typically include the following: Determining the purpose of the evaluation (eg accountability, improvement, learning): Understanding the program or intervention to be evaluated. The purpose of the evaluation and understanding the program will determine the next three parts of the plan: What is the evaluation question, what is the evaluation to find out. How will the evaluation answer the question: what is the evaluation design, how will the data be collected. Resources needed: , personnel, resources, skills, time Finally, the plan should include: How to engage the stakeholders.

8 Helps determine the questions and that the results will be used. Making the results useful, how will the results be reported so that they can be used by the organization to make improvements. Who will own the data. Sources and additional resources about planning evaluations : Better evaluation . Manage evaluation . The program Manager's Guide to evaluation , Second Edition, 2010. guide-to- evaluation -second-edition Centers for Disease Control and Prevention. Framework for program evaluation in Public Health. MMWR 1999;48(No. RR-11).

9 Electronic copy available at: What is program evaluation 7. A key part of evaluation is to determine the question. Evaluations can generally answer two types of questions: 1. Outcome evaluation : What is the outcome of the program ? Did the program have any impact, was there any improvement in people's lives? 2. Process evaluation : How did the program get to that outcome? Did the program have some set of procedures? Were these procedures followed, were the procedures reasonable, was there a better way to get to the outcomes? The next section focuses on how to determine the question for your evaluation , describing a few tools which might be useful.

10 Additional resources: Approaching An evaluation -- Ten Issues to Consider Brad Rose Consulting, Inc. Electronic copy available at: What is program evaluation 8. Logic Model One way to figure out the questions is for the evaluator and program people to develop a very good description of: What the program /intervention is doing What the outcomes should be, How the program /intervention is supposed to get there, and Why the program leads to the outcome. This description helps to identify how the program should lead to the outcome, why the program activities should lead to the outcomes, and where to evaluate the program to check whether it does.