Program Evaluation Methods

1 Program Evaluation Methods :Measurement and Attribution of Program ResultsThird EditionReview Practices and StudiesGovernment Review and Quality ServicesDeputy Comptroller General BranchTreasury Board of Canada, Secretariat galement disponible sous le titre :M thodes d valuation des programmes Mesure et attribution des r sultats desprogrammes Minister of Public Works and Government ServicesPublished by Public Affairs BranchTreasury Board of Canada, SecretariatiTable of ContentsPageChapter 1 Objectives and Organization of this The Evaluation Evaluation Issues3 Chapter 2 Evaluation Causal Inference in Causal Evaluation Developing Credible research Criteria .. Decision Environment The Need for Multiple Strategies .. Summary35 Chapter 3 Evaluation Randomized Experimental Quasi-experimental Pre- Program /Post- Program Designs .. Historical/Time Series Designs .. Post- Program -only Designs.

2 Implicit Use of Causal Models in Evaluation Summary59iiChapter 4 DATA COLLECTION Literature File Expert Case Summary82 Chapter 5 ANALYTICAL Statistical Analysis of qualitative Analysis of Further Program The Use of Simulation Models .. Input-output Models .. Micro-economic Macro-economic Statistical Models .. Cost-benefit and Cost-effectiveness Summary113 Chapter 6 CONCLUSIONS114 Appendix 1 SURVEY Survey Measurement Estimating Survey Strengths and Weaknesses122 Appendix 2 GLOSSARY OF TERMS125 Appendix 3 BIBLIOGRAPHY133 Appendix 4 ADDITIONAL REFERENCES146 Figures and TablesFigure 1 The Flow of Evaluation Tasks2 Figure 2 Experimental Designs Give the Best Estimate ofIncremental Effects15 Figure 3 Different Issues Demand Different Strategies20 Figure 4 Model of the Effect of an Excise Tax101 Table 1 Basic Program Evaluation Issues4 Table 2 Considerations in Developing Credible Evaluations22 Table 3 Example of Descriptive Statistics84 Table 4 Further Descriptive Data87 Table 5 Example of Descriptive Statistics89 Chapter 1 Introduction1 Chapter 1 INTRODUCTIONE valuating Program performance is a key part of the federal government sstrategy to manage for results.

3 The Program cycle (design, implementation andevaluation) fits into the broader cycle of the government s Expenditure ManagementSystem. Plans set out objectives and criteria for success, while performance reportsassess what has been performance is an essential link in this cycle. Evaluations shouldproduce timely, relevant, credible, and objective findings and conclusions on programperformance, based on valid and reliable data collection and analysis. Ideally,evaluations should present these findings and conclusions in a clear and balancedmanner that indicates the reliability of the document discusses the appropriate Methods for achieving these analyticalobjectives. In large part, of course, the challenges faced by the evaluator are typical ofall social science research . The relevant literature is full of excellent descriptions ofthe use and abuse of Evaluation Methods . Note that the literature on social scienceresearch techniques and issues covers the methodological issues discussed in thispublication in much greater detail.

4 Note also that few of the Methods discussed hereshould be used without consulting additional reference material or experiencedpractitioners. For this reason, most of the sections in this guide include a list ofadditional and Organization of this TextIt is generally difficult to determine the appropriate Methods for carrying out aprogram Evaluation . The task is complicated by the many specific Evaluation issuesthat may require attention, by the numerous Methods that could be used to gather andexamine information given the resources and time available, and by the need to ensurethat all relevant issues are 12 Figure 1 The Flow of Evaluation Tasks(A) Evaluation AssessmentFamiliarization( Program profile andgeneral literature review)Rationale analysis(Logic model)Formulation and analysis ofevaluation study options(Issues, Methods and costs)(B)EvaluationEvaluation design for the selected optionSub-Study 1 Sub-Study 2 Sub-Study 3 Synthesis of findingsRecommendationsChapter 1 Introduction3 This publication helps practitioners and other interested parties to understandthe methodological considerations involved in measuring and assessing programoutcomes.

5 It places particular emphasis on the strengths and weaknesses of each of thevarious Methods discussed. The publication is not meant to serve as a set of guidelinesthat provide step-by-step instructions for evaluators. Rather, it deals with themethodological considerations present in the development of a credible study that willassess Program Evaluation ProcessThere are three phases to an Evaluation (represented graphically in Figure 1): Evaluation assessment or framework (the planning phase); Evaluation study; and decision- making based on findings and Evaluation assessment phase identifies the main issues and questions to beaddressed in the study and develops appropriate Methods for gathering evidence onthese. This information is presented to the client for the Evaluation in the form ofoptions from which the most appropriate can be selected. Once specific terms ofreference are developed, the Evaluation study can begin. Data are collected andanalyzed to produce findings about the Evaluation issues ( sub-studies 1, 2 and 3 inFigure 1).

6 These findings and subsequent recommendations form the basis on whichdecisions about the future of the Program are made. The reporting of these findingshelps maintain accountability for IssuesIn discussing Evaluation issues and Methods for addressing them, it is usuallyuseful to distinguish between two levels of Program results: operational outputs; and outcomes, which include benefits to Program clients (and unintendednegative effects on clients and others) and related outcomes linked tothe Program s objectives (such as job creation; improvements in health,safety, and welfare; and national security).Evaluations typically cover many issues. While the specific details will beunique to a Program , issues can often be grouped into the following classes. Continued Relevance: The extent to which the Program continues tobe relevant to government priorities and the needs of 14 Results: The extent to which the Program meets its objectives, withinbudget and without causing significant unwanted results.

7 Cost Effectiveness: The extent to which the Program involves the mostappropriate, efficient and cost-effective method to meet 1 Basic Program Evaluation RELEVANCEP rogram Rationale To what extent are the objectives and mandate of the Program stillrelevant? Are the activities and operational outputs consistent with the Program smandate and plausibly linked to the objectives and the other intendedresults? RESULTSO bjectives Achievement In what manner and to what extent were appropriate objectivesachieved as a result of the Program ?Impacts and Effects What client benefits and broader outcomes, both intended andunintended, resulted from carrying out the Program ? In what manner and to what extent does the Program complement,duplicate, overlap or work at cross purposes with other programs ? Alternatives Are there more cost-effective alternative ways to programs that mightachieve the objectives and the intended results? Are there more cost-effective ways of delivering the existing Program ?

8 Chapter 1 Introduction5 From the point of view of Evaluation Methods , two groups of evaluationissues can be usefully distinguished. First, there are issues related to the theory andstructure of the Program , the Program s rationale and possible alternatives. Consider,for example, an industrial assistance Program where the government gives grants on aproject-by-project basis. The rationale question in this instance would be Why doesthe government want to encourage firms to undertake projects that they would notordinarily undertake? For the Program to pass this test, there must be a compellingpublic policy rationale behind the Program . The social benefits to Canada must exceedthe social costs, making the project worthwhile from the broad Canadian perspective,even if the private returns are not sufficient for an individual firm to invest. Such asituation could arise because of the government s ability to diversify risk over a largenumber of projects which, if taken individually, would prove too risky for anyindividual private firm to a second example of Program rationale and alternatives issues, consider aspecial educational Program set up to instruct immigrants in French or questions might focus on possible deficiencies in the current school is there a need for the federal government to run such a Program ?

9 Is it becauseschools are overcrowded, or is it because only private schools are available and theyare too expensive for many immigrants? One may note that more English coursesshould be available to immigrants, but conclude that direct aid to existing schoolswould be a more effective other class of Evaluation issues (achievement of objectives, and programimpacts and effects) relates to the Program s results. What happened because of theprogram? Returning to the industrial assistance Program example, suppose agovernment grant was given to a project that involved hiring 10 new employees. Can itbe said, in relation to the job creation objective underlying the Program , that theprogram was successful because it created these 10 jobs? Before we can make acredible statement about the Program s accomplishment of this objective, thefollowing questions must be answered: Would the project have proceeded without government assistance? Ifso, would it have been pursued on a smaller scale?

10 Were the people hired unemployed at the time, or did they simplytransfer from other jobs? If these other jobs were left vacant or if theyalso were filled only by individuals who were otherwise employed, thenthere may be no net job creation related to the project. If this were so,the job creation objective would not have been 16 Evaluation must deal with both the intended and unintended impacts of theprogram. Intended impacts might be, in this instance, higher personal incomes orincreased Canadian exports. Unintended consequences could be increasedsubsidization of foreign firms at the expense of Canadian firms or a continuation ofactivities inconsistent with needed restructuring in the industry. If the project wouldhave gone ahead without government assistance, the credit (or the blame) for positive(or negative) impacts cannot be attributed to the assistance the Program from our second example, the primary objective might beto increase the reading ability of participating immigrants.