evaluation frameworks excerpts - idmbestpractices.ca

1 excerpts from Review of evaluation frameworks prepared for the Saskatchewan Ministry of Education by Barbara Kahan, Kael Consulting March 2008 Saskatchewan Ministry of Education TABLE OF CONTENTS EXECUTIVE Underlying Values and principles ..2 Concepts ..2 Approaches ..2 Benefits, Challenges, Suggestions For Success ..2 Incorporating evaluation into an Purpose of this Review ..4 Methods ..4 GUIDING Values and evaluation Evaluator Ethics ..9 Research Organizational Practice evaluation Defining evaluation ..11 evaluation Other evaluation -Related Concepts ..19 OVERVIEW OF Steps to Guide evaluation Practice ..23 Summary of Major evaluation Approaches ..26 Results-Focused evaluation ..28 Utilization-Focused Collaborative evaluation ..40 Balanced Appreciative Other Summary Note on evaluation evaluation IN Benefits.

2 56 Challenges ..56 Suggestions for Success ..57 Incorporating evaluation into an Review of evaluation frameworks (March 2008) 1 NOTES This review was prepared for the Saskatchewan Ministry of Education in 2008 to assist the development of a guiding framework for the Saskatchewan Ministry of Education s evaluation activities. The Saskatchewan Ministry of Education holds the copyright to this document. Results were based on several key informant interviews and a literature review. The assistance of people in a number of ministries, in particular the Saskatchewan Ministry of Education, is gratefully acknowledged. excerpts from the original document are included here with permission of the Saskatchewan Ministry of Education. Discussion of the specific Saskatchewan context and a possible Saskatchewan Ministry of Education framework are omitted from the current document.

3 Review of evaluation frameworks (March 2008) 2 EXECUTIVE SUMMARY Underlying Values and principles Every framework is based on a set of underlying values and principles . Making values and principles explicit increases the likelihood they will be integrated into practice and promotes a reflective approach, resulting in greater understanding of, and ongoing revisions to, values and practice. Some documents translate values and principles into sets of guidelines, ethical principles or standards. For example, the Joint Committee on Standards for Educational evaluation s most recent draft of Program evaluation Standards (2007) focuses on four categories: feasibility; propriety (ethics); accuracy; and utility (usefulness). Concepts evaluation concepts are understood in very different ways, making it important to define them in order to ensure high quality communication.

4 The concepts include evaluation itself, the various types of evaluation (ranging from process and outcome evaluations to meta- evaluation ), and other evaluation -related concepts, such as: baseline, stakeholder, qualitative, quantitative. Approaches Many evaluation approaches exist. The literature suggests that no one approach is best for all situations as each approach has its own particular set of strengths and cautions. Often two or more approaches are combined when conducting an evaluation . Each approach has an associated set of steps designed to guide evaluation processes and activities; there is a large degree overlap in the suggested steps, although the nature of the methods and tasks related to each step vary. Major evaluation approaches include: results focused (goals based, goals free, theory based); utilization focused; collaborative; balanced scorecard; appreciative inquiry.

5 Benefits, Challenges, Suggestions For Success Considering evaluation -related benefits, challenges and suggestions for success increases the likelihood of an evaluation framework successfully guiding an evaluation . Potential benefits range from increased capacity to improved outcomes. Challenges range from lack of resources to measurement issues. Examples of suggestions for success include planning for evaluation from the start of a project, reviewing processes as well as outcomes, and using a variety of methods and designs. Review of evaluation frameworks (March 2008) 3 Incorporating evaluation into an Organization Examples of guidelines for incorporating evaluation into an organization include the following: take context into account; ensure adequate resources; foster an evaluation -friendly culture; integrate evaluation into ongoing processes and activities; define roles and responsibilities.

6 Review of evaluation frameworks (March 2008) 4 INTRODUCTION Purpose of this Review This review provides preliminary information to support the development of a guiding framework for Saskatchewan Ministry of Education evaluation processes and activities. This review uses the term framework to mean a set of steps to operationalize a particular evaluation model or approach. In this review the terms approach and model are used interchangeably. Methods Two methods were used to collect information for this review: a literature review and key informant interviews. literature review The following databases and resource websites were searched for potential sources of relevant information: Education Resources Information Centre (ERIC) database Web of Science database Google general Google Scholar websites of several provincial ministries of education: British Columbia, Alberta, Manitoba, Ontario and Quebec Government of Canada website numerous other websites, using a cascade approach (following one link leads to other links which lead to more links) Where possible the key phrase evaluation framework (s) or evaluation model(s) was used.

7 In some cases other key terms were added to the search such as education or business. The searches were restricted for the most part to the last 10 years (1998-2008) and English language documents. Key informants suggested some documents; others were identified by reviewing the author s personal collection. The documents included in this review were selected to ensure information related to a variety of perspectives: business, human service and education sectors Saskatchewan government ministries Review of evaluation frameworks (March 2008) 5 other provinces other countries overviews of evaluation approaches, models and frameworks individual frameworks with defined steps (See the Reference section for the complete list of documents referred to in this review.) key informant interviews Six key informants were interviewed, representing the Ministry of Education and other Saskatchewan ministries.

8 The purpose of the interviews was to gain information about the use of evaluation frameworks in the provincial government, ideas about evaluation frameworks in general, and best ways to implement an evaluation framework in a ministry. Key informant statements included in this review were checked with key informants to ensure interview comments were interpreted accurately. limitations Due to short timelines an in-depth review was not possible. Some information may be incomplete. For the most part secondary sources were reviewed rather than original sources. (Secondary sources refer to material whose authors are not the creators of the content which they are discussing.) Context In many sectors program or policy evaluation is more the exception than the rule for a number of reasons, ranging from time and resource issues to a perception that evaluation is not worth the effort.

9 At the same time, recognition is growing that evaluation is important to meet accountability requirements on the one hand and to improve policies and programs on the other. That governments should apply themselves to continuously improving their performance, particularly with respect to the management of public funds and the stewardship of public assets, appears now to be a generally accepted principle of good governance. (Davies1999) Contents This review discusses the key components that compose a comprehensive evaluation framework : guiding principles concepts models/approaches Review of evaluation frameworks (March 2008) 6 steps for operationalizing the models or approaches understanding evaluation in practice: positives to encourage and negatives to avoid, challenges and solutions, and prerequisites for success Only the basic evaluation model or approach and the operational steps are strictly necessary to a framework .

10 However, each of the other components adds considerably to the understanding and appropriate use of a framework . Review of evaluation frameworks (March 2008) 7 GUIDING principles Every framework is based on a set of underlying values which translate into principles . Often these values and principles are implicit. The benefits of making them explicit are two-fold: Naming values and principles makes it easier to ensure that they are integrated into processes and activities. The process of identifying and defining values and principles promotes a reflective approach where underlying values and principles are examined. In some cases they are revised as a result of this reflection; in other cases a deeper understanding and commitment to them results. Further, processes and activities may change as a result of this reflection.