Overcoming Program Evaluation Challenges

1 Overcoming Program Evaluation ChallengesRobin ShepardAssistant Professor of Life Sciences Communication andExtension Water Quality Coordinator, University of Wisconsin-MadisonEvaluation of education is commonly approached byexamining the differences that our efforts have made inthose who take part in our programs. But this is anoverly simplistic view of both the process of evaluationand its purpose. Evaluation involves a systematiccollection of information about the activities,characteristics, and outcomes of programs, personnel,and products for use by specific people to reduceuncertainties, improve effectiveness, and make decisionswith regard to what those programs or products are doingand affecting (Patton, 1982). It is the process ofcomparing evidence with criteria in order to assess thevalue of a Program , activity, or OF EDUCATIONAL Program EVALUATIONE valuating educational programming can target a numberof important purposes, including: assisting in planning and setting Program objectives; assessing Program procedures and tasks as theyoccur; assessing specific Program short-term effects; and assessing long-range of these purposes actually relies on different typesof Evaluation , especially in the techniques TYPES OF EVALUATIONP erhaps the most overlooked aspect of Evaluation is howit fits into Program planning.

2 Evaluation often addressescritical questions about accountability, effectiveness, andefficiency. But in developing the criteria by whichprograms are measured we also collect information aboutwhat a Program can and/or should Evaluation TECHNIQUES. Theseevaluations are aimed at providing information forprogram planning, improvement, modification, andmanagement. Formative evaluations often focus onidentifying audience needs and/or issues, problems,behaviors, etc., that a Program should address. Whendone at the beginning of a project , they form the basis forwhy and how the project proceeds. They also provide abaseline of information from which changes can MONITORING TECHNIQUES. Theseevaluations vary widely from periodic checks ofcompliance with policy to routine tracking of servicedelivered to counting the number of clients.

3 Theseevaluations most often include post-workshop and post-field day questionnaires and Program participant surveysthat focus on who attended and how they felt about theprogram they Evaluation TECHNIQUES. These evaluations are aimed at determining programresults and effectiveness, especially for the purpose ofmaking major decisions about Program continuation,expansion, redirection, and/or funding. This type ofevaluation often focuses on what happened as a result ofthe Program . Such Evaluation usually requires data frommultiple points in time so that changes can be the title suggests, summative evaluations are done atthe end of a project and focus on impacts. In manyinstances, summative evaluations should be based onearlier data collection efforts. They may incorporateformative Evaluation principles as part of acomprehensive Evaluation YOUR EVALUATIONE valuation should not be an afterthought.

4 It is anessential component of a Program and should be carefullyintegrated into a project from the very beginning. As youbegin the task of determining how to evaluate, try walkingthrough the following five steps:STEP #1. Begin with a basic review of the project 'soverall purpose, its objectives, the topics or issuesaddressed by the project , and its target #2. Consider that Evaluation can have one or morespecific purposes; it is important that your evaluationstrategy flows directly from those purposes. Forexample, an Evaluation may: show changes in knowledge or awareness of anissue; provide information to specific audiences; show changes in attitudes; show changes in behavior; document practice adoption; ascertain monetary impacts on farmers who adoptspecific practices; or show changes in the condition of natural #3.

5 In order to make the results of the final reportuseful, consider who holds a stake in the project and itsoutcomes. This illustrates the importance of identifyingspecific information that stakeholders want or #4. Take stock of the information you already haveand what you need to collect. An Evaluation of a projectrarely relies on a single data source or single #5. Select the appropriate Evaluation method and/or methods after you address: Program purpose; Evaluation purpose; stakeholders and their needs; and the information you have in hand versus what youneed to often the methods for Evaluation are determinedbefore those points are what to measure or observe is perhaps the mostcritical question that the Evaluation planning process mustaddress. It is important that Evaluation planning not jumpto "what and how to measure" too quickly.

6 There is arange of Program characteristics that are commonlyconsidered as items to measure. These include: thesetting or context in which the Program occurs, programparticipant reactions, the process of implementation, Program outcomes or reaching specific Program goals, oreven measuring Program costs and/or savings to programparticipants (Herman, Morris, and Fitz-Gibbon, 1987).As the Evaluation is planned it is helpful to ask additionalprogram stakeholders about the impacts they are mostinterested in. Impact-focused evaluations require morethan just recording participation numbers or participantreactions through workshop questionnaire (Mohr, 1995).Impact-focused evaluations consider what happenedafter participants left the workshop or demonstration andimplemented what they SUGGESTED PLANNING PROCESSB efore setting out to evaluate an educational Program , trywriting down some Evaluation goals and objectives.

7 Thisshould actually be done early in the Program designprocess, before implementation and during the actualprogram planning process. This important step will notonly clarify the purpose of the Evaluation , but it will helpexplain your intentions to administrators, staff, and evenprogram participants. This goal-setting also leads to astaff commitment to action and a feeling that evaluationis not an afterthought but part of Program specifically, planning an Evaluation should focus on: what information is important to collect over the lifeof the project ( , knowledge, skills, attitudes,and/or behaviors); how the information should be collected ( , sur-veys, focus groups, interviews, meeting ques-tionnaires, etc.); who will collect the information ( , project staffor an external professional); the time frame for data collection ( , weeks,months, is it a one- or time-two comparison);and how the results will be communicated ( , report,newsletter, news releases, memos, personal dis-cussions, etc.)

8 Programs or projects with external advisory committeesmay be able to take advantage of such a group forevaluation planning. Evaluation is important foraccountability purposes, and those who establish theterms of accountability should take part in determiningthe what and how of information collection regarding thesuccesses and obstacles of the Program . This group willlikely want to know what the Program accomplished, andtherefore should have opinions about what are or are notimportant measures of Program SPECIFIC BARRIERSAs you approach the Evaluation of a project there are anumber of common issues that should be addressed bythose responsible for planning the Evaluation . Many ofthese issues need to be addressed at the Program deliverylevel, or directly by staff conducting the of the most common barriers to Evaluation include: Lack of familiarity with the project being evaluated.

9 Limited Evaluation skills in those conducting theevaluation. Inability to write for lay audiences and translateevaluation findings. Limited design/desktop publishing support for sum-marizing findings into reports. The investigative skills of those conducting theevaluation being too narrowly focused. Lack of familiarity with both Evaluation processesand the subject being evaluated ( , a specificaspect of farm management). Inattention to detail in collecting information. Lack of commitment to working with project part-ners to determine collaborative impacts. Failure to plan funding for the Evaluation in the gen-eral implementation budget. Staff turnover - by Evaluation time, those responsi-ble for implementing the education Program mayhave moved on to other , one of the most pervasive and difficult problemsto address is when those in charge of a Program seeevaluation as a threat (Van den Ban and Hawkins, 1996).

10 This can be a serious problem, especially in agencycultures where criticism might cause loss of face and isnot seen as a positive way to help staff improve theirwork. The issue of perceived threat must be addressedby administrators and organization leaders. The feedbacksystem of an agency or institution must be supportive andencouraging - rather than responding negatively toevaluation Evaluation CHALLENGESD uring the past decade, Evaluation measures associatedwith educational programs have become morechallenging and sophisticated. For example, in the 1980sand early 1990s, the United States Department ofAgriculture required cooperative state extension servicestaff to record Program participant or attendancenumbers and report them annually. While such numbersare still required in certain Program areas, the type ofeducational programs and the issues that extension staffaddress cannot be adequately judged solely onparticipation rates.