Planning a Program Evaluation - The Learning Store

1 Ellen Taylor-PowellSara SteeleMohammad DouglahFebruary 1996 Program Development and EvaluationPlanning a Program EvaluationG3658-1 AcknowledgementsFor their timely and thoughtful review of thispublication, the authors wish to thank Phil Holman, Stephen Kluskens, Greg Lamb,Joan LeFebvre, Dave Sprehn, NancyStoutenborough, Jack Trzebiatowski and Kathi Vos. n n the are you going to evaluate?..2 What is the purpose of the Evaluation ?..3 Who will use the Evaluation ? How will they use it?..3 What questions will the Evaluation seek to answer?..5 What information do you need to answer the questions?.. of information: numerical and is the Evaluation needed?..10 What resources do you need time, money, people?..10 Collecting the sources of information will you use?..11 What data collection method(s) will you use?

2 11 What collection procedures will you use?..12 Using the will the data be analyzed?..13 How will the information be interpreted By whom?..14 How will the Evaluation be communicated and shared?..14 Managing the the plan: timeline and the OF CONTENTSI ntroductionThere is no blueprint or recipe for conducting agood Evaluation . Because the term evaluationissubject to different interpretations, a programcan be evaluated in a variety of ways. ManyExtension professionals evaluate their pro-grams informally on an ongoing basis throughcasual feedback and observation. The resultinginformation is often very useful and may be allthat is needed to keep a Program relevant andoperating efficiently. You can enhance thevalue of information garnered from an evalua-tion however, if you devote sufficient fore-thought and Planning to the evaluationprocess.

3 As the term is used in this guide, Program evaluationrefers to the thoughtfulprocess of focusing on questions and topics ofconcern, collecting appropriate information,and then analyzing and interpreting the infor-mation for a specific use and guide is designed to help you plan aprogram Evaluation . It is organized into fourmajor sections:nFocusing the evaluationnCollecting the informationnUsing the informationnManaging the evaluationEach section presents a series of questions andconsiderations for you to adapt to your ownneeds and situation. You ll discover that Evaluation is more thanjust collecting information. It involves seriousreflection on questions such as:nWhat is the purpose of the Evaluation ?nWhat do I want to know?nWhat do I intend to do with the information?Answers to these questions are crucial if yourevaluation is to produce useful guide will help you think about andanswer these and other questions as you plan aprogram the evaluationWhat are you going to evaluate?

4 Define what you intend to evaluate. This mayor may not be easy depending upon howclearly defined your Program is. For example,you might wish to evaluate a financial man-agement Program . Think about the Program spurpose and content. Do you want to examinethe whole Program or just a particular compo-nent of it? Briefly describe what you want toevaluate purpose, expected outcomes,intended beneficiaries and activities. What isthe planned link between Extension s inputsand the desired outcomes?The Evaluation effort should fit the Program -ming effort. In some cases, you may only beinterested in finding out how peopleresponded to your teaching style, or how satis-fied they were with a particular event. In othercases, you may want to document behavioralchanges or impacts which require a more com-prehensive Evaluation and level of effort.

5 Thepoint is to tailor your Evaluation to fit theprogram. Don t expect to measure impact froma single workshop or behavioral changes froma limited media , not all extension work needs to beformally evaluated. Formal evaluations requiretime, money and resources. Sometimes pro-grams lack sufficient substance to warrant aformal Evaluation , or it may be too costly tocollect the evidence needed to demonstrateimpact. It may be possible that no one is inter-ested in the findings. We also need to consider our clientele. Peopleget tired of filling out end-of-session forms oranswering surveys. Be selective, considerateand think about what is needed and what willbe DEVELOPMENT AND Evaluation n n nWhat is the purpose of the Evaluation ?The fundamental purpose of Evaluation is tocreate greater understanding.

6 WithinExtension, Program evaluations are conductedlargely to improve educational efforts and toaddress accountability. In action, these pur-poses translate into more specific reasons forconducting an is the purpose(s) of the Evaluation youpropose? For example, will it:nHelp others (taxpayers, administrators,participants, colleagues, committeemembers) understand the Program and itsresults?nImprove the Program ?nImprove your teaching?nMeasure whether the Extension programmade a difference in people s lives?nDetermine if the Program is worth thecost?nAnswer questions posed by funders andinfluential members of the community? nMeet rank and tenure requirements?nMeet administrative requirements?nOther?It is important to clearly articulate the evalua-tion s purpose. Otherwise, it will lack directionand the resulting information will not be asvaluable as it could will use the Evaluation ?

7 How will they use it?Have you ever completed an Evaluation andasked, What shall I do with it? Or talliedthe results but never really used the findings?If we want to collect relevant data and makethe best use of limited resources, we mustthink about how we ll use our Evaluation rightfrom the , we conduct an Evaluation only forour own use. Usually, however, there areothers who have requested or could use theresulting information. Any of the groups listedbelow might be interested in the evaluationresults of an Extension affected in some way by theprogram (either directly or indirectly) suchas Program participants, nonparticipants,criticsnCounty board members, elected officials nCommunity leadersnColleagues, volunteers, collaborators, sup-portersnExtension administratorsnMedianTenure committeesnGrantorsnAgencies, firms, interest groupsIdentify potential users of the information.

8 Findout whatthey want to know, and howthey willuse the information (see table 1). If you don tknow, ask. This will help you to clarify thepurpose(s) of the Evaluation , build commit-ment for it and fine-tune the particular ques-tions the Evaluation will AN EVALUATIONn n nInvolving othersAs with Program Planning , involving intendedusers of the information in an Evaluation leadsto greater commitment to the evaluationprocess, helps ensure that relevant questionsare asked, and increases the chances that find-ings are listened to and input may be included throughout theentire Evaluation process or just at specificstages, such as when you set the Evaluation sfocus, determine the information needs, orcollect and interpret data. In recent years, con-siderable emphasis has been placed on involv-ing stakeholders as partners in the evaluationprocess to ensure that the information col-lected is relevant and that there is a commit-ment to use , however, when we include users, it is as helpers or data collectors, while we remainin control of the Evaluation .

9 Alternativeapproaches in the field of Evaluation aim tochange the center of control. Participatoryapproachesand empowerment Evaluation enablepeople to conduct and use their appropriately constituted advisory groupor a co-sponsor can be a strong asset. Theseparties can serve as advocates for the evalua-tion, see that tasks are completed and helpmake resources available. As a result, morepeople respond, the findings receive moreattention, and the results are disseminatedmore DEVELOPMENT AND Evaluation n n nTable wants to know what? Howwill the information be used?Who might use What do they want How will they use the Evaluation *to know?the results?YouIs the Program meeting To make decisions about clientele needs?modifying the programIs my teaching effective?To influence decisions about tenureor meritCounty boardWho does the Program serve?

10 To make decisions about budget Is the Program cost-effective?allocationsProfessional Are you an effective educator?To make rank and promotionreview committeedecisionsExtension administrationHas the Program achieved To justify extension programs its expected outcomes?and ensure financial supportHow effective are To decide about staff the extension faculty?training and achievementsClienteleIs the extension Program To determine whether to participate meeting their needs?in other extension programs *Examples of broad user categories are listed here. Be as specific as possible when you identify potential users and their a list of the questions and topics thatyou and the individuals or groups youhave listed want to address. As you do so,review the Program s elements. Sometimesprograms change as they are implemented;sometimes not all the intended activities arecarried out.