What Makes Good Evaluation? - Purdue Agriculture

1 what Makes good Evaluation? Developed by Susan Barkman , 4-H Department, Purdue UniversityGearing Up to Report Program Impact Thesystematiccollection and analysis of information to determine the worth of a curriculum, program or activity. Alkin 1990 Worth can be a measure of: The quality or satisfaction level leading to program improvement. The effectiveness as a means to demonstrate the impactof the program or that the targeted outcomes have been Improvement evaluation what did you like best (least) about the program? How would you improve the program?Program Improvement evaluation Please rate the following sessions?Poor Fair good ExcellentOpening Session1234 Applying Pesticides1234 Planning Your Budget1234 Gardening with Kids1234 Closing Session1234 Program Impact evaluation Has the program achieved it s targeted outcomes?

2 How have the participants changed as a result of this program? what Programs Should Be Evaluated? Are participants involved in at least 2 or more hours of educational programming? Does the program have clearly stated outcome(s)? Do your stakeholders care about whether this outcome is achieved? Is the program addressing an issue that the community cares about?Targeted OutcomeThe change in participants you would expect to see as a result of participating in the are the participants going to:know,feelor do different after the programAwareness CreatedKnowledge GainedAttitudes ChangedSkills DevelopedAspirations SparkedBehavior ChangedPractice AdoptedDecisions MadePolicies Changed or AdoptedSocial Action InitiatedSocial Impact Economic ImpactCivic ImpactEnvironmental ImpactLEARNING ACTION IMPACTT argeted OutcomesFarmers adopt new management practices that reduce risk to water quality (ACTION OUTCOME practice adopted)Water quality improved(IMPACT OUTCOME environmental)Teen mothers ate healthier and did not consume alcohol, tobacco and other drugs(ACTION OUTCOME behavior changed)

3 Babies liveIMPACT OUTCOME socialBabies have to stay less days in hospitalIMPACT OUTCOME economicYou Be the Judge? Master Gardeners will increase their knowledge about garden management. Increase the number of schools that offer school enrichment programs. Adults will develop a more positive attitude toward foods that contain products derived from Be the Judge? Adults will adopt practices that will decrease the amount of drift when applying pesticides. The educator will create an awareness and knowledge of gang related symbols. Farmers will improve their farm financial management Be the Judge? Parents will use positive parenting techniques. Youth will improve their classroom behavior. Increase involvement of citizens in community Methodology(when and how you are going to administer) Post-Test only Pre and Post-TestEvaluation Methodology(when and how you are going to administer)

4 Post/Post Test Post-Retro PreMeasuring Gains in Knowledge True-False Matching Multiple-Choice CompletionMeasuring Attitudinal ChangeBELIEF Genetically-modified foods are inferior to foods from plants developed using conventional like to do science Attitudinal ChangeBEHAVIOR INTENTION or PREFERENCEW ould I would serve genetically-modified foods to my The government should ban production of genetically-modified Skills and Behavior Changes Pre/post or post/retro pre questionnaire Observe the participantsMeasuring Skills DevelopedHow comfortable do you feel about your ability to lead a small group uncomfortableSomewhat uncomfortableSomewhat comfortableVery comfortableMeasuring Skills DevelopedPlease circle the letter that best describes your ability to do the following things:1.

5 Finding someone s email address on the Using HTML to design a simple web Adding motion and sound to your web Much help Maybe Little help Can Do It needed needed Do ItMeasuring Skills DevelopedPlease rate your skills after the training and what you thought they were before the trainingBEFORE AFTERTRAININGTRAININGNone A lot None A lotAbility to design a 12345123 45quantitative instrument Ability to conduct a pilot test 12345123 45 Measuring BehaviorThe following statements describe how you might communicate with others in your everyday life. Place an X in the box that best describe how often you have done what is described over the past Rarely Sometimes Often AlwaysI find it easy to get my point use body language to help get my message use my tone of voice to reinforce what I am trying to Behavior ChangesBehavior 1stRating with task until in class on = never, 2 = occasionally, 3 = 50% of time, 4 = frequently, 5= neverMeasuring Intentionsto ActDo you plan to soil test for phosphorus?

6 Yes NoAs a result of this program, how willing are you to soil test for phosphorus this coming season?Not willing Reluctant Somewhat willingQuite willing Very willingOpen-ended Questions Advantage reveals what is on the participant s mind, instead of what the evaluator thinks is on their mind Disadvantage takes time to analyze data and participants don t always take time to fill outNot very effective with youthOpen-ended Questions In the next 3 months, what is one thing you will do as a result of participating in this training? Name one or more ways you learned to economize your finances? what is one positive ways you can express your emotions? Describe any changes in your child s behavior that has occurred since he/she has been participating in the ____ this brief session has helped you to GEAR UP to design good evaluation toolsthat will yield valuable impact more info or help in designing evaluation instruments contact Susan Barkman, 4-H Department, Purdue University