Evaluation Plan Rubric - core.human.cornell.edu

1 Cornell Office for Research on Evaluation January 2010 Page 1 Evaluation plan Rubric Organization & Program Name: Rubric completed by (Name and Title): Date: This Rubric is intended to be a tool for external reviewers to use in providing systematic feedback on Evaluation plans. It is also meant to provide useful guidance to Evaluation plan authors themselves as they develop or revise their own Evaluation plans. This document has three parts. Part I is an overall assessment, Part II lists characteristics of a well written Evaluation plan , and Part III refers to writing and formatting.

2 Directions: 1. Read through entire Evaluation plan being reviewed 2. Provide general overarching comments in Part I 3. Complete Part II as directed 4. Revisit and revise your comments in Part I 5. Complete Part III as directed Part I: Overall Assessment: Evaluation plans should: Provide an accurate, concise and coherent description of the program Explain what Evaluation work is being planned and how the work will be accomplished Be internally consistent (the planned Evaluation should be appropriate for the program s content, lifecycle stage, and stakeholder needs; and the elements of the Evaluation plan should be consistent with each other ( Evaluation purpose, scope, questions, measures, sampling strategy, design and analysis plans).)

3 With these things in mind, please give some overall comments on the plan , an explanation of your assessment and any further suggestions: Cornell Office for Research on Evaluation January 2010 Page 2 Part II: Section by Section Assessment The categories below correspond to Evaluation plan Sections. Each item within a category provides a specific criterion for quality of work. The five-code scale is intended for at-a-glance feedback, rather than numerical scoring. For each item, please check the box corresponding to your assessment of that item, using the numerical codes listed below.

4 Please include comments for more specific feedback. incomplete developing acceptable well done exceptional Category/Criteria 1 2 3 4 5 Comments Program Mission Statement Statement is concise Statement is understandable Statement is specific to the program being evaluated (not the larger organization) Statement conveys the big picture goals of the program Program Description Description is concise Description is understandable to a program outsider Description is complete ( can stand alone without logic model) Description provides basic information about program implementation Description provides information about program context Description provides information about program goals Evaluation Purpose Statement Statement is concise Statement is understandable Statement identifies specific program elements that are the Evaluation s focus Statement articulates the main goal(s)

5 Of the Evaluation Evaluation goals are appropriate given the program lifecycle Statement describes how the Evaluation results will be used Statement explains how the current Evaluation plan fits in with any other Evaluation work on this program. Cornell Office for Research on Evaluation January 2010 Page 3 incomplete developing acceptable well done exceptional Category/Criteria 1 2 3 4 5 Comments Evaluation Questions Questions are concise Questions are understandable Questions are appropriate to current lifecycle stage of the program If appropriate, the basis for comparison is indicated Questions fit the Evaluation purpose (including scope)

6 Questions are clearly related to the program s logic model and/or pathway model The number of questions appears to be feasible Evaluation Sample Sample section is understandable For each Evaluation question, the population of interest is described For each Evaluation question, the sample is described For each sample, the recruitment strategy is described Each sampling strategy chosen is appropriate for generating evidence on the desired claims For each Evaluation question, the sample size is appropriate for generating evidence for the desired claims Cornell Office for Research on Evaluation January 2010 Page 4 incomplete developing acceptable well done exceptional Category/Criteria 1 2 3 4 5 Comments Evaluation Measures Measures are listed by Evaluation question Description of each measure includes measure type ( , survey, observation, interview, etc,)

7 For each measure, the focal construct is clearly identified For each measure, it is clear whether it is already in hand or needs to be located or developed For existing measures, any need for modification has been indicated For measures obtained from elsewhere, proper references are cited For each Evaluation question, the type of measure selected ( , survey, observation, interview, etc,) is appropriate for generating appropriate evidence The measures selected are appropriate given program description The measures selected are appropriate given stakeholder needs For existing measures, validity and reliability evidence is briefly described.

8 Evaluation Design Description of design is concise Description of design is understandable Designs are listed by Evaluation questions Any non-text notation used provides clarity, and all symbols are defined For each Evaluation question, the design type is indicated ( post-only, pre/post, pre/post with comparison group, etc.) Design(s) are appropriate given program lifecycle For each Evaluation question, the chosen design is appropriate for generating evidence for the desired claims Cornell Office for Research on Evaluation January 2010 Page 5 incomplete developing acceptable well done exceptional Category/Criteria 1 2 3 4 5 Comments Data Collection and Management Data collection plan is concise For each measure, the plan for how it will be administered is complete For each measure, the plan for how data will be collected is complete For each measure.

9 The plan for how data will be handled and stored is complete If applicable, the plan for how data will be coded and/or scored is clear A plan for how data will be organized in preparation for analysis is clear Evaluation Analysis: Data Analysis Analysis strategies are listed by Evaluation question Data analysis strategies are concise Data analysis strategies are understandable The analysis strategy is appropriate for generating evidence to answer the related Evaluation question(s) For each Evaluation question, there is a description of how that data will be used to answer that question Data analysis strategies are discussed in terms of stakeholder needs Evaluation Reporting Reporting plans are listed by Evaluation question Each reporting plan is concise Each reporting plan is understandable For each Evaluation question.

10 The method and frequency of reporting is described Each reporting plan addresses stakeholder needs Reporting plans fit purpose and scope of current Evaluation Cornell Office for Research on Evaluation January 2010 Page 6 incomplete developing acceptable well done exceptional Category/Criteria 1 2 3 4 5 Comments Evaluation Timeline Timeline includes program and/or activity dates (whichever is relevant) Timeline events are given in calendar time (not just in relative terms) For each Evaluation question, estimated start and end dates for sample identification and recruitment are indicated (if applicable) For each Evaluation question, estimated start and end dates for measure development are indicated (if applicable)