The 2010 User-Friendly Handbook for Project Evaluations

1 Directorate for Education and Human Resources Division of Research and Learning in Formal and Informal Settings National Science Foundation The 2010 User-Friendly Handbook for Project evaluation The 2010 User-Friendly Handbook for Project evaluation Prepared under Contract REC 99-12175. by Joy Frechtling Westat with contributing authors of three special sections: Melvin M. Mark Debra J. Rog Veronica Thomas, Henry Frierson, Stafford Hood, & Gerunda Hughes Elmima Johnson Program Officer and COTR. Division of Research and Learning in Formal and Informal Settings National Science Foundation December 2010. NOTE: Any views, findings, conclusions, or recommendations expressed in this report are those of the authors and do not necessarily represent the official views, opinions, or policy of the National Science Foundation.

2 TABLE OF CONTENTS. Chapter Page Introduction .. 1. 2. 1. Reasons for Conducting Evaluations .. 3. 5. 2. evaluation Prototypes .. 6. The Different Kinds of evaluation .. 7. Formative evaluation .. 8. Summative evaluation .. 10. evaluation Compared to Other Types of Data Collection Activities .. 11. Using evaluation Information .. 13. Summary .. 14. References .. 14. 3. The evaluation Process Getting Started .. 15. Develop a Conceptual Model of the Project and Identify Key evaluation Points .. 15. Develop evaluation Questions and Define Measurable Outcomes .. 24. Develop an evaluation Design .. 30. Determining the Type of Design Required to Answer the Questions Posed .. 31. Selecting a Methodological Approach .. 31. Selecting a Comparison 32.

3 34. Timing, Sequencing, and Frequency of Data Collection .. 36. References .. 37. iii TABLE OF CONTENTS (CONTINUED). Section Page 4. The evaluation Process: Carrying Out the Study and 39. Conducting the Data Collection .. 39. Analyzing the Data .. 42. Reporting the 44. Background .. 44. evaluation Study Questions .. 44. evaluation Procedures .. 45. Data 45. Findings .. 46. Conclusions (and Recommendations) .. 46. Other Sections .. 46. How Do You Develop an evaluation Report? .. 47. Disseminating the Information .. 48. References .. 51. 5. Data Collection Methods: Some Tips and Comparisons .. 52. Theoretical Issues .. 52. Value of the 52. Scientific Rigor .. 52. Philosophical Distinction .. 53. Practical 54. Credibility of Findings .. 54. Staff Skills.

4 54. Costs .. 55. Time Constraints .. 55. Using the Mixed-Methods 56. References .. 57. iv TABLE OF CONTENTS (CONTINUED). Section Page 6. Review and Comparison of Selected Techniques .. 58. Surveys .. 58. When to Use Surveys .. 59. Interviews .. 59. When to Use Interviews .. 60. Social Network Analysis .. 61. When to Use Social Network Analysis .. 63. Advantages and Disadvantages of SNA .. 63. Focus Groups .. 64. When to Use Focus Groups .. 64. Observations .. 66. When to Use Observations .. 66. Tests .. 67. When to Use 68. Other Methods .. 69. Document Studies .. 69. Key Informant .. 71. Case Studies .. 73. Summary .. 73. References .. 74. 7. A Guide to Conducting Culturally Responsive Evaluations .. 75. The Need for Culturally Responsive evaluation .

5 77. Preparing for the evaluation .. 79. Engaging Stakeholders .. 81. Identifying the Purpose(s) and Intent of the evaluation .. 83. Framing the Right Questions .. 84. Designing the evaluation .. 85. v Selecting and Adapting 86. Collecting the Data .. 87. Analyzing the Data .. 89. Disseminating and Using the Results .. 91. Ethical Considerations and Cultural Responsiveness .. 92. 93. References .. 93. 8. Ensuring Rigor in Multisite 97. 97. Defining Multisite evaluation .. 97. Advantages and Disadvantages of Multisite Evaluations .. 98. Multisite Approaches and Designs .. 99. Factors That Determine the MSE 99. Sampling Sites .. 100. Laying the Foundation for a Multisite evaluation .. 100. Multisite Data Collection: Developing a Common Protocol.

6 101. Assessing the Interventions .. 103. Monitoring Fidelity .. 103. Assessing Comparison as Well as Treatment Sites .. 103. Maintaining the Rigor of the Study Design .. 103. Quality Control in MSE Data Collection .. 104. Selecting, Hiring, and Training Data Collectors .. 104. Ongoing Data Collection 106. MSE Quantitative Analysis .. 106. Preparatory 106. Pooling Data .. 107. Maintaining Independence in the Data .. 108. Design Sensitivity .. 108. Qualitative Analysis Strategies .. 108. Strategies for Reporting and Briefing .. 109. vi Conclusion .. 110. References .. 110. 9. Project evaluation for NSF-Supported Projects in Higher Education .. 112. An Early Consideration: evaluation Purpose .. 113. evaluation Design: It Depends .. 115. Thinking about Tradeoffs.

7 119. A Brief Review of the Gold Standard Debate .. 120. Alternative, Related Methods .. 123. 126. References .. 127. Appendix A. Finding an Evaluator .. 128. Appendix B. Glossary .. 129. Appendix C. Bibliographies .. 136. Annotated Bibliography on Readings in 136. Annotated Bibliography on Readings on Cultural Context, Cultural Competence, and Culturally Responsive evaluation .. 139. Other Recommended Reading .. 146. vii TABLE OF CONTENTS (CONTINUED). List of Exhibits Exhibit Page 1 The Project development/ evaluation cycle .. 4. 2 Levels of evaluation .. 7. 3 Types of 8. 4 Types of data collection 12. 5 Logic model .. 16. 6 Conceptual model for Local Systemic Change (LSC). Initiatives .. 18. 7 Logic model for ADVANCE IT Program .. 21. 8 Identifying key stakeholders.

8 26. 9a Goal and objective writing worksheet .. 28. 9b Sample goal and objective writing workshop for an LSAMP goal .. 29. 10 Three types of errors and their 35. 11a Matrix showing crosswalk of study foci and data collection activities .. 37. 11b Crosswalk of study sample and data collection activities . 37. 12 Formal report outline .. 49. 13 Example of mixed-methods 56. 14 Advantages and disadvantages of surveys .. 59. 15 Advantages and disadvantages of interviews .. 61. 16 Which to use: Focus groups or in-depth interviews? .. 65. 17 Advantages and disadvantages of observations .. 67. 18 Advantages and disadvantages of tests .. 69. 19 Advantages and disadvantages of document studies .. 71. 20 Advantages and disadvantages of using key informants.

9 72. 21 Advantages and disadvantages of using case 73. viii I. NTRODUCTION. T. his Handbook was developed to provide Project directors and principal investigators working with the National Science Foundation (NSF) with a basic guide for evaluating NSF's educational projects. It is aimed at people who need to learn more about both the value of evaluation and how to design and carry out an evaluation , rather than those who already have a solid base of experience in the field. It builds on firmly established principles, blending technical knowledge and common sense to meet the special needs of NSF and its stakeholders. The Handbook discusses quantitative and qualitative evaluation methods, suggesting ways in which they can be used as complements in an evaluation strategy.

10 As a result of reading this Handbook , it is expected that principal investigators will increase their understanding of the evaluation process and NSF's requirements for evaluation , as well as gain knowledge that will help them to communicate with evaluators and obtain data that help them improve their work. To develop this Handbook , we have drawn on the similar handbooks and tools developed for the National Science Foundation (especially the 1993. User-Friendly Handbook for Project evaluation : Science, Mathematics, Engineering and Technology Education, the 1997 User-Friendly Handbook for Mixed-Method Evaluations , and the 2002 User-Friendly Handbook for Project evaluation ) and the National Aeronautics and Space Administration. However, special attention has been given to aligning the Handbook to NSF's unique needs and experiences.