The 2010 User-Friendly Handbook for Project Evaluations

1 Directorate for Education and Human Resources Division of research and Learning in Formal and Informal Settings National Science Foundation The 2010 User-Friendly Handbook for Project Evaluation The 2010 User-Friendly Handbook for Project Evaluation Prepared under Contract REC 99-12175. by Joy Frechtling Westat with contributing authors of three special sections: Melvin M. Mark Debra J. Rog Veronica Thomas, Henry Frierson, Stafford Hood, & Gerunda Hughes Elmima Johnson Program Officer and COTR. Division of research and Learning in Formal and Informal Settings National Science Foundation December 2010.

2 NOTE: Any views, findings, conclusions, or recommendations expressed in this report are those of the authors and do not necessarily represent the official views, opinions, or policy of the National Science Foundation. TABLE OF CONTENTS. Chapter Page Introduction .. 1. 2. 1. Reasons for Conducting Evaluations .. 3. 5. 2. Evaluation Prototypes .. 6. The Different Kinds of Evaluation .. 7. Formative Evaluation .. 8. Summative Evaluation .. 10. Evaluation Compared to Other Types of Data Collection Activities .. 11. Using Evaluation Information .. 13. Summary.

3 14. References .. 14. 3. The Evaluation Process Getting Started .. 15. Develop a Conceptual Model of the Project and Identify Key Evaluation Points .. 15. Develop Evaluation Questions and Define Measurable Outcomes .. 24. Develop an Evaluation Design .. 30. Determining the Type of Design Required to Answer the Questions Posed .. 31. Selecting a Methodological Approach .. 31. Selecting a Comparison 32. 34. Timing, Sequencing, and Frequency of Data Collection .. 36. References .. 37. iii TABLE OF CONTENTS (CONTINUED). Section Page 4. The Evaluation Process: Carrying Out the Study and 39.

4 Conducting the Data Collection .. 39. Analyzing the Data .. 42. Reporting the 44. Background .. 44. Evaluation Study Questions .. 44. Evaluation Procedures .. 45. Data 45. Findings .. 46. Conclusions (and Recommendations) .. 46. Other Sections .. 46. How Do You Develop an Evaluation Report? .. 47. Disseminating the Information .. 48. References .. 51. 5. Data Collection Methods: Some Tips and Comparisons .. 52. Theoretical Issues .. 52. Value of the 52. Scientific Rigor .. 52. Philosophical Distinction .. 53. Practical 54. Credibility of Findings .. 54.

5 Staff Skills .. 54. Costs .. 55. Time Constraints .. 55. Using the Mixed-Methods 56. References .. 57. iv TABLE OF CONTENTS (CONTINUED). Section Page 6. Review and Comparison of Selected Techniques .. 58. Surveys .. 58. When to Use Surveys .. 59. Interviews .. 59. When to Use Interviews .. 60. Social Network Analysis .. 61. When to Use Social Network Analysis .. 63. Advantages and Disadvantages of SNA .. 63. Focus Groups .. 64. When to Use Focus Groups .. 64. Observations .. 66. When to Use Observations .. 66. Tests .. 67. When to Use 68. Other Methods.

6 69. Document Studies .. 69. Key Informant .. 71. Case Studies .. 73. Summary .. 73. References .. 74. 7. A Guide to Conducting Culturally Responsive Evaluations .. 75. The Need for Culturally Responsive Evaluation .. 77. Preparing for the Evaluation .. 79. Engaging Stakeholders .. 81. Identifying the Purpose(s) and Intent of the Evaluation .. 83. Framing the Right Questions .. 84. Designing the Evaluation .. 85. v Selecting and Adapting 86. Collecting the Data .. 87. Analyzing the Data .. 89. Disseminating and Using the Results .. 91. Ethical Considerations and Cultural Responsiveness.

7 92. 93. References .. 93. 8. Ensuring Rigor in Multisite 97. 97. Defining Multisite Evaluation .. 97. Advantages and Disadvantages of Multisite Evaluations .. 98. Multisite Approaches and Designs .. 99. Factors That Determine the MSE 99. Sampling Sites .. 100. Laying the Foundation for a Multisite Evaluation .. 100. Multisite Data Collection: Developing a Common Protocol .. 101. Assessing the Interventions .. 103. Monitoring Fidelity .. 103. Assessing Comparison as Well as Treatment Sites .. 103. Maintaining the Rigor of the Study Design .. 103. Quality Control in MSE Data Collection.

8 104. Selecting, Hiring, and Training Data Collectors .. 104. Ongoing Data Collection 106. MSE Quantitative Analysis .. 106. Preparatory 106. Pooling Data .. 107. Maintaining Independence in the Data .. 108. Design Sensitivity .. 108. Qualitative Analysis Strategies .. 108. Strategies for Reporting and Briefing .. 109. vi Conclusion .. 110. References .. 110. 9. Project Evaluation for NSF-Supported Projects in Higher Education .. 112. An Early Consideration: Evaluation Purpose .. 113. Evaluation Design: It Depends .. 115. Thinking about Tradeoffs .. 119.

9 A Brief Review of the Gold Standard Debate .. 120. Alternative, Related Methods .. 123. 126. References .. 127. Appendix A. Finding an Evaluator .. 128. Appendix B. Glossary .. 129. Appendix C. Bibliographies .. 136. Annotated Bibliography on Readings in 136. Annotated Bibliography on Readings on Cultural Context, Cultural Competence, and Culturally Responsive Evaluation .. 139. Other Recommended Reading .. 146. vii TABLE OF CONTENTS (CONTINUED). List of Exhibits Exhibit Page 1 The Project development/evaluation cycle .. 4. 2 Levels of evaluation .. 7.

10 3 Types of 8. 4 Types of data collection 12. 5 Logic model .. 16. 6 Conceptual model for Local Systemic Change (LSC). Initiatives .. 18. 7 Logic model for ADVANCE IT Program .. 21. 8 Identifying key stakeholders .. 26. 9a Goal and objective writing worksheet .. 28. 9b Sample goal and objective writing workshop for an LSAMP goal .. 29. 10 Three types of errors and their 35. 11a Matrix showing crosswalk of study foci and data collection activities .. 37. 11b Crosswalk of study sample and data collection activities . 37. 12 Formal report outline .. 49. 13 Example of mixed-methods 56.