1 Institute for Law and justice Alexandria, Virginia Training Evaluation model : Evaluating and improving criminal justice Training Final Report September 2007. Submitted to National Institute of justice Prepared by Kelly Bradley Edward Connors Institute for Law and justice Acknowledgements The authors are deeply appreciative of the cooperation and support of all the individuals who made this project possible. We are indebted to Mark Gage, Marcia Williams, and Jim Foley of the National White Collar Crime Center; William Brunson of the National Judicial College and Nancy Yeend and John Paul Jones of the John Paul Jones Group; Liebe Geft, Sunny Lee, and Mark Katrikh of the Simon Wiesenthal Center Museum of Tolerance; and Steve Morrison, Michael Logsdon, Randall Milks, Regina Patrosi, and Michael Hummel of the National Corrections and Law Enforcement Training and Technology Center.
2 These individuals showed the courage to open their programs to outside evaluators. We are also truly grateful for the cooperation of the Training participants who agreed to be interviewed multiple times and complete surveys for this Evaluation . We are also genuinely indebted to Detective Dave D'Amico of the Monmouth County, New Jersey Prosecutor's Office, and Detective Alix Olson of the Madison, Wisconsin, Police Department, for allowing us to learn more about their efforts to combat crimes of hate and terror. A study of this depth and length does not thrive without strong support from the Office of justice Programs' staff who provided guidance and oversight to the project, including Dr.
3 Edwin Zedlewski, Dr. Katherine Browning and Maggie Heisler from NIJ, and Todd Brighton and Elaine Snyder from BJA. Additionally, we would like to thank Dr. James Wells, Dr. Kevin Minor, and Dr. Gary Cordner of Eastern Kentucky University for conducting and writing the Evaluation of the NCLETTC Training . They also contributed significantly to the development of the Training Evaluation model . Finally, we would like to thank the ILJ staff who worked so hard on this project, including Chera Baity, Tiffaney Hall, and Joan Peterschmidt for assistance in data collection; Deborah Spence for assistance with the literature review; and Dr.
4 Thomas McEwen, Barbara Webster, and Laurie Samuel for their contributions in writing this report. This project was supported by grant number 2003-DD-BX-K101 awarded to the Institute for Law and justice by the National Institute of justice , Office of justice Programs, Department of justice . Points of view in this document are those of the authors and do not necessarily represent the official position or policies of the Department of justice Table of Contents Chapter 1: Project Background and Report Overview Project Need for a criminal justice Training Evaluation model ..2. Overview of the Chapter 2: Planning for Evaluations Types of Evaluation Planning Steps.
5 9. Identify Program Goals, Objectives, and Evaluation Develop Conceptual Framework and Logic Design Evaluation Methodology ..11. Conduct the Analyze and Communicate Evaluation Evaluating criminal justice Training Programs ..20. Opportunities for Control and Comparison Challenges in Evaluating criminal justice Training ..20. Chapter 3: Factors That Contribute to Successful Practitioner Training Outcomes Training Adult Learning Concepts ..23. Instructional Methods ..25. Practical Training Facilitation Skills ..28. Communication Active Body Language ..32. Sensitivity to Adult Students' Cultural Chapter 4: criminal justice Training Evaluation model Kirkpatrick's Training Evaluation Customizing and Expanding on Kirkpatrick's Evaluation model for criminal justice Training .
6 37. Conduct Needs Assessment ..40. Design Training Plan ..41. Develop and Test the Curriculum ..43. Deliver the Curriculum ..47. Evaluate the Training and Trainers and Revise ..48. Chapter 5: Project Methodology Key Decision Processes for Site Selection ..65. Overview of Methods ..68. Chapter 6: Cross-site Comparisons and Findings Summary of the Training Evaluation model 's Applications ..71. Needs Training Develop and Test Pilot Test ..77. Trainer Training Course Evaluation ..79. Recommendations and Lessons Learned: Tips for Evaluating and improving criminal justice Costs of Chapter 7: National White Collar Crime Center's Foundations of Intelligence Analysis Training The National White Collar Crime Center.
7 99. History and Background ..99. Center Services ..100. Review of the Intelligence Literature ..103. Intelligence-led Policing ..104. National Intelligence Core Standards ..106. Foundations of Intelligence Analysis Program Overview ..108. Evaluation Evaluation Questions ..110. Data Collection Methods and Framework ..111. Study Strengths and Evaluation Findings ..114. Participant Reaction ..115. Knowledge and Skills Behavior Changes ..121. Discussion ..127. Strengths of the Recommendations for APPENDIX 7-A: FIAT Development SME APPENDIX 7-B: NW3C FIAT Course Training Evaluation APPENDIX 7-C: Pre-Post FIAT Participant Self-assessment of Course Comfort Level.
8 145. APPENDIX 7-D: Matched Pairs T-test Results of Pre/Post FIAT Course Comfort Chapter 8: Simon Wiesenthal Center's National Institutes Against Hate Crimes and Terrorism Training Simon Wiesenthal Center ..148. Review of Hate Crimes and Terrorism Literature ..148. Hate Crime Defined ..148. Statistics ..151. Teaching Tools for Tools for Tolerance National Institutes Against Hate Crimes and Terrorism ..154. Evaluation Evaluation Questions ..157. Data Collection Methods and Tools ..158. Strengths and Weaknesses ..167. Evaluation Findings ..168. Participant Reaction ..169. Learning/Knowledge Gained.
9 178. Attitude and Behavior Changes ..182. Organizational Discussion ..195. APPENDIX 8-A: SWC Training Evaluation APPENDIX 8-B: Case Study of Monmouth County, New Jersey ..216. APPENDIX 8-C: Case Study of Madison, Wisconsin ..229. Chapter 9: NCLETTC Advanced Leadership Techniques Training for First Responders, Corrections, and Security Officers Context for Overview of Literature Relevant to the Research Method ..245. Trainings and Design, Instrumentation, and Data Level 1 Reaction Results ..258. Level 2 Knowledge Results ..266. Level 3 Behavior Change Level 4 Organizational Impact Discussion.
10 286. Main Findings ..286. Strengths, Limitations, and Recommendations ..291. APPENDIX 9-A: NCLETTC Study Timeline, Milestone, and Workplan Chart ..300. APPENDIX 9-B: NCLETTC Training Evaluation Materials ..305. Chapter 10: National Judicial College's Civil Mediation Training National Judicial Literature Civil Mediation Training ..333. Program Overview ..333. Evaluation Evaluation Questions ..335. Data Collection Methods and Framework ..336. Evaluation Findings ..340. Participant Reaction ..340. Knowledge and Skills Behavior Discussion ..345. APPENDIX 10-A: NJC Training Evaluation Materials.