Training Evaluation Model: Evaluating and …

1 Institute for Law and justice Alexandria, Virginia Training Evaluation model : Evaluating and improving criminal justice Training Final Report September 2007. Submitted to National Institute of justice Prepared by Kelly Bradley Edward Connors Institute for Law and justice Acknowledgements The authors are deeply appreciative of the cooperation and support of all the individuals who made this project possible. We are indebted to Mark Gage, Marcia Williams, and Jim Foley of the National White Collar Crime Center; William Brunson of the National Judicial College and Nancy Yeend and John Paul Jones of the John Paul Jones Group; Liebe Geft, Sunny Lee, and Mark Katrikh of the Simon Wiesenthal Center Museum of Tolerance; and Steve Morrison, Michael Logsdon, Randall Milks, Regina Patrosi, and Michael Hummel of the National Corrections and Law Enforcement Training and Technology Center. These individuals showed the courage to open their programs to outside evaluators. We are also truly grateful for the cooperation of the Training participants who agreed to be interviewed multiple times and complete surveys for this Evaluation .

2 We are also genuinely indebted to Detective Dave D'Amico of the Monmouth County, New Jersey Prosecutor's Office, and Detective Alix Olson of the Madison, Wisconsin, Police Department, for allowing us to learn more about their efforts to combat crimes of hate and terror. A study of this depth and length does not thrive without strong support from the Office of justice Programs' staff who provided guidance and oversight to the project, including Dr. Edwin Zedlewski, Dr. Katherine Browning and Maggie Heisler from NIJ, and Todd Brighton and Elaine Snyder from BJA. Additionally, we would like to thank Dr. James Wells, Dr. Kevin Minor, and Dr. Gary Cordner of Eastern Kentucky University for conducting and writing the Evaluation of the NCLETTC Training . They also contributed significantly to the development of the Training Evaluation model . Finally, we would like to thank the ILJ staff who worked so hard on this project, including Chera Baity, Tiffaney Hall, and Joan Peterschmidt for assistance in data collection; Deborah Spence for assistance with the literature review; and Dr.

3 Thomas McEwen, Barbara Webster, and Laurie Samuel for their contributions in writing this report. This project was supported by grant number 2003-DD-BX-K101 awarded to the Institute for Law and justice by the National Institute of justice , Office of justice Programs, Department of justice . Points of view in this document are those of the authors and do not necessarily represent the official position or policies of the Department of justice Table of Contents Chapter 1: Project Background and Report Overview Project Need for a criminal justice Training Evaluation model ..2. Overview of the Chapter 2: Planning for Evaluations Types of Evaluation Planning Steps ..9. Identify Program Goals, Objectives, and Evaluation Develop Conceptual Framework and Logic Design Evaluation Methodology ..11. Conduct the Analyze and Communicate Evaluation Evaluating criminal justice Training Programs ..20. Opportunities for Control and Comparison Challenges in Evaluating criminal justice Training .

4 20. Chapter 3: Factors That Contribute to Successful Practitioner Training Outcomes Training Adult Learning Concepts ..23. Instructional Methods ..25. Practical Training Facilitation Skills ..28. Communication Active Body Language ..32. Sensitivity to Adult Students' Cultural Chapter 4: criminal justice Training Evaluation model Kirkpatrick's Training Evaluation Customizing and Expanding on Kirkpatrick's Evaluation model for criminal justice Training .37. Conduct Needs Assessment ..40. Design Training Plan ..41. Develop and Test the Curriculum ..43. Deliver the Curriculum ..47. Evaluate the Training and Trainers and Revise ..48. Chapter 5: Project Methodology Key Decision Processes for Site Selection ..65. Overview of Methods ..68. Chapter 6: Cross-site Comparisons and Findings Summary of the Training Evaluation model 's Applications ..71. Needs Training Develop and Test Pilot Test ..77. Trainer Training Course Evaluation ..79. Recommendations and Lessons Learned: Tips for Evaluating and improving criminal justice Costs of Chapter 7: National White Collar Crime Center's Foundations of Intelligence Analysis Training The National White Collar Crime Center.

5 99. History and Background ..99. Center Services ..100. Review of the Intelligence Literature ..103. Intelligence-led Policing ..104. National Intelligence Core Standards ..106. Foundations of Intelligence Analysis Program Overview ..108. Evaluation Evaluation Questions ..110. Data Collection Methods and Framework ..111. Study Strengths and Evaluation Findings ..114. Participant Reaction ..115. Knowledge and Skills Behavior Changes ..121. Discussion ..127. Strengths of the Recommendations for APPENDIX 7-A: FIAT Development SME APPENDIX 7-B: NW3C FIAT Course Training Evaluation APPENDIX 7-C: Pre-Post FIAT Participant Self-assessment of Course Comfort Level ..145. APPENDIX 7-D: Matched Pairs T-test Results of Pre/Post FIAT Course Comfort Chapter 8: Simon Wiesenthal Center's National Institutes Against Hate Crimes and Terrorism Training Simon Wiesenthal Center ..148. Review of Hate Crimes and Terrorism Literature ..148. Hate Crime Defined ..148. Statistics ..151. Teaching Tools for Tools for Tolerance National Institutes Against Hate Crimes and Terrorism.

6 154. Evaluation Evaluation Questions ..157. Data Collection Methods and Tools ..158. Strengths and Weaknesses ..167. Evaluation Findings ..168. Participant Reaction ..169. Learning/Knowledge Gained ..178. Attitude and Behavior Changes ..182. Organizational Discussion ..195. APPENDIX 8-A: SWC Training Evaluation APPENDIX 8-B: Case Study of Monmouth County, New Jersey ..216. APPENDIX 8-C: Case Study of Madison, Wisconsin ..229. Chapter 9: NCLETTC Advanced Leadership Techniques Training for First Responders, Corrections, and Security Officers Context for Overview of Literature Relevant to the Research Method ..245. Trainings and Design, Instrumentation, and Data Level 1 Reaction Results ..258. Level 2 Knowledge Results ..266. Level 3 Behavior Change Level 4 Organizational Impact Discussion ..286. Main Findings ..286. Strengths, Limitations, and Recommendations ..291. APPENDIX 9-A: NCLETTC Study Timeline, Milestone, and Workplan Chart ..300. APPENDIX 9-B: NCLETTC Training Evaluation Materials.

7 305. Chapter 10: National Judicial College's Civil Mediation Training National Judicial Literature Civil Mediation Training ..333. Program Overview ..333. Evaluation Evaluation Questions ..335. Data Collection Methods and Framework ..336. Evaluation Findings ..340. Participant Reaction ..340. Knowledge and Skills Behavior Discussion ..345. APPENDIX 10-A: NJC Training Evaluation Materials ..348. Appendices Appendix A: NW3C Fiat Course: Instructor Classroom Training Observation Assessment Instrument ..358. Appendix B: Training Evaluation model Project Evaluability Questions for Site Selection Screening ..363. Appendix C: Synthesis Report on Evaluability Assessments of Training Programs .365. Appendix D: NW3C FIAT Training Evaluation Plan 371. Appendix E: Memorandum of Understanding Between Institute for Law and justice and National White Collar Crime Center .379. References ..384. Chapter 1. Project Background and Overview The purpose of this project was to produce a Training Evaluation model that can guide evaluations of a wide range of criminal justice Training programs.

8 The study was conducted by the Institute for Law and justice in partnership with Eastern Kentucky University. It was sponsored by the National Institute of justice (NIJ) with funding from the Bureau of justice Assistance (BJA). The project's overall goal was to help the Office of justice Programs (OJP), Department of justice , achieve more consistency and control over the hundreds of Training programs for which it provides funding, and at the same time, increase the capacity of other criminal justice programs federal, state, and local to conduct their own Training evaluations. Project Summary This study had two major objectives: (1) develop a flexible model for Evaluating criminal justice Training programs, and (2) test the model by applying it in the field to four Training programs. The four programs that were evaluated to test the model had received BJA. discretionary grant funding for Training (commonly known as earmarks ). They were selected in part because they permitted a test of the model in diverse environments: the programs were different in terms of learning objectives, intended audiences, instructional methods, subject matter, and other factors.

9 The four participating Training programs were Foundations of Intelligence Analysis Training (FIAT) offered by the National White Collar Crime Center. This was a basic analytical intelligence Training curriculum for law enforcement and regulatory personnel. Tools for Tolerance Institutes offered by the Simon Wiesenthal Center. The purpose of this Training was to give participants new perspectives on hate crime and terrorist acts, help them form multi-agency collaborations, and foster the development of strategic action plans. Advanced Leadership for Law Enforcement and Corrections Professionals offered by the National Corrections and Law Enforcement Training and Technology Center. This course was focused on teaching values-based leadership skills to agency leaders who are responsible for first responders and correctional and security officers. Chapter 1: Project Background and Overview 1. Civil Mediation Training offered by the National Judicial College. This course familiarized participants with the civil mediation process and qualified them for certification in states that require it.

10 The research study teams began with a widely accepted Training Evaluation model (Kirkpatrick 1998) that was originally developed for the private business sector. The Kirkpatrick model was then refined to address Training Evaluation needs in criminal justice . The lessons learned by applying the model in the field were of great benefit in shaping the final model , helping to ensure that it would be applicable to all criminal justice Training programs. The final model retains Kirkpatrick's Evaluation framework but places greater emphasis on conducting a Training needs assessment and on planning for a Training Evaluation when the Training is geared for criminal justice audiences; and it makes changes to the Level 4 application. Need for a criminal justice Training Evaluation model Compared to many other topics in the criminal justice Evaluation research literature, little attention has been devoted to Evaluating Training programs. This holds true despite the facts that: (a) high caliber Training programs are essential if criminal justice personnel are to perform their duties in a professional manner; and (b) each year, much time, effort, and money are devoted to staff Training (Minor, Wells, Cobb, Lawrence, & Cox 2005).