USE OF PHILLIPS’S FIVE LEVEL TRAINING EVALUATION AND …

1 USE OF phillips 'S five LEVEL TRAINING EVALUATION AND RETURN ON. INVESTMENT FRAMEWORK IN THE U. S. NON-PROFIT SECTOR. Travis K. Brewer, , , Dissertation Prepared for the Degree of DOCTOR OF PHILOSOPHY. UNIVERSITY OF NORTH TEXAS. August 2007. APPROVED: Jerry Wircenski, Major Professor Mark Davis, Minor Professor Jeff Allen, Committee Member Jack J. phillips , Committee Member Robin Henson, Interim Chair of the Department of Technology and Cognition M. Jean Keller, Dean of the College of Education Sandra L. Terrell, Dean of the Robert B.

2 Toulouse School of Graduate Studies Brewer, Travis K. Use of phillips 's five LEVEL TRAINING EVALUATION and ROI framework in the nonprofit sector. Doctor of Philosophy (Applied Technology and Performance Improvement), August 2007, 163 pp., 29 tables, references, 100 references. This study examined TRAINING EVALUATION practices in nonprofit sector organizations. It offered a framework for evaluating employee TRAINING in the nonprofit sector and suggested solutions to overcome the barriers to EVALUATION . A mail survey was sent to 879 individuals who were members of, or had expressed an interest in, the American Society for TRAINING and Development.

3 The membership list consisted of individuals who indicated association/nonprofit or interfaith as an area of interest. Data from the survey show that TRAINING in the nonprofit sector is evaluated primarily at LEVEL 1 (reaction) and LEVEL 2 (learning). It also shows decreasing use from LEVEL 3 (application). through LEVEL 5 (ROI). Reaction questionnaires are the primary method for collecting LEVEL 1. data. Facilitator assessment and self-assessment were listed as the primary method for evaluating LEVEL 2. A significant mean rank difference was found between LEVEL 2 (learning) and the existence of an EVALUATION policy.

4 Spearman rho correlation revealed a statistically significant relationship between LEVEL 4 (results) and the reasons TRAINING programs are offered. The Kruskal-Wallis H test revealed a statistically significant mean rank difference between academic preparation of managers with LEVEL 3 EVALUATION . The Mann-Whitney U test was used post hoc and revealed that master's degree had a higher mean rank compared to bachelor's degree and doctorate. The Mann-Whitney U test revealed that there were statistically significant mean rank differences on LEVEL 1, LEVEL 2, LEVEL 3, and LEVEL 5 EVALUATION use with the barriers little perceived value to the organization, lack of TRAINING or experience using this form of EVALUATION , and not required by the organization.

5 Research findings are consistent with previous research conducted in the public sector, business and industry, healthcare, and finance. Nonprofit sector organizations evaluate primarily at LEVEL 1 and LEVEL 2. The existence of a written policy increases the use of LEVEL 2 EVALUATION . TRAINING EVALUATION is also an important part of the TRAINING process in nonprofit organizations. Selecting programs to evaluate at LEVEL 5 is reserved for courses which are linked to organizational outcomes and have the interest of top management.

6 Copyright 2007. by Travis K. Brewer ii ACKNOWLEDGEMENTS. As a long journey comes to a close, I would like to acknowledge the following people and organizations for their love, support, sacrifice, and assistance as I pursued my doctoral degree. To my committee, professors Jerry Wircenski, Jeff Allen, and Mark Davis, I. thank you for your support and encouragement. And to committee member Jack phillips , thank you for agreeing to sit on my committee. I appreciate your devotion and expertise to TRAINING EVALUATION and ROI.

7 To my partner, Dirk, for supporting and encouraging me when times were tough. Your deep devotion and undying love kept me going when I really wanted to quit. I. would not be where I am today without your support. To my parents, Jerry and Sherlene Brewer, I thank you for your love and support. Mom, I admire you for the obstacles you have overcome in life. I get my strength and perseverance from you. Dad, thank you for helping me to understand that in school I. would learn how to learn.. To my mentor and friend, Diane Culwell, I would like to thank you for your wisdom, guidance and support of me during my journey.

8 You allowed me to develop and grow as a person and as a trainer. I would also like to acknowledge Patti phillips and the ROI Institute. Patti, thank you for giving me advice and direction during my journey, which kept me focused on TRAINING EVALUATION . Every time I bugged you with questions, you were more than happy to give me answers. I also appreciate the ROI Institute for the generous grant to support my study and for the work they do with TRAINING EVALUATION and ROI. iii TABLE OF CONTENTS. Page LIST OF Chapter 1.

9 INTRODUCTION ..1. Theoretical Framework Significance of the Study Purpose of the Study Research Questions and Hypotheses Limitations Delimitations Definition of Terms Summary 2. REVIEW OF RELATED LITERATURE ..13. Introduction Employer-Sponsored TRAINING Definition Need for TRAINING TRAINING in Nonprofit Sector TRAINING EVALUATION Definition of TRAINING EVALUATION Frameworks of EVALUATION phillips 's five - LEVEL TRAINING EVALUATION Framework Use of phillips 's Framework Findings on Use Barriers to Use 3. METHODOLOGY.

10 35. Introduction Research Design Population Sample Instrumentation Variables Validity Reliability Data Collection Procedures iv Data Analysis Procedures Summary 4. Overview Demographics Research Questions Analysis Hypotheses Analysis General Comments Summary of Findings 5. SUMMARY, CONCLUSIONS, AND Overview Summary of Findings Discussion of Findings Limitations of the Results Conclusions Recommendations APPENDICES ..101. REFERENCES ..154. v LIST OF TABLES. Table Page 1. five - LEVEL ROI 2. Use of EVALUATION at Each 3.