Evaluation of the Teacher Incentive Fund: Final …

1 Evaluation of the Teacher Incentive Fund: Final report on implementation and Impacts of Pay-for-Performance Across Four Years December 2017 Department of Education THIS PAGE IS INTENTIONALLY BLANK Evaluation of the Teacher Incentive Fund: Final report on implementation and Impacts of Pay-for-Performance Across Four Years December 2017 Hanley Chiang Cecilia Speroni Mariesa Herrmann Kristin Hallgren Paul Burkander Alison Wellington Mathematica Policy Research Elizabeth Warner Project Officer Institute of Education Sciences NCEE 2018-4004 DEPARTMENT OF EDUCATION Department of Education Betsy DeVos Secretary Institute of Education Sciences Thomas W. Brock Commissioner of the National Center for Education Research Delegated Duties of the Director National Center for Education Evaluation and Regional Assistance Ricky Takai Acting Commissioner December 2017 The report was prepared for the Institute of Education Sciences under Contract No.

2 ED-IES-14-C-0115. The project officer is Elizabeth Warner in the National Center for Education Evaluation and Regional Assistance. IES Evaluation reports present objective information on the conditions of implementation and impacts of the programs being evaluated. IES Evaluation reports do not include conclusions or recommendations or views with regard to actions policymakers or practitioners should take in light of the findings in the reports. This report is in the public domain. Authorization to reproduce it in whole or in part is granted. While permission to reprint this publication is not necessary, the citation should be: Chiang, Hanley, Cecilia Speroni, Mariesa Herrmann, Kristin Hallgren, Paul Burkander, Alison Wellington.

3 (2017). Evaluation of the Teacher Incentive Fund: Final report on implementation and Impacts of Pay-for -Performance Across Four Years ( NCEE 2017- 4004). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, Department of Education. This report is available on the IES website at Upon request, this report is available in alternate formats such as Braille, large print, audiotape, or computer diskette. For more information, please contact the Department s Alternate Format Center at 202-260-9895 or 202-205-8113. ACKNOWLEDGMENTS This study would not have been possible without the contributions of many individuals. We are grateful for the cooperation of many TIF administrators, teachers, principals, district leaders, and central office staff who assisted with the study s data collection and provided important information that shaped the study.

4 A dedicated technical assistance team helped TIF districts implement the programs examined in this study. This team was led by Duncan Chaplin and Jeffrey Max and included Lauren Akers, Kevin Booker, Julie Bruch, Albert Liu, Allison McKie, Debbie Reed, Alex Resch, Christine Ross, and Margaret Sullivan from Mathematica and Patrick Schuermann and Eric Hilgendorf from the Peabody College of Education at Vanderbilt University. Several individuals made enormous efforts to collect data successfully for this study. Sheila Heaviside and Annette Luyegu provided excellent leadership over our administration of Teacher , principal, and district surveys, and Kathy Shepperson oversaw the design of key systems for collecting this survey data.

5 Lauren Akers, Nickie Fung, Chris Jones, Margaret Sullivan, Sarah Wissel, and Claire Smither Wulsin patiently conducted and summarized numerous interviews with TIF administrators. Acquiring and processing administrative data required a large effort led by Jacqueline Agufa and Mary Grider with assistance from Michael Brannan, Kai Filipczak, Chris Jones, William Leith, Mickey McCauley, Sarah Osborn, and Juha Sohlberg. Many people contributed to the analysis and interpretation of the study s data and the production of this report . The study received useful advice from our technical working group, consisting of David Heistad, James Kemple, Daniel McCaffrey, Anthony Milanowski, Richard Murnane, Jeffrey Smith, and Jacob Vigdor.

6 At Mathematica, Jill Constantine and Steven Glazerman helped shape the Evaluation and provided expert advice. Sarah Osborn helped with a variety of critical tasks including facilitating the management of the project. The analysis was made possible by an excellent team of programmers, consisting of Ra l Torres Aragon, Michael Brannan, Molly Crofton, Kai Filipczak, and John Hotchkiss. John Kennedy edited the report , and Jill Miller carefully and patiently prepared the report for publication. iii THIS PAGE IS INTENTIONALLY BLANK CONTENTS EXECUTIVE SUMMARY .. xxi I INTRODUCTION .. 1 II STUDY SAMPLE, DESIGN, DATA, AND METHODS .. 11 III PROGRAMS AND EXPERIENCES OF ALL 2010 TIF DISTRICTS .. 29 IV TIF implementation IN Evaluation DISTRICTS.

7 39 V IMPACTS OF PAY-FOR-PERFORMANCE ON EDUCATORS ATTITUDES AND BEHAVIORS .. 75 VI IMPACTS OF PAY-FOR-PERFORMANCE ON EDUCATOR EFFECTIVENESS AND STUDENT ACHIEVEMENT .. 89 VII EDUCATORS ATTITUDES AND DISTRICTS PLANS TOWARD CONTINUING KEY TIF COMPONENTS .. 111 REFERENCES .. 119 APPENDIX A SUPPLEMENTAL INFORMATION ON STUDY SAMPLE, DESIGN, DATA, AND METHODS FOR CHAPTER II .. APPENDIX B SUPPLEMENTAL INFORMATION ON ANALYTIC METHODS FOR CHAPTER II .. APPENDIX C SUPPLEMENTAL FINDINGS ON PROGRAMS AND EXPERIENCES OF ALL 2010 TIF DISTRICTS FOR CHAPTER III .. APPENDIX D SUPPLEMENTAL FINDINGS ON TIF implementation IN Evaluation DISTRICTS FOR CHAPTER IV .. APPENDIX E SUPPLEMENTAL FINDINGS ON IMPACTS OF PAY-FOR-PERFORMANCE ON EDUCATORS ATTITUDES AND BEHAVIORS FOR CHAPTER V.

8 APPENDIX F SUPPLEMENTAL FINDINGS ON IMPACTS OF PAY-FOR-PERFORMANCE ON EDUCATOR EFFECTIVENESS AND STUDENT ACHIEVEMENT FOR CHAPTER VI .. APPENDIX G SUPPLEMENTAL FINDINGS ON FACTORS ASSOCIATED WITH DIFFERENCES IN IMPACTS FOR CHAPTER APPENDIX H COST-EFFECTIVENESS METHODS AND DETAILED FINDINGS FOR CHAPTER VI .. v THIS PAGE IS INTENTIONALLY BLANK TABLES Main Data Sources for This report .. xxiv Number of Districts that Implemented TIF and Responded to the District Survey, by Year .. 11 Comparison of TIF Evaluation Districts and Non- Evaluation Districts (Percentages Unless Otherwise Noted) .. 13 Number of Schools in the Evaluation , by Cohort and Treatment Status .. 16 Characteristics of Students Enrolled in Treatment and Control Schools in the Pre- implementation School Year (2010 2011) (Percentages Unless Otherwise Indicated).

9 18 Characteristics of Educators in Treatment and Control Schools in Year 1 (Percentages Unless Otherwise Noted) .. 19 Data Sources for This report .. 20 TIF Districts Reported implementation of TIF Required Components for Teachers and Principals (Percentages) .. 30 Staff Eligibility for Pay-for-Performance Bonus, Year 4 (Percentages) .. 33 Additional Pay Opportunities for Teachers and Principals, Year 4 .. 34 Planned Professional Development Activities for Teachers (Percentages) .. 34 Evaluation Districts Reported implementation of TIF Required Components for Teachers and Principals (Percentages) .. 41 Measures of Student Achievement and Observations of Practices Used to Evaluate Teachers and Principals, as Reported by Evaluation Districts (Percentages).

10 43 Key Features of Evaluation Districts Teacher Pay-for-Performance Bonus Programs in Year 4 .. 48 Additional Pay Opportunities, as Reported by Evaluation Districts, Year 4 .. 58 Professional Development Activities for Teachers Planned Under TIF, as Reported by Evaluation Districts (Percentages) .. 59 vii Tables Mathematica Policy Research Information Districts Provided to Teachers About Actual Pay-for-Performance Bonuses from Years 2 and 3 (Percentages) .. 61 Teachers Reports of the Measures Used to Evaluate Teachers (Percentages) .. 64 Principals Reports of the Measures Used to Evaluate Principals (Percentages) .. 64 Actual and Reported Receipt of Pay-for-Performance Bonus from Year 3 for Teachers in Treatment Schools in Year 4.