TALIS 2018 Technical Report - OECD

1 TALIS 2018 Technical ReportTEACHING AND LEARNING INTERNATIONAL SURVEY 1 TALIS 2018 Technical Report OECD 2020 000 TALIS TALIS 2018 Technical Report PUBE This work is published under the responsibility of the Secretary-General of the OECD. The opinions expressed and arguments employed herein do not necessarily reflect the official views of the OECD member countries. This document, as well as any data and any map included herein, are without prejudice to the status of or sovereignty over any territory, to the delimitation of international frontiers and boundaries and to the name of any territory, city or area. The statistical data for Israel are supplied by and under the responsibility of the relevant Israeli authorities. The use of such data by the OECD is without prejudice to the status of the Golan Heights, East Jerusalem and Israeli settlements in the West Bank under the terms of international law.

2 Note by Turkey The information in this document with reference to Cyprus relates to the southern part of the Island. There is no single authority representing both Turkish and Greek Cypriot people on the Island. Turkey recognises the Turkish Republic of Northern Cyprus (TRNC). Until a lasting and equitable solution is found within the context of the United Nations, Turkey shall preserve its position concerning the Cyprus issue . Note by all the European Union Member States of the OECD and the European Union The Republic of Cyprus is recognised by all members of the United Nations with the exception of Turkey. The information in this document relates to the area under the effective control of the Government of the Republic of Cyprus. Photo credits: Cover Hill Street Studios/Gettyimages You can copy, download or print OECD content for your own use, and you can include excerpts from OECD publications, databases and multimedia products in your own documents, presentations, blogs, websites and teaching materials, provided that suitable acknowledgement of OECD as source and copyright owner is given.

3 All requests for public or commercial use and translation rights should be submitted to Requests for permission to photocopy portions of this material for public or commercial use shall be addressed directly to the Copyright Clearance Center (CCC) at or the Centre fran ais d exploitation du droit de copie (CFC) at 3 TALIS 2018 Technical Report OECD 2019 Acknowledgements The OECD Teaching and Learning International Survey ( TALIS ) is the outcome of collaboration among the participating countries and economies, the OECD Secretariat, the European Commission and an international consortium led by the International Association for the Evaluation of Educational Achievement (IEA). The OECD recognises the significant contributions of members of the IEA Hamburg, Germany, IEA Amsterdam, the Netherlands, as well as Statistics Canada, who authored this Report .

4 The principal authors of the respective chapters were as follows: Chapter 1: Steffen Knoll and Ralph Carstens Chapter 2: John Ainley and Ralph Carstens Chapter 3: Ralph Carstens Chapter 4: Michelle Djeki , David Ebbs, Sandra Dohr, Jan-Philipp Wagner, Steffen Knoll and Viktoria B hm Chapter 5: Jean Dumais and Yves Morin Chapter 6: Viktoria B hm, Alena Becker and Christine Busch Chapter 7: Michelle Djeki , David Ebbs, Sandra Dohr, Jan-Philipp Wagner, Viktoria B hm, Alena Becker and Steffen Knoll Chapter 8: Alena Becker and Christine Busch Chapter 9: Jean Dumais and Yves Morin Chapter 10: Jean Dumais, Alena Becker, Jan-Philipp Wagner, David Ebbs and Steffen Knoll Chapter 11: Agnes Stancel-Pi tak, Justin Wild, Minge Chen, Mojca Rozman, Plamen Mirazchiyski and Hynek Cigler Chapter 12: Ann-Kristin Koop, Falk Brese and Agnes Stancel-Pi tak. Paula Wagemaker copy-edited the Report on behalf of the IEA.

5 OECD Directorate for Education and Skills (EDU) analysts Pablo Fraser, Gabor Ful p and No mie Le Donn provided additional editorial support. Timothy L. Kennel, Bart Meuleman and Christian Monseur provided valuable comments during the development of some chapters. Emily Groves (EDU) provided additional proofreading and managed the layout process, with support from Eleonore Morena, who verified the final layout, and from Henri Pearson (EDU communications team). Emily Groves also provided administrative assistance. We would like to gratefully acknowledge the contribution to TALIS of the late Fons van de Vijver, who was Chair of the Technical Advisory Group and an advisor to TALIS since the first cycle in 2008. 4 TALIS 2018 Technical Report OECD 2019 Table of contents Acknowledgements .. 3 Abbreviations and acronyms .. 13 Three-digit operational codes of TALIS 2018 participants.

6 15 Executive summary .. 17 Governance, management and structure .. 17 Participants, international options and survey phases .. 18 Framework and questionnaire development, translation .. 18 Sampling of schools and teachers .. 19 Main survey administration and quality observations .. 19 Data collection, processing, weighting and adjudication .. 20 Scaling, analysis and results table production .. 20 Differences between TALIS 2018 and the 2008 and/or 2013 21 Chapter 1. Introduction .. 22 TALIS in brief .. 23 Participating countries and economies .. 24 Managing the survey internationally .. 26 Working with national project managers .. 27 Standardised procedures .. 29 Key survey phases .. 30 References .. 33 Notes .. 33 Chapter 2. Development of the conceptual framework .. 34 Introduction .. 35 General purpose of TALIS and policy focus for 2018 .. 35 Developing and refining the conceptual framework.

7 37 Connecting the conceptual framework with proposed analyses .. 46 References .. 48 Notes .. 49 Chapter 3. Development of the teacher and principal questionnaires .. 50 General aims and principles .. 51 Timeline .. 54 Questionnaire Expert Group (QEG) role, membership and collaboration .. 56 Pilot phase .. 58 Field trial phase .. 62 Main survey phase .. 68 References .. 78 Notes .. 79 Chapter 4. Preparation of national survey instruments .. 81 Overview of adaptation and translation verification .. 82 Instruments requiring adaptation and translation .. 83 5 TALIS 2018 Technical Report OECD 2019 Identifying the target language .. 83 Adaptations .. 85 National adaptation forms .. 88 Hierarchy of international options during instrument preparation .. 88 Engaging translators .. 90 Producing translations .. 91 International translation verification .. 92 Feedback from NPMs on international translation and translation verification.

8 93 Layout verification: Paper and online data collection .. 94 References .. 95 Notes .. 95 Chapter 5. Sample design .. 96 Overview .. 97 International sampling 97 Target population and survey population: International requirements and national implementations .. 97 Sample size requirements .. 99 National sampling strategies .. 101 ISCED levels 2, 1 and 3 samples, by participating country and economy .. 103 Reference .. 108 Notes .. 108 Chapter 6. Field operations procedures .. 109 Technical standards, manuals and software .. 110 Administering the TALIS core and additional international options .. 112 Contacting schools and within-school sampling procedures .. 113 Administering the questionnaires and conducting national quality observations .. 116 Monitoring the online questionnaires .. 119 Material receipt and preparing for data 119 Notes .. 120 Chapter 7. Quality assurance procedures for the TALIS data collection.

9 121 Quality control in survey administration .. 122 International quality control programme .. 122 School visit design .. 125 Interviews with the SCs .. 127 translation verification Report (s) .. 133 Extra quality control questions .. 133 Survey activities questionnaire .. 133 Notes .. 138 Chapter 8. Creating and checking the international database .. 139 Overview .. 140 Online data collection and verification .. 140 Data entry and verification of paper questionnaires at national centres .. 142 Data checking, editing and quality control at the IEA Hamburg .. 144 Interim data products .. 152 Building the international database .. 153 Reference .. 155 6 TALIS 2018 Technical Report OECD 2019 Note .. 155 Chapter 9. Estimation weights, participation rates and sampling error .. 156 Overview .. 157 Estimation weights .. 157 Weights for school and principal 158 Weights for teacher data.

10 160 Weights for the TALIS -PISA link 163 Participation rates .. 163 Sampling error with balanced repeated replication (BRR) .. 165 References .. 175 Notes .. 175 Chapter 10. Data adjudication .. 176 The general adjudication process in TALIS .. 177 What was adjudicated in TALIS ? .. 177 The criteria for assessment .. 179 Recommended usage ratings for participants .. 179 Adjudicating the TALIS -PISA samples .. 188 References .. 190 Notes .. 190 Chapter 11. Validation of scales and construction of scale scores .. 191 Overview .. 192 Computation of simple indices .. 192 Scaling procedures .. 198 Results from scales evaluation and scale score creation .. 217 References .. 431 Notes .. 433 Chapter 12. Table production and verification, analyses .. 437 Introduction .. 438 Responsibilities .. 438 Populating the table shells: General procedures .. 438 Estimating standard errors using weights.