P I SA - oecd.org

1 PISA NATIONAL PROJECT MANAGER MANUAL Doc.: 10 March 2017 Produced by PISA 2018 International Contractors P 2 I S A 0 1 8 Changes from Part 1 (October 2015): - Chapters 7-10 are new and the text is shown in black. - Major changes made to Chapters 1-6 are noted in green text 2 TABLE OF CONTENTS CHAPTER 1 INTRODUCTION TO PISA 2018 AND PISA 2018 NPM MANUAL .. 7 Introduction .. 7 PISA: An Overview .. 7 PISA 2018: An Overview .. 9 PISA: International Management Structure ..10 Accommodations for Students with Disabilities or Special Needs ..12 Further Reading ..13 The NPM Manual ..13 Purpose of the NPM Manual ..13 Structure of the NPM Manual ..14 Schedule of Tasks for PISA 2018.

2 15 CHAPTER 2 COMMUNICATION PROTOCOLS ..17 Overview ..17 Communications among the PISA Stakeholders ..17 The PISA Governing Board, the OECD Secretariat, and the International Contractors ..17 Subject Matter Expert Groups ..18 The General Public ..18 The Academic and Research Community ..18 Communication between National Centres and International Contractors ..19 Participant Contact Details ..19 Communication CHAPTER 3 THE PISA PORTAL ..31 Overview ..31 National Centre Website Access ..31 Website Structure ..32 Questions, Comments, or Suggestions ..33 3 CHAPTER 4 FIRST REVIEW PHASE ..34 Overview of Tasks ..34 Key Tasks during the First Review Establish a National Centre.

3 35 Participate in the Instrument Development Process ..36 Initiate Preparations for the Field Trial Sampling: Establishing Test Dates, Population Definition, and Language Distribution ..37 Negotiate Details about the Study Programmes ..38 Negotiate Your Translation Plan with the International Translation and Verification Referee 38 Indicate your Participation in International and/or National Options ..39 Prepare for the Translation and Verification Phases ..41 Attend the First NPM Meeting in March 2016 ..47 CHAPTER 5 FIELD TRIAL PREPARATION PHASE ..49 Overview of Tasks ..49 Key Tasks during the Field Trial Preparation Phase ..49 Further Specify your Field Trial Sampling Plan.

4 49 Finalise Your Field Trial Sampling Plan ..51 Assemble Test and Questionnaire Materials ..51 Work with Schools ..57 Recruit and Train Test Administrators ..61 Plan for National Quality Monitors to Visit Schools to Monitor Testing Sessions ..63 Plan Staffing and Resources for Coding and Data Entry Operations ..63 Organise for Staff to Attend the Field Trial International Training ..64 Install and Test Software, Data Codebooks, and Coding Documents ..65 CHAPTER 6 FIELD TRIAL PHASE ..66 Overview of Tasks ..66 Key Tasks during the Field Trial Phase ..67 Attend the Second NPM Meeting in November 2016 ..67 Dispatch Test and School-level Materials ..67 Continue Liaising with Schools.

5 69 4 Organise Follow-Up Sessions ..70 Monitor Data Collection of Online Questionnaires ..71 Return Materials to National Centre ..71 Understand the Process of Coding Cognitive items ..71 Understand the Process of Coding Occupations ..73 Attend the Field Trial International Training in January 2017 ..75 Understand Data Entry Operations ..75 Submit Sampling Checks to Core C Sampling ..76 Submit Database to Core A Data Management ..76 Be Available During Post-Submission Data Archive your Field Trial Materials ..78 Complete and Submit the Field Trial Review Questionnaire ..79 Monitor Data Collection of Online Questionnaires ..79 Begin Main Survey Sampling Preparation.

6 79 CHAPTER 7 SECOND REVIEW PHASE ..81 Overview of Tasks ..81 Key Tasks during the Second Review Phase ..81 Review of Field Trial Data and Data Analysis Reports ..81 Review of Operational Procedures ..83 Begin Main Survey School Sampling Work ..83 Review Main Survey Testing Period ..86 Make Preparations for attending the NPM Meeting in November 2017 ..86 CHAPTER 8 MAIN SURVEY PREPARATION PHASE ..87 Overview of Tasks ..87 Key Tasks during the Second Review Phase ..87 Update and/or prepare more promotional material for education authorities and schools 87 Attend the NPM Meeting and KeyQuest trainings ..88 Negotiate Your Main Survey School Sampling Tasks ..88 Prepare the Main Survey Instruments and School-level Materials.

7 88 Finalise Non-Verified Minority Languages Instruments for the Main Survey ..92 Nominate International PISA Quality Monitors (PQMs) ..92 5 Install Main Survey Version of KeyQuest ..94 Liaise with Schools; Selection of the Student Samples ..94 Recruit and Train Test Administrators ..95 Plan Staffing and Resources for Coding ..96 Attend Coder Training ..96 CHAPTER 9 MAIN SURVEY PHASE ..97 Overview of Tasks ..97 Key Tasks during the Main Survey Phase Prior to the Testing Period ..98 Print, Package, and Distribute Main Survey Materials to Schools and Test Administrators as Appropriate ..98 Key Tasks during the Main Survey Phase During the Testing Co-ordinate Activities of Test Administrators and International PISA Quality Monitors (PQMs).

8 99 Continue Liaising with Schools ..99 Monitor School and Student Response Rates .. 100 Organise Follow-Up Sessions .. 101 Monitor Data Collection of Questionnaires .. 102 Key Tasks during the Main Survey Phase Post Testing Period .. 102 Return Materials to National Centre .. 102 Submit Your Sampling Data to Core C Sampling .. 103 Review Item Coding and Data Entry Operations .. 103 Continue to Monitor Data Collection of Online Questionnaires (after the testing period) 104 Recruit Item Coders .. 104 Review Guidelines and Plan for Item Coding Procedures and Coding Design .. 105 Review Guidelines and Plan for Occupation Coding Procedures .. 105 Process Data Entry in the 105 Submit Database to Core A Data Management.

9 106 Be Available During Post-Submission Data 107 Complete and Submit the Main Survey Review Questionnaire .. 107 Archive Your Main Survey Materials .. 107 Review Your National Data and Analysis Reports .. 108 Follow the Preparation of the International Database .. 109 6 CHAPTER 10 REVIEW AND REPORTING PHASE .. 110 Overview of Tasks .. 110 Key Tasks during the Review and Reporting Phase .. 110 Review International Databases and Prepare National Reports .. 110 Analyse National Data .. 111 Prepare for and Document Data Adjudication Process .. 111 Prepare International Report and the Public Use Files .. 113 Coordinate National Dissemination .. 114 APPENDIX A. THE PISA NATIONAL PROJECT MANAGER ROLES AND RESOURCES.

10 115 Introduction .. 115 Profile of NPM .. 115 The PISA Survey Period .. 116 Required Resources PISA Core and Optional Components .. 117 Human Resources .. 117 Facilities .. 121 Production of Materials .. 121 Dissemination .. 122 Meeting and Training Attendance .. 122 APPENDIX B. CONFIDENTIALITY AGREEMENT .. 123 OECD/PISA 2018 Confidentiality 123 APPENDIX C. PISA ACRONYMS AND GLOSSARY .. 124 List of Acronyms .. 124 Glossary .. 128 APPENDIX D. TENTATIVE SCHEDULE OF TASKS, BY PHASES AND AREA .. 140 APPENDIX E. TENTATIVE SCHEDULE OF TASKS, CHRONOLOGICAL .. 152 7 1 INTRODUCTION TO PISA 2018 AND PISA 2018 NPM MANUALMANUAL Introduction This chapter provides a brief overview of the rationale and design of the Programme for International Student Assessment (PISA).