AHELO g/edu/ahelo - OECD

1 ASSESSMENT OF HIGHER EDUCATIONLEARNING OUTCOMESAHELOFEASIBILITY STUDY REPORTVOLUME 1 DESIGN AND IMPLEMENTATIONK arine TremblayDiane LalancetteDeborah RoseveareOver the past 5 years, the OECD has carried out a feasibility study to see whether it is practically and scientifically feasible to assess what students in higher education know and can do upon graduation across diverse countries, languages, cultures and institution types. This has involved 249 HEIs across 17 countries and regions joining forces to survey some 4 900 faculties and test some 23 000 students. This report presents the design and implementation lessons learnt from this unprecedented experience, as the AHELO Feasibility Study concludes in December 2012. A second volume will be published in February 2013 that will delve further in the analysis of the data and national experiences, while a third volume to be published in April 2013 will present the discussions and insights from the AHELO Feasibility Study Conference (taking place in March 2013).

2 ContentsChapter 1 The rationale for an AHELO : higher education in the 21st century contextChapter 2 The beginning of AHELO : decisions and challengesChapter 3 Design and management of the feasibility studyChapter 4 Instrument developmentChapter 5 ImplementationChapter 6 Lessons learnt on design and implementationMore information on us: photo OECD 2012 Assessment of Higher Education Learning Outcomes Feasibility Study Report Volume 1 Design and Implementation karine Tremblay Diane Lalancette Deborah Roseveare Introduction 2 OECD 2012 This work is published on the responsibility of the Secretary-General of the OECD. The opinions expressed and arguments employed herein do not necessarily reflect the official views of the Organisation or of the governments of its member countries.

3 This document and any map included herein are without prejudice to the status of or sovereignty over any territory, to the delimitation of international frontiers and boundaries and to the name of any territory, city or area. You can copy, download or print OECD content for your own use, and you can include excerpts from OECD publications, databases and multimedia products in your own documents, presentations, blogs, websites and teaching materials, provided that suitable acknowledgement of OECD as source and copyright owner is given. All requests for public or commercial use and translation rights should be submitted to Requests for permission to photocopy portions of this material for public or commercial use shall be addressed directly to the Copyright Clearance Center (CCC) at or the Centre fran ais d exploitation du droit de copie (CFC) at 3 AHELO Feasibility Study Report - Volume 1 OECD 2012 ACKNOWLEDGEMENTS AND CREDITS This report has been written by karine Tremblay, Diane Lalancette and Deborah Roseveare of the OECD Directorate for Education.

4 The authors wish to acknowledge the extensive contributions from the AHELO Consortium report authors and contributors: Hamish Coates, Sarah Richardson, Yan Bibby, Stephen Birchall, Eva van der Brugge, Rajat Chadha, Steve Dept, Jean Dumais, Daniel Edwards, Thomas van Essen, Peter Ewell, Andrea Ferrari, Jacob Pearce, Claire Melican, Xiaoxun Sun, Ling Tan, Rebecca Taylor and Don Westerheijden. The authors also wish to acknowledge the contributions of participating countries National Project Managers and National Experts who drafted the boxes on national experiences. Valuable comments on draft chapters were provided by members of the AHELO Group of National Experts and the Technical Advisory Group. Special thanks are due to Leslie Diamond who edited the first drafts of this report, as well as to C cile Bily who edited the final version, drafted the introduction, readers guide and annexes, and prepared this report for publication.

5 Thanks are also due to the many other OECD colleagues who contributed to this project at different stages of its development including Barbara Ischinger, Andreas Schleicher, Richard Yelland, Fabrice H nard, Val rie Lafon and Sabrina Leonarduzzi. Alenoush Saroyan also provided insightful feedback on the draft chapters during a sabbatical stay at the OECD. The authors are extremely grateful for her time and valuable input to the drafts. The AHELO feasibility study also benefited from the contributions of the following consultants, seconded staff and interns: Rodrigo Casta eda Valle, HoonHo Kim, Claire Leavitt, Eleonore Perez Duarte, Tupac Soulas, Takashi Sukegawa and Mary Wieder. The Secretariat would also like to express its gratitude to the sponsors who, along with the participating countries, generously contributed to this project and without whom the AHELO feasibility study would not have been possible: Lumina Foundation for Education (USA), Compagnia di San Paolo (Italy), Calouste Gulbenkian Foundation (Portugal), Riksbankens Jubileumsfund (Sweden), the Spencer and Teagle Foundations (USA) as well as the higher Education Founding Council HEFCE (England) and the Higher Education Authority HEA (Ireland).

6 The William and Flora Hewlett Foundation also provided support for participation in the study. And finally a special word of thanks to Jan Levy, the Chair of the AHELO GNE, who provided invaluable guidance and support to the Secretariat throughout the feasibility study. Introduction 4 OECD 2012 TABLE OF CONTENTS INTRODUCTION 9 READERS GUIDE 10 CHAPTER 1 THE RATIONALE FOR AN AHELO : HIGHER EDUCATION IN THE 21ST CENTURY CONTEXT 15 Global trends in higher education 16 Expansion of higher education systems 16 Wider participation 18 Emergence of new players 19 More diverse profiles of institutions, programmes and students 20 Continuing advancement and rapid integration of new technology 22 Greater internationalisation 23 Increasing pressures on costs and new modes of financing 26 Growing emphasis on market forces.

7 Competition and signalling mechanisms 27 New modes of governance stressing performance, quality and accountability 29 The quality challenge and limitations of diverse attempts to fill the quality information gap 30 Concerns for drop-out and its implications for equity and efficiency 30 Insufficient information on higher education quality 32 The rationale for an AHELO 34 Moving beyond collegial approaches 34 Growing focus on student learning outcomes 35 Central emphasis on student centered learning and research on teaching-learning processes 37 AHELO within the broader movement towards competencies and learning outcomes 38 CHAPTER 2 THE BEGINNING OF AHELO : DECISIONS AND CHALLENGES 55 Questioning feasibility: Key challenges in developing and operationalising an AHELO 57 Risk of ranking 57 Misuse of results 58 Complexity of comparisons across diverse institutions 59 Contesting of standardised tests 60 Potential impact for institutional autonomy and academic freedom 61 Relevance of generic skills in different systems 62 Methodological and practical questions 63 5 AHELO Feasibility Study Report - Volume 1 OECD 2012 Initial expert meetings to frame a roadmap 64 Washington meeting 64 Paris meeting 67 Seoul meeting 69 CHAPTER 3 DESIGN AND MANAGEMENT OF THE FEASIBILITY STUDY 77 Survey design 78 Study design emphasising a proof of the concept 78 Four distinct but coherent strands of work 80 Participating countries 82 Constraints and implications on the survey design 83 Implications of the

8 Global financial crisis and other constraints on the study 83 Closing the funding gap 84 Impacts of a phased approach on the AHELO Feasibility Study timeframe 86 Phases of work 88 Phase 1 instrumentation and initial proof of concept 88 Phase 2 - practical implementation and psychometric analysis of results 91 Phase 3 value-added methodologies and approaches 95 Study management and actors 96 International management 96 Expert groups involved 99 National and institutional coordination 100 CHAPTER 4 INSTRUMENT DEVELOPMENT 105 The instrumentation process 106 Step A: Developing assessment frameworks 107 Step B: Developing assessment instruments 109 Step C: Translating and adapting assessment instruments and surveys 111 Step D: Small-scale validation of the assessment and survey instruments 111 Step E.

9 Final review of the assessment and survey instruments 112 The AHELO feasibility study Generic Skills Assessment 112 Development of the Generic Skills assessment framework 113 Development of the Generic Skills instrument 114 The AHELO feasibility study Economics Assessment 116 Development of the Economics assessment framework 116 Development of the Economics instrument 118 The AHELO feasibility study Engineering Assessment 121 Development of the Engineering assessment framework 121 Development of the Engineering instrument 123 Introduction 6 OECD 2012 The AHELO feasibility study Contextual Surveys 128 Development of the Contextual dimension framework 129 Development of the Contextual dimension surveys 132 Localisation of assessment and survey instruments 135 CHAPTER 5 IMPLEMENTATION 147 Management of AHELO implementations in countries and institutions 148 Selection of institutions and securing institutional engagement 151 Sampling of students and faculty 153 Student sampling 154 Faculty sampling 156 Electronic delivery 157 Study delivery mechanism and preparation 158 Response rates and incentives 161 Student response rates 161 Response rates and possible non-response bias 163 Student engagement and effectiveness of incentives 165 Faculty responses 172 Scoring student responses 173 Scorer technical standards and training 173 Scoring experience 174 Institution Context Instrument 175 CHAPTER 6 LESSONS LEARNT ON DESIGN AND IMPLEMENTATION 181 Purpose of AHELO to support improvement in learning outcomes 182 Feasibility study design 183 Diversity 183

10 Higher education institutions as the unit of analysis 183 Engaging key stakeholders and experts 184 Assessment strands 184 Management of the feasibility study 184 Funding 185 Contracting arrangements 185 Timelines 186 Instrument development 186 Assessment frameworks 187 7 AHELO Feasibility Study Report - Volume 1 OECD 2012 Matrix sampling 187 Multiple-choice questions and constructed response tasks 188 Contextual dimension 188 Instrument localisation 188 Field implementation 188 Timelines 189 Capacity, readiness and quality control 189 Electronic delivery 190 Sampling frames and student response rates 190 Scoring 191 Estimating costs of participation 192 Extra value for participating institutions and faculties 192 ANNEX A LIST OF CONTRIBUTORS 197 ANNEX B ILLUSTRATIVE ITEMS 219 Tables Table - Country localisation workflows 138 Table - Key AHELO feasibility study implementation activities and timelines 150 Table - Number of participating institutions per field implementation 152 Table - Student response rates by strand and country 162 Table - Comparison of population and response data (standard deviation)