Competency Assessments for Nuclear Industry Personnel

1 Competency Assessments forNuclear Industry PersonnelCOMPETENCY Assessments FORNUCLEAR Industry PERSONNELThe following States are Members of the international atomic energy agency :The agency s Statute was approved on 23 October 1956 by the Conference on the Statute of the IAEA held at United Nations Headquarters, New York; it entered into force on 29 July 1957. The Headquarters of the agency are situated in Vienna. Its principal objective is to accelerate and enlarge the contribution of atomic energy to peace, health and prosperity throughout the world .AFGHANISTANALBANIAALGERIAANGOLAARGENTIN AARMENIAAUSTRALIAAUSTRIAAZERBAIJANBANGLA DESHBELARUSBELGIUMBENINBOLIVIABOSNIA AND HERZEGOVINABOTSWANABRAZILBULGARIABURKINA FASOCAMEROONCANADACENTRAL AFRICAN REPUBLICCHADCHILECHINACOLOMBIACOSTA RICAC TE D IVOIRECROATIACUBACYPRUSCZECH REPUBLICDEMOCRATIC REPUBLIC OF THE CONGODENMARKDOMINICAN REPUBLICECUADOREGYPTEL SALVADORERITREAESTONIAETHIOPIAFINLANDFRA NCEGABONGEORGIAGERMANYGHANAGREECEGUATEMA LAHAITIHOLY SEEHONDURASHUNGARYICELANDINDIAINDONESIAI RAN, ISLAMIC REPUBLIC OF IRAQIRELANDISRAELITALYJA M A I CAJA PANJORDANKAZAKHSTANKENYAKOREA.

2 REPUBLIC OFKUWAITKYRGYZSTANLATVIALEBANONLIBERIALI BYAN ARAB JAMAHIRIYALIECHTENSTEINLITHUANIALUXEMBOU RGMADAGASCARMALAYSIAMALIMALTAMARSHALL ISLANDSMAURITANIAMAURITIUSMEXICOMONACOMO NGOLIAMOROCCOMYANMARNAMIBIANETHERLANDSNE W ZEALANDNICARAGUANIGERNIGERIANORWAYPA K I S TA NPA NA M APA R AG UAYPERUPHILIPPINESPOLANDPORTUGALQATARREP UBLIC OF MOLDOVAROMANIARUSSIAN FEDERATIONSAUDI ARABIASENEGALSERBIA AND MONTENEGROSEYCHELLESSIERRA LEONESINGAPORESLOVAKIASLOVENIASOUTH AFRICASPAINSRI LANKASUDANSWEDENSWITZERLANDSYRIAN ARAB REPUBLICTAJIKISTANTHAILANDTHE FORMER YUGOSLAV REPUBLIC OF MACEDONIATUNISIATURKEYUGANDAUKRAINEUNITE D ARAB EMIRATESUNITED KINGDOM OF GREAT BRITAIN AND NORTHERN IRELANDUNITED REPUBLIC OF TANZANIAUNITED STATES OF AMERICAURUGUAYUZBEKISTANVENEZUELAVIETNAM YEMENZAMBIAZIMBABWECOMPETENCY Assessments FORNUCLEAR Industry PERSONNELI N T E R NAT I O NA L ATO M I C E NE RG Y AG E N C YVIENNA, 2006 IAEA Library Cataloguing in Publication DataCompetency Assessments for Nuclear Industry Personnel .

3 Vienna : international atomic energy agency , ; 24 cm. + 92 0 110105 8 Includes bibliographical Nuclear Industry Employees Evaluation. I. international atomic energy 00434 COPYRIGHT NOTICEAll IAEA scientific and technical publications are protected by the terms of the Universal Copyright Convention as adopted in 1952 (Berne) and as revised in 1972 (Paris). The copyright has since been extended by the World Intellectual Property Organization (Geneva) to include electronic and virtual intellectual property. Permission to use whole or parts of texts contained in IAEA publications in printed or electronic form must be obtained and is usually subject to royalty agreements. Proposals for non-commercial reproductions and translations are welcomed and will be considered on a case by case basis. Enquiries should be addressed by email to the Publishing Section, IAEA, at or by post to:Sales and Promotion Unit, Publishing SectionInternational atomic energy AgencyWagramer Strasse 5P.

4 O . B o x 1 0 0A-1400 ViennaAustriafax: +43 1 2600 29302tel.: +43 1 2600 22417 IAEA, 2006 Printed by the IAEA in AustriaApril 2006 STI/PUB/1236 FOREWORDIn 1996, the IAEA published Technical Reports Series No. 380, Nuclear Power Plant Personnel Training and its Evaluation: A Guidebook. This publication provides guidance for the development, implementation and evaluation of training programmes for all Nuclear power plant Personnel using the systematic approach to training (SAT) methodology. The SAT methodology has since been adopted and used for the development and implementation of training programmes for all types of Nuclear facility and activities in the Nuclear Industry . The IAEA Technical Working Group on Training and Qualification of Nuclear Power Plant Personnel recommended that an additional publication be prepared to provide further guidance concerning Competency Assessments used for measuring the knowledge, skills and attitudes of Personnel as the result of training.

5 This publication has been prepared in response to that critical component of SAT (as part of the implementation phase) is the assessment of whether Personnel have achieved the standards identified in the training objectives. The Nuclear Industry spends a significant amount of resources conducting Competency Assessments . Competency Assessments are used for employee selection, trainee assessment , qualification, requalification and authorization (in some Member States the terminology may be certification or licensing ), and job advancement and promotion. Ineffective testing methods and procedures, or inappropriate interpretation of test results, can have significant effects on both human performance and Nuclear safety. Test development requires unique skills and, as with any skill, training and experience are needed to develop and improve them. Test item and examination development, use, interpretation of results and examination refinement, like all other aspects of SAT, should be part of an ongoing, systematic process.

6 This publication is primarily intended for use by Personnel responsible for developing and administering tests. However, it should also be of value to line managers as well as to managers of training and human resources units. An understanding and appreciation of the importance of using valid and reliable methods for testing Personnel will enhance the benefits and results of training, as well as ensuring that appropriate persons are selected for positions in the Nuclear industryThe IAEA wishes to thank the participants and their Member States who contributed examples for this publication and for their valuable review and critique of it. Particular thanks are due to J. Yoder (United States of America) for the initial drafting and editing. The IAEA officer responsible for this publication was T. Mazour of the Division of Nuclear NOTEA lthough great care has been taken to maintain the accuracy of information contained in this publication, neither the IAEA nor its Member States assume any responsibility for consequences which may arise from its use of particular designations of countries or territories does not imply any judgement by the publisher, the IAEA, as to the legal status of such countries or territories, of their authorities and institutions or of the delimitation of their mention of names of specific companies or products (whether or not indicated as registered) does not imply any intention to infringe proprietary rights, nor should it be construed as an endorsement or recommendation on the part of the.

7 Objective .. Scope .. Relevant IAEA publications .. Terminology .. PURPOSES OF TESTING .. Employee selection and job placement .. Trainee assessment .. Qualification, requalification and authorization .. Job advancement/promotion .. Other purposes and uses of testing .. Trainee motivation .. Instructional improvement .. Programme evaluation .. An instrument to provide teaching and feedback .. FOR EXAMINATIONS .. Job/task analysis and job Competency analysis .. Training objectives .. Classification of training objectives .. Selection and overview of assessment methods .. Supporting material .. EXAMINATIONS General discussion .. Development of written test items .. Formats of test items .. Short answer test items .. Multiple-choice test items .. Essay format test items .. Open reference testing .. Test item answers and point value .. Cognitive level versus point value.

8 Scoring essay format test items .. Examination specifications .. Developing examination specifications .. Examination construction .. Examination layout and assembly .. Examination banks .. Trainee access to examination banks .. Written examination administration .. Establish environment .. Test directions .. Test monitoring .. Use of handouts .. EXAMINATIONS .. Benefits of oral assessment .. Types and uses of oral examinations .. One-on-one oral checkout .. Oral (or examination) board .. Other uses of oral boards .. General guidance for oral Assessments .. Procedures for oral examination .. Examination intent .. Development of oral questions .. Types of question .. Questioning process .. Administration of oral checkouts .. Assessor actions at completion of an oral checkout .. Adminstration of oral boards .. Oral board preparation .. Participation on oral boards.

9 Oral board facilities .. Conduct of oral boards .. Oral board grading .. Assessments .. Developing performance Assessments .. Determine test limitations .. Determine elements to be tested .. Determine conditions and standards .. Determine method of accomplishment .. Construct the performance test .. Develop scoring procedures .. Piloting the performance test .. Performance test adminstration .. Establish environment .. Test directions .. Conducting the performance test .. Assessing the performance test .. Debriefing the trainee .. Other types of performance assessment .. assessment centres .. Performance Assessments of professional technical staff Simulator Assessments .. BASED EXAMINATIONS .. METHODS TO IMPROVE THE EFFICIENCY AND EFFECTIVENESS OF TRAINING .. Some Member State approaches .. Sharing of training and examination materials .. Sharing of examination banks.

10 Industry -wide examination process .. Reciprocity agreements .. Nuclear training groups .. AND VALIDITY .. Reliability .. Validity .. Content validity .. Concurrent validity .. Construct validity .. Predictive validity .. Methods to improve validity .. AND TEST ITEM ANALYSIS .. Examinations .. Test item analysis ..79 APPENDIX I:DEFINITION OF TERMS ..81 APPENDIX II:ACTION VERBS ..88 APPENDIX III:TEST ITEM DEVELOPMENT FORM ..91 APPENDIX IV:WRITTEN TEST ITEM REVIEW CHECKLIST..92 APPENDIX V:EXAMPLES OF MULTIPLE-CHOICE TEST ITEMS ..97 APPENDIX VI:EXAMPLE DIRECTIONS TO TRAINEES FOR A WRITTEN EXAMINATION.. 104 APPENDIX VII: EXAMPLE FIELD CHECKLIST FOR A MAINTENANCE TASK .. 106 APPENDIX VIII: EXAMPLE WORKPLACE assessment .. 110 APPENDIX IX:EXAMPLE ORAL BOARD INSTRUCTIONS .. 116 APPENDIX X:METHOD FOR ORAL BOARD GRADING .. 118 APPENDIX XI:PERFORMANCE TEST CONSTRUCTION CHECKLIST .. 122 APPENDIX XII: EXAMPLE PERFORMANCE TEST FOR A MAINTENANCE INSPECTION TASK.