Competency-based training and assessment

1 Competency-based training and assessment Henry Defalque TO/PEL HIGH LEVEL SEMINAR ON LOC-I AND UPRT (Johannesburg, 19 October2017) 2 Outline HIGH LEVEL SEMINAR ON LOC-I AND UPRT What is the difference between CBTA & traditional training What is ICAO policy Scope of amendment 5 to PANS- training Competency-based training and assessment (CBTA) principles CBTA benefits The ICAO Competency framework CBTA workflows Consequential amendments to Annex 1 and PANS-TRG + Doc 9379 (PEL system), 9841 (ATO), 10011 (UPRT), 9625 (FSTD) CAA safety oversight of CBTA Conclusions Differences between CBTA & Traditional training 3 Traditional training qualification -based: Meeting/maintaining the qualifications requirements (Annex 1/ national regs) expressed in quantitative terms that prescribe training programme inputs ( required hours of study, hours of practice, etc.)

2 Programme design and content are also influenced by the CAA s testing criteria and methods Competency-based training programme: an integrated and outcomes-focused training programme aimed at providing the graduates with the competencies to be safe, highly effective and efficient in the performance of their duties Programme design requirements: training needs analysis; adapted competency model; training and assessment plans; LMS Feedback on training & graduate s performance is critical for evaluation of course HIGH LEVEL SEMINAR ON LOC-I AND UPRT Experienced course developers Differences between CBTA & Traditional training 4 HIGH LEVEL SEMINAR ON LOC-I AND UPRT ICAO Policy 5 A38-12: Consolidated statement of continuing ICAO policies and associated practices related specifically to air navigation: APPENDIX D - Qualified and Competent Aviation Personnel Whereas the satisfactory implementation of SARPs and PANS is contingent upon having qualified and competent personnel.

3 Associated practices The Council should assist Member States to harmonize aviation professionals levels of competency. These efforts should be based on: .. a Competency-based approach. HIGH LEVEL SEMINAR ON LOC-I AND UPRT Competency-based approach - latest changes 6 amendment 5 to the Procedures for Air Navigation Services training (PANS-TRG, Doc 9868) Applicable 5 November 2020 Definitions Interlinked concepts Design methodology Need for change PANS- training currently contains General provisions for Competency-based training and assessment training and assessment for: multi-crew pilot licence EBT aircraft maintenance personnel air traffic management personnel Other disciplines to be developed, aerodrome personnel However: Some training and assessment are task-oriented MPL Some training and assessment are based on observing behaviours that contribute to a superior performance on the job Need for a system of competence measurement that looks at the total performance, rather than singles tasks 7 HIGH LEVEL SEMINAR ON LOC-I AND UPRT Scope of the amendment 8 amendment 5 to the Procedures for Air Navigation Services training (PANS-TRG, Doc 9868) provides: Revision/removal/addition of definitions for terms related to competencies a description of how competency-related concepts are interlinked a generic methodology to design Competency-based training and assessment HIGH LEVEL SEMINAR ON LOC-I AND UPRT Key component.

4 Instructional systems design (ISD) methodologies What is a competency? Competency. A dimension of human performance that is used to reliably predict successful performance on the job. A competency is manifested and observed through behaviours that mobilize the relevant knowledge, skills and attitudes to carry out activities or tasks under specified conditions. HIGH LEVEL SEMINAR ON LOC-I AND UPRT Revised 9 Explaining the terms 10 Internal qualities Can be learned & developed Competency actions Competency is manifested and observed through Observable behaviours are single job-related behaviours that can be observed and may or may not be measurable .. those behaviours mobilize the relevant knowledge, skills and attitudes to carry out activities or tasks under specified conditions Knowledge is specific information used to enable a person to apply skills and attitudes and to recall facts, identify concepts, apply rules, procedures or principles Skill is an ability to perform an activity or action.

5 Attitude is a persisting internal mental state or disposition that influences an individual s choice of personal action toward some object, person or event and that can be learned. HIGH LEVEL SEMINAR ON LOC-I AND UPRT 11 1)Relevant competencies are clearly defined for a particular role within an aviation discipline 2)Competencies can be trained for, observed and assessed consistently 3)Common understanding of the competency requirements. 4)Clear performance criteria are established by the training provider for assessing competence. 5)Evidence of competent performance is valid and reliable. 6)Link between competencies and training , required performance and assessment 7) assessment based on multiple observations across multiple contexts. 8)Demonstration of an integrated performance of all the required competencies Competency-based training and assessment (CBTA) principles HIGH LEVEL SEMINAR ON LOC-I AND UPRT Why CBTA matters?

6 1)Enables individuals to reach their highest level of operational capability while ensuring a basic level of competence as a minimum standard. 2)Enables individuals to cope with predictable and unforeseen situations. 3)Is relevant to the job and the context in which the job will be performed 4)Is geared towards learning rather than passing a test 5)Makes full use of available training tools and methodologies 6)Supports continuous learning and performance improvement HIGH LEVEL SEMINAR ON LOC-I AND UPRT 12 First step 13 HIGH LEVEL SEMINAR ON LOC-I AND UPRT ICAO Competency framework 14 The first step is for ICAO to define a Competency framework for a given aviation discipline 1)An ICAO competency framework is a selected group of competencies for a given aviation discipline. 2)The competency framework consists of a set of competencies for one specific discipline 3)It supports the establishment of a systematic approach by providing a generic model that should be adapted to suit the variety of situations/contexts that exist worldwide 4)Each competency has an associated description and observable behaviours 5)It is a generic blueprint to guide the development and maintenance of a strong workforce HIGH LEVEL SEMINAR ON LOC-I AND UPRT ICAO Competency framework for Aeroplane Pilots 15 Structure of an ICAO Competency framework HIGH LEVEL SEMINAR ON LOC-I AND UPRT ASSUMPTIONS 1)Competencies are applied to individuals 2)All tasks require the application of a set of competencies 3)

7 Same set of competencies throughout a career (but with different degrees of performance) 16 HIGH LEVEL SEMINAR ON LOC-I AND UPRT urs Discipline Setting the scene for training 17 HIGH LEVEL SEMINAR ON LOC-I AND UPRT CBTA Workflows HIGH LEVEL SEMINAR ON LOC-I AND UPRT 18 Workflow 1: training specifications Workflow 2- : Adapted competency model Relationship between workflow 1 and 2 Purpose of training Task list Requirements Input: ICAO Competency framework and training specifications 19 20 Adapted competency model. A group of competencies with their associated description and performance criteria adapted from an ICAO competency framework and that an organization uses to develop Competency-based training and assessment for a given role. Adapted Competency Model HIGH LEVEL SEMINAR ON LOC-I AND UPRT Components of an adapted competency model 21 HIGH LEVEL SEMINAR ON LOC-I AND UPRT Workflow 1: training specifications Workflow 2- : Adapted competency model Workflow 2- : training Plan Workflow 2- : assessment plan training required to achieve the competencies Relationship between workflow 1 and 2 Process and tools for gathering reliable evidence 22 Consequential amendments to align with amendment 5 23 Annex 1: Definitions: related to competencies References to PANS-TRG MPL provisions.

8 OAlignment of text with Amdt 5 oReplacement of MPL competency units [phases of flight] by an adapted competency framework based on the ICAO aeroplane pilot competency framework oReview of FSTD descriptions for phases of training oDeletion of Attachment B (levels of competency not aligned with Amdt 5) New provisions for the remote pilot licence HIGH LEVEL SEMINAR ON LOC-I AND UPRT Consequential amendments to align with amendment 5 24 PANS-TRG: New ICAO competency framework for aeroplane pilot Rework entirely the MPL Chapter, deleting some material transferring other Revise the Evidence-based training (EBT) Chapter Revise the UPRT Chapter (minor) New TEM Chapter (from existing attachment) New Remote Pilot Licence competency framework Rework entirely the maintenance personnel part Minor update to ATM personnel part New ICAO flight dispatcher competency framework HIGH LEVEL SEMINAR ON LOC-I AND UPRT Consequential amendments to align with amendment 5 25 Align manuals with Amdt 5.

9 Doc 9379 - Manual of Procedures for Establishment and Management of a State's Personnel Licensing System + Remote pilot licence + guidance on examiners Doc 9841 - Manual on the Approval of training Organizations + recognition of the approval of foreign ATOs Doc 10011 - Manual on Aeroplane Upset Prevention and Recovery training + Rev 3 of the Airplane Upset Prevention and Recovery training Aid Doc 9625 - Manual of Criteria for the qualification of Flight Simulation training Devices Doc 9995 - Manual on Evidence-Based training HIGH LEVEL SEMINAR ON LOC-I AND UPRT Consequential amendments Timeline 26 Annex 1 and PANS-TRG: applicable November 2020 All manuals: before Q3 2018 HIGH LEVEL SEMINAR ON LOC-I AND UPRT CAA Oversight of CBTA 27 Establish a regulatory framework for the CBTA to be conducted in an ATO, including the ATO s duties and role Guidance material to cover: MPL and RPL CBTA design conduct of CBTA and final assessment CBTA instructor qualifications, including assessment CBTA designated examiner qualifications CBTA inspector qualifications Inspector procedures for C BTA oversight (approval/surveillance) HIGH LEVEL SEMINAR ON LOC-I AND UPRT CAA Oversight of CBTA 28 Qualifying inspectors for approving ATO s CBTA training programmes: specific training C BTA principles in PANS-TRG.

10 Generic methodology, ICAO Competency framework , adapted competency model - including performance criteria Regulations on C BTA Understanding and use of CAA guidance material Approval of CBTA training programme: adapted competency model, training and assessment plans, training conduct and course evaluation Approval of qualifications for the CBTA instructors and ATO designated examiners Duties: conduct a normal ATO approval process (as for traditional ATOs) PLUS Approve CBTA training , including the adapted competency model and the training and assessment plans, and inspection of actual conduct of CBTA Approve qualifications programmes for CBTA instructors & designated examiners Qualify FSTDs and approve systems used for CBTA training , if applicable Conduct surveillance of the ATO and its training programme + monitor graduates performance on the job HIGH LEVEL SEMINAR ON LOC-I AND UPRT + OJT Proof-of -concept trial ICAO Competency framework CAA Oversight of C BTA The burden for the CAA is high in terms of new regulations and producing guidance material Inspector training is paramount and significant CBTA is a new domain The inspector job is more complex than for traditional training .