Best Practices in Pilot Selection

1 DOT/FAA/AM-19/6 Office of Aerospace Medicine Washington, DC 20591 Best Practices in Pilot Selection Dana Broach David Schroeder Kevin Gildea February 13, 2019 Final Report NOTICE This document is disseminated under the sponsorship of the Department of Transportation in the interest of information exchange. The United States Government assumes no liability for the contents thereof. _____ This publication and all Office of Aerospace Medicine technical reports are available in full-text from the Civil Aerospace Medical Institute s publications website: Technical Documentation Accession 's Catalog and SubtitleBest Practices in Pilot DateFebruary 13, Organization (s) Organization Report Broach, Kevin Gildea, & David J.

2 Organization Name and Address10. Work Unit No. (TRAIS)Aerospace Human Factors Research LaboratoryAAM-500 Box 25082 Oklahoma City, OK 73125 11. Contract or Grant Sponsoring Agency Name and AddressOffice of Aerospace MedicineFederal Aviation Administration800 Independence Avenue, S. , DC 2059113. Type of Report and Period CoveredFinal 15. Supplementary Notes16. AbstractDemand for new civilian pilots continues to grow as the world s aviation system expands. and regional air carriers will need about 1,900 to 4,500 new pilots per year ( Accountability Office, 2014).

3 Selection of the new pilots is a critical human resourcemanagement challenge for operators. We review civilian Pilot Selection procedures relative to a setof seven best Practices : 1) Conduct a job analysis; 2) Define measurable, observable jobperformance metrics; 3) Identify and use reliable and valid predictors; 4) Conduct an appropriatevalidation study; 5) Determine cut-scores (pass/fail) on tests based on predicted job performance;6)Evaluate the fairness of tests and cut-scores; and 7) Document the analyses. We reviewed 15pilot test batteries and Selection processes for 2 and multiple foreign airlines based on publiclyavailable information.

4 Overall, adherence to best Practices as defined by the relevant legal andprofessional guidelines, standards, principles, and Practices for the development, validation, anduse of employee Selection procedures, was inconsistent. Relatively few of the test batteries werebased on a recent or current job analysis. There is evidence of psychometric reliability and usefulvalidity for the test batteries. However, there is no evidence on the reliability or validity of theinterviews used in Pilot Selection . The primary job performance criterion is success or failure intraining.

5 No data are available on the comparative Selection (pass) rates by demographic groupson the test batteries. Technical reporting and documentation on Pilot test batteries is better inEurope than in the Overall, we agree with the 2012 conclusion reached by the International AirTransport Association (IATA) that Pilot Selection programs worldwide and in the do not appearto have a strong scientific basis. We make two recommendations to improve the state-of-the-art incivilian Pilot Key WordsPilotPersonnel SelectionJob PerformanceJob AnalysisAbility Tests18.

6 Distribution StatementDocument is available to the publicthrough the National TechnicalInformation Service, Springfield, VA 2216119. Security Classif. (of this report)UNCLASSIFIED 20. Security Classif. (of this page)UNCLASSIFIED 21. No. of Pages 7322. PriceForm DOT F (8-72) Reproduction of completed page authorized i ACKNOWLEDGMENT Research report in this paper was conducted under the Air Traffic Program Directive/Level of Effort Agreement between the Human Factors Division (ANG-C1), FAA Headquarters, and the Aerospace Human Factors Research Division of the FAA Civil Aerospace Medical Institute.

7 Ii TABLE OF CONTENTS Acknowledgment .. i Table of Contents .. ii Table of Figures .. iv Table of Tables .. v Introduction .. 1 Organization of report .. 4 Terminology .. 4 Chapter 1: Best Practices in personnel Selection processes .. 7 Conduct a job analysis .. 7 Define job performance measures .. 8 Identify predictors .. 9 Evaluate relationship of KSAO measures to job performance criteria .. 9 Set cut-scores and/or decision rules .. 10 Evaluate the fairness of the Selection process .. 10 Assess Selection process utility.

8 11 Challenges in applying these best Practices .. 12 Chapter 2: Pathways for Becoming a Pilot .. 15 Pathway 1: University-based Pilot training programs .. 15 Pathway 2: Independent flight schools/flight academies .. 16 Pathway 3: Former military Pilot .. 17 Pathway 4: Multi-Crew Pilot License (MPL) .. 17 Pathway 5: Flow-through agreements .. 18 Chapter 3: Civilian Pilot Selection .. 20 Previous reviews of (civilian) Pilot Selection 20 Review of Pilot aptitude test batteries .. 20 Best practice #1: Conduct a job 21 Best practice #2: Define job performance measures.

9 27 Best practice #3: Identify predictor measures.. 27 Best practice #4: Evaluate relationship of KSAO measures (predictors) to job performance criteria.. 27 Best practice #5: Set cut-scores and/or decision rules.. 28 Best practice #6: Evaluate the fairness of the Selection procedures.. 30 Best practice #7: Assess Selection procedure utility.. 30 Innovations in Pilot aptitude assessments .. 30 Personality assessment.. 30 Social skills or competencies.. 30 Assessment Centers.. 32 Interviews .. 33 Chapter 4: Screening in Other Transportation-related Occupations.

10 35 Air traffic control specialist (ATCS; FG-2152) .. 35 Railroad dispatcher .. 39 Comparison of Selection of pilots with Selection in other transportation occupations .. 40 Chapter 5: Summary, Recommendations, and Research Questions .. 41 iii Summary of best Practices & issues .. 41 Recommendations .. 41 Future research questions .. 42 References .. 44 Footnotes .. 59 Appendix A: Best Practices checklist for evaluation of Pilot Selection / Selection 1 Job Analysis .. 1 Predictors .. 1 Criteria .. 1 Validation .. 1 Decision Rules.