C-2 S - Defense Technical Information Center

1 AFHRL-TR-86- 22ON-THE-JOB training : DEVELOPMENT ANDASSESSMENT OF A METHODOLOGY FORGENERATING TASK PROFICIENCYEVALUATION INSTRUMENTS .''.',HSU CV)mRonnie Warm-mTl J. Thomas RothMJean A. Fitzpatrick Applied Science Associates, Box 1072 IButler, Pennsylvania 16003 training SYSTEMS DIVISIONR Lowry Air Force Base, Colorado 80230-5000ES September 1986 Final Report for Period January -October 19850 URD Approved for public release. distribution is .EC-2 S LABORATORYOCT 2101986 AIR FORCE SYSTEMS COMMANDBROOKS AIR FORCE BASE, TEXAS " b% %%'NOTICE-When Government drawings, specifications, or other data are used for anypurpose other than in conrection with a definitely Government-relatedprocurement, the United States Government incurs no responsibility or anyobligation whatsoever.

2 The fact that the Government may have formulated or 4'-in iny '. y supplic th, s.'id c-rawing , ,veciflcatiuns, or o~her data, to be regarded by implication, or otherwise in any manner construed, aslicensing the holder, or any other person or corporation; or as conveyingany rights or permission to manufacture, use, or sell any patented -invention that may in any way be related thereto,The Public Affairs Office has reviewed this report, and it is releasable tothe National Technical Information Service, where it will be available to,dthe general public, including foreign nationals, .This report has been reviewed and is approved for S.

3 WALKER '-Contract Moni torI,JOSEPH Y. YASUTAKE, Technical DirectorTraining Systems DivisionDENNIS W, JARVI, Colonel, USAFC omnmander" .r.* .-de -47. -' -*. CLASSIFICATION OF THIS PAGEREPORT DOCUMENTATION PAGEla. REPORT SECURiTY CLASSIFICATION lb RESTRICTIVE SECURITY CLASSIFICATION AUTHORITY 3 DISTRIBUTION AVAI-ABILITY OF REPORTA pproved for public release; distribution is DECLASSIFICATION / DOWNGRADING SCHEDULE4 PERFORMING ORGANIZATION REPORT NUMBER(S) 5 MONITORING ORGANIZATION REPORT NUMBER(S)AFHRL-TR-86-22 %"..".6a. NAME OF PERFORMING ORGANIZATION 6b OFFICE SYMBOL 7a NAME OF MONITORING ORGANIZATION "(if applicable)-" "".

4 Applied Science Associates, Inc training Systems Division6c. ADDRESS (City, State, and ZIP Code) 7b ADDRESS (City, State and ZIP Code) Box 1072 Air Force Human Resources LaboratoryButler, Pennsylvania 16003 Lowry Air Force Base, Colorado 80230-5000Ba. NAME OF FUNDING ISPONSORING 8b OFFICE SYMBOL 9 PROCUREMENT INSTRUMENT DENTIFICATION NUMBERORGANIZATION (If applicable)* Air Force Human Resources Laboratory HQ AFHRL F3361S-82-C-00048c. ADDRESS (City, State, and ZIP Code) 10 SOURCE OF FUNDNG NUMBERSB rooks Air Force Base, Texas 78235-5601 PROGRAM PROJECT TASK WORK UNITELEMENT NO NO NO ACCESSION NO62205F 1121 09 1011 TITLE (Include Security Classitfication)On-the-Job training : Development and Assessment of a Methodology for Generating Task Proficiency EvaluationInstruments12.

5 PERSONAL AUTHOR(S)Warm, R.; Roth, Fitz artick! ,.13a. TYPE OF REPORT 13b. TIME COVERED 14 DATE OF REPORT (Year, Month, Day) 15 PATt COUNTF inal FO Af~Tfl~jSeptember 19861316. SUPPLEMENTARY NOTATION17. COSATI CODES 18 SUBJECT TERMS (Continue on reverse if necessary, and identify by block number)FIELD GROUP SUB-GROUP assessment instrument OJT evaluation05 09 assessment methodology performance assessmenton-the-job training performance evaluation (Continued)19 ABSTkACT (Continue on reverse if necessary and identify by bloxk number)"This document describes the development and assessment of a methodology for generating on-the-job training (OJT} task proficiency assessment in-truments.)

6 The Task Evaluation For', (TEF) develprpment i cc: rLs werederived to address previously identified deficiencies in the evaluation of OlJT task proficiency. The TEFdevelopment procedures allow subject-matter experts (SMEs), without experience in assessment methodology, toconstruct task proficiency assessment instruments which can be used by OJT supervisors to assess traineeproficiency at specific tasks. The reliability and validity of the procedures and of the forms resulting fromprocedural application were examined in two Air Force career fields (Aircraft Maintenance and SecurityPolice/Law Enforcement). The results of the procedural assessment indicated that SMEs could reliably applythe development procedures to construct TEFs which accurately and completely depict the crltir l ,N-Pcts oftask performance.

7 The operational ass-'_ent demonstrated that evaluator subjectivity in the assignment .-nrcs and pa sifail decisions is minimized when TEFs are used. In addition, it was shown thatevAluations conducted with TEFs provide end scores and pass/fail decisions which accurately reflect the numberand types of errors oetected in task performance. The combined results of the two assessments demonstrate the20 DISTRIBUTIONI AVAILABILITY OF ABSTRACT 21 ABSTRACT SECURITY UNCLASSIFIED/UNLIMITED 0 SAME AS RPT [ 1 TDC USERS22a NAME OF RESPONSIBLE INDIVIDUAL 22b TFLEPHONt (Include Ar'a Corde) 22c OFFICE SYMBOLAncX A- Perrion STINFO Office (512) 536-3877 AFHRI_/TSRDO FORM 1473,84 MAR 83 APR edition may be used until exhausted SECURITY CLASSIFICATION OF THIS PAGEAll other editions are obsolete ip -4 Unclassified".]

8 - ", .-18. (Concluded)performance standards-task evaluation-task performancetask proficiency19. (Concluded)potential of the TEFs in the OJT environment. This document includes an overview of the theoretical approach, anddetailed descriptions of the assessment methodologies and results. The final section of this document contains .recommenlations for the integration of the TEF development procedures and the TEFs into the OJT ,.-pp -R' "),'-S-e ",..* ..S, .---. \. -- .~. -SUMMARYThe ability to accurately and objectively assess an individual's of performance on the job is important to Air Force systems forpersonnel selection, assignment, training , and utilization.

9 Currently,the on-the-job training (OJT) supervisor is responsible for theevaluation of an individual trainee's task proficiency. Concern has expressed within the Air Force training community regarding thevariability which exists among supervisors in the evaluation of OJT 9task proficiency. In particular, there exists a lack ofstandardization with regard to the construction of assessmentinstruments, administration of performance evaluations, the scoring ofresults, and providing Task Evaluation Form OJT task proficiency assessment system _(TEF System) which was developed and evaluated by Applied ScienceAssociates, Inc. (ASA), enables subject-matter experts (SMEs), without --experience or training in assessment methodology, to developinstruments which can be used to assess OJT task proficiency.

10 SMEsapply prescribed development procedures to a specific task in order togenerate a Task Evaluation Form (TEF). Evaluators use the TEFs toconduct evaluations of OJT task performance. The TEF developmentprocedures combine logical analysis and a critical incident technique,resulting in the identification and description of task elements which --are critical to successful task performance. Potential performeractions and outcomes are divided into evaluation areas: Time/Speed;End Product; Sequence-Following; Safety; and Tools. Equipment, and4 Materials Use. The TEF development procedures guide the developer indetermining which evaluation areas are critical to successful taskperformance.