Measuring ROI: The Fif - Complete Learning Solutions

1 By Jack J. PhillipsMeasuring ROI:The Fif10 Technical & Skills Training t April 1996 Training departments are increasingly being asked to justifyevery dollar spent. Here s an approach to calculating return oninvestment using a time-honored training too long, the training and developmentprocess has escaped the scrutiny of account-ability. While expenditures have grown,many training departments have not takenthe extra step to show the payoff of theirefforts particularly the more elaborate,comprehensive, and expensive and techniques to measure return oninvestment (ROI) are now available, and the pro-cess has become reliable andacceptable. Measuring thereturn on training investmentshould be a requirement inmost organizations, at least forsome dismissing the ideaof Measuring your company sreturn on training investment,consider the rewards of such anevaluation. An example of howvaluable training can be is evi-dent in the literacy training pro-gram at Magnavox Electronics Systems Company,West Coast Division, in Torrance, literacy programs are undertakenbecause they are needed, but little attention ispaid to the program s economic returns.

2 Mag-navox wanted to know if this type of trainingreturned an economic dividend. After its 18-week literacy program was initiated, whichcovered verbal and math skills, it was shown tohave a significant payoff for the company. Thebenefits of the program (reduced scrap,rework, and increased productivity) were con-verted to dollar values. The program yielded acost-benefit ratio of :1 and a 741 percentreturn on training investment! And while thesenumbers are impressive, the program designersconsider them every corner of the trainingand development field, the pres-sure to measure the return oninvestment is increasing. Atsome time or another, virtuallyevery organization will face thisimportant issue. Many progres-sive organizations are taking alogical and methodical approachto developing ROI for a limitednumber of programs, using asampling basis. Six trends haveincreased the interest in, and use of, ROI measure-ment in training and development:tTraining and development budgets are con-tinuing to grow, which creates more pressure and development are linked to com-petitive strategies, which make them importantareas for Measuring program programs have failed to deliver what was expected; consequently, program sponsors have requested ROI concern for accountability in all functionsin an organization is increasing; thus, the trainingand development function becomes one of manysupport efforts under executives in a large number of organiza-tions now require ROI justify their contribution,trainers have increased theirinterest in the ROI NEWMODELFOREVALUATIONLEVELSD eveloping ROI for trainingrequires a key modification ofa classic model .

3 The four-levelframework developed by Don-ald kirkpatrick in 1959 does not focus directly onthe ROI issue. As shown in figure 1 (page 12), kirkpatrick defines Level 4 evaluation as theresults linked to training. These results could takethe form of reduced absenteeism and turnover,quality improvement, productivity, or even costreduction. But this level of evaluation does notrequire a specific monetary value (cost savings) tobe determined. To obtain a true ROI evaluation ,the monetary benefits of the program should becompared to the cost of implementation in order tovalue the investment. In effect, this process movesevaluation to the next level Level 5 in our revisedKirkpatrick model . Thus, the fifth level of evalua -tion is developed by collecting Level 4 data, con-verting the data to monetary values, and comparingthem to the cost of the program to represent thereturn on training practice, many organiza-tions are taking evaluation tothis new level for a few selectedcourses, often using some formof sampling.

4 When the ROIformula is developed, eval-uation is conducted at all fivelevels. For example, in the Magna-vox case, Level 1 evaluation reaction of employees wasmeasured by post- course sur-veys. Level 2 Learning was measured by TABE(Test of Adult Basic Education) scores beforeand after training. At Level 3, changes in thebehavior of employees were measured by dailyefficiency ratings. At Level 4, business resultswere measured through improvements inproductivity and reductions in scrap and , at Level 5, ROI was calculated byTechnical & Skills Training t April 1996 11fth Level of EvaluationIn This Storytcost/benefit analysistevaluationIllustration: Rob Colvin/SISconverting productivity and quality im-provements to monetary values and com-paring these to the full program costs toyield an ROI PROCESSC alculating ROI requires a process model ,as depicted in figure 2 below. The variouselements of evaluation (design instru-ments, levels, and purposes) form thespecific data collection plans.

5 A variety of data collection tools, ranging fromquestionnaires and surveys to monitoringon-the-job performance, are available data is collected, the next step ofthe ROI analysis begins with deliberateattempts to isolate the effects of trainingon the data items. At least ten strategieshave been used to accomplish this:tuse of controlsttrend line analysis (time series)tforecasting methodstparticipant estimates of training impacttsupervisor estimates of training impacttmanagement estimates of trainingimpacttcustomer inputtexpert estimates of training impacttsubordinate input on training impacttcalculations/estimations of the impactof other next step is to convert collecteddata to monetary values. This requires adirect conversion of hard data, such asquantity, quality, cost, or time, which is aneasy task for some programs such as tech-nical training. For soft data, the task ismore difficult, although a variety of tech-niques are used to place values on theimprovements.

6 Among the techniquesused arethistorical coststsupervisor estimationtmanagement estimationtexpert opiniontparticipant estimationtexternal next stepis to calculate thecosts for the pro-gram. Althoughthere has alwaysbeen a need tocapture trainingcosts, the need isamplified withmore attention onaccountabilityand the ROI cal-culation. The ROIformula is theannual net pro-gram benefits divided by program costs,where the net benefits are the monetaryvalue of the benefits minus the costs of theprogram. The ROI formula is as follows:ROI (%) = Benefits Costs x 100 CostsThis model also recognizes that thereshould be intangible benefits that will bepresented along with the ROI STRATEGIES/BESTPRACTICESA lthough it is difficult to uncover compa-nies precise strategies, a recent searchidentified over 2,000 organizations thatcould be contacted in an effort to deter-mine the nature and status of the ROI pro-cess.

7 Although this was not a carefullydesigned research project, it presented areview of the efforts of many progressiveorganizations. Several common strategiesbegan to emerge that can be consideredbest practices for calculating an ROI intraining and development. A review ofsuch best practices is as follows:Set targets for each evaluation the complexity of the eval-uation levels described earlier. Some12 Technical & Skills Training t April & Planned Action2. Learning3. Job Applications4. Business Results 5. Return on InvestmentQuestionsWhat are participants reactionsto the program, and what do theyplan to do with the material?What skills, knowledge, orattitudes have changed and by how much?Did the participants apply whatthey learned on the job?Did the on-the-job applicationproduce measurable results?Did the monetary value of theresults exceed the cost for theprogram?Five Levels of EvaluationFigure 1 evaluation InstrumentsEvaluation LevelsEvaluation DesignEvaluation PurposesIsolatingthe Effectsof TrainingCollectingPost-Program DataConverting Data to Monetary ValueCalculating the Return on InvestmentIdentifyingIntangible BenefitsTabulating Program CostsPolicy StatementProcedures and GuidelinesStaff SkillsManagement SupportTechnical SupportOrganizational CultureSignificant InfluencesROI Process in HRDF igure 2organizations attempt to manage the pro-cess by setting targets for each level.

8 Atarget is the percentage of training pro-grams measured at that level. For exam-ple, at Level 4, where it is difficult tomeasure, organizations have a low levelof activity usually less than 20 5 evaluation ROI is even lesslikely to occur usually around five per-cent, reflecting the complexity of a pro-cess that commands significant resourcesand process of establishing evaluationtargets has two important advantages. First,it provides measurable objectives for thetraining staff to clearly measure progressfor all programs or any segment of the pro-cess. Second, adopting targets focusesmore attention on the accountability pro-cess, communicating a strong message tothe training staff about the commitment tomeasurement and at the micro and evaluation usuallyfocuses on an individual program or a fewtightly integrated courses. The ROI pro-cess is more effective when assessing thedirect payoff of an individual to evaluate a group of coursesconducted over long periods of time isquite difficult.

9 The cause and effect rela-tionship becomes more confusing andcomplex. Also, it is inappropriate toattempt to evaluate an entire function suchas quality training or technical this reason, evaluation must be amicro-level sampling for ROI the desired level of ROI cal-culations is an important issue. There is noprescribed formula and the numberdepends on many variables, includingtstaff expertise on evaluationtnature and type of training programstresources that can be allocated to the processtsupport from management for trainingand developmenttorganization s commitment to mea-surement and variables specific to the orga-nization may enter the process. Mostorganizations settle on evaluating one ortwo sessions of their most popular pro-grams. For example, the federal govern-ment s Office of Personnel Managementhas developed an ROI calculation for oneof its most popular courses Introductionto Supervision.

10 Still others may select aprogram from each of their major trainingsegments. In a large bank with six trainingacademies, a program is selected fromeach academy, each year, for an ROIcalculation. For organizations implementing theROI concept for the first time, only onecourse should be selected for a calculationas part of the ROI Learning curve. In thefinal analysis, the selection of programsfor ROI calculations should yield a level ofsampling where top management is com-fortable in its assessment of the trainingand development function. nEditor s Note:The Magnavox case is justone of 18 cases reported in MeasuringReturn on Investment,recently publishedby the American Society for Training and Development as volume one of its & Skills Training t April 1996 13 Jack J. Phillipsis president of Perfor-mance Resources Organization ( Box1969, Murfreesboro, TN 37133-1969;615/896-7694).