Can We Really Measure Training ROI?

1 UT Center for Industrial ServicesCan We Really Measure Training ROI?(Do We Really Want To?)Bill StetarUniversity of Tennessee2003 DOE Annual Training Symposium "Transformation for Success"Stetar - DOE Training Symposium - Aug 20032UT Center for Industrial ServicesPurpose of this Session.. advancements in models that approach Training ROI systematically and rigorously in order to gain a truer measurement of Training ways for Training providers to confidently provide data that supports justification of Training from a business how Training program design should start with business results and not learning between predictive and retrospective - DOE Training Symposium - Aug 20033UT Center for Industrial ServicesTangible results cost, quality, quantityResults4 ResultsChanges in job behaviorBehavioral change3 BehaviorPrinciples, facts, techniquesLearning, Attitudes2 LearningLearner satisfaction with the programSatisfaction1 ReactionFocusType of MeasureLevelThe Kirkpatrick ModelStetar - DOE Training Symposium - Aug 20034UT Center for Industrial ServicesOrigin of " Training ROI" Key proponent Jack Phillips Expressed as a ratio / percentage % ROI = (Net Benefits / Costs)

2 X 100 Level 5 measurement Effort to speak the vernacular Gain acceptance & credibilityStetar - DOE Training Symposium - Aug 20035UT Center for Industrial ServicesWhy ROI?About $ billion to $ billion is wasted annually on ineffective Training programs American industry is spending billions and billions on trainingprograms and doing no evaluation of their effectiveness. You have to Measure it. -Cary Cherniss, Rutgers University10/07/1998 Stetar - DOE Training Symposium - Aug 20036UT Center for Industrial ServicesWhy ROI?A study of 15 countries in the Organization for Economic Cooperation and Development found that the majority of enterprises believe employee Training is responsible for "productivity improvements, greater workforce flexibility, savings on materialand capital costs, improved quality of the final product or service,and a more motivated workforce."However, many companies have not measured the benefits and related them to the cost of Training in a way that reveals the rate of return on a firm's investment.

3 Apparently there is no other workplace issue on which so much money is spent with as little accountability as & Realities No. 16: Return on Investment in Training ,Bettina L. Brown, Center on Education & Training for Employment, Ohio State University, ERIC/ACVE, 2001 Stetar - DOE Training Symposium - Aug 20037UT Center for Industrial ServicesCritics of Level 5 Say.. No more than a restatement of Level 4 results Executive management has not bought into the concept ROI does not Measure worth or "soft" benefits, it is only one tool that can Measure results (Ernst & Young) Accounting world does not view Training as a capital investmentStetar - DOE Training Symposium - Aug 20038UT Center for Industrial ServicesROI vsCost-Benefit AnalysisSource:Measuring Return on Investment (ROI), David , , Proceedings of the Fourth Annual National Human Services Training Evaluation Symposium, May 23-25, 2001, UC Berkeley"Doing Training ROI analysis is a fancy way of doing cost-benefit analysis.

4 " -Olga Bulatova, Direct of Ernst & Young Professional Education Center in the CIS CountriesStetar - DOE Training Symposium - Aug 20039UT Center for Industrial ServicesNo two models are alike..Showing That Enterprise Training Pays Off, Janelle Moy, NCVERS tetar - DOE Training Symposium - Aug 200310UT Center for Industrial ServicesMost Have These In Common Direct & Indirect Costs Attributable to the Project or Program Benefits Must be Converted to Equivalent Monetary Units Measurement is Absolutely Critical Baseline & Post-Mortem ROI or CBA must be reserved to mission-critical programs Assumptions, assumptions, & assumptionsStetar - DOE Training Symposium - Aug 200311UT Center for Industrial ServicesMerck Model Developed early-1990's in response to CEO's inquiry: "What are We Getting for Our Money?" Situation at Merck: 1986 $25 million spent on Training 1991 -$70 million spent on trainingStetar - DOE Training Symposium - Aug 200312UT Center for Industrial ServicesMerck Model (cont'd) Constructed its own version of the "Utility Model" Value of an activity is derived based on statistical information and a clear understanding of how the business - DOE Training Symposium - Aug 200313UT Center for Industrial ServicesKey Terms: Validity & Reliability Valid: measures what it is supposed to Measure Reliable: consistent and reproducibleStetar - DOE Training Symposium - Aug 200314UT Center for Industrial Services The basic Kirkpatrick, Cost-Benefit model appears to have widespread usage or application Predictive measurement is as crucial as retrospective measurement Causal analysis methods being used to link results to action Training design predicated by quantifying desired business results What are the trends?

5 Stetar - DOE Training Symposium - Aug 200315UT Center for Industrial ServicesTrends (cont'd) Shift in emphasis:From " Training " to "learning" Key proponents:Motorola, IBM & Xerox(to name a few) - DOE Training Symposium - Aug 200316UT Center for Industrial ServicesWhere's the Low Hanging Fruit?Business improvement Training provided to increase skill capacity, counteract performance gaps, support business goals Due Diligence Training you are required to do either by law or mandateStetar - DOE Training Symposium - Aug 200317UT Center for Industrial ServicesFor "Due Diligence" Activities, typically, the measurement is internally focusedFor per studentTravel cost reductionsMaterial costsSupport costsEmphasis is on "Cost-Avoidance" ROIS tetar - DOE Training Symposium - Aug 200318UT Center for Industrial Services"Business Improvement" Activities are Ideal Candidates for ROI AnalysisROI Perspective Specifies desired outcomes beforehand Captures baseline data Establishes measurement criteria what & when Identifies potential variablesStetar - DOE Training Symposium - Aug 200319UT Center for Industrial ServicesBenefits Target Key Areas Productivity & Efficiency Sales & Profitability Quality of Products & Services Customer Service & Satsifaction Health & Safety Organizational Learning & Development Organizational Climate, Culture & PracticesSource.

6 Showing That Enterprise Training Pays Off, Janelle Moy, NCVERS tetar - DOE Training Symposium - Aug 200320UT Center for Industrial ServicesProductivity & Efficiencyoproduction costs per unitoproductivity targets met/exceededoproduction/completion time per unit ( forms, loans, clients,projects)ooutput (per worked hour, per shift, per machine, or per annum)oovertime (quantity, cost)oimproved innovation in products/servicesoinduction time for new employeesoproductivity of new employeesoequipment/facility/assetutilis ation( down time due to machine stoppages, shiftochangeover time)oequipment maintenance (costs or repair time), or replacement costsoresponse time ( to service calls or orders)ocapacity of staff to solve routine and non-routine problems ( supervision time required)ostaffing requirements and workforce flexibility ( dependenceon casual/contractlabour)Source: Showing That Enterprise Training Pays Off, Janelle Moy, NCVERS tetar - DOE Training Symposium - Aug 200321UT Center for Industrial ServicesSales & Profitability overhead costs operating costs operating costs as a percentage of total costs/revenue value of contracts won, loans processed revenue/income/sales (monthly, annually, per employee, per team,per branch or store) market share (number of customers, dollars spent, unit volume sold) sales to new customers group operating profit profit per employee stock market performance ( shareholder return)Source.

7 Showing That Enterprise Training Pays Off, Janelle Moy, NCVERS tetar - DOE Training Symposium - Aug 200322UT Center for Industrial ServicesQuality of Products & Servicesoon time provision of products/servicesowastage, reject, error or rework ratesoconformance record with quality specifications ( batch yields, throughput of invoices)oachievement/maintenance of quality ratingocompliance with quality, legal and/or ethical requirementsoachievement of quality awardSource: Showing That Enterprise Training Pays Off, Janelle Moy, NCVERS tetar - DOE Training Symposium - Aug 200323UT Center for Industrial ServicesCustomer Satisfaction & Servicesocustomer satisfaction levels (with timeliness, availability, quality and price of goods and services)ocustomer relationships and experiencesorepeat business (customer retention or loyalty)onew business resulting from client referralsomore/new customers or markets ( contracts won, loans processed, funding awarded)olost businessonumber of complaintsSource: Showing That Enterprise Training Pays Off, Janelle Moy, NCVERS tetar - DOE Training Symposium - Aug 200324UT Center for Industrial ServicesHealth & Safetyoaccidents or injuries (number, time lost, compensation costs, premium cost/rating)osafety critical incidents (number, cost)ocompliance with safety and health requirements oviolation of safety rulesoimproved response to crisesSource.

8 Showing That Enterprise Training Pays Off, Janelle Moy, NCVERS tetar - DOE Training Symposium - Aug 200325UT Center for Industrial ServicesOrganization Learning/Developmentoperformance appraisal ratingsoachievement oforganisationalcompetency profile requirements ( to meet accreditation or licensing requirements, new operating environments or facilitateorganisationalexpansion)onumbe r/percentage of employees with nationallyrecognisedqualificationsointer nal promotions resulting from employee competence and performanceotraining awards receivedoemployee perceptions of Training and development opportunitiesoalignment with human resources, business and strategic planningSource: Showing That Enterprise Training Pays Off, Janelle Moy, NCVERS tetar - DOE Training Symposium - Aug 200326UT Center for Industrial ServicesOrganization Culture/Climateoemployee retention/turnover/recruitment ( numbers, costs)oabsenteeismodisputes/grievances (number, cost or time lost)onumber of employee suggestions (submitted or implemented)oemployee satisfaction and motivationointerpersonal relationships and commitment to team goalsoparticipation in teams and committeesoteam performanceointernal communication and information systemsoimplementation of new work practicesostandardisationof work practicesoimplementation/maintenance of a service cultureocontribution to re-engineering andrefocussingof enterpriseSource: Showing That Enterprise Training Pays Off, Janelle Moy, NCVERS tetar - DOE Training Symposium - Aug 200327UT Center for Industrial ServicesROI Starts with FEA First & foremost.

9 Determine if there is a problem or a performance gap worth doing something about Quantify the cost of the problem Specify a solution that costs less than the problem Specify the outcomes of any Training intervention to be applied Set up your measurement methodology validity & reliabilityIt's like risk analysisStetar - DOE Training Symposium - Aug 200328UT Center for Industrial ServicesTimet1t2 Program BenefitMeasuring ROI at time t1 will not be able to justify HRD investmentProgram Cost$Benefit0$CostSource: Greg Wang, PhD, DPT Consulting Group, Inc. 2000 Stetar - DOE Training Symposium - Aug 200329UT Center for Industrial ServicesPay me now or pay me & ImprovealysisAnStetar - DOE Training Symposium - Aug 200330UT Center for Industrial ServicesTarget Audience Does Matter!A FRESH LOOK AT RETURN ON INVESTMENTA White Paper by Jay Cross, 2000, SmartForceStetar - DOE Training Symposium - Aug 200331UT Center for Industrial ServicesThank you.

10 Bill StetarUniversity of