Metrics to Evaluate R&D Groups - mit.edu

1 Metrics to Evaluate R&D GroupsPhase I: Qualitative InterviewsFlorian ZettelmeyerandJohn R. Hauser March 7, 1995 UpdateCOMMENTS WELCOME!Florian Zettelmeyer is a student at the Massachusetts Institute of Technology, Sloan Schoolof Management, 38 Memorial Drive, E56-345C, Cambridge, MA 02142, (617) 253-3198, (617)258-7597 fax, John R. Hauser is the Kirin Professor of Marketing, MassachusettsInstitute of Technology, Sloan School of Management, 38 Memorial Drive, E56-314, Cambridge,MA 02142, (617) 253-2929, (617) 258-7597 fax, research was funded by the International Center for Research on the Management ofTechnology, Sloan School of Management, We wish to thank the managers, scientists, andengineers who donated their time to talk to us about this important to Evaluate R&D GroupsPhase I: Qualitative InterviewsAbstractWe describe the results of in-depth qualitative interviews with Chief Executive Officers,Chief Technical Officers, and researchers at ten large research-intensive organizations.

2 In theseinterviews we explored how these organizations measure success in the R&D mission and how theyprovide incentives to managers and researchers in R&D. We gained insight into the three tiers ofthe R&D mission: exploring the tools of the future, creating the tools, and pioneering the use of thetools. We learned how these tiers relate to the university and the business units. Metrics of successvary by tier, in part, because the role of corporate investment, the role of managers, and the role ofinternal customers vary by report provides an initial perspective on R&D Metrics from the viewpoint of practicingmanagers and researchers. This perspective, which is Phase I of our research project, suggestsresearch topics which we will explore in greater depth in subsequent Phase II. We summarize theseresearch directions at the end of this 1990, in the alone, private corporations spent over $70 billion on research anddevelopment (R&D).

3 This was approximately of total sales and of total profits. Similar spending occurred in Japan (over $27 billion) and Germany (over $19 billion). In order tojustify investments of this magnitude, private industry must believe that the return on investment(ROI) for R&D exceeds the firms' , investments in R&D are inherently difficult to Evaluate . While the costs areclearly visible and are recorded as they incur, the return on R&D investment may occur many yearsin the future and may be hard to attribute to a specific project. This is particularly true if R&Ddevelops a strategic technical competence that is applied across many projects and pervadeseverything that the organization does. Furthermore, decisions on which projects to fund or whichstrategic competence to pursue must be made under considerable market uncertainty andtechnology long-term goal is to understand how best to allocate R&D investments, but in order toallocate investments we must first understand how firms Evaluate the R&D function.

4 We want toknow how firms decide whether a project, a program, or a strategic direction succeeds and we wantto know how top management rewards and motivates R&D scientists, engineers, and managersbased on their past performance and/or their potential. Once we understand how firms nowperform these tasks we will be better able to develop a theory to describe and to improve this paper we describe phase I of our research. In this phase we spoke to managers andresearchers at a variety of firms who invest heavily in R&D. We report here what they told us. Wehave purposefully chosen not to structure this description within any previously publishedframework. Instead we have attempted to allow the managers and researchers themselves todescribe the world in which they operate. We chose this strategy because many of the organizationsin our sample have recently changed the way they Evaluate R&D or are in the process of doing so1.

5 1 For example, one organization, which was once run as a central laboratory, is now aligned with the business units,another organization has given the business units more power by allowing them to fund R&D directly, anotherorganization reorganized so the manufacturing and R&D were "laid together," another organization moved its R&Dfrom a center of excellence to the divisions, and another organization has restructured its R&D division to focus them ondeveloping projects that result in profitable products and to Evaluate R&D GroupsPage 2 This report provides a snapshot of the current beliefs; later phases of our research will evaluatethese beliefs from the perspectives of history and paper is structured as follows. We provide a brief description of the sample and ourmethods. We then describe a tiered perspective on the R&D mission and within that structuredescribe the role of customers, managers, and corporate investment.

6 We describe the Metrics thatfirms use and relate those Metrics to motivation and incentives. We close with a set of suggestedresearch directions for the continuation of this research and MethodsThis report is based on qualitative interviews at AT&T Bell Laboratories, Robert BoschGmbH, Cable & Wireless, Chevron Petroleum Technology Company, Electricit de France(Direction des Etudes et Recherches), Hoechst Celanese Advanced Technology Group, PolaroidCorporation, Schlumberger (Measurement and Systems), the Army (Missile RD&E Center,Army Research Lab), and Varian Vacuum Systems. We feel that this group of firms is sufficient toraise many of the issues of measuring R&D success. Later phases of this research will attempt toquantify these issues through a larger scale survey. At each firm we attempted to speak to the ChiefTechnology Officer (CTO), the person(s) to whom the CTO reported, the person(s) who reported tothe CTO, and researchers within the R&D organization.

7 In total we interviewed 43 managers interviews varied from approximately one hour to a full day of interviews. In somesituations the interviewee showed us around the facility and introduced us to many people in theorganization. In each interview we sought to understand how the interviewee affected and wasaffected by the organization, how the interviewee believed that the organization measured R&Dsuccess, and how he or she believed it should measure R&D success. We discussed the motivationand evaluation of employees and how formal and informal incentives (if any) affected employeebehavior and the firm's profitability. Because the interviews were exploratory, they werefreewheeling. We allowed each interviewee to explore the topics in any way that he or she foundcomfortable and we encouraged each person to speak about related topics which they foundMetrics to Evaluate R&D GroupsPage of the seniority of the people in our sample and because of the sensitive nature ofthe interviews, we did not use audio recording.

8 We promised that we would not tie any interviewee(or organization) directly to a quote. (The only exceptions are quotes from widely circulateddocuments that were given to us by the interviewees.) Instead the interviewer kept detailed noteswhich where later transcribed. An analysis of the content suggested 37 interrelated topics. Weattempt to cover each of these topics in the following of the R&D MissionTo understand Metrics and incentives we begin by describing a tiered structure of the R&Dmission. We draw the word, "tier," from one of our interviewees, but we found the basic structureat each interview site. For example, the Army uses funding numbers such as , , .., todescribe their tiers. We feel that this tiered structure is important to our story because Metrics andincentives vary as the mission of R&D varies from one tier to to Evaluate R&D GroupsPage 4 Table 1. Tiers of the R&D MissionTier 0 Tier 1 Tier 2 Tier 3 Tier 4 ActivityLaying the foundationsExploring the tools of the futureCreating the toolsPioneering the use of the toolsUsing the tools routinelyPerformed byUniversity orBasic Research LabR&DR&DR&DBusiness UnitsExampleMathematics for codingAlgorithm developmentAlgorithm implementationPilot systemTechnology transfer to business unitsMetrics to Evaluate R&D GroupsPage 5 The tiered structure is described in Table 1.

9 Tiers 1, 2, and 3 describe the missions of theR&D laboratory. Tiers 0 and 4 are included to describe the interface between R&D and itssuppliers and customers. (The boundaries vary by firm, , in some cases tier 0 is the university;in some cases it is a basic facility within the firm.)We illustrate the role of each tier through an example from one of our interviewees. Consider an organization that wants to communicate thousands of high-resolution detailed 3-dimensional (3D) images to and from a remote field site. If the firm could do this then the firmcould do its job more effectively and with significantly lower cost than having to analyze theimages on-site. However, the sheer volume of data means that today's technology does not have thebandwidth to accomplish the 0 is the basic research that lays the foundations for later tiers. In the case of 3D images,tier 0 might be the development of the fractal mathematics that allows these images to be coded fortransmission.

10 These mathematics may or may not have been developed for this application, butthey prove important to its solution. Most likely they were developed at a university or a centralfacility. The R&D laboratory may not need to develop these tools, but it must have the ability toidentify whether these mathematics exist, to find out where they exist, and access the 1 uses basic foundations to explore the tools of the future. In the case of 3D images,tier 1 might include the development of algorithms that use fractal mathematics to code the images. Once developed, the algorithms might have other uses within the firm and, once developed, thealgorithms might give the firm a competitive advantage. The tier 1 researchers might have focusedonly on 3D imaging or they may have been trying to explore tools that could solve a portfolio 2 creates the tools. For example, tier 2 researchers might write the software anddevelop (or buy) the hardware to implement the algorithms.