Technology Readiness Assessment Guide

1 From August 11, 2016 to August 10, 2017, GAO is seeking input and feedback on this Exposure Draft from all interested parties. See page 9 for more 2016 GAO-16-410 GAssessment GuideTechnology ReadinessBest Practices for Evaluating the Readiness of Technology for Use in Acquisition Programs and Projects Page 1 DRAFT Preface .. 8 Introduction .. 11 The Guide s Case Studies .. 13 The Guide s Readers .. 13 Acknowledgments .. 14 Chapter 1 .. 15 What Is a Technology Readiness Assessment (TRA)? .. 15 Definitions and Overview .. 16 TRAs Inform Technology Development and Identify Potential Concerns .. 18 Overview of Technology Development and Its Relationship to Acquisition Programs .. 18 Technology Development Models.

2 23 TRA s Relationship to Program Management and Oversight .. 24 Tailoring TRAs for Different Purposes .. 25 Chapter 2 .. 26 Why TRAs Are Important and Understanding Their Limitations .. 26 Maturity of Technology at Program Start Is an Important Determinant of Success .. 28 TRA Process Is a Mechanism That Informs Important Acquisition Functions .. 29 Understanding TRAs Can Help Practitioners and Consumers of Information .. 31 TRAs Are Snapshots in Time .. 33 Advancements in Technology Can Pose Challenges in Applying 33 Organizational Experience, Culture, and Bias Can Affect TRAs .. 34 Page 2 DRAFT TRAs Depend on the Quality and Availability of Credible Data .. 36 Chapter 3 .. 38 Best Practice: A Reliable Process for Conducting Credible TRAs.

3 38 More Frequent Evaluations of Technology Maturity .. 40 High Quality TRAs .. 41 Chapter 4 .. 43 Best Practice: Including Technology Maturity Assessments in the Program Strategy, Designing the TRA Plan, and Determining the 43 Technical Maturity Assessment Strategy .. 43 The Assessment Team .. 46 The TRA Plan .. 48 Chapter 5 .. 52 Best Practice: Selecting Critical 52 Critical Technologies Defined .. 52 Challenges in Selecting Critical Technologies .. 54 Steps for Selecting Critical Technologies .. 55 Identifying Other Important Technologies and Programmatic Issues .. 65 Chapter 6 .. 68 Best Practice: Evaluating Critical Technologies .. 68 Steps for Evaluating Critical Technologies .. 68 Relevant Information Must Be Used to Evaluate Critical Technologies.

4 74 Operational Environment Is Key to Evaluating Critical Technologies .. 74 Creating Critical Technology Subsets for Exceptionally Large or Complex Programs .. 75 Page 3 DRAFT Chapter 7 .. 77 Best Practice: Preparing the TRA Report .. 77 The TRA Report .. 77 Steps for Preparing and Coordinating the TRA Report .. 80 Response to the TRA Report .. 84 How Dissenting Views Are Documented and Submitted .. 84 Chapter 8 .. 86 Best Practice: Using the TRA Results .. 86 How TRA Reports Are Used .. 86 TRAs for Governance Decisions .. 87 TRAs as Knowledge-building Exercises .. 88 Identification of Potential Areas of Concern and Risk .. 89 Early Technology Development .. 91 TRA Process Facilitates Information Sharing Opportunities.

5 92 Basis for Developing a Technology Maturation Plan (TMP) for Immature Technologies .. 93 Chapter 9 .. 95 Best Practice: Preparing a Technology Maturation Plan (TMP) .. 95 Steps for Preparing a Technology Maturation Plan .. 96 Updating a Technology Maturation Plan .. 98 The Technology Maturation Plan Template .. 101 Chapter 10 .. 106 Practices Are Evolving in Evaluating Software Systems and Systems Integration Using TRAs .. 106 Applying TRAs to Software Systems .. 106 Software Embedded Technologies versus Software-only Technologies .. 108 Page 4 DRAFT Development of System-level Readiness Metrics .. 109 Appendix I: Key Questions to Assess How Well Programs Followed the Six Step Process for Developing Credible TRAs.

6 112 Appendix II: Auditing Agencies and Their Websites .. 117 Appendix III: Case Study Backgrounds .. 119 Case Study 1: Immature Technologies Increases Risk, GAO-08-408 119 Case Study 2: Assessments Provide Key Information, GAO-10-675 . 120 Case Study 3: Space Programs Often Underestimate Costs, 120 Case Study 4: Program Updates Can Change Critical Technologies, GAO-02-201 .. 121 Case Study 5: Identifying Back-up Critical Technologies, GAO-08-467SP .. 121 Appendix IV: Experts Who Helped Develop This Guide .. 123 Appendix V: Contacts and Acknowledgments .. 130 GAO Contacts .. 130 Other Leadership Provided for This Project .. 130 Acknowledgments .. 130 Appendix VI: Examples of Various TRL Definitions and Descriptions by Organization.

7 131 Appendix VII: Other Types of Readiness Levels .. 137 Appendix VIII: Agency Websites Where TRA Report Examples Can Be Found .. 142 References .. 143 Image Sources .. 146 Page 5 DRAFT Tables Table 1: Cost and Schedule Experiences for Products with Mature and Immature Technologies 29 Table 2: Six Steps for Conducting a Technology Readiness Assessment (TRA) 39 Table 3: Characteristics of High Quality Technology Readiness Assessments (TRA) 41 Table 4: Software Implementation Project Work Breakdown Structure 60 Table 5: Technology Readiness Levels (TRLs) Supporting Information for Hardware and Software 72 Table 6: Example Program Management Tools Used with Technology Readiness Assessments (TRAs) 90 Table 7: Auditing Agency Websites 117 Table 8.

8 GAO Reports Used As Case Study in the TRA Guide 119 Table 9: Experts Who Made Significant Contributions 123 Table 10: Experts Who Made Noteworthy Contributions 124 Table 11: DOD Technology Readiness Levels (2011) 131 Table 12: DOD Software Technology Readiness Levels (2009) 132 Table 13: NASA Hardware Technology Readiness Levels (2013) 133 Table 14: NASA Software Technology Readiness Levels (2013) 134 Table 15: DOE Technology Readiness Levels (2011) 135 Table 16: DOD Manufacturing Readiness Levels 137 Table 17: Integration Readiness Levels 140 Table 18: System Readiness Levels 141 Figures Figure 1: Technology Readiness Levels 17 Figure 2: Phased Acquisition Cycle with Decision Points 19 Page 6 DRAFT Figure 3: Technology Readiness Assessment and Technology Readiness Level Limitations 32 Figure 4: Notional Depiction of the Integrated Schedule for a Program 46 Figure 5: Four Steps for Selecting Critical Technologies 56 Figure 6: Common Elements of a Work Breakdown Structure 57 Figure 7: A Contract Work Breakdown Structure 59 Figure 8: A Process Flow Diagram (simplified) 62 Figure 9: Four Steps to Evaluate Critical Technologies 69 Figure 10: Technology Readiness Assessment Report Template 79 Figure 11.

9 Five Steps to Prepare the Technology Readiness Assessment Report 81 Figure 12: Acquisition Cycle with Technology Readiness Assessments at Decision Points for Governance 88 Figure 13: Five Steps to Prepare the Technology Maturation Plan 96 Best Practices Checklists Best Practice Checklist: TRA Team and Purpose, Scope, and Schedule for TRA Plan 50 Best Practice Checklist: Selecting Critical Technologies 66 Best Practice Checklist: Evaluating Critical Technologies 76 Best Practice Checklist: Preparing the TRA Report 85 Best Practice Checklist: Using the TRA Results 93 Best Practice Checklist: Preparing a TMP for Immature Technologies 104 Best Practice Checklist.

10 Evaluating Software Systems Using TRAs 111 Abbreviations AD2 Advancement Degree of Difficulty COTS commercial off the shelf DDR&E Deputy Director for Research and Engineering DOD Department of Defense Page 7 DRAFT DOE Department of Energy DTRA Defense Threat Reduction Agency IRL integration Readiness level ISRACO International Systems Readiness Assessment Community of Interest IT information Technology MAIS major automated information system MDA Milestone Decision Authority MRL Manufacturing Readiness Level NASA National Aeronautics and Space Administration NPOESS National Polar-orbiting Operational Environmental Satellite System R&D3 research and development degree of difficulty RI3 risk identification, integration, and ilities SEMP systems engineering master plan SRL system Readiness level TBD technical baseline description TMP Technology maturation plan TE Technology element TPMM Technology program management model TRA Technology Readiness Assessment TRL Technology Readiness level USASMDC Army Space and Missile Defense Command WBS work breakdown structure This Guide is a work of the government and is not subject to copyright protection in the United States.