TEST AND EVALUATION MANAGEMENT GUIDE - DAU

1 TEST AND EVALUATION MANAGEMENT GUIDE DoD Test and EVALUATION MANAGEMENT GUIDE Table of Contents 1 Contents Foreword 8 Chapter 1: 9 Introduction 9 Congress 9 OSD Oversight Structure 10 Service T&E MANAGEMENT Structures 14 Summary 22 Chapter 2: The T&E Process 23 Introduction 23 Defense Systems Acquisition Process 23 T&E And SE Processes 35 Five-Step Illustrative T&E Process 40 Scientific Test and Analysis Techniques (STAT) 42 Summary 44 Chapter 3: Program Office Responsibilities for T&E 45 Introduction 45 Relationship To The PM 47 Early Program Stages 48 PMO/Contractor Test MANAGEMENT 51 Test Program Funding/Budgeting 52 Contractor Testing 53 Specifications 53 Independent Test And EVALUATION Agencies 54 PMO Involvement In Operational T&E Activities 54 Summary 60 Chapter 4: Test - Related Documentation 61 Introduction 61 Requirements Documentation 62 Program Decision Documentation 64 Program MANAGEMENT Documentation 65 Test Program Documentation 68 Other Test-Related Status Reports 74 Summary 75 Chapter 5.

2 Test & EVALUATION 76 Introduction 76 "Test" and " EVALUATION " 76 The EVALUATION Process 76 Distinction Between Issues and Criteria 77 MOEs 81 EVALUATION Planning 82 DoD Test and EVALUATION MANAGEMENT GUIDE Table of Contents 2 Evaluating Developmental And Operational Tests 84 Summary 85 Chapter 6: Introduction to DT&E 86 Introduction 86 DT&E and The System Acquisition Cycle 87 DT&E Responsibilities 90 Test Program Integration 92 Areas Of DT&E Focus 92 System Design For Testing 94 DT&E OF Limited Procurement Quantity Programs 95 Summary 95 Chapter 7: DT&E Support of Technical Reviews and Milestone Decision 96 Introduction 96 DT&E and The Review Process 96 Tailoring Reviews to Specific Needs 106 Summary 107 Chapter 8: Integrated Testing 108 Introduction 108 Integrated Testing 108 Early Involvement 109 EVALUATION Framework 112 Test Data 115 Summary 115 Chapter 9: Production - Related Testing Activities 116 Introduction 116 Quality In Design 116 Production MANAGEMENT 116 Role Of The PRR 117 Production Qualification Testing 117 Transition To Production 119 LRIP 120 PAT&E 120 Summary 121 Chapter 10.

3 Introduction to Operational Test and EVALUATION 122 Introduction 122 Purpose and Scope of OT&E 122 Test Participants 124 OT&E as it Relates to DT&E 125 Types of OT&E 125 Test Planning 128 Test Execution 130 Test Reporting 130 Summary 130 DoD Test and EVALUATION MANAGEMENT GUIDE Table of Contents 3 Chapter 11: OT&E to Support Decision Reviews 132 Introduction 132 OT&E During the TD Phase 132 OT&E During the EMD Phase 132 OT&E During the P&D Phase 133 OT&E During the O&S Phase 133 Summary 135 Chapter 12: TEMP 136 Introduction 136 Temp Development 136 Temp Requirements 141 Temp Format 141 Summary 143 Chapter 13: Test Resources 144 Introduction 144 Obtaining Test Resources 144 Test Resource Planning 149 Test Resource Funding 154 Summary 157 Chapter 14: Introduction to Acquisition of Defense Business Systems (DBSs) 159 Introduction 159 DBSs 159 BCL Model 159 Incremental Acquisition Approach For DBSs 160 Summary 164 Chapter 15.

4 Software and IT Testing Considerations 165 Introduction 165 Role Of The Software Specification Review 165 Role Of The TRR For Software 169 Software Development Process 170 Potential Power Of Human-Based Testing 171 "Black Box" Versus "White Box" Testing 172 On the Impossibility of Exhaustive Software Testing 173 Software Error Categorization And Priority 174 Overview of Software Measurement With T&E Applications 175 Independent Verification and Validation (IV & V) 184 T&E Issues Associated With Spiral and agile Development Approaches 185 COTS Considerations 188 Mature Software 189 Summary 190 DoD Test and EVALUATION MANAGEMENT GUIDE Table of Contents 4 Chapter 16: Testing for Vulnerability and Lethality 191 Introduction 191 LFT 191 Nuclear Weapons Testing 196 Testing For NH&S 196 Summary 198 Chapter 17: Logistics Support T&E 200 Introduction 200 Planning For Logistics T&E 204 Aspects of LOG T&E 209 Limitations to LOG T&E 217 Summary 217 Chapter 18: M&S Support to T&E 218 Introduction 218 Types of Models and Simulations 218 Scope of M&S 221 Support to Test Design and Planning 222 Support to Test Execution 224 Support to Analysis and Test Reporting 224 Simulation Integration 226 Simulation Planning 227 Summary 227 Chapter 19.

5 Electromagnetic Environmental Effects (E3) Testing and Spectrum Supportability (SS) 228 Introduction 228 Key Definitions and Concepts 228 E3/SS Planning, Analysis, Systems Engineering, and T&E For the Systems Acquisition Process 233 Summary 244 Chapter 20: Special Topics: Interoperability and Cyber Security 248 Policy 248 Interoperability Testing 248 "Operationalizing" the NR KPP 251 Role of JITC 252 Supporting Integrated Architecture Products 253 Cyber Security and IA Requirements 255 agile Development and T&E 259 Summary 260 Chapter 21: Multi-Service and Joint Testing 263 Introduction 263 Background 263 Test Program Responsibilities 264 Test Team Structure 265 Test Planning 265 Deficiency Reporting 266 DoD Test and EVALUATION MANAGEMENT GUIDE Table of Contents 5 Test Reporting 267 Joint Test and EVALUATION 267 Summary 269 Chapter 22: International T&E Programs 270 Introduction 270 FCT Program 270 NATO Comparative Test (NCT) Program 272 T&E MANAGEMENT in Multi-National Programs 272 and NATO Acquisition Programs 273 Summary 273 Chapter 23.

6 Commercial Off-The-Shelf and Non-Developmental Items Considerations 274 Introduction 274 Market Investigation and Procurement 276 COTS and NDI Testing 277 Resources and Funding 279 Summary 279 Chapter 24: Testing the Special Cases 280 Introduction 280 Testing with Limitations 280 Space System Testing 283 Cyber Warfare 284 Operations Security and T&E 288 Joint Concept Technology Demonstrations 289 Chemical and Biological Agent (CBA) Testing 289 Summary 290 Appendix A - Acronyms 291 Appendix B - References 307 DoD Test and EVALUATION MANAGEMENT GUIDE Table of Contents 6 List of Figures Figure 1-1 DoD Test and EVALUATION Organization 10 Figure 1-2 Army Test and EVALUATION Organization 15 Figure 1-3 Navy Test and EVALUATION Organization 18 Figure 1-4 Air Force Test and EVALUATION Organization 21 Figure 2-1 Defense Acquisition System Model 24 Figure 2-2 Improving Milestone Process Effectiveness 25 Figure 2-3 Key Technical Activities and Technical Reviews/Audits during the Acquisition Lifecycle 30 Figure 2-4 Illustrative Systems Engineering Activities (EMD Phase)

7 38 Figure 2-5 Impact of Decision Timing on Lifecycle Costs 40 Figure 2-6 Five Step Test and EVALUATION Process 41 Figure 3-1 T&E WIPT Focus Areas 47 Figure 3-2 Examples of T&E Lessons Learned 55 Figure 4-1 Examples of T&E-related Program Plans and Documentation 61 Figure 4-2 Example of Requirements Evolution 63 Figure 4-3 Example of Typical Test Plan Format 71 Figure 4-4 Types of Information in a Final Test Report 73 Figure 5-1 Illustrative Generic EVALUATION Process 77 Figure 5-2 Example of Key Topics in an EVALUATION Plan 78 Figure 5-3 Example of Dendritic Analysis Showing Critical Issue Decomposition 83 Figure 6-1 Examples of Testing Activities Throughout the Acquisition Life Cycle 87 Figure 7-1 Example DT&E Activities Across Phases of the Acquisition Life Cycle 01 Figure 7-2 Example of Technical Review Timing Across the Acquisition Life Cycle 98 Figure 7-3 Specifications Summary 99 Figure 8-1 Integrated Testing as Part of an Overall T&E Continuum 111 Figure 8-2 Top-Level EVALUATION Framework Matrix 114 Figure 9-1 Typical Criteria Evaluated as Part of a PRR 118 Figure 10-1 Illustrative Scope of the Operational Test and EVALUATION Effort 123 Figure 10-2 Comparisons of DT and IOT&E Differences Highlighted 125 Figure 12-1 TEMP Format 142 Figure 13-1 DoD MRTFB by DoD Component 148 Figure 13-2 TEMP Part IV - Resource Summary 150 Figure 13-3 Resourcing Cycle Overlaps in the DoD Budget Process 155 Figure 14-1 BCL Acquisition Business Model 161 Figure 15-1 Error Avalanche Showing How Software Errors Can Accumulate 164 Figure 15-2 Software Error Distribution Summary (Notional)

8 166 Figure 15-3 Illustrative Software Test Planning Activities 168 Figure 15-4 Illustrative Software Development Activities in System WBS Context 171 Figure 15-5 Testing Example 174 Figure 15-6 Priority Classification Scheme for Software Errors 175 Figure 15-7 Example of Requirements Stability Measure 177 Figure 15-8 Example of Computer Resource Utilization Measure 178 Figure 15-9 Test Progress Measure Indicating Schedule Slip 179 Figure 15-10 Test Facility Under-Utilization Example 180 Figure 15-11 Problem Report Closure Rate Status Problems 181 Figure 15-12 Problem Report Aging Example 182 DoD Test and EVALUATION MANAGEMENT GUIDE Table of Contents 7 Figure 15-13 Problem Report Open Status Example 183 Figure 15-14 Tracking Problem Report Severity 183 Figure 15-15 Spiral Model Software Development Approach 186 Figure 15-16 Notional agile Development Model Depicting Testing 198 Figure 15-17 Example of Software Maturity Criteria 190 Figure 16-1 Summary of Key Parts of the Regarding LFT 192 Figure 16-2 Summary of Degrees of LFT 193 Figure 16-3 Illustrative LFT&E Planning and Execution Process 195 Figure 16-4 Nuclear Weapons Effects vs.

9 System Survivability 197 Figure 17-1 Illustrative Scope of LOG T&E 200 Figure 17-2 IPS Elements 202 Figure 17-3 Logistics Supportability Objectives in the Overall T&E Program 206 Figure 17-4 Example RGC 213 Figure 18-1 The Simulation Spectrum 219 Figure 18-2 Criteria Values Conducive to Use of M&S 221 Figure 18-3 Illustrative Use of M&S in T&E 224 Figure 19-1 Key E3/SS Terms and Definitions 232 Figure 19-2 S-Diagram 233 Figure 20-1 RMF Process 257 Figure 20-2 Procedures for OT&E for IA in Acquisition Programs 258 Figure 21-1 Multi-Service OT&E (MOT&E) Team Composition 261 Figure 21-2 Deficiency Report Summary Format 266 Figure 21-3 JT&E Process 267 Figure 23-1 Spectrum of COTS/NDI Applicability (not to scale) 274 DoD Test and EVALUATION MANAGEMENT GUIDE & 8 Test & EVALUATION MANAGEMENT GUIDE Foreword This 6th edition of the Test and EVALUATION MANAGEMENT GUIDE (TEMG) was updated with the cooperation and support of the Deputy Assistant Secretary of Defense for Developmental T