Lesson 18: Test and Evaluation Overview - AcqNotes

1 Lesson 18: Test and Evaluation Overview Why Test and Evaluate? Information is critical to managing an acquisition program. This includes information on the capabilities and limitations of the system and risks affecting system cost, development schedule, and performance. The overall goal of T&E is to reduce risk by providing crucial information to decision makers. T&E Mandates DODD provides principles and policies governing test and Evaluation in the Defense Acquisition System. DODI identifies the flow of the T&E activities within the development process and states that test activities shall be part of a Test and Evaluation strategy to: l Provide information regarding risk and risk mitigation.

2 L Provide empirical data to validate models and simulations. l Permit an assessment of the attainment of technical performance specifications and system maturity. T&E Customers These Customers:Use T&E Information to:Acquisition Managersl Evaluate the acquisition program for risk management. l Determine if system performance meets thresholds. l Determine if the system has the potential to meet objectives. l Determine the technical changes needed to keep the program on track. Designers and Engineersl Evaluate system design and performance. l Identify if design changes are needed. l Provide feedback into Systems Engineering Process.

3 L Update modeling and simulation. End-usersl Learn how the system performs under operational conditions. l Utilize operational performance characteristics to improve tactics and warfighting capability. l Obtain information to help plan for training and logistics support. l Understand full performance envelope and capabilities. Defining Test and Evaluate TERMDEFINITIONTestA program, procedure, or process to obtain, verify, or provide data for determining the degree to which a system (or subsystem) meets, exceeds, or fails to meet its stated requirements. These requirements can be stated in terms of thresholds and objectives. Testing: l Obtains raw data.

4 L Measures specific, individual performance factors ( , measuring range). l Is resource intensive. Evaluate Reviewing, analyzing, and assessing data obtained from testing to project system performance under operational conditions. Evaluating: l Produces analyzed information from test data, modeling and simulation, or other sources. l Draws conclusions by looking at how the factors interact. l Is intellectually intensive. T&E Feedback Loop T&E is the essential feedback loop contained within the Systems Engineering Process. T&E ensures that the system meets the capability needs. The T&E process is repeated as the system evolves from models to components to production articles and complete systems.

5 Test and Evaluation Process Overview The T&E process has five steps that occur as we evaluate important information within the acquisition process. Five Steps in the T&E Process STEPDESCRIPTION1 Identify critical issues and data engineering analysis by the Evaluation and the development of an Evaluation plan. Expected outcomes are test is planned and conducted. Data are retrieved and analyzed. Data may be collected from other means such as Modeling and Simulation (M&S), training, post-synthesis step in which an Evaluation report of data is compiled. Predictions from Step 2 are compared with actual measured results are balanced with other available program information.

6 The appropriate programming decision is and the Acquisition Life Cycle T&E activities occur throughout the acquisition life cycle. T&E reveals information about a program and measures performance of a system against established capability needs. The PM coordinates T&E activities appropriate to each phase and milestone of an acquisition program. Pre-Systems Acquisition and Milestone A The Role of T&E in support of Milestone A: l No actual system testing occurs before Milestone A. l Projects that undergo a Milestone A decision have an Evaluation strategy. The Evaluation strategy addresses modeling & simulation, including identifying and managing associated risk, and early T&E strategy to evaluate system concepts against mission requirements.

7 L The most promising system concepts are defined in terms of initial, broad objectives and an overall test and Evaluation strategy (including Developmental Test and Evaluation (DT&E), Operational Test and Evaluation (OT&E), and, if applicable, Live Fire Test and Evaluation (LFT&E)). l The initial TEMP is developed during the Technology Development Phase. l For ACAT I programs, the PM forms the T&E Working-Level Integrated Product Team (WIPT). l A T&E WIPT is useful for Pre-Systems Acquisition activities that are likely to become acquisition programs. l Immediately upon forming, the T&E WIPT develops an Evaluation strategy that describes how the capabilities in the ICD will be evaluated once the system is developed.

8 L The Evaluation strategy is approved by the Director, Operational Test & Evaluation (DOT&E). Milestone B and the Endineering and Manufacturing Development & Demonstration Phase At Milestone B, the MDA: l Approves the acquisition strategy. l Approves the acquisition program baseline. l Approves low-rate initial production quantities (where applicable). l Approves EMD exit criteria. l Considers any early operational assessments or test and Evaluation results. The Role of T&E in Milestone B: l At Milestone B or beyond, a Test Evaluation Master Plan (TEMP) is required in lieu of the Evaluation strategy. l The TEMP focuses on the overall structure, major elements, and objectives of the T&E program.

9 L If applicable, the TEMP must contain a mature strategy that commits to full-up, system-level, live fire testing, or a waiver request is submitted and approved for an alternative Live Fire Test & Evaluation (LFT&E) plan. l The TEMP is updated at program major decision points, when the program baseline has been breached, when the associated CDD/CPD has been significantly modified, or on other occasions when the program has changed significantly. l Evolutionary acquisition programs may require additional updates to ensure that the TEMP reflects the currently defined program. l The concept of early and integrated T&E emphasizes testing of prototypes or production representative articles during EMD.

10 L Early Operational Assessments may be included as appropriate. Results are reported to the Service Chief by the appropriate operational test activity and used by the MDA in support of decisions. l System modeling, simulation, test, and Evaluation activities are integrated into an efficient continuum planned and executed by a test and Evaluation integrated product team (T&E IPT). l Modeling, simulation, and development tests are under the direct responsibility of the PM or a designated test agency. For ACAT I programs, the DOT&E and the cognizant Overarching Integrated Product Team Leader approve the TEMP (including the LFT&E strategy, if applicable) for all OSD test and Evaluation oversight programs.