2-27-13 The Drive Toward Data Driven Estimating …

1 1 The Drive Toward data Driven EstimatingBoeing/Lockheed Martin/Northrop GrummanOriginal March 2011 Updated February 2013 BOE Training Material2 Background/Objectives Background:There is a significant industry effort to increase the quality of contractor Estimating processes. The industry has applied the term data Driven Estimating to this new method of developing and documenting our basis of estimates(BOEs). data Driven Estimating methods have certain characteristics that are starting to emerge as standards within our industry. Objectives: What is driving our industry Toward the use of data Driven Estimating methods and why is this a business imperative Review the current primary Estimating methods Be able to identify the characteristics of good data Driven Estimating processes and how they are represented in our BOEs What are objective-scope-measures and how are they used in a data Driven estimate Provide examples of data Driven versus non- data Driven BOEs3 data Driven Estimating - Contents BOE - Description data Driven Estimating - Definition The Future Fundamentals of data Driven Estimating Developing Relationships from History The Cost Estimating Relationship (CER)

2 Process data Driven Estimate Examples We are the Customer Summary Backup Charts/Supplemental Resources4 Purpose of a BOE Documents and substantiates effort required to perform tasks Documents basis and assumptions Details understanding of requirements Conveys credibility by clearly defining tasks and assumptions Demonstrates that estimates are credible, defendable and realistic (historical data ) Provides evidence that estimate is consistent with other proposal volumes Time-phased resources consistent with IMP/IMS Allocates skill mix appropriate for task Aligns with Work Breakdown Structure (WBS), SOW, and other product structures Meet customer funding, timing, and/or scheduling constraints5 BOEs The Components A BOE consists of the following main Element ID WBS, SOW, Location, CLIN, the products or services delivered and the tasks to be method, rationale, assumptions, and computations used to develop the estimate Method of historical dataand comparability of the historical data to the current program Explanation of any adjustments made to historical cost or programmatic data ( , use of modifiers) labor resourcesto perform the defined profile(time-phasing) of labor elementsas required by the cost proposal instructions6 BOE Estimating Methods Labor estimates are developed using one or these typical Estimating methods Judgment based on subject matter expert (SME)

3 Experience with similar efforts. Level-of-Effort/Support- uses resource-based estimates that are often based on a predetermined level of support for a given period of time Analogy/Comparison Method estimate based on historical frame of reference from a similar effort with adjustments for programmatic or technical differences Standard Time Estimating - Labor Standards and Realization/ Performance Factors - time necessary to complete a defined element of work following a prescribed (recognized) technique or method. Unit Method generally a simple ratio (not always statistically validated) Follow-on Based On Actuals extrapolation of actuals (CEAC, production) Parametric Cost Models use of calibrated, internally developed, or commercially available cost models Cost Estimating Relationships (CERs)- equations developed based on historical, statistically correlated relationships Note: Material or Other Direct Costs may be estimated by data such as quotes or catalogs.

4 See FAR for specific cost element and degree of competition requirements7 Customer Expectations Customers/DCAA/DCMA expect contractors to begin adopting improved Estimating methods to support TINA-compliant proposals and other Estimating productsLeast PreferredJudgmentMost Preferred Fact Based Estimating Methods Move To Fact Based ASAP8 BOE Methodologies Credibility ScaleHistorical resource based estimates are easier to support than judgmental type CredibilityLaborHighFollow-on Projectionsbased on ActualsAnalogy/ComparisonCER sParametric Estimates / Cost ModelsStandardsUnit MethodsLevel Of EffortLowJudgment 9 Customer Consequences Poor Estimating Failure to be responsive to customer Estimating processes could be grounds for Non-compliance with RFP or government regulations Rejection of proposal and possibly defective pricing Estimating System Inadequacy Penalties for bad BOEs include: Questioned costs DCAA/Government Deficiency Report Negotiation losses Negative factor in competitive evaluations Defective pricing and fraud prosecution Suspension of bidding and debarment Holdbacks10 data Driven BOEs Basic Definition Fact Based Estimating : Estimating processes/BOEs that provide the supporting information and rationale necessary for accurate, credible and traceable estimates Information includes: Use of defined Estimating methods Historical data from past programs Use of objective scope measures and complexity factors where appropriate Rationale addresses.

5 Complete description of the Estimating approach Explanation of the applicability of the historical information Estimating process must be documented, consistent and repeatable11 Fact Based BOEs The Fundamentals develop using a defined and documentedestimating rely on scope measures(attributes) and complexity factors(parametrics) when estimate development process is explained and documentedin the BOE Professional judgments, modifiers/adjustment factors, and historical context is adequately explained Process is consistent and relies on relevant historical dataand context that is accessible and verifiable Shows use of latest and most current relevant data Supports all resource estimates with verifiable processes are monitored for learning and improvementas required to support future estimates12 Objective Work-Scope Measures (OSM) Scope Measures: Estimating predictive measures technical measures relating to the size of the effort (attributes) Generally physical and technical measures Sized at different product levels - system, subsystem, box, etc.

6 Example: Work Products: # drawings by type, sheet counts, # documents & sheet count, SLOC by category, # requirements, # reviews, # test & type, weight, quantities, etc. Traits/Complexity Factors: characteristics that can improve the accuracy of relationships developed for the OSMs Example: Scope Measures = Drawings Traits: (numeric and descriptive): # Drawings, Type of Drawing; Drawing Heritage; Item Next Higher Assembly/Element; Number of Sheets; # of Revisions Example: Scope Measures = Source Lines of Code (SLOC) Traits: Software Language; New SLOC; Modified SLOC; Reuse SLOC; Actual Total Delivered SLOC, Deleted SLOC, Auto Generated Code, etc13 Why Use Scope Measures and Traits in BOEs? Scope Measures- help to scope the size of the effort to be estimated Objective quantitative measures of scope to be performed Directly correlate to (predictors) resource estimates for work products and tasks Understanding the size (scope) of the proposed effort is critical to developing an accurate estimate (BOE) The BOE author should explain the derivation, source, and application of the scope measures in the BOE Traits/Complexity Modifiers address unique characteristics of the scope measures that can affect required effort to produce Capture the effects of technical and programmatic differences between historical programs and the current proposed effort.

7 The produceability can vary widely for any scope measure - traits can be used to explain the differences Sometimes requires judgment14 Objective Scope Measures - Examples Source Lines of Code (SLOC) Function points Computer Software Configuration Item (CSCI) Number of drawings Power requirements Number/type of subsystems, subassemblies, components Volume (Dimensions) Weight Number/type of reviews Number of end items Number/type of tools Number of Contract data Requirements Lists (CDRLs) Number of Subcontract data Requirements Lists (SDRLs) Number of system requirements Number of Interface Control Drawings ( ICDs) Number of payloads Number of program segments Number of internal interfaces Number of external interfaces Number of risks Number of architectural tradesCandidate Estimating AttributesSoftwareHardwareSystems Engineering15 Traits/Complexity Factors - Examples Design reuse Performance factors Complexity Reuse and pedigree New technology Components/functional complexity Schedule Requirements maturity Technology Readiness Level (TRL) requirements/maturity path Multi functional analyses Testing (qualification; prototype protoqualification; delta qualification; static.)

8 Modal) Assembly, Integration and Test (AI&T)) Training requirements Mission profile and environments Mission design ( data rates, pointing stabilities, Technical Performance Measurements (TPMs)) Oversight and analyses of subcontracted elements Risk Program maturity/life cycle Complexity Schedule Heritage Required documentationEstimating ModifierSoftwareHardwareSystems Engineering16 Explaining Adjustment to a BOE Provided below are some examples of adjustments that may be required to improve the accuracy of the estimate. The list is not meant to be totally inclusive of adjustments an estimator has available in their tool box. Technical Complexity ~ Labor Estimate Improvement Curves ~ Labor Estimate Schedule adjustments ~ Labor Estimate Quote Decrement ~ Material Estimate Quantity Adjustment ~ Material Estimate Escalation ~ Material Estimate Exchange Rate ~ Material EstimateIn order to use an objective or subjective adjustment, an estimator must explain it thoroughly in the BOE.

9 17 Using Historical data in a BOE Use of Historical data Must Address: Source of data generally from the organization s certified historical data repositories Complete identification of the nature, source and details of relevant historical data Ensure compliance with by documenting the following: Actual historical hours Charge numbers included Technical data used Time frame of historical data Repository or source of historical data 18 Developing a Relationship from Historical data The statistical analysis process used in the development of a cost Estimating relationship (CER) is documented in numerous industry and statistical sources The objective of the analysis is to validate that a significant statistical relationship exists for the CERs and to demonstrate that the CERs accurately predict effort or costs CER Strengths Past performance reflective of future costs Lean and credible Estimating method in preparing BOEs More scientific and objective than judgement19 Processs Estimate of Effort/Cost of Equation of the Cost Estimating Relationship (CER)

10 The a data Driven Relationship (CER)numeratordenominatoritem to estimateknown itempoolbasedependent variableindependent variable20 Developing a data Driven Relationship (CER) Regression Analysis: Develop mathematical relationship to make inferences about a future probabilistic event based on a sample of historical data . If relationship exist between two or more variables, such as weight and hours of effort/cost required, regression analysis is a powerful tool for the cost estimator/analyst. Regression analysis consists of curve fitting and the analysis of the descriptive and inferential statistics that result from the fit of the a data Driven Relationship (CER)YXYXP ositive Linear RelationshipPositive Increasing Curve-linear RelationshipYXYXN egative Linear RelationshipYXPositive Decreasing Curve-linear RelationshipNegative Increasing Curve-linear Relationshipxbby10 xbby10 baxy )1b( baxy )1b0( baxy )1b( 22 Preparation of the Historical data (CER) Step 1: Select the dependent variable to be estimated ( results = hours, cost) Step 2: Determine the auditable source of historical data (both cost and technical measures) Step 3: Define the population(s), subpopulations, and sample(s) of data considering methods of collection (sites, programs, stratifications) Step 4: S