Measures for Success: Lessons Learned from …

1 PSMCRS Software and Systems MeasurementPSM - IFPUG, 1 PSM All rights for success : Lessons Learned from applying PSMP ractical Software and Systems MeasurementObjective Information for Decision MakersDepartment of Defense US ArmyPractical Software and Systems MeasurementPSM - IFPUG, 2 PSM All rights Principles Measurement Is a Consistent but Flexible Process that Must Be Tailored to the Unique Information Needs and Characteristics of the Project or Organization Decision Makers Must Understand What is Being Measured and Trust the Information Measurement Must Be Used to be MeaningfulPSMCRS Software and Systems MeasurementPSM - IFPUG, 3 PSM All rights - ISO Standards - CMMIP ractical Softwareand Systems MeasurementISO/IEC SC7 Standards12207(revision - supporting process)15288 (measurement concepts)9126 (terminology coordinated)14598(terminology coordinated)-----ISO/IEC 15939, Software Measurement ProcessCMMIM easurement and AnalysisPractical Software and Systems MeasurementPSM - IFPUG, 4 PSM All rights Management Requires Measurement Information to Support Critical Project and Organizational Business and Technical Decisions Measurement Results Must Be Communicated and Used Across the Entire Corporate Entity Measurement Results Must Be Incorporated with Risk Management and Context InformationPSMCRS Software and Systems MeasurementPSM - IFPUG, 5 PSM All rights but Flexible ProcessPractical Software and Systems MeasurementPSM - IFPUG, 6 PSM All rights ActivitiesCore Measurement ProcessInformation NeedsEvaluateMeasurementImprovementActio nsAnalysisResults and Performance MeasuresScope of PSMUser FeedbackAnalysis ResultsEstablish andSustain CommitmentPlanMeasurementPerformMeasurem entTechnical and ManagementProcessesMeasurement Plan NewIssuesPSMCRS Software and Systems MeasurementPSM - IFPUG.

2 7 PSM All rights Measurement Measures Are Defined to Provide Insight into a Project or Organization s Information Needs by-Identifying what the decision makers need to know-Relating these to entities that can be measured-Selecting and specifying Measures based on project and organizational processesPractical Software and Systems MeasurementPSM - IFPUG, 8 PSM All rights Categories Schedule and Progress Resources and Cost Product Size and Stability Product Quality Process Performance Technology Effectiveness Customer SatisfactionPSMCRS Software and Systems MeasurementPSM - IFPUG, 9 PSM All rights Measurement - Lessons Learned Start by Implementing a Small Set of Measures -Reduces the level of change, resources required, and impact on the workload-Add additional Measures later, incrementally The Measurement Plan Should Specify Both What Will be Measured and How the Process Will WorkPractical Software and Systems MeasurementPSM - IFPUG, 10 PSM All rights Measurement RequirementsEnterprise ManagementOrganizationalManagementProjec tManagementPerformance MeasurementNormative Performance BaselinesTechnical and Business PolicyInvestment Decisions & AnalysisProcess ImprovementProject Planning GuidelinesPerformance Based GuidelinesOrganizational Norms & BenchmarksProject Estimation & PlanningProject Performance TrackingProject Tradeoff AnalysisResource ManagementRiskManagementProcessInformati on - DrivenMeasurement ProcessPSMCRS Software and Systems MeasurementPSM - IFPUG, 11 PSM All rights Measurement - Lessons Learned (cont).

3 Workshops Are Good For Defining Information Needs Allow Tailoring of Organizational Measures Successful Measurement Programs Integrate the Needs of all Decision-Makers-This simplifies data collection and reduces duplicationPractical Software and Systems MeasurementPSM - IFPUG, 12 PSM All rights Measurement Collecting Measurement Data, Performing Analysis, and Presenting Results Analysis Can Include Estimation, Feasibility Analysis of Plans, and Performance Analysis of Actual Data Against PlansPSMCRS Software and Systems MeasurementPSM - IFPUG, 13 PSM All rights Analysis ModelTechnology EffectivenessProcessPerformanceProduct Size and StabilityResourcesand CostSchedule and ProgressProductQualityCustomer SatisfactionPractical Software and Systems MeasurementPSM - IFPUG, 14 PSM All rights - EffortEstimating , , KFunction PointsStaff Month-1 Standard DeviationProject: PSMData as of 31 Mar 98+1 Standard DeviationAverageProject under considerationMeasurement Feasibility AnalysisPSMCRS Software and Systems MeasurementPSM - IFPUG, 15 PSM All rights Learned - Perform Measurement Automate Data Collection Whenever Possible It Take About 6 to 9 Months to Establish Measures -Initial focus is on ensuring data is provided-Next focus in on data problems-After these are resolved, focus can move to performance issues Clearly Identify the Reporting Mechanisms at Both the Project and Organizational Levels Aggregation Approaches Need to be Specified, Especially When Measures Are TailoredPractical Software and Systems MeasurementPSM - IFPUG, 16 PSM All rights and Sustain Commitment Includes Establishing the Resources, Training.

4 And Tools to Implement a Measurement Program Effectively and Ensuring Commitment to Use the Information that is Produced Need to have Commitment Reinforced and Demonstrated Regularly Need to Adopt an Action OrientationPSMCRS Software and Systems MeasurementPSM - IFPUG, 17 PSM All rights Support for Measurement The Resulting Culture Shock from Implementing any new Process causes a Natural Reaction of Personal Resistance-Overcome resistance to change Provide the Participants with an Understanding of the Measurement Process and the Benefits to Their Projects-Training programs should help project representatives identify information needs and Measures -Planning workshops should include representatives at all levels of the organizationPractical Software and Systems MeasurementPSM - IFPUG, 18 PSM All rights Learned - Evaluate Measurement Both the Measurement Process and the Specific Measures Should be Periodically Evaluated and Improved Measurement is an Iterative Process: the Measures are Refined as Information Needs Change and the Organization Implements Improvement ActionsPSMCRS Software and Systems MeasurementPSM - IFPUG, 19 PSM All rights and Trust Measurement InformationPractical Software and Systems MeasurementPSM - IFPUG, 20 PSM All rights Model.

5 High-Level ViewAdapted from ISO/IEC 15939 - Software Measurement ProcessInformationProductInformation NeedMeasurementConstructMeasurableConcep tAttributeEntityPSMCRS Software and Systems MeasurementPSM - IFPUG, 21 PSM All rights Construct - LevelsInformationProductDerivedMeasureIn dicatorBase MeasureAttributeMeasurement ConstructPractical Software and Systems MeasurementPSM - IFPUG, 22 PSM All rights or Evaluation thatProvides a Basis for DecisionMaking-Operations Quantifying anAttribute Against a ScaleIndicatorDerived MeasureDerived MeasureBaseMeasureAlgorithm Combining Two or More Base Measures A measure of a Single AttributeBy a Specific Method Quantity Defined as a Function ofTwo or More MeasuresAlgorithm Combining Measures and Decision CriteriaMeasurementFunctionEntitiesInfor mation NeedsAdapted from ISO/IEC 15939 - Software Measurement ProcessBaseMeasureInterpretationAnalysis ModelInformationProductAttributeAttribut eProperty Relevant to Information NeedsMeasurement ConstructPSMCRS Software and Systems MeasurementPSM Version.

6 23 Measurement Construct - ProductivityAnalysisModelCompute Mean and StandardDeviation for Historical ProjectCompared to Estimate for newProjectsAdjusted Mean and TwoStandard Deviation Confidence LimitsAT T R IB UT E SBASE MEASURESINDICATORSDERIVED MEASURESC ount TotalHoursCount SemicolonsDivide SizeBy Effort Repeat for Each Historical ProjectValue for Project XValue for Project XValue for Project XProject X Actual ProductivityMeasurementFunctionSizeProje ct XMeasurementMethodMeasurementMethodCount TotalHoursCount inputs, outputs, inquires, files, and interfacesDivide SizeBy Effort Repeat for Each Historical ProjectValue for Project XValue for Project XValue for Project XProject X Actual ProductivityMeasurementFunctionSizeProje ct XSourceCodeSemicolonHoursTimecardsFuncti on PointsHoursTimecardsMeasurementMethodMea surementMethodValue for Project XValue for Project XSizeProject XValue for Project XValue for Project XSizeProject XCount TotalHoursDivide SizeBy Effort Value for Project YValue for Project YValue for Project YProject X Actual ProductivityMeasurementFunctionSizeProje ct XMeasurementMethodMeasurementMethodCount TotalHoursDivide SizeBy Effort Repeat for Each New ProjectProject YEstimatedProductivityMeasurementFunctio nSizeProject XDesign SpecificationSemicolonHoursEffort PlannedFunction PointsHoursMeasurementMethodMeasurementM ethodSizeProject XSizeProject YCount inputs, outputs, inquires, files, and interfacesPractical Software and Systems MeasurementPSM - IFPUG.

7 24 PSM All rights Learned Provide Well Defined Base Measures to Ensure Consistency Present Information in a Format that is Understandable to the Decision Maker People Doing Measurement Aren t Generally Making the Decisions - It Is Important to Present Information Clearly and Concisely Just Because the Measurement Information is Accurate and Objective, Does Not Mean It Will be Acted OnPSMCRS Software and Systems MeasurementPSM - IFPUG, 25 PSM All rights Must be UsedPractical Software and Systems MeasurementPSM - IFPUG, 26 PSM All rights Learned The Measurement Process Should be an Integral Part of the Way Business Is Conducted Data Must be Provided Early Enough to Allow Managers to Take Action Results Must Be Communicated Throughout the Organization in a Timely Manner Decisions Should Not Wait for Perfect Data, but Should be Based on Accurate Data, Augmented by Risk Management and Context InformationPSMCRS Software and Systems MeasurementPSM - IFPUG, 27 PSM All rights Learned (cont.)

8 The Measurement Results Should Help Decision-Makers Optimize Overall Performance Most Organizations are Composed of a Portfolio of Distinct Projects, So Project-Level Information Must Be Aggregated to Appropriate Levels of the Organization to be Used EffectivelyPractical Software and Systems MeasurementPSM - IFPUG, 28 PSM All rights FocusDiscovered vs. Closed Defects 050100150200250300350400450 Nov-02 Dec-02 Jan-03 Feb-03 Mar-03 Apr-03 May-03 Jun-03 Jul-03 Aug-03 Sep-03 Date# of DefectsDiscoveredClosed-ImplementedPSMCR S Software and Systems MeasurementPSM - IFPUG, 29 PSM All rights Data Highlights IssuesTime to Fix vs. Time to CloseClosed DefectsAs of 25 Sep 0316141986051015202530354045 Calendar DaysFixedReworkTime to CloseTime to FixTime to FixTime to Rework Time to FinishTime to Finish93% of the PCRs follow the Normal Flow7% of the PCRs involve ReworkPractical Software and Systems MeasurementPSM - IFPUG, 30 PSM All rights Software and Systems MeasurementPSM - IFPUG, 31 PSM All rights / Threats More Use/Mis-Use of Measurement Continuous Requests for the List of Best Measures Constrained Environments that Don t Let You Act on the Measurement Results Changing DoD Requirements for Measurement-Greatly reduced requirements/standards ( DoDI-5000) -Continued emphasis on performance requirementsPractical Software and Systems MeasurementPSM - IFPUG, 32 PSM All rights for Improvements Align Information Needs Across All Perspectives Put Your Energy into Making Sure Information Is the Best It Can Be (Versus The Most It Can Be)

9 -Fewer but better Measures Design the Measurement Process to React to Continuous Change Use the measurement results-Use the measurement results-Use the measurement results Software and Systems MeasurementPSM - IFPUG, 33 PSM All rights Products Practical Software and Systems MeasurementPSM - IFPUG, 34 PSM All rights Products Available PSM Book - Version 5 - Published by Addison Wesley PSM Insight PSM Guidebook - Version 4 1 and 2 day Training Sessions Measurement Workshops for Programs or Organizations Measurement Experience Reports Sample Measurement SpecificationsPSMCRS Software and Systems MeasurementPSM - IFPUG, 35 PSM All rights Products Available (cont) DoD Implementation Guidance Measurement White Papers-Evolutionary Acquisition-Interoperability-Object-Orie nted Design-Product Lines applying PSM to Enterprise Measurement (PSM and the Balanced Scorecard) Rational Unified Process (RUP) Measurement Plug-InPractical Software and Systems MeasurementPSM - IFPUG, 36 PSM All rights Products Under Development Measurement White Papers-Acquisition Measurement-Safety and Security Measurement (UK/Australian collaboration)-Systems Engineering Technical Measures (INCOSE collaboration)-System of System Measurement (DAU collaboration)-Process Improvement Measurement-Organizational / Enterprise MeasurementPSMCRS Software and Systems MeasurementPSM - IFPUG, 37 PSM All rights More InformationCheryl JonesQuality Engineering & System Assurance (QA&SA)US Army ARDECAMSRD-AAR-QEW-DBuilding 62 Picatinny Arsenal, NJ 07806-5000(973) 724-2644 (Voice)(973) 724-2382 Home Page.

10 Support Center47196 Middle Bluff PlacePotomac Falls, VA 20165 Practical Software and Systems MeasurementPSM - IFPUG, 38 PSM All rights Material PSMCRS Software and Systems MeasurementPSM - IFPUG, 39 PSM All rights Participating Organizations US Navy NAVSEA US Navy NAWC US Navy NRL US Navy NSWC US Navy NUWC US Navy OPTEVFOR OSD NII OSD PA&E ODUSD (S&T) Aerospace Corp. Dept of Homeland Security, Customs & Border Protection DoD and Government US Air Force AFMC US Air Force SSSG US Air Force STRATCOM US Air Force STSC US Army AMC US Army CECOM US Army RDECOM-ARDEC US Army SAALT US Army SMDC USMC MCTSSA US Navy AEDC US Navy ASN RDA US Navy FMSO US Navy NAVAIR DAU-DSMC DCMA DFAS DISA DLA FAA IDA MITRE Corp. NASA Sandia National Lab Soc. Sec. Admin. Software Engineering InstitutePractical Software and Systems MeasurementPSM - IFPUG, 40 PSM All rights Participating Organizations Hawaiian Electric IEEE IFPUG IITRI INCOSE ITABHI Independent Engineering, Inc.