Current Defect Density Statistics - SoftRel, LLC

1 You may not reprint or copy any part of this presentation withouYou may not reprint or copy any part of this presentation without express and written t express and written consent from Ann Marie consent from Ann Marie NeufelderNeufelderCurrent Defect Density Current Defect Density StatisticsStatisticsAnn Marie Ann Marie NeufelderNeufelderCopyright SoftRel, LLC 2007 Copyright SoftRel, LLC 2007 Copyright SoftRel, LLC 2007 Copyright SoftRel, LLC 2007 Actual fielded Defect Density from 90+ projects Actual fielded Defect Density from 90+ projects spanning nearly every industryspanning nearly every industryDefect Density groupAverage delivered normalized Defect densityWorld clas sVery goodGoodAverageFairPoorUglyThis data is in terms of fielded (escaped) defects per 1000 lines of effective code normalized to clusters are visible.

2 A method to predict which cluster your project will fall into was developed from this SoftRel, LLC 2007 Copyright SoftRel, LLC 2007 How to determine normalized effective sizeHow to determine normalized effective sizePredict/count new and modified lines of codePredict/count new and modified lines of codePredict/count deleted lines Predict/count deleted lines Multiply existing but unchanged code by 10%Multiply existing but unchanged code by 10%Entire functions deleted reduce existing sizeEntire functions deleted reduce existing sizeEffective size = Modified + New + Lines subtracted + (10% of exiEffective size = Modified + New + Lines subtracted + (10% of existing code)sting code)Multiply effective size by conversion ratio to Multiply effective size by conversion ratio to assembler using industry tables as summarized assembler using industry tables as summarized belowbelow Second generation (C, Fortran) Second generation (C, Fortran) 33 Object oriented (Java, C++, Object oriented (Java, C++, AdaAda9x) 9x)

3 66 Visual Basic Visual Basic 1010 Copyright SoftRel, LLC 2007 Copyright SoftRel, LLC 2007 Use the SoftRel Survey to predict the best Use the SoftRel Survey to predict the best clusterclusterAppropriate Appropriate ClusterCluster determined by these thingsdetermined by these things Inherent stability of existing design and codeInherent stability of existing design and code Methods and techniques use to prevent defects and develop Methods and techniques use to prevent defects and develop softwaresoftware Application typeApplication type Existence of major obstacles (new technology, new Existence of major obstacles (new technology, new environments, etc.)environments, etc.) Existence of major opportunities (end user domain experts Existence of major opportunities (end user domain experts available to project, etc.))

4 Available to project, etc.) Inherent stability of development processInherent stability of development processProcess alone will not guarantee a world class cluster!Process alone will not guarantee a world class cluster! An SEI CMM level 1 organization can be world class An SEI CMM level 1 organization can be world class An SEI CMM level 4 or 5 does not guarantee world class An SEI CMM level 4 or 5 does not guarantee world class Copyright SoftRel, LLC 2007 Copyright SoftRel, LLC 2007 How the survey score maps to the clusters How the survey score maps to the clusters SoftRel survey score versus Defect Density groupSoftRel Survey scoreWorld classVery goodGoodAverageFairPoorUglyThe most variation exists in the world class cluster, however, this cluster is easily predictable because of void of major obstacles and presence of major opportunities as shown SoftRel, LLC 2007 Copyright SoftRel.

5 LLC 2007 The World Class Cluster had no major The World Class Cluster had no major obstaclesobstaclesMajor project obstacles versus Defect Density groupNumber of major project obstaclesWorld clas sVery goodGoodAverageFairPoorUglyObstacles are defined specifically as new technology, new operating system, new development environment, new compiler, new target hardwareCopyright SoftRel, LLC 2007 Copyright SoftRel, LLC 2007 The The UglyUgly group had no opportunities group had no opportunities Major project opportunities versus Defect Density groupNumber of major project opportunitiesWorld classVery goodGoodAverageFairPoorUglyOpportunity Explicitly defined as the degree to which end user domain experts are available to the software engineers for this projectCopyright SoftRel, LLC 2007 Copyright SoftRel, LLC 2007 Average Defect Density by system typeAverage Defect Density by system special target hardwaren/a systemsn/a stationery capitol ground , control and and controlRatio of test to fieldAverage testing Defect densityAverage fielded Defect densitySystem application typeCopyright SoftRel, LLC 2007 Copyright SoftRel.

6 LLC 2007 Average Defect Density by software typeAverage Defect Density by software process HW is new or servern/a knowledge can be acquired via public domain in short period of capabilitiesRatio of test to fieldAverage testing Defect densityAverage fielded Defect densitySoftware application typeThis is the same set of data sliced a different SoftRel, LLC 2007 Copyright SoftRel, LLC 2007 Average Defect Density by risk levelAverage Defect Density by risk risks (loss of product with monetary value) risks (banking, etc) risk (occupational, regional, national or global)Ratio of test to fieldAverage testing Defect densityAverage fielded Defect densityRisk levelThis is the same set of data sliced a different SoftRel, LLC 2007 Copyright SoftRel, LLC 2007 You can also predict the risk of a late deliveryYou can also predict the risk of a late deliveryNormalized Fielded Defect Density Percentile Ratio of testing to fielded Defect Density Ave Min Max Stddev Probability of a late delivery (%) Margin of error when delivery is late (%) World Class.

7 011 .006 10 Very Good .060 .0172 20 25 Good .112 .0169 25 25 Average .250 .0590 36 41 Fair .618 .177 85 125 Poor .081 100 100 Ugly .5 .524 83 75 Probability of late delivery If your organization makes 10 releases and the probability of being late is 10% then 1 out of 10 will be lateMargin of error Measured as a percentage of the original schedule predictionCopyright SoftRel, LLC 2007 Copyright SoftRel, LLC 2007 How to predict your clusterHow to predict your clusterAnswer a survey based onAnswer a survey based on RisksRisks Product characteristics Product characteristics Application typeApplication type ResourcesResources Practices, techniques and methodsPractices, techniques and methods Process stabilityProcess stabilityDetermine a baseline cluster for your Determine a baseline cluster for your typicaltypical projectprojectEach project specific additional obstacle lowers Each project specific additional obstacle lowers the cluster while adding domain expertise raises the cluster while adding domain expertise raises the clusterthe clusterCopyright SoftRel, LLC 2007 Copyright SoftRel.

8 LLC 2007 Lessons LearnedLessons LearnedRisks cannot be overcome by any of the followingRisks cannot be overcome by any of the following New expensive automated tools to theoretically speed up New expensive automated tools to theoretically speed up development (this will actually increase the risk level for the development (this will actually increase the risk level for the first first time project)time project) Wishful thinkingWishful thinkingRisks can be minimized byRisks can be minimized by More granular milestonesMore granular milestones Addressing high risk items before everything else in the Addressing high risk items before everything else in the scheduleschedulesoftware engineers tend to work on the low risk tasks firstsoftware engineers tend to work on the low risk tasks first Design prototyping when design is a riskDesign prototyping when design is a risk Requirements prototyping when end user requirements are Requirements prototyping when end user requirements are volatilevolatile Defect prevention techniques such as formal unit testingDefect prevention techniques such as formal unit testing Increasing the end user domain knowledge

9 Of the team Increasing the end user domain knowledge of the team This does not mean software experience This does not mean software experience this means application this means application experienceexperienc