The Impact of Test Case Prioritization on Test Coverage ...

1 The Impact of Test case Prioritization on Test Coverage versus Defects FoundRamadan AbdunabiYashwant K. MalaiyaComputer Information Systems DeptComputer Science DeptColorado State UniversityColorado State UniversityFort Collins, CO 80523 Fort Collins, CO Prior studies demonstrate the importance ofthe relationship between code Coverage and defects found todetermine the effectiveness of test inputs. The variation ofdefect Coverage with code Coverage has been studied for a fixedexecution sequence of test cases. We conducted an experimentto evaluate hypotheses expressed in two research first question addresses the relationship between defectcoverage with code Coverage for different execution sequencesof test cases. The second research question evaluates theeffectiveness and expensiveness of employing the prioritizationtechniques. The study confirms that altering the test casesexecution order can affect the relationship between thecode Coverage and defects found.

2 The results show that theoptimal Prioritization outperforms the st-total and the Terms Defect Coverage , Statement Coverage , Empir-ical Study, Test Cases, PrioritizationI. INTRODUCTIONThe relationship between test Coverage and the numberof defects found is an indicator of software testing and London [9] showed that code Coverage is anindicator of testing effectiveness and completeness. A highcode Coverage with low fault rate indicates a high this work, we conducted an experiment to investigatethe relationship between test Coverage and number of defectsfound for varying execution order of test cases. The testcases are ordered in each execution sequence based on TestCase Prioritization (TCP) techniques defined in the study byElbaum at al [7]. We propose an approach that can be usedto improve the selection of the most cost-effective research results provide insight into the tradeoffsbetween techniques, and the conditions underlying thosetradeoffs, relative to the programs, test suites, and modifiedprograms that we examine.

3 If these results generalize toother workloads, they could guide the informed selectionof techniques by practitioners. The analysis strategy we usedemonstrably improves the Prioritization technique selec-tion process, and can be used by practitioners to evaluatetechniques in a manner appropriate to their chosen experimental studies [4], [10] showed that at thebeginning of testing, the number of detected faults growsslowly, which appears askneeshape in a plot of the defectsfound versus test Coverage . Following theknee, when thefault detection has been already started, the number ofdetected faults grows linearly with test Coverage . Saturationor acceleration is unlikely, and have not been , all the studies to date presented in section 2 ignorethe role of the ordering of test cases in the growth of numberof detected faults and test test cases might have different fault detectingability and by altering the execution order of the test caseschanges the curve of defects found versus test Coverage .

4 Thefirst hypothesis in this study is that the most effective testcases will give high test Coverage and number of defectsfound than less effective test cases. However, we cannotdistinguish between test cases unless we execute them andthen investigate the reported hypothesis is that altering the execution se-quences of test cases will change the position of other important expected results in this study is a betterunderstanding of how to choose the test sequence that leadsto maximum defect Coverage growth. Further expectation ofthe possible effects is the change in the shape of the curvefollowing theknee. The future goal of the study is to obtaina better insight into the behavior of the number of defectsfound with test Coverage in order to propose more accuratefaults prediction important consideration in our empirical study offault detection is whether to use natural, manually seeded,or automatically seeded faults.

5 Using automatically seededfaults was the only feasible option. Even apart from re-source considerations, automatically seeded faults offer someadvantages for experimentation: unlike hand-seeded faults,automatically seeded faults are not influenced by the personseeding the created faulty versions using mutation-based faultinjection to non-faulty programs. This was done to enlargeour data sets and because mutation-based faults have widelyused for analyzing test effectiveness [6], [2], [13]. For all ofour subject programs, any faulty versions that did not leadto at least one test case failure in our execution environmentwere initially had a concern that the effectiveness mea-sure may not capture the artificially seeded faults andreal effectiveness of a test suite and; its fault detectioncapability.

6 Nevertheless, structural Coverage and mutationscore has been widely used as a successful surrogate offault detection capability in software testing literature [6],[2], [13]. Andrews et al. [6] reported that faults generated10 Int'l Conf. Software Eng. Research and Practice | SERP'17 |ISBN: 1-60132-468-5, CSREA Press with mutation operators are similar to hand seeded faults forseven programs and natural faults in a space , a mutation testing tool for Java programs, is usedin our study to generate faulty version programs to simulatereal faults. Mutant programs are commonly used as practicalreplacements for real faults. In these studies [14], [20], [1],a mutant program has shown that mutation faults can berepresentative of real rest of the paper is organized as the background material about the relationshipbetween code Coverage and defects found and Test CasePrioritization (TCP) techniques.

7 The challenges in the study,experimental approaches, and data sets are discussed the results with respect tothe research questions. The potential threats to validity arediscussed inSection5. Finally,Section6summarizes theresults and discusses future research RELATEDWORKP rior studies showed that the test Coverage is a good es-timator of the defects after achieving high Coverage . Bishop[3] used test Coverage to estimate the number of residualfaults. This model is applied to a specific data set with knownfaults, and the results agreed well with the model. Cai et al.[5] proposed a reliability growth model that combines testingtime and test Coverage Malaiya Li Bieman Karcich SkibbeMLBKS model[15] relates test Coverage and defects found. This studydemonstrated the early applicability of their model to de-scribe the relationship between the test Coverage and thedefects found.

8 Furthermore, the study showed that the initialdefect density decides the position of thekneeof the curvedescribing the and Lyu [4] studied the relationship between codecoverage and fault detection using various scenarios: dif-ferent test case Coverage techniques, functional testing testing, normal operational testing vs. exceptionaltesting, and in different combinations of Coverage overall result showed code Coverage is a good indicatorfor testing contrast to our results, these studies empirically showedthat random test case Prioritization ( random ordering)can be ineffective. It has been a long tradition to deemrandom ordering as the lower bound control technique . Ifrandom ordering is indeed ineffective, we would like to askthe question: Why are other techniques not used to resolvetie cases?

9 Moreover, none of these studies are concerned about therelationship between code Coverage and defect coverageunder different execution sequences of test cases. In ourstudy, the test cases are given particular order in eachexecution to examine the Impact on the Coverage and defectrelationship. These prior studies have investigated smallprograms than the number of programs. Thus, we suspectthat the results in these prior studies are limited and cannotbe , a number of test case Prioritization approacheshave been proposed [11], [23]. Similar to our work, these ap-proaches compare Test case Prioritization (TCP) techniquesin terms of their effectiveness, similarity, efficiency, andperformance degradation. While these studies are consideredamong the most complete studies in terms of evaluationdepth, they ignored the static techniques considered in thispaper.

10 Thus, our study is differentiated by the unique goal ofunderstanding the relationships between purely static et al. [23] studied the regression Prioritization andFast Mutation Testing Prioritization , which are similar to ourstudy: to reorder the test cases to make regression/mutationtesting faster. However, their mechanisms are different. Re-gression test Prioritization aims to cover more program unitsfaster, thus increasing the probability of revealing unknownfaults earlier; whereas the locations of mutation faults ( ,mutated statements) are known for mutation testing and asimple strategy can merely execute the test cases that reachthe mutation faults, making the Coverage -based regressiontest Prioritization techniques are not suitable for et al. [19], [7] studied the Total and Ad-ditional approaches that utilize program dynamic coverageinformation.