Combinatorial Testing and Covering Arrays

1 Combinatorial Software TestingCovering ArraysCombinatorial Testing and Covering ArraysLucia MouraSchool of Electrical Engineering and Computer ScienceUniversity of 2017 Combinatorial Testing and Covering ArraysLucia MouraCombinatorial Software TestingCovering ArraysSoftware and Network TestingWe want to test asystem:a programa circuita package that integrates several pieces of softwaredifferent platforms where a package needs to run correctlya highly configurable softwarea GUI interfacea cloud applicationWe would like atest suitethat gives a goodcoverageof theinput parameter space in order to detect the maximum number oferrors/ Testing and Covering ArraysLucia MouraCombinatorial Software TestingCovering ArraysCombinatorial Software TestingFirst we isolate the systemparametersand its possiblevaluesthe input parameters of a program and its possible values5, 10, 20, 25, 30<100,000[100,000-250,000)[250,000-450,000)> 450,00011 op es6 op es17 op es(5,4,11,17,6)the inputs of a circuit.]]

2 5 binary inputsIntroduction6 Example our second example, we look at the circuit diagram in Figure is a small circuit, with only five inputs -a,b,c,d,ande-eachofonebit,andthree outputs. This circuit could be exhaustively tested using 25=32tests,butbasedon the fact that some of the inputs do not interact, doing so is unnecessary. We insteadshow that we can fully test the circuit using only eight tests by using a variable strengthcovering array . We represent the circuit as a hypergraph, with one vertex for each input,and one hyperedge for each output, as shown in Figure : An example of a small circuit with five inputs and three : The hypergraph that arises from the circuit in Figure , taking each inputas a vertex and each output as a hyperedge over the inputs contributing to its now give a variable strength Covering array for the circuit that requires only eightdifferent test cases, with the guarantee that for each output, every possible combinationof inputs have been tested together in some test third example demonstrates the use of variable strength coveringarrays to systems Testing .

3 Assume that we have a simple 3-tier architecture system as(2,2,2,2,2)the components of a platform and its configurationsComponentWeb BrowserOperatingConnectionPrinterSystemT ypeConfigConfig:Netscape(0)Windows(0)LAN (0)Local (0)IE(1)Macintosh(1)PPP(1)Networked(1)Ot her(2)Linux(2)ISDN(2)Screen(2)Table 3: Four components, each with 3 configurationsTest CaseBrowserOSConnectionPrinter1 NetScapeWindowsLANL ocal2 NetScapeLinuxISDNN etworked3 NetScapeMacintoshPPPS creen4 IEWindowsISDNS creen5 IEMacintoshLANN etworked6 IELinuxPPPL ocal7 OtherWindowsPPPN etworked8 OtherLinuxLANS creen9 OtherMacintoshISDNL ocalTable 4: Test Suite to Cover all Pairs from Table 3agriculture and manufacturing [64]. It has entered the software Testing community, appear-ing in practitioner s guidebooks [70, 89], and provided in simple spreadsheet formats [40, 41].

4 The use of Covering Arrays in software Testing was pioneeredby Mandl [83] and Brownlie etal. [9, 102], and statistical foundations were explored in [46, 47, 48, 49, 82, 91]. Empiricalresults indicate that Testing of all pairwise interactionsin a software system indeed finds alarge percentage of existing faults [45, 78]. Indeed, Burr etal. [11]providemoreempiricalresults to show that this type of test coverage leads to usefulcodecoverage as well. empirical results to argue that the Testing of all pairwise interactions in asoftware system finds a large percentage of the existing faults [45]. theeffectiveness of these methods to software code coverage. They show that high code blockcoverage is obtained when Testing all two-way interactions,buthighersubsetsizesareneed edfor good path coverage [54].

5 Fault reports for three software show that 70% of faults can be discovered by Testing all two-way interactions, while90% can be detected by Testing all three way interactions. Six-way coverage was required inthese systems to detect 100% of the faults reported [78]. Thisstudywasfollowedbysimilarexperiments , such as one of 109 software-controlled medicaldevicesthatwererecalledbythe Food and Drug Administration (FDA) [79]. These experiments found that 97% ofthe flaws in these 109 cases could be detected with pair-wise Testing of parameter three devices required coverage higher than et al. [132] quantify the coverage for a particularinteraction strength. Forinstance, if we have four factors, any new test case can contribute at most(42),or6new31(3,3,3,3) Combinatorial Testing and Covering ArraysLucia MouraCombinatorial Software TestingCovering ArraysPairwise TestingTesting a system withk= 4components each havingv= 3values:ComponentWeb BrowserOperatingConnectionPrinterSystemT ypeConfigConfig:Netscape(0)Windows(0)LAN (0)Local (0)IE(1)Macintosh(1)PPP(1)Networked(1)Ot her(2)Linux(2)ISDN(2)Screen(2)Table 3: Four components, each with 3 configurationsTest CaseBrowserOSConnectionPrinter1 NetScapeWindowsLANL ocal2 NetScapeLinuxISDNN etworked3 NetScapeMacintoshPPPS creen4 IEWindowsISDNS creen5 IEMacintoshLANN etworked6 IELinuxPPPL ocal7 OtherWindowsPPPN etworked8 OtherLinuxLANS creen9 OtherMacintoshISDNL ocalTable 4.

6 Test Suite to Cover all Pairs from Table 3agriculture and manufacturing [64]. It has entered the software Testing community, appear-ing in practitioner s guidebooks [70, 89], and provided in simple spreadsheet formats [40, 41].The use of Covering Arrays in software Testing was pioneeredby Mandl [83] and Brownlie etal. [9, 102], and statistical foundations were explored in [46, 47, 48, 49, 82, 91]. Empiricalresults indicate that Testing of all pairwise interactionsin a software system indeed finds alarge percentage of existing faults [45, 78]. Indeed, Burr etal. [11]providemoreempiricalresults to show that this type of test coverage leads to usefulcodecoverage as well. empirical results to argue that the Testing of all pairwise interactions in asoftware system finds a large percentage of the existing faults [45].

7 Theeffectiveness of these methods to software code coverage. They show that high code blockcoverage is obtained when Testing all two-way interactions,buthighersubsetsizesareneed edfor good path coverage [54]. fault reports for three software show that 70% of faults can be discovered by Testing all two-way interactions, while90% can be detected by Testing all three way interactions. Six-way coverage was required inthese systems to detect 100% of the faults reported [78]. Thisstudywasfollowedbysimilarexperiments , such as one of 109 software-controlled medicaldevicesthatwererecalledbythe Food and Drug Administration (FDA) [79]. These experiments found that 97% ofthe flaws in these 109 cases could be detected with pair-wise Testing of parameter three devices required coverage higher than et al.

8 [132] quantify the coverage for a particularinteraction strength. Forinstance, if we have four factors, any new test case can contribute at most(42),or6new31 Test all possibilities:34= can be done with only 9 BrowserOperatingConnectionPrinterSystemT ypeConfigConfig:Netscape(0)Windows(0)LAN (0)Local (0)IE(1)Macintosh(1)PPP(1)Networked(1)Ot her(2)Linux(2)ISDN(2)Screen(2)Table 3: Four components, each with 3 configurationsTest CaseBrowserOSConnectionPrinter1 NetScapeWindowsLANL ocal2 NetScapeLinuxISDNN etworked3 NetScapeMacintoshPPPS creen4 IEWindowsISDNS creen5 IEMacintoshLANN etworked6 IELinuxPPPL ocal7 OtherWindowsPPPN etworked8 OtherLinuxLANS creen9 OtherMacintoshISDNL ocalTable 4: Test Suite to Cover all Pairs from Table 3agriculture and manufacturing [64]. It has entered the software Testing community, appear-ing in practitioner s guidebooks [70, 89], and provided in simple spreadsheet formats [40, 41].

9 The use of Covering Arrays in software Testing was pioneeredby Mandl [83] and Brownlie etal. [9, 102], and statistical foundations were explored in [46, 47, 48, 49, 82, 91]. Empiricalresults indicate that Testing of all pairwise interactionsin a software system indeed finds alarge percentage of existing faults [45, 78]. Indeed, Burr etal. [11]providemoreempiricalresults to show that this type of test coverage leads to usefulcodecoverage as well. empirical results to argue that the Testing of all pairwise interactions in asoftware system finds a large percentage of the existing faults [45]. theeffectiveness of these methods to software code coverage. They show that high code blockcoverage is obtained when Testing all two-way interactions,buthighersubsetsizesareneed edfor good path coverage [54].

10 Fault reports for three software show that 70% of faults can be discovered by Testing all two-way interactions, while90% can be detected by Testing all three way interactions. Six-way coverage was required inthese systems to detect 100% of the faults reported [78]. Thisstudywasfollowedbysimilarexperiments , such as one of 109 software-controlled medicaldevicesthatwererecalledbythe Food and Drug Administration (FDA) [79]. These experiments found that 97% ofthe flaws in these 109 cases could be detected with pair-wise Testing of parameter three devices required coverage higher than et al. [132] quantify the coverage for a particularinteraction strength. Forinstance, if we have four factors, any new test case can contribute at most(42),or6new31(example from Colbourn 2004) Covering Arrayswith strengtht= 2,k= 4parameters,v= 3values foreach, can cover all pairwise interactions withN= Testing and Covering ArraysLucia MouraCombinatorial Software TestingCovering ArraysPairwise TestingCovering array :strengtht= 2,k= 5paramters, values(3,2,2,2,3),N= 10testsPractical Combinatorial Testing _____ 7 selecting input values to exercise the application in each scenario, possibly supplementing these tests with unusual or suspected problem cases.