Measuring Combinatorial Coverage of Manual Testing

1 M lardalen University School of Innovation, Design and Engineering V ster s, Sweden Thesis for the Degree Master of Science in Software Engineering credits Measuring Combinatorial Coverage of Manual Testing Miraldi Fifo Examiner: Associate Professor Wasif Afzal M lardalen University, V ster s, Sweden Supervisor: Dr. Eduard Paul Enoiu M lardalen University, V ster s, Sweden 1 Abstract introduction : Software Testing is a very important activity which assures the quality of the software under test. It becomes crucial in safety-critical systems, where an unexpected behavior of the software can even cause loss of human life or environmental disasters.

2 However, in such complex systems it becomes infeasible to test all possible software scenarios for possible faults. Experience shows that software faults, which can cause unexpected software behavior, are caused by the interactions of variables of the tests. Combinatorial Testing is the technique which focuses on the variable interactions of the tests and aims to reduce the number of tests needed to cover all software scenarios while still preserving a high fault detection rate. Background: Manual Testing is the technique used to assure software quality in Bombardier Transportation AB, a Swedish company whose focus is on rail transport and development of trains. Since this process depends on the skills of the engineers, it can result in a large portion of tests not created and consequently in a large number of scenarios uncovered with tests.

3 Therefore, Combinatorial Testing technique is used to measure the Combinatorial Coverage of these tests created from experienced engineers. Many comparisons of Manual Testing and other Testing techniques in terms of test Coverage , code Coverage or mutation analysis. To the best of our knowledge there are no other studies in literature that have measured the Combinatorial Coverage of Manual tests designed from experienced engineers, for different strength interactions of the variables of the tests nor other available tools that generate the number of missing tests to achieve full Combinatorial Coverage for specific interactions. Aim: The goal of this thesis is to answer the two research questions: RQ1. What is the Combinatorial Coverage achieved by tests manually created by experienced engineers in industry?

4 RQ2. Can the effectiveness of manually created tests be improved in terms of Combinatorial Coverage using Combinatorial Testing ? Method: In this thesis we investigate the Combinatorial Coverage of manually created tests by engineers working in industry and the implications of using Combinatorial Testing in practice. The Combinatorial Coverage Measurement 2 (CCM) NIST tool is used to measure the test Coverage achieved. The research questions are answered by the following steps: 1) Review the scientific literature for related work, 2) Refine thesis research questions based on the previous step, 3) Propose the case study design and perform the measurements needed for data analysis, 4) and the results are analyzed and discussed in terms of test efficiency ( , number of test cases) and effectiveness ( , achieved Combinatorial Coverage ).

5 Results: The 2-way interaction Combinatorial Coverage achieved by Manual tests is on average, 57% for 3-way Combinatorial Coverage , for 4-way Combinatorial Coverage , for 5-way Combinatorial Coverage and 13% for 6-way Combinatorial Coverage . Full Combinatorial Coverage can be achieved for 2-way and 3-way interactions by adding eight and 66 missing tests on average, respectively. For 4-way interactions full Combinatorial Coverage can achieved by adding 658 missing tests. For 5-way and 6-way interactions full Combinatorial Coverage can be achieved by adding 5163 and 6170 missing tests on average respectively. Conclusion: The Combinatorial Coverage decreases as the strength of interactions increases. The effectiveness of the tests can be improved for 2-way and 3-way interactions and fully or partially improved for 4-way interactions, depending on the decision of engineers to add all missing tests or part of them, since the number of missing tests is increasing significantly, thus resulting in a very large number of tests to be added.

6 It is not particularly efficient to improve test effectiveness by augmenting Manual tests cases using Combinatorial Testing for higher strength interactions, since in most of the tests suites we studied one would need to generate additional missing tests. This is explained by the exponential growth of the number of variable combination for such interactions. 3 Acknowledgments First of all, I want to thank my supervisor Eduard Paul Enoiu for his patience and support throughout this thesis. I am sure that without his suggestions and help, the process of writing this thesis would have been much more difficult than it actually was. Secondly, I want to thank my family for their unconditional support and love. These two elements were crucial for me to overcome the difficulties. Last but not least I want to dedicate this work to my beloved grandparents, whose principal dream during this period was to see me achieve this goal.

7 4 5 Table of Contents Table of Contents introduction .. 9 Problem Description .. 10 Contribution .. 10 Overview .. 12 Background .. 13 Programmable Logic Controllers and Function Block Diagrams .. 14 Related Work .. 17 Combinatorial Testing .. 17 Combinatorial Coverage .. 27 Case Study Design .. 31 Problem Formulation .. 31 Method .. 32 Measuring Test Efficiency and Effectiveness .. 33 Manual Testing .. 33 NIST tool for Measuring Model Coverage .. 34 Results and Analysis .. 39 Results Q1: What is the Combinatorial Coverage achieved by tests manually created by experienced engineers in industry? .. 39 Results Q2: Can the effectiveness of manually created tests be improved in terms of Combinatorial Coverage using Combinatorial Testing ? .. 50 Discussion and Validity Threats.

8 65 Discussion .. 65 Validity Threats .. 68 Conclusion and Future Work .. 71 Conclusion .. 71 Future Work .. 72 Bibliography .. 74 Appendix .. 81 6 List of Figures The truth table for a given predicate (a b) 100% 2-way Combinatorial Architecture of a PLC Simple Function Block Hardware structure of a Example of a test array with four binary variables taken The levels of Coverage for all variable combinations based on an The graph with the results of 2-way Combinatorial Coverage for a test array An example of an input file for the CCM Graphical representation of CCM tool developed by Box plot of Combinatorial Coverage for all t-way Scatter plot of number of variables and number of Manual tests for each test Bar plot of number of variables for each test Bar plot of number of tests for each test Scatter plot of number of variable/values for 2-way interactions and 2-way Combinatorial Coverage for each Manual Scatter

9 Plot of the number of variable/values for 3-way interactions and 3-way Combinatorial Coverage for each Manual Scatter plot of the number of variable/values for 4-way interactions and 4-way Combinatorial Coverage for each Manual Scatter plot of the number of variable/values for 5-way interactions and 5-way Combinatorial Coverage for each Manual Scatter plot of the number of variable/values for 6-way interactions and 6-way Combinatorial Coverage for each Manual Statistics of a test suite with mean values for nr. of variables, nr. of tests and nr. of Statistics of a test suite with mean values for nr. of tests and nr. of variable-values and an additional variable to its mean Statistics of a test suite with mean values for nr. of variables and nr.

10 Of variable values and a missing test to its mean Statistics of a test suite with mean values for nr. of variables and nr. of tests and an additional variable-value to its mean Graphical representation of the number of missing tests for each t-way 7 Example with 8 Manual tests and 4 8 List of Tables Table : Descriptive statistics for Combinatorial Coverage of each t-way Table : Inferential statistics for Combinatorial Coverage of each t-way Table : Statistics regarding average number of missing tests to achieve full Combinatorial Coverage on each t-way 9 Chapter 1 introduction Software programs nowadays are widely used in a large number of domains to facilitate the working process and working procedures.