Data-Driven and Keyword-Driven Test Automation …

1 HELSINKI UNIVERSITY OF TECHNOLOGYD epartment of Computer Science and EngineeringSoftware Business and Engineering InstitutePekka LaukkanenData- [~ keyword :7~] and [~ keyword :2~] TestAutomation FrameworksMaster s thesis submitted in partial fulfillment of the requirements for the degreeof Master of Science in , February 24, 2006 Supervisor:Professor Reijo SulonenInstructor:Harri T oh onen, UNIVERSITYABSTRACT OF THEOF TECHNOLOGYMASTER S THESISA uthor:Pekka LaukkanenName of the thesis: [~ keyword :1~] and [~ keyword :2~] Test [~ keyword :0~] FrameworksDate:February 24, 2006 Number of pages:98 + 0 Department:Department of ComputerProfessorship:T-76 Science and EngineeringSupervisor:Prof. Reijo SulonenInstructor:Harri T oh onen, growing importance and stringent quality requirements of software systems are in-creasing demand for efficient software testing.

2 Hiring more test engineers or lengtheningthe testing time are not viable long-term solutions, rather there is a need to decrease theamount of resources needed. One attractive solution to this problem is test [~ keyword :0~] , allocating certain testing tasks to computers. There are countless approaches to testautomation, and they work differently in different contexts. This master s thesis focuseson only one of them, large-scale frameworks for automated test execution and reporting,but other key approaches are also briefly thesis opens its discussion of test [~ keyword :0~] frameworks by defining their high-levelrequirements. The most important requirements are identified as ease-of-use, maintain-ability and, of course, the ability to automatically execute tests and report results. Moredetailed requirements are derived from these high-level requirements: [~ keyword :1~] andkeyword- [~ keyword :7~] testing techniques, for example, are essential prerequisites for both ease-of-use and next step in the thesis is constructing and presenting a framework concept fulfillingthe defined requirements.

3 The concept and its underlying requirements were testedin a pilot where a prototype of the framework and some automated tests for differentsystems were implemented. Based on the pilot results, the overall framework conceptwas found to be feasible. Certain changes to the framework and original requirementsare presented, however. The most interesting finding is that it is possible to cover allthe [~ keyword :1~] testing needs with the [~ keyword :2~] approach : test [~ keyword :0~] , test [~ keyword :0~] framework, [~ keyword :1~] testing, [~ keyword :2~] testingiiTEKNILLINEN KORKEAKOULUDIPLOMITY ON TIIVISTELM ATekij a:Pekka LaukkanenTy on nimi:Aineisto- ja avainsanaohjatut testiautomaatioj arjestelm atP aiv am a ar :98 + 0 Osasto:Tietotekniikan osastoProfessuuri:T-76Ty on valvoja:Prof. Reijo SulonenTy on ohjaaja:DI Harri T oh onenOhjelmistoj arjestelmien merkityksen sek a laatuvaatimusten kasvaminen aiheuttaa pai-neita ohjelmistojen testaukselle.

4 Testaajien m a ar an lis a aminen tai testausajan pi-dent aminen ei ole loputtomasti mahdollista, pikemminkin resursseja halutaan v ahent a houkutteleva ratkaisu on testauksen automatisointi eli osan testausty on antami-nen tietokoneiden hoidettavaksi. Erilaisia tapoja testauksen automatisointiin on lukui-sia ja ne toimivat eri tavoin erilaisissa tilanteissa ja ymp arist oiss a. T am a diplomity ok asittelee tarkemmin vain yht a l ahestymistapaa, laajoja automaatioj arjestelmi a testienautomaattiseen suorittamiseen ja raportoimiseen, mutta my os muut tavat ovat t arkeit arjestelmien k asittely aloitetaan m a arittelem all a niille korkean tasonvaatimukset. T arkeimmiksi vaatimuksiksi todetaan helppok aytt oisyys, yll apidett avyyssek a tietenkin kyky automaattisesti suorittaa testej a ja raportoida niiden aiden vaatimusten kautta p a ast a an tarkempiin vaatimuksiin ja todetaan mm.

5 Ett aaineisto-ohjattu ( [~ keyword :1~] ) ja avainsanaohjattu ( [~ keyword :2~] ) testaustekniikkaovat edellytyksi a sek a helppok aytt oisyydelle ett a yll apidett ty oss a suunnitellaan m a aritellyt vaatimukset toteuttava testiautomaa-tioj arjestelm a. J arjestelm an toimintaa sek a sen pohjana olleita vaatimuksia testataanpilotissa, jossa toteutetaan sek a prototyyppi itse j arjestelm ast a ett a automatisoitu-ja testej a erilaisille ohjelmistoille. Pilotin tuloksien perusteella suunnitellun automaa-tioj arjestelm an voidaan todeta olevan p a aperiaatteiltaan toimiva. Lis aksi kokemus-ten perusteella j arjestelm a an sek a alkuper aisiin vaatimuksiin esitet a an joitain muutok-sia. Mielenkiintoisin l oyd os on se ett a kaikki aineisto-ohjatut testit voidaan toteuttaak aytt aen ainoastaan avainsanaohjattua l : testiautomaatio, testiautomaatioj arjestelm a, aineisto-ohjattu testaus,avainsanaohjattu testausiiiAcknowledgementsThis master s thesis has been done for a Finnish software testing consultancy com-pany Qentinel mainly during the year 2005.

6 I wish to thank my instructor HarriT oh onen, and all other Qentinelians for comments, feedback and the Department of Computer Science and Engineering I first of all want tothank my instructor Professor Reijo Sulonen. Additionally I am grateful for JuhaItkonen and Casper Lassenius for their support and valuable also want to express my gratitude to Mark Fewster who was kind enough toreview the thesis in its early form. Mark s comments and positive feedback mademe believe that the ideas I present are valid and the remaining hard work is worththe effort. Also I want to thank Petri Haapio who has been in charge of both theautomation project where I got the original idea for this thesis and a new one wherethe [~ keyword :0~] framework presented in this thesis has been successfully taken intoreal , I would like to thank my family and my wonderful girlfriend Sonja foreverything.

7 , February 24, 2006 Pekka LaukkanenivContentsTermsix1 Promises and Problems of Test [~ keyword :0~] .. Different Test [~ keyword :0~] Approaches .. Dynamic vs. Static Testing .. Functional vs. Non-Functional Testing .. Granularity of the Tested System .. Testing Activities .. Small Scale vs. Large Scale Test [~ keyword :0~] .. Scope .. Methodology .. Goals .. Structure .. 132 Requirements for Test [~ keyword :0~] High Level Requirements .. Framework Capabilities .. Executing tests Unattended .. Starting and Stopping Test Execution .. Handling Errors .. Verifying Test Results .. Assigning Test Status .. Handling Expected Failures .. Detailed Logging .. Automatic Reporting .. Modularity.

8 Linear Test Scripts .. Test Libraries and Driver Scripts .. Promises and Problems .. [~ keyword :1~] Testing .. Introduction .. Editing and Storing Test [~ keyword :4~] .. Processing Test [~ keyword :4~] .. Promises and Problems .. [~ keyword :2~] Testing .. Introduction .. Editing and Storing Test [~ keyword :4~] .. Processing Test [~ keyword :4~] .. Keywords in Different Levels .. Promises and Problems .. Other Implementation Issues .. Implementation Language .. Implementation Technique .. Testware Architecture .. Testability .. Control .. Visibility .. Roles .. Test [~ keyword :0~] Manager .. Test [~ keyword :0~] Architect .. Test Automator .. Test Designer .. Detailed Requirements .. Chapter Summary.

9 423 Concept for Large Scale Test [~ keyword :0~] Framework Structure .. Test Design System .. Test Monitoring System .. Test Execution System .. Presenting and Processing [~ keyword :1~] Test [~ keyword :4~] .. Presenting Test Cases .. Using Test [~ keyword :4~] .. Example .. Presenting and Processing [~ keyword :2~] Test [~ keyword :4~] .. Presenting Test Cases .. Presenting User Keywords .. Using Test [~ keyword :4~] .. Chapter Summary .. 614 Implementation and Implementation Decisions .. Technical Decisions .. Decisions Regarding the Pilot .. Implementing Reusable Framework Components .. Test [~ keyword :4~] Parser .. Logger .. Summary .. [~ keyword :1~] Windows Application Testing .. Test [~ keyword :4~] .. Driver Script .. Test Library.

10 Test Log .. Summary .. [~ keyword :2~] Windows Application Testing .. Test [~ keyword :4~] .. Driver Script .. Test Library .. Test Log .. Summary .. [~ keyword :2~] Web Testing .. Test [~ keyword :4~] .. Driver Script .. Test Library .. Test Log .. Summary .. Chapter Summary .. 855 Feasibility of the Framework Concept .. Changes to the Framework and Requirements .. Using Only [~ keyword :2~] Approach .. Set Up and Tear Down .. Test Suites .. Generic Driver Script .. Revised List of Requirements .. Chapter Summary .. 916 Conclusions93 Bibliography95viiiTermsAcceptance TestingA level of testing conducted from the viewpoint of the customer,used to establish the criteria for acceptance of a system.