Capture and replay is an efficient possibility to create automated regression tests. The tests that are generated in this way are considered as fragile and have a high need of maintenance. A big problem is the recognition of components of the graphical user interface. Often manual created test scripts and laborious patterns like page objects are used to reduce the usage of assets. Scripts require programming skills which complicates the testing by domain experts. Additional costs can arise due to high communication need.
The goal is to examine the strengths and weaknesses of the capture-replay approach, therefore four commercial and open source tools for the Java interface technology Swing are evaluated. For this purpose a test scenario is designed that addresses typical deficits of such tools and helps simulating daily development processes. This report shows that the results can vary a lot and which kind of advantages and disadvantages capture and replay offers in comparison to manual created test scripts.
The results of this report show that the effectiveness of capture-replay-tools varies a lot. In contrast to relevant publications the compatibility regarding GUI components and the corresponding events was checked and also the behavior of the tools in their discipline was evaluated – regression testing. Four tools were chosen: Pounder, Marathon, QF-Test and Ranorex that have different backgrounds. Pounder as open source and free tool that isn’t actively maintained anymore but has a good reputation. Marathon however isn’t open source and freely available, but is maintained continuously by salaried developers. QF-Test is a proprietary tool which is highly mature and offers a huge amount of additional features. Ranorex’s profile is similar, but in this evaluation it is the only tool that is not implemented in Java.
These differences are to be taken into account for the valuation of the results. QF-Test is the tool that could finish the most tests successfully, but it is compared to the second winner Marathon a closed solution and fee-based. With MarathonITE an extended, promising variant of Marathon is available. As compromise you must forego open source and free availability. Pounder and Ranorex on third and fourth place seem absolutely unsuitable. Punder needs too big adaptions due to its restrictions to start and for administration of complex applications, Ranorex decelerates the execution time enormously and came off badly doing the tests. So it seems advantageous when SUT and tool use the identical platform as it is the case here: Java and Java Virtual Machine (JVM).
Thus two tools, QF-Test and Marathon are an efficient possibility of generating automated regression tests. Domain experts without programming skills can create their own functional tests without having to develop written specifications that must be implemented by developers tediously. Both tools offer the connection of scripting languages, by what capture and replay can be used in early stages of development efficiently. Furthermore adaptions for optimizing the application which is to be tested via capture-and-replay tools (e.g. allocation of explicit names) seem less time-consuming, like special patterns (e.g. page objects), that are frequently used for testing with manually created test scripts.
This thesis needs further empiric studies to compare the efficiency of capture and replay directly with alternative solutions. The results can be considered for the selection of an appropriate tool. The designed test scenario offers a good point of departure for further comparisons. But at this point further adaptions are necessary. This evaluation has shown that diverse tests depend too much from each other. Further, more independent tests enable an easier analysis of the actual error cause. Furthermore the test scenario or SwingSet3 can be extended to apply typical characteristics of software projects for being able to evaluate the suitability for daily use or an alternative test strategy. Here for example the connection of external data sources or the integration in build processes is imaginable.
The whole extract from the evaluation report concerning QF-Test can be found here (German PDF only).
