Software technologies have developed so quickly and extensively in recent years that developers have hardly had any time left to additionally take care of software quality. Therefore, testing has become an increasingly important part of software development to relieve developers.
Since in today's world people's lives depend on computer software, it is necessary if professional implementation is done. This is one reason why software testing is of utmost importance. Testing must be done before the software is deployed, as this helps to detect bugs in time and ensure that the functionalities of the software work as required. Software testing is an important part of software quality assurance, an activity to evaluate and improve software quality (Cf. F. Okezie et al 2019 p. 1).
In recent years, it often happened that software was almost finished before it was tested because many companies did not yet have defined testing departments. After some negative experience gained from this, there is now a consensus on how significant training and awareness of the importance of software testing during development has become. Software testing is necessary to meet the demand for high delivery quality, whether the software is delivered to customers or internal users (Cf. Witte (2019), p. 4.).
In the context of this work, the question is pursued as to where the differences, similarities, problems and limitations lie between the individual test tools. In summary, the test tools are compared with each other. Thereby the achievement of the goals, quality, and time must be possible, whereby also the economic aspects must be considered.
In addition, different test tools are compared with each other using different criteria. In chapter 2 the terms relevant for this work, like software quality, tests, errors, test tools and test expenditure are exactly defined. In chapter 3 the basic test procedures and test types are presented. The test procedures include the static and the dynamic test. In static testing, tool-assisted analysis and structured group testing - also called reviews - are considered in more detail. In dynamic testing, black box and white box procedures are explained. Test types include unit testing, integration testing, system testing, acceptance testing, functional testing, non-functional testing, negative testing, regression testing, and end-to-end testing. In chapter 4, defect management is explained in detail. Thereupon, a categorization of the test tools is presented in chapter 5. A distinction is made between tools for static testing and tools for dynamic testing. Chapter 6 compares the common tools with each other. Thereby it is defined, what is understood by these test tools and how they function. Since the license of three tools was not accessible, among them "Ranorex", "TestComplete", "UFT (Unified Functional Testing)", it is dealt in this work only on the description of these tools. Furthermore, in chapter 7 it is shown where the test tools can be used by system type. Finally, recommendations are presented to users in Chapter 8, taking into account programming skills and cost effectiveness considerations.
After extensive research in the literature on the following testing tools, the table below summarizes the required information about the tools based on certain selected criteria. This analysis can help testers to select the best tool for testing software. The table once again summarizes the features of the tools.
| Criteria/Tools | TestCafé | Selenium Web-Driver | Appium | Selenium IDE | QF-Test | Ranorex | TestComplete | UFT |
|---|---|---|---|---|---|---|---|---|
| Programming language | JavaScript, Type-Script, CoffeeScript | Java, C#, Perl, Python, JavaScript, Ruby, PHP | Java, C#, Perl, Python, JavaScript, Ruby, PHP | Java, C#, Perl, Python, JavaScript, Ruby, PHP | Java, Python, Groovy, JavaScript | C #, Python, C++, VB.net and XML | VbScript , Delphi, C++, C# und JScripts | VbScri pt |
| Supported platforms | Web | Web | Mobile, Windows | Web | Java, Web, Desktop (Win), Mobile (Android) | Mobile, Web, Desktop (Win) | Mobile, Web, Desktop (Win) | Web, Desktop (Win) |
| Reports on test results | yes | yes | yes | yes | yes | yes | yes | yes |
| Re- reusability | yes | yes | yes | no | yes | yes | yes | yes |
| Recording/ Playback | no | no | no | yes | yes | yes | yes | yes |
| easy to learn | yes | yes | no | yes | yes | |||
| Programming skills | yes | yes | yes | yes | no | no | no | no |
| Tools test types | End-to-End Tests | Funktional Unit Test, End-to-End Test | GUI Test, Functional Unit Test | Functional Unit Test | GUI Test | GUI Test | Funkcional Unit Test, GUI Test, Unit Test | Functional Unit Test, Regression Test |
| License | Free | Free | Free | Free | Not free | Not free | Not free | Not free |
| Sources | DevExpress - TestCafe | Thought-Works - Selenium Documentation (2021) | Tom Christie Appium - Documentation (2014) | Thought-Works - Selnium Dokumentation (2021) | QF-Test Documentation (2020) | Ranorex GmbH - Documentation (2021) | Smart-Bear Software - Documentation (2021) | MicroFocus - Documentation (2021) |
In software development, quality is the most important goal of any project, so when choosing a tool, it is recommended to take into account the scope of the project and the costs estimated for testing, Also, the platform on which the project will be deployed should be reflected in the criteria for selecting a testing tool. Based on the results obtained in this work, Ranorex, TestComplete, UFT and QF-Test are recommended for testing on all platforms and for large projects. Since these are licensed tools, the budget for testing should be considered. Appium is recommended for mobile applications, while TestCafe, Selenium Webdriver, Selenium IDE are available for web testing (Cf. F. Okezie et al. 2019 p. 9).
A goal of this work was it to compare individual test tools on the basis different criteria within the range of the software test with one another. For the comparison test tools were selected from the Internet and tested by means of a cinema web page, which was programmed in the context of the module "Web technology" during the study.
It is essential for a tester to know some important terms from the field of software testing and their definitions. This includes the term software quality, because quality and its assurance are also part of the tester's tasks. Likewise, a tester must master the definition of tests in order to be able to do the job with knowledge. Also the concept of error, which can have different definitions depending on what the error refers to, must be known. Since a tester has only a certain amount of time to test in a project, he must be able to estimate how long the test will take. In addition, the main types of tests that are suitable for implementation differ in their objectives. These include the objectives and methods of functional testing, where the functionality of the system as described in a specification is tested. Conversely, there are also the non-functional tests, which deal with aspects such as usability and user acceptance. In order for testing to be performed, procedures are needed to create test cases or to be able to get a view of the test object. Procedures such as the black box procedure, which performs testing based on specifications, and the white box procedure, which takes the program text as a reference for the tests, must be considered. In both procedures, different views of the test object can be considered, which differ greatly and provide a good overview of the system.
Software tests are an important part of the software development life cycle. They ensure that the software made available on the market is free of errors. Therefore, a tester must be able to distinguish the different software testing methods as well as the categories of tools and their areas of application. Tester should always be aware of the use of these tools in the development process.
During the work it was found that some tools are suitable only for testing web applications and others can be used for testing web, mobile, desktop - applications. Also most of the tools were "capture/replay tools" that can be used by users without programming skills.
What makes a good tester is the accumulated experience in software testing, because only through a lot of experience a tester can develop a good feeling for finding bugs in software. Another requirement for a good tester is having a lot of programming skills because the scripts are generated in different programming languages. Since a tester is the one who delivers the bug caused by another to the developer, he must have a certain sensitivity to convince the developers to fix the bug he found.
Because of the points discussed in this paper, software testing has become indispensable in software development. As a software tester, one is responsible for ensuring the quality of the product as well as the documentation. This allows developers to focus on programming, designing and correcting errors in the program. They no longer have to search for the errors themselves, but they are presented with them by the testers. Therefore, the cooperation and mutual support between them must work well.
This work would be useful for testers and developers to know which tool is best for a particular project, more tools can be compared and more criteria can be highlighted for comparison. This would allow software testers to choose perfect tools for testing applications. This will save more time and reduce costs if the right testing tool is selected. It can be concluded that there is no one perfect tool for testing, but for a particular testing purpose, compromises can be made depending on the size of the project to select the best tool that takes into account the planned cost of testing and allows for the platform of the application and also the language in which the project is developed.
The whole bachelor's thesis concerning QF-Test can be found here (German PDF only).
Bachelor's thesis: Software Testing: Common test tools in comparison, October 2021 - Fachhochschule Südwestfalen - University of Applied Sciences, Iserlohn, Germany.
