iSYS Software GmbH, based in Munich, is a software and consulting company that has established itself as a specialist in the field of software project management for more than 25 years. iSYS develops individual software solutions and offers consulting in the entire project environment. By designing, implementing, operating and maintaining IT solutions based on future-proof platforms and standards, iSYS increases the value and competitiveness of its customers and partners.
Erzsébet Nicole Harmat , working in the quality management department at iSYS, designed a methodology for the systematic selection of a test automation tool for a software development project as part of her bachelor thesis. The core of the methodology is the creation of an individual, project-related criteria catalog. This serves as a basis for the evaluation of test automation tools. The results, which are calculated on the basis of a formula designed in the methodology, show to what extent a test automation tool meets the requirements placed on a test automation tool by the project.
| Criteria | Weight-ing | Question | QF‑Test | Selenium | TestCafe | TestCafe Studio |
|---|---|---|---|---|---|---|
| Evaluation in the operational environment | 2% | Can the evaluation take place on site and on the object to be tested? | 2 | 2 | 2 | 2 |
| Market presence | 2% | Has the TaT (test automation tool) been on the market for at least 5 years? | 2 | 2 | 2 | 2 |
| Community | 3% | Is there any activity in the community in the last week or month? | 2 | 2 | 2 | 2 |
| Support | 4% | Is there support possibility via e-mail and telephone? | 2 | 0 | 1 | 1 |
| Update | 4% | Has there been an update in the last four months or year? | 2 | 1 | 2 | 2 |
| Documentation | 4% | Is sufficient TaT documentation (manuals/tutorials) available for self-study in German and/or English? | 2 | 2 | 1 | 1 |
| Video tutorials | 4% | Are there at least 10h video tutorials on the Tats website or on YouTube in German and/or English? | 2 | 2 | 2 | 1 |
| Human resources | 3% | Are there at least three staff members who can work with the evaluated TaT? | 2 | 0 | 0 | 0 |
| Record/Playback functionality | 6% | Does the TaT have a record/playback function? | 2 | 2 | 0 | 2 |
| Exception handling | 6% | Does exception handling exist? | 2 | 2 | 2 | 2 |
| Waiting for events | 6% | Is there a possibility to wait for an event (e.g. document loaded or component exists)? | 2 | 2 | 2 | 2 |
| Test-suite structure | 6% | Is there a possibility to build the test suite modularly? | 2 | 2 | 2 | 2 |
| Log file | 2% | Is it possible to create a configurable log file? | 2 | 1 | 1 | 1 |
| Reporting | 3% | Does the TaT also have a report generator with its own design options? | 2 | 0 | 0 | 0 |
| User interface (UI) | 6% | Does the TaT have a GUI? | 2 | 2 | 0 | 2 |
| Usability_01 | 2% | Is it possible to insert elements or lines with different functionalities in the test case? | 2 | 0 | 0 | 2 |
| Usability_02 | 2% | Is it possible to move lines in the test-suite? | 2 | 0 | 0 | 2 |
| Usability_03 | 2% | Is it possible to undo the deletion in the test suite? | 2 | 2 | 2 | 0 |
| Maintainability | 4% | Is there a log in tree view? | 2 | 0 | 0 | 0 |
| Version Control System (VCS) | 6% | Are the TaT project artifacts manageable with git? | 2 | 2 | 2 | 0 |
| Interface to Jira | 2% | Does the TaT have a Jira interface? | 2 | 2 | 0 | 0 |
| PDF check | 2% | Can PDF files be checked for content and design? | 2 | 0 | 0 | 0 |
| Developer license costs in the deployment period | 6% | Are the developer license costs in the deployment period below 1,500 EUR/year per developer? | 1 | 2 | 2 | 1 |
| Floating licenses | 2% | Is there an easy way (e.g. via license file) to unlock any workstations for the TaT? | 2 | 2 | 2 | 0 |
| Runtime license costs | 2% | Are the runtime license costs within the specified range? | 1 | 2 | 2 | 1 |
| Runtime environment license duration | 2% | Are there runtime licensing options per month? | 0 | 2 | 2 | 0 |
| Runtime environment | 3% | Is there any way to run the automated tests independent of the development environment? | 2 | 2 | 2 | 2 |
| Virtualized environment | 4% | Can the test cases run in Docker? | 2 | 2 | 2 | 0 |
| Result | 100% | 94% | 77% | 65% | 59% |
(page 34-39 and 50f)
The whole extract from the bachelor's thesis concerning QF-Test can be found here (German PDF only).
Bachelor's thesis: Conception of a methodology for the systematic selection of a test automation tool for a software development project using the example of real estate management, August 2021 - Erzsébet Nicole Harmat, Faculty of Computer Science and Mathematics, Hochschule München, Munich, Germany.
1. Functional cookies
We use functional cookies to ensure the basic functionality of the website.
2. Performance and statistic cookies
We use Matomo for analyzing and optimizing our website. Cookies permit an anonymous collection of information that help us offering you a clear and user-friendly visit of our web pages.
This cookie contains a unique, pseudonymized visitor ID internal to Matomo for recognizing returning visitors.
This cookie is used to track from which website the anonymized user proceeded to our website.
The Matomo session cookie is used to track the visitor's page requests during the session.
is created and should be then directly deleted (used to check whether the visitor’s browser supports cookies).
short lived cookies used to temporarily store data for the visit.
short lived cookies used to temporarily store data for the visit.
