For its customer, DVZ GmbH has compared the test automation tools QF-Test and Tosca and concluded that QF-Test is the better automation tool for desktop applications, both for individual and multiple projects. The customer is advised to use QF-Test.
The IT service provider for the state administration of Mecklenburg-Vorpommern is a long-standing and strong partner of the public sector. With the expertise of approximately 700 employees and state-of-the-art technologies, they shape the future of digital administration together with their customers. Below we reproduce the detailed comparison report by Martin König with slight editorial changes. You can find the original report in the German PDF version.
DVZ compared the two automation tools to provide its customer with a recommendation on which tool is better suited for their automated desktop application tests. The evaluation was based on a Java Swing application with a browser component for login.
The comparison focused on usability, required training time, and various deployment possibilities. Both solutions are similar in terms of application but follow different approaches.
Different applications were used for various tests, with a focus on desktop applications.
Mobile applications for Android and iOS were not considered, as there were no internal use cases.
Maintainability and expansion possibilities were also evaluated.
The comparison was conducted on a laptop with an i7 CPU.
Tests within virtual machines (VMs) were also performed, but they had minimal impact on the evaluation.
Installing QF-Test is significantly easier and more intuitive compared to Tosca.
Both tools benefit from basic knowledge of test automation and software development.
Both tools support various technologies, facilitating their use in different projects. The primary project for comparison was a Java SWING application with a browser component for login.
Both tools were able to automate and implement the test cases.
With QF-Test, on the other hand, you can start the tutorial directly in the tool and view the demo suites without much effort for further settings.
The tutorials are also very different.
The QF-Test tutorial takes considerably less time and no additional registration is required. In addition, the manual is very well prepared and even clearer in the newer version. You can also start with QF-Test directly by “Learning-By-Doing” and is supported by “Step-By-Step” instructions directly in the tool.
Maintenance of the components in Tosca is very time-consuming, as they have to be completely re-recorded if they are not recognized. The hierarchy of recognition in Tosca is: IDs, properties (combination), path, pixels (positions X/Y). The recognition is about 80% correct, but requires a lot of manual effort.
In QF-Test, the elements can be distributed over several .qft files, which simplifies recognition and finding. Depending on the application, the manual effort can of course also increase here. However, component recognition is significantly higher than with Tosca.
For larger test scenarios, such as process chains, the complexity can increase enormously.
Tosca Testsuite | QF-Test |
---|---|
Technology support (test options)
| |
Infrastructure
|
|
CI/CD Integration
|
|
Extensibility / Plug-Ins
|
|
Requirements
|
|
Installation
|
|
Test Script Creation
|
|
Repositories
|
|
User-Friendliness
|
|
Training / Research
|
|
Support
|
|
Maintenance
|
|
Miscellaneous
|
|
QF-Test provides a better overall experience in handling, test script creation, installation, and maintenance than Tricentis Tosca. The added flexibility through scripting in JavaScript or Jython makes QF-Test significantly more adaptable compared to Tosca’s rigid drag-and-drop system.
Component recognition in QF-Test is about 20-30% more accurate and easier to understand, which significantly improves maintainability.
! Overall, QF-Test is the superior automation tool for desktop applications, both for single and multiple projects. DVZ recommends QF-Test for automation.
Tricentis Tosca | QF-Test | |
---|---|---|
Installation | 3 | 5 |
Infrastructure | 2 | 5 |
Test Script Creation | 3 | 4 |
Komponent Recognition | 3 | 5 |
Tutorials / Documentation | 2 | 5 |
Learning Curve | 4 | 4 |
Training Time | 2 | 4 |
Maintainability | 2 | 4 |
Extendibility | 5 | 2 |
Support | 1 | 5 |
Total Score | 27/50 | 43/50 |
Very good = 5 points, good = 4, middle= 3, sufficient = 2, unsatisfactory = 1
Martin König, test automation engineer, DVZ Datenverarbeitungszentrum Mecklenburg-Vorpommern GmbH, Germany