For its customer, DVZ GmbH has compared the test automation tools QF-Test and Tosca and concluded that QF-Test is the better automation tool for desktop applications, both for individual and multiple projects. The customer is advised to use QF-Test.

About DVZ, the IT Service Provider for Public Administration

The IT service provider for the state administration of Mecklenburg-Vorpommern is a long-standing and strong partner of the public sector. With the expertise of approximately 700 employees and state-of-the-art technologies, they shape the future of digital administration together with their customers. Below we reproduce the detailed comparison report by Martin König with slight editorial changes. You can find the original report in the German PDF version.

Why Did DVZ Compare QF-Test and Tosca?

DVZ compared the two automation tools to provide its customer with a recommendation on which tool is better suited for their automated desktop application tests. The evaluation was based on a Java Swing application with a browser component for login.

Initial Situation

The comparison focused on usability, required training time, and various deployment possibilities. Both solutions are similar in terms of application but follow different approaches.

  • Different applications were used for various tests, with a focus on desktop applications.

  • Mobile applications for Android and iOS were not considered, as there were no internal use cases.

  • Maintainability and expansion possibilities were also evaluated.

  • The comparison was conducted on a laptop with an i7 CPU.

  • Tests within virtual machines (VMs) were also performed, but they had minimal impact on the evaluation.

Prerequisites and Installation

Installing QF-Test is significantly easier and more intuitive compared to Tosca.

  • For Tosca, infrastructure must be set up in advance for the on-premise solution. A server is required to install the "Tosca Server," which manages agent distribution and repository uploads. Additional components include a database for repositories and a license server.
  • Installing Tosca Commander, the actual automation application, is straightforward, but integrating it with various servers and parts of the Tosca infrastructure requires manual configuration.
  • Tosca has higher hardware requirements due to its infrastructure.

Both tools benefit from basic knowledge of test automation and software development.

  • According to Tricentis, this is not necessary for Tosca, but internal experience has shown that without prior knowledge, the process and procedure can be misunderstood. To explain this: 3-4 people without a programming background were introduced to the tool over the deployment period of Tosca. Unfortunately, none of the people involved were able to use Tosca completely and they refused to continue using it because Tosca is too complex.

Use Cases

Both tools support various technologies, facilitating their use in different projects. The primary project for comparison was a Java SWING application with a browser component for login.

Both tools were able to automate and implement the test cases.

  • It should also be mentioned that Tosca cannot be integrated into a pipeline, at least not without major effort. The infrastructure of the tool is too complex and the hardware requirements of the client are too high to enable quick and easy integration. The automated execution takes place via a call to the server, which in turn instructs the agent to start the execution. Although this works, it is a duplication within the pipeline and not an agile solution in the modern Dev-Ops environment, as activations etc. are required.

First Launch, Usability, and Overview

  • The first start with Tosca is very extensive, as many parameters have to be set, such as creating the repository and connecting it to the servers. 

With QF-Test, on the other hand, you can start the tutorial directly in the tool and view the demo suites without much effort for further settings.

  • The interface of Tosca Commander is relatively confusing and packed with several functions via an old ribbon, which makes the application itself appear very inaccessible. It should of course be noted that Tosca also provides other parts for software testing in Tosca Commander. For example, requirements can be created directly in the tool or more complex data tables can be realized via TestDesign.

The tutorials are also very different.

  • The tutorial series from Tricentis cost around 20 to 40 hours in total. These are necessary in order to understand how the tool works. Unfortunately, the tutorials are only prepared in English and sometimes contain errors, for example in the punctuation. For the Tosca tutorial, an account must also be set up on the support site and the manual unfortunately does not provide a good basis for getting started with the tool.

The QF-Test tutorial takes considerably less time and no additional registration is required. In addition, the manual is very well prepared and even clearer in the newer version. You can also start with QF-Test directly by “Learning-By-Doing” and is supported by “Step-By-Step” instructions directly in the tool.

Maintainability

  •     Maintenance of the components in Tosca is very time-consuming, as they have to be completely re-recorded if they are not recognized. The hierarchy of recognition in Tosca is: IDs, properties (combination), path, pixels (positions X/Y). The recognition is about 80% correct, but requires a lot of manual effort. 

In QF-Test, the elements can be distributed over several .qft files, which simplifies recognition and finding. Depending on the application, the manual effort can of course also increase here. However, component recognition is significantly higher than with Tosca.

For larger test scenarios, such as process chains, the complexity can increase enormously.

  • With Tosca in particular, the test cases are then very difficult to read and understand, as the components cannot be accessed directly, but are usually nested in a “shell”. The automatic designation of the “component shell” usually has little relation to the specialist application and must be described manually.
  • Tosca also offers an AI component recognition called “Ara”. This was tried out with a test account, but provided little to no added value. Furthermore, an additional login is required here and the “Ara” can then only be used by certain people. Extra costs are incurred for use.

The comparison

Tosca TestsuiteQF-Test

Technology support (test options)

  • API
  • Web
  • Windows
  • Mobile
  • Datenbank

Infrastructure

  • Hardware requirements relatively high for local execution
  • Hardware requirements for server high
  • Tosca Server / License server (order distribution, license distribution, repository provision)
  • Datenbase server
  • Agents for test run

 

  • Hardware requirements moderate
  • Lizenz as a file

CI/CD Integration

  • Challenging integration due to required port approvals for server/agents

  • Batch mode available

  • Execution only possible on Tosca Agents (Tosca Testsuite must be installed)

  • Local execution via TC-Shell (included in Tosca) possible but relatively complex

  • Local execution via Task Scheduler (Windows) also possible

  • Feasible via Jenkins

  • Also possible with Azure

  • Local execution via Task Scheduler (Windows) also supported

Extensibility / Plug-Ins

  • Not extendable

  • Compatible with other Tricentis tools, such as NeoLoad or qTest

  • Extendable

Requirements

  • Basic knowledge of software testing (terminology)

  • Basic Tosca Testsuite course, Path 1 + 2 (Support Hub Tricentis)

  • Estimated effort: approx. 20–40 hours

  • Tutorial course available (developer documentation)
  • Estimated effort: approx. 6–8 hours

Installation

  • Windows only

  • Windows
  • macOS
  • Linux

Test Script Creation

  • GUI scanning fully supported

  • Drag & drop test case creation is simple but unnecessarily complex and unclear

  • No "real" scripting possible

Repositories

  • Test scripts stored in a database

  • Versioning possible, but rollback is error-prone

  • Git, local storage
  • Versioning available with Git

User-Friendliness

  • Medium user-friendliness: unclear interface, relatively easy operation after extended training

  • High user-friendliness: intuitive interface, clear layout

Training / Research

  • Steep learning curve

  • Good (English) workshops directly from Tricentis, but not adapted for the German market (e.g., rounding, time formats, etc.)

  • No additional (free) resources available

  • Good learning curve

  • Learning by experimentation possible

  • User manual available in German and English

Support

  • Long response times

  • Only an English community hub with limited usefulness

  • Relatively high effort required for error documentation

  • Within 12 hours

Maintenance

  • Time-consuming, as scans must be repeated

  • Drag & drop further complicates maintenance

  • Maintenance is sometimes impossible for beginners

  • References or module links must sometimes be manually adjusted

  • Tosca Server and Agents must be reviewed and maintained weekly if necessary

  • Currently difficult to estimate and highly project-dependent (can be simplified with modular libraries)

 

Miscellaneous

  • Version updates can render repositories unusable if technologies are deprecated

Conclusion by DVZ: QF-Test is the Better Choice for Test Automation

QF-Test provides a better overall experience in handling, test script creation, installation, and maintenance than Tricentis Tosca. The added flexibility through scripting in JavaScript or Jython makes QF-Test significantly more adaptable compared to Tosca’s rigid drag-and-drop system.

  • Although Tosca covers a broader range with additional tools like qTest or NeoLoad, these are costly and offer minimal added value for agile projects. Moreover, Tosca's total cost is approximately 5–6 times higher than QF-Test, even when purchasing five QF-Test technology licenses.

Component recognition in QF-Test is about 20-30% more accurate and easier to understand, which significantly improves maintainability.

! Overall, QF-Test is the superior automation tool for desktop applications, both for single and multiple projects. DVZ recommends QF-Test for automation.

Comparison QF-Test vs. Tosca in German

The advantages

 Tricentis ToscaQF-Test
Installation35
Infrastructure25
Test Script Creation34
Komponent Recognition35
Tutorials / Documentation25
Learning Curve44
Training Time24
Maintainability24
Extendibility52
Support15
Total Score27/5043/50

Very good = 5 points, good = 4, middle= 3, sufficient = 2, unsatisfactory = 1

Martin König, test automation engineer, DVZ Datenverarbeitungszentrum Mecklenburg-Vorpommern GmbH, Germany

 

Interested in QF-Test?

Tell us about yourself and we'll connect you with a QF-Test expert who can share more about our product.

Contact our service teamProduct booklet