Course: Verification and Validation of Software

It was easy to learn how to start using the tool by making simple tests.

The tool was really useful for Model-based testing, but there were some problems along the way that were mainly due to bugs. The application we were testing had some features that involved the icon tray menu. Recorded actions that involved interaction with the tray icon were not reliable since they would only work sometimes.

The main pain-point of the using QF-Test is the difficulty in sharing the test with a teammate. The jar file location is hard-coded as an absolute path and we haven't found a way to make this dynamic between systems/computers. As such, when coding in a different machine, we had to reconfigure these paths. Overall, the experience was positive with the exception of the application's jar file path problem.

The QF-Test tool appeared to be promising when automating tests of user interfaces. One problem that is not directly related to the test creation, but we ran into, was that with high DPI monitors, some workarounds were needed to run the application decently.

We felt that the QF-Test tool is pretty good and the installation process was straightforward. We didn't manage to add the tool to PATH with the setup tool, so we resorted to finding the executable in the project's files and executing it using the terminal. We feel that this part of the user experience could be improved upon.

The tool felt very intuitive to use. However, we could not figure out how to check if the search bar was invisible since the variable visible still appeared as true even when there was no bar on the screen.

We can easily test transitions and states, and every single detail about what we do during recording (clicks, checks, inputs, etc.). Our group didn't use all the features of the tool, but the ones we used looked perfect for what we needed to do.
We would like to share a suggestion about the tool: we believe it’s a symbiotic improvement if your QF-Test tool can be integrated in an IDE. We`re not saying the tool is hard to understand and work (it's super intuitive by instance), but we think that might facilitate and incentive its use for many other people that might not even know the existence of the tool.

After using QF-Test to create test suites for the project, we consider it very well-prepared and useful. Despite some initial confusion regarding the setup, by using the Quickstart Wizard, we managed to use the tool to its full potential in the scope of our project. The only issue we encountered was that we had to change the path of the executable file every time we switched between different devices, due to using the complete path, because we were unaware how to use a relative path.

The initial setup and integration with our project were easy and the wizard worked as intended. We could not find a way to use a relative path for the jar path. This meant that on the different PCs, there was a need to run the wizard every time someone changed the tests. After that, step recording was intuitive, and we did not have any issues. We found that dragging-and-dropping test steps on the left menu worked as intended as well as copying, cutting and pasting them.

Overall, QF-Test proved to be very useful when creating test cases that require interaction with the UI. However, one problem that we found is that the configuration file used makes it harder to collaborate with other people when using the tool. The configuration file contains information that is specific to each developer's machine. We edited this file in an attempt to use it in a different machine. Unfortunately, QF-Test_ signs the configuration file and checks its integrity upon loading, making it impossible to use a file that's been edited. Perhaps, if the test suite data was stored in a separate file, it could be used with VCS, while the file with user specific information is ignored.

In general, the tool is really intuitive, however:

• It was hard to understand if a specific test passed. This information is only available near a small "plus" icon on the bottom right corner of the tool. It would be better to also have a message similar to the JUnit tool, that indicates if the test passed or failed in a clear way. This might not make difference to old users, but it would improve the experience for new users like us.
• The design is another point. It wouldn't make a difference to old customers that are already used to the tool but improving the design would make more attractive.
• In our case, it was also difficult to create some of the checks. Particularly, the App that we were testing mainly consists of a table, and it was hard to select a particular cell of that table without accidentally selecting other parts of the table.
• If we include multiple sequences within the same test-case, it is difficult to understand in which of them the test failed.
• The checks are based on the text content of the inputs and text elements, but sometimes this is not enough. For instance, in our project, when a row is selected, its colour changes. It would be useful to be able to capture this modification, as well as other visual modifications of the UI that are not text related.

Our experience with the QF-Tool was generally positive. We will now highlight some features that we think would be excellent additions to the tool, as they would make the user's life easier in some cases that we came across throughout this work:

• When using the tool to test a webpage, in the Open Browser Window action there should be an option to open the window in full screen or fully expanded, since by default it does not fill the entire screen, which can lead to disappearance of some buttons in the interface due to the responsiveness of the site to be tested.
• To solve this problem, it was necessary to find out the width and height of the screen and change these fields, but it would be useful for the program to do this automatically when we select a checkbox to open the already expanded window.
• When selecting a table, there should be a button to count its number of rows, without having to specify its content.
• It should be possible to test the bar on top of application windows, with the name and icon of the window.
• When recording double-clicks, the click coordinates should be saved, as it may have a different effect depending on the location of the click. For example, if we edit text input, the click location would indicate the cursor positioning in that input.

Test portability is an issue. Since QF-Test Setup Wizard generates a dependency on the local filesystem, we were forced to manipulate this dependency every time we committed information towards Github since another team member was running in a different filesystem. 

While the tool seems interesting for developing UI testing, it also comes with some difficulties for developers, such as:

• Outdated interface, which leads to confusing methodologies on implementation of methods, for example, lack of checking options and inability to navigate when in checking mode (i.e. opening dropdown menus).
• While there's best practices for organisation of test-suites, it provides little to no version control, making the development in pairs or in team not friendly, and also requires manual management of paths.
• Poor documentation / tutorial on how dependencies work, their order of execution and how to setup multi-step dependency trees, each with their own setup and cleanup processes.

We have worked with similar tools in the past (like cypress) and there are a few differences to note. To start, cypress has a cleaner/more modern GUI, and I felt like it has a better user experience. The learning curve for the QF-Test seems much steeper than other tools, but it looks super powerful. The manual / tutorials seem very helpful and complete, which is something that is necessary when there isn't much community online to get support from.

One feature that we noticed that it was missing is a toggle button for quickly disabling/enabling test case steps. Sometimes when developing a test, we implement verifications that aren't finished, as a placeholder, and it's a bit annoying to get errors for those each time we do a test run.

Otherwise seems a slick tool. A must-have for desktop java applications!

The QF-Test tool allowed us to fulfil the basic usage of writing simple GUI tests for a Java application. In a couple of hours with the help of QF-Test tool beginner tutorials we were able to understand how to use it within our basic application. However, we found the controls not so intuitive and the UI outdated and confusing. A better UI and tooltips would greatly improve beginner level user experience.

By watching the tutorials we learned that this tool enables much more powerful testing with integration in CI/CD pipelines, multiplatform testing (not specific to Java applications), video recording, etc. We didn't test these features (Out of the scope of the assignment). Also, the platform from our experience appears to really stable and ready for professional/production environments. We tested on Windows and MacOS without any issue. Really curious to see how the QF-Test runner would perform on docker/Linux machines for the CI/CD integration (not tested).

For this project the QF-Tool for Windows platform, desktop version was used. The installation process was very simple and very intuitive, the use is also simple, with a graphical interface where the elements are well distributed, which makes the learning curve to work with the tool very flat. In terms of tests it covered all the scenarios we identified during the project. One negative aspect of the tool is that it is necessary to invest time to explore and read the documentation to be fully acquainted with the tool, because the community is really small, so any problems could take a lot of time to resolve.