35

How to start a testing project

This chapter talks about the most important aspects that should be considered before you start to use QF-Test widely in your testing project. It mostly raises questions and gives general answers with references to more detailed information.

The aim of this chapter is to provide hints about issues which you should take care of in order to make your GUI tests reliable, stable, repeatable and especially maintainable.

35.1

Infrastructure and testing environment

Before you start creating and running automated tests you should think about some general matters pertaining to the environment where the tests have to run. In order to make tests reliable and repeatable you have to take into account that you must be able to bring your SUT into a well-defined state, which includes the state of its backend, e.g. a server and/or a database. If you do not think about such aspects it might become very difficult and sometimes quite tricky to rerun a test or simply to analyze test results and maintenance of tests can become a nightmare.

Please consider the following topics:

1. What is the initial state of your SUT?
   • Which user is the actual user running the tests in your SUT? Most projects work with dedicated test users for running tests. Another approach could be to have one test user per test engineer.
   • Which language setting of your SUT is the primary one? Is it really required to reach a full coverage of all supported languages or is it sufficient to run the bulk of the tests in one primary language and create only a few tests to specifically test localization? In most cases repeating tests in several languages just covers the same functionality, so you gain no real new information after running them. However, unless you take precautions, the language setting will influence the component recognition of QF-Test, please see section 5.3 for details.
2. What is the initial state of your database?
   • Can you work with an extra test database or do you have to use a production database? Test databases contain test data with designed and planned content whereas production databases contain real-life data. Is the latter predictable and reliable? What about the danger that tests can mess with and possibly destroy production data? If at all possible you should avoid running automated tests in a production environment.
   • Can you clean up or reset the environment after one test run for rerunning the test? Is it possible to undo changes in the database or is it required to use new test data for the next regression phase?
   • How can you read or write test data? Do you want to use standard SQL scripts or can you reuse libraries from development? Some projects even reinstall the whole database before every test run because they cannot reuse any test data or clean the database correctly.
3. Do you want to integrate QF-Test with other tools, e.g. build tools or test management tools?
   • How to integrate QF-Test with a test management tool? If you can reuse already planned test steps you can avoid redundant work in planning tests and creating them. For the standard integration for such tools, please see chapter 27.
   • Should test's be launched by a build-tool? If you have created tests you can run them unattended and trigger the run by a build-system like Ant or CruiseControl. Please see chapter 24 for details about test execution.
   • Should test results be uploaded to a reporting system or into a test management system or is it more sufficient to put the HTML reports and run logs on a centralized HTTP-server?
4. Who will work with QF-Test?
   • Do only one or two engineers work with QF-Test or do all developers and business testers participate in test development? You can find some hints about working in a team with different roles in section 36.5.
   • What are the skills of the engineers? It is recommended to have at least one dedicated person with a good QF-Test knowledge in the team, who is also capable of implementing scripts and understanding software development principles.

Of course there will be more issues to take care about which are specific for your project. Try to figure them out.

35.2

Location of files

You should also think about following aspects of saving or installing files:

1. Where to install QF-Test to? QF-Test can be installed locally on every system but this forces you to update every system manually whenever you need to upgrade to a new version. You can also install QF-Test on a shared network drive, if your network is reliable, see subsection 35.2.1 for details.
2. Where to store the configuration file qftest.cfg? Among other things that file contains information about how QF-Test should recognize components or what should go into the run log. These options have to be the same for every QF-Test user, otherwise you cannot share tests in your team. To ensure that you can either use a shared network installation for QF-Test or specify the config file via command-line parameters when launching QF-Test. Make sure the shared config file is write-protected unless you explicitly want to change it. For details, see section 1.6.
3. Where to store the license file license? You should put the license file to a central place in order to update the license only once when you receive an update for it. Again, you can either have a shared network installation for QF-Test or can use command-line parameters to specify the location of that file when launching QF-Test. For details, see section 1.5.
4. Where to store the test suites? The best place to store test suites is a version management system where you can track the changes and access any version of the files. If this is not possible you should store them on a shared network drive.
5. Where to store the test data files? Test data files are associated with test suites so you should store them closely to the suites, i.e. either in the same version management system or on a shared network drive.
6. Where to store the HTML reports and run logs? You should put those files in a centralized place where any engineer can take a look at them to evaluate the test results. Most people tend to use an HTTP server or a shared network drive for that.

35.2.1

Network installation

If you plan to install QF-Test on a shared network drive you have to take care about some specific things.

The main source of conflict is the system settings file qftest.cfg. It is actually a good (and necessary) thing to have all users use the same system settings, especially for the recognition options. Sharing the system settings file facilitates this. However, this file should be made read-only so that one user will not inadvertently change the system settings for everyone. If the file is read-only, QF-Test will not save the system settings upon exit. Any change to these settings will have to be made by explicitly making that file writable, then exiting QF-Test and then making it read-only again. Alternatively each user could specify a different system settings file via -systemcfg <file> but that's not advisable.

The running QF-Test instances will also share the log directory (internal logging, not a problem) and the Jython package cache which can occasionally cause problems so that QF-Test cannot initialize its Jython interpreter. This doesn't happen often and can be fixed by clearing (not removing) the Jython cachedir.

For Windows, each user should also execute the setup.exe for the primary QF-Test version, located in the installed qftest-x.y.z directory, to get proper registry settings and documentation links on his machine.

In the rare case when a QF-Test patch overwrites existing jar files of QF-Test, running instances based on those jars may crash on Windows.

35.3

Component Recognition

The most important aspect of a GUI testing tool is a stable and reliable recognition of the graphical components. In that area QF-Test is very flexible and can be configured in several ways. In most cases the default configuration for the component recognition works well, but sometimes you may have to change it.

If you change the component recognition options after creating lots of test cases, those test cases may break. Therefore you should try to find the most appropriate settings for your project as early as possible. It is worth spending time in that area before starting to implement a huge amount of tests because in the worst case you might have to re-record or at least update all or most of the existing test cases after a critical change of the recognition options.

Best start by recording some demo test cases and figure out how QF-Test recognizes your SUT's components. The recognition mechanism is described in chapter 5 and section 5.9. If you rerun those demo test cases - ideally on different versions of your SUT - and run into recognition problems, you have to ask yourself following questions about those tests:

1. Are there enough synchronization points, like 'Wait for component to appear' or 'Check' nodes with timeouts to execute test steps only if the SUT is ready for them?
   1. Sample 1: After opening a window you can only work in that window, if it is really there -> Use a 'Wait for component to appear' node.
   2. Sample 2: After pressing on a search button, you can only continue with the test when the search is really over -> Use a 'Check' node with a timeout.

Another important aspect besides synchronization points is the correct approach of recognizing components. You have to ask yourself the following questions to determine, which recognition approach might be the most appropriate one:

1. Do the developers use unique and stable names for their components? Please take a closer look at section 41.13.
2. Perhaps it is sufficient to use a regular expression for the 'Feature' attribute of the component of the main window under the 'Windows and components' node. Please see subsection 5.4.3.1 for details.
3. If development did not set useful or even dynamic names it may be required to implement a NameResolver. Please take a closer look at section 5.3.
4. Do any of the QF-Test recognition options need to be changed? These are described in section 5.3.
5. Is it possible to use generic components? See section 5.8 for details.

In some cases it is sufficient to change the default configuration. Let us assume the developers have set unique and stable names for the target components, i.e. buttons, textfields, checkboxes etc. In such cases it may be sufficient to just change the 'Name override mode' setting of QF-Test to 'Name overrides everything'. This setting tells QF-Test to ignore any changes in the component hierarchy and just work with the target components and the window directly.

Note You have to change this option in two places: Once at 'Record' -> 'Components' -> 'Name override mode' and at 'Replay' -> 'Recognition' -> 'Name override mode'. See section 5.3 for more details.

Last update: 9/10/2024 Copyright © 1999-2024 Quality First Software GmbH