26

Distributed test development

The previous chapters all focused on creating and organizing sets of reliable tests in a single test suite. However, when testing a large application, a single suite may not be enough. There are at least two scenarios where splitting tests into multiple test suites becomes essential:

• Multiple developers are creating and editing tests. To avoid redundancy and duplication of code, separately developed tests should use common Procedures and Components where possible, but only one person can edit a test suite at a time.
• Tests are simply getting too large. Run logs for extensive test runs may cause the system to run out of memory and organizing a large number of tests in a single suite is difficult. Things may become unwieldy. It may also be desirable to be able to run some of the tests as part of the whole test as well as standalone.

QF-Test provides a set of advanced features that make it possible to split and arrange tests across a set of test suites. Multiple developers can work on separate parts of a test, then coordinate their efforts, merge the Components of their suites and create libraries of shared Procedures.

This chapter first explains the various mechanisms for distributed test development and how they interact. The final section then summarizes these in concise step-by-step instructions on how to approach large testing efforts with QF-Test.

26.1

Referencing nodes in another test suite

It is possible to reference Procedures and Components in a test suite other than the current one. These references can be explicit or implicit through included files:

• Explicit references use a syntax similar to the one used in URLs to specify an item inside a web page. The referenced suite must be prepended to the Procedure name attribute of a Procedure call of the QF-Test component ID attribute of a Component dependent node, separated by a '#' character. The usual packagepath.procedure becomes suite#packagepath.procedure.
• Implicit references make use of the Include files attribute of the Test suite node. Whenever a node is not found in the current suite, QF-Test will search for a matching Procedure or Component within all the suite's directly or indirectly included files (a file is considered indirectly included by a suite if it is found as an included file within one of the suite's own included files; for example, if suite A includes B, and suite B includes C, then C is indirectly included by A).

A test suite that references a node in another test suite becomes dependent on that suite. If the Name of a Procedure or the QF-Test ID of a Component in the referenced suite changes, the suite with the reference must get updated, otherwise the link is broken and the suite will no longer work correctly. In such cases QF-Test will automatically update references if it knows about them. The best way to ensure that is to have both test suites in a common project because QF-Test automatically tracks all includes and all explicit references within a project. Alternatively you can list the calling suite in the Dependencies (reverse includes) attribute of Test suite root node of the referenced suite.

While implicit references are more convenient in most cases, they can make tests harder to understand because it is not immediately obvious where the Procedure or Component referenced by some node is actually located. One possibility to find out is to select "Locate procedure" (Ctrl⁠+⁠P) or "Locate component" (Ctrl⁠+⁠W) from the context menu. Additionally, QF-Test provides the menu items »Operations«-»Make references explicit« and »Operations«-»Make references implicit« which let you toggle quickly between the two modes without changing the actually referenced nodes.

In both cases, the referenced suite can either be given a relative or absolute filename. Relative filenames will be resolved relatively to the directory of current suite, or - if that fails - relatively to the directories on the library path (see option Directories holding test suite libraries). Always use the forward '/' as the directory separator, even under Windows. QF-Test will map it to the correct character for the system it runs on. This keeps your test suites independent from the operating system.

Note Your Package and Procedure names as well as Component QF-Test IDs should not contain any '\' or '#' characters. If they do, you need to include an escape character in the Procedure call or the QF-Test component ID attribute. See section 49.5 for details about escaping and quoting special characters.

When choosing the Procedure for a Procedure call or the Component for some event in the dialog, QF-Test offers a selection of all currently opened test suites. If a Procedure or Component from another test suite is selected, QF-Test automatically creates the correct reference, taking included suites into account. When the test is run at a later time, the referenced test suite is loaded automatically if necessary.

During execution QF-Test keeps a stack of currently executing suites. Whenever a Procedure is called in another suite, the called suite is pushed on to the top of this stack and removed when execution returns to the calling suite. Whenever during the execution of a Procedure a Window or Component is referenced by its QF-Test ID, QF-Test searches through this stack of suites from the top to the bottom, i.e. first in the test suite of the called Procedure and then in the calling suite, always checking any included files along the way. This process is quite complicated and you should take care to keep your include hierarchies simple. In case you encounter problems anyway, a detailed explanation is given in section 49.6.

26.2

Managing Components

As we have emphasized in chapter 5, the Components are the essential part of a test suite. If the SUT changes between releases, these will likely be affected most. If changes are so massive that QF-Test cannot adapt automatically, the Components will have to be updated manually. This is why you should try to avoid redundancy in the Component hierarchy of your tests more than in any other part.

Therefore, when splitting your tests across multiple test suites you should try to keep the Components together in one central test suite and include this suite from the other suites. For very large applications you may want to split the Component hierarchy into parts, each related to a separate part of the SUT's GUI.

Maintaining this central Component library is not trivial. The problems that will arise can be resolved with QF-Test as follows:

• When multiple test developers are recording new Components simultaneously, they cannot be integrated immediately into the central suite, because only one user can edit the central suite at a time. Instead, Components must be integrated later by importing them into the central suite when the new tests have stabilized. This is explained in the following section.
• When the SUT changes, Components in the central suite may need to be updated. If this involves changing any Component QF-Test IDs, this will break any references to these Components from other suites. To avoid that, QF-Test must update those references and it will do so, provided that the suites that depend on the central suite are currently loaded, belong to the same project or are listed in the Dependencies (reverse includes) attribute of the Test suite node of the central suite.

26.3

Merging test suites

test suites can be merged by importing one test suite into another with the »File«-»Import...« menu item.

You can select the areas of the test suite, which should be imported.

You have to take care about a correct Include/Reverse-Include of your test suites to ensure, that all calls and component references are still valid. See chapter 37 for details.

26.3.1

Importing Components

During import, all Windows and Components of the imported test suite are integrated into the component hierarchy of the importing suite. Components that already exist are not copied. A QF-Test ID conflict (identical components with different QF-Test IDs or differing components with identical QF-Test IDs) is resolved automatically by changing the QF-Test ID of the imported component.

Afterwards, all Windows and Components are removed from the imported suite. All nodes in the imported suite that referred to these Components are updated accordingly. Ideally, the imported suite should include the importing suite so no explicit suite references will have to be created.

3.3+26.3.2

Importing Procedures and Testcases

As you can import Components QF-Test also allows to import Procedures, Packages, Dependencies and Test cases as well as Test sets by choosing 'Procedures' or 'Tests' in the import dialog. You should take care about keeping all calls consistent, e.g. in most cases it does not make sense to import Procedures without their required Components.

In case you only want to import one dedicated Procedure or Test case you can use the button 'Detailimport' on the importdialog. Here you can choose any node you want to import separately.

26.4

Strategies for distributed development

There is no single best way of test development or organization, but one approach that works well is the following:

• Start with a central test suite that has the functionality needed to start and stop the SUT and a basic set of Tests and Procedures. This will become your master suite which will contain all Components.
• Make sure that your developers have understood the importance of assigning names with setName() and that unique names are assigned consistently where needed. Where setName() is not an option, try to implement ComponentNameResolvers to achieve this (see subsection 54.1.7). You should be able to record and replay sequences without much ado and without "polluting" the Component hierarchy after trivial changes in the user interface.
• Move as much functionality as possible into Procedures, especially commonly-used stuff and the setup and cleanup routines for the SUT.
• To create new tests, start with an empty test suite. Include the master test suite by editing the Include files attribute of the Test suite node of the new suite. Create the Setup and Cleanup nodes to start and stop the SUT by calling the respective Procedures in the master suite.
• Create your tests as required. When recording sequences, the Components of the master-suite will be used if possible. New Components are added to the new suite, so the master suite will not be modified at this stage.
• Where possible, call Procedures in the master suite for common operations.
• When your new set of tests is complete and you are satisfied that they work well, import any required nodes of your new test suite into the master suite. You have to ensure that all new Component nodes that you recorded are imported into the master suite's Component hierarchy in any case. The master suite's existing Components will not be affected by this, so other suites that depend on the master suite will not need to be modified.
• After importing Components you can import all or only the required Procedures into the master suite.
• You now have various options of how to arrange the actual sequences of events and checks that form your tests. In any case it is a good idea to move everything to Procedures and Packages structured after your test-plan. Then the top-level Test set or Test case nodes of the master suite and your new suite will only contain the required hierarchy of Test set, Test case, Test step and Sequence nodes filled with Procedure calls to the actual test cases. Such an arrangement has several advantages:
  • All your tests are structured cleanly.
  • You can easily create different sets of tests with varying complexity and run-time.
  • You have the option to keep the test cases in separate test suites and have the master suite call them. These "test case-libraries" must include the master-suite, so they need not contain any Components themselves. You can organize your tests so that the master-suite will run the whole set of tests, while each separate suite can also be run standalone.
  • The tests can be maintained by several developers as long as modifications to the master suite are coordinated.
• If you decide to keep your new tests in the newly created test suite instead of moving them to the master suite, modify the master suite to tell QF-Test that there is a new test suite that depends on it. To do so, either ensure that both test suites belong to the same project or add the new test suite to the Dependencies attribute of the master suite's Test suite node.
• If you need to modify or extend the new test suite later, proceed as before. You can record new sequences as needed. When you are done, merge any newly created Components back into the master suite.
• If your SUT changes in a way that requires updates or adaptions to the master-suite's Component hierarchy, you must coordinate your test developers. Before you start updating the Components, make sure that all suites that directly or indirectly include the master suite belong to the same project as the master suite or are listed in the Dependencies attribute of the master suite's Test suite node. If modifying the Components of the master suite involves any QF-Test component ID changes, QF-Test will update the depending test suites accordingly, so they should not be edited simultaneously by others.
• The file format for QF-Test test suites is XML and thus plain text. As a result, test suites can be managed very well by version control systems like CVS. Changes to some QF-Test component ID attributes of the depending suites can typically be merged with other changes without conflicts, alleviating the need for coordination.

Of course, the above scheme can be extended to have several master suites for testing different parts or aspects of an application. It may be a good idea to ensure that the component hierarchies in these suites don't overlap too much though. This will save you the effort of maintaining all these hierarchies in case the user interface of the SUT changes significantly.

3.1+26.5

Static validation of test suites

Working in a project over time will cause modifications, refactoring or deletion of steps in your test suite structure, e.g. you may consider renaming Procedures or simply removing them once they are not required anymore.

26.5.1

Avoiding invalid references

In such cases it is quite important that you adapt all references of the according Procedure in order to guarantee that the remaining tests keep running. For this purpose QF-Test automatically updates all references during the process of renaming or moving elements on demand.

If you want to ensure that your created test structure doesn't contain any call of non-existing Procedures anymore, you can also use the "Analyze references" command of QF-Test in order to perform a static validation of your test suite. This command will open a dialog showing all references and whether they are still okay or something is missing.

You can trigger the analysis via a right mouse-click and selecting »Additional node operations«-»Analyze references...« or selecting the according entry from the main menu under »Operations«. This method is also available in batch mode.

3.5+ QF-Test also provides features to search through your test suites for duplicate nodes, empty Packages or Procedures or to analyze for nodes having invalid characters in their names.

This kind of static validation is available for Procedures, Dependencies, Test cases, Test sets and Components and their references.

4.0.3+26.5.2

Unused procedures

During test development it could happen that procedures, which were used in the first version of your tests will not be used in newer versions due to re-factoring of tests. If those procedures won't get deleted immediately they will stay in the test suite and the test suite will grow and grow. Sometimes you could get the feeling that you have too many procedures or that you have lost the overview of your procedures. In order to check for such unused procedures or dependencies in your test suite you can open the context menu via a right mouse click at Test suite or Procedures and select »Additional node operations«-»Find unused callables...«. This operation creates a report showing any procedures or dependencies which had been created but haven't been used yet. Now you could decide what you want to do with those.

Sometimes you might simply remove all of those unused nodes immediately via »Additional node operations«-»Remove unused callables«.