29

Dependencies: Automatically ensuring correct prerequisites for each Test case

Video: Dependencies

This chapter explains the dependency concept of QF-Test. That concept is important for creating robust test cases and for defining recovery activities, if a single test case crashes during the test run. It has been introduced to guarantee that any test case's prerequisites are fulfilled before running it.

You can find the following examples in the file qftest-9.0.0/doc/tutorial/ advanced-demos/en/dependencies.qft. There is also a second test suite qftest-9.0.0/doc/tutorial/ advanced-demos/en/dependencies_work.qft to perform your own implementations. Please take care to copy all test suites to a project-related folder first and modify them there.

29.1

General

Please copy the test suite qftest-9.0.0/doc/tutorial/ advanced-demos/en/dependencies_work.qft to a project-related folder first and open it there. Take a look at the first Test set 'Discount Tests'. It contains three different Test case nodes and the respective Setup and Cleanup to launch and stop the SUT before each Test case. That is a typical test suite you might have created yourself.

Let us assume you just want to launch one specific Test case, because that one failed or it verifies a specific defect. If you want to do so, you have either to run the whole Test set, because you have to ensure that the prerequisites are fulfilled, or you run the Setup manually and run the according Test case then.

As this is a situation which is very common but not easily solvable with the concepts we know already, QF-Test introduces the Dependencies concept. It is responsible for managing prerequisites of Test cases and allows you to run one Test case directly. In that case QF-Test will take care of the prerequisites of the Test case, e.g. launching the SUT, selecting a vehicle and so on.

A Dependency can contain a Setup, a Cleanup, an Error handler and a Catch node. The Setup of a Dependency will be executed before each Test case. This is because the correct pre-conditions of a Test case are a very important aspect of a robust test run. You could imagine a situation where one Test case closes the SUT because of a mistake. Then the next test case has to launch the SUT again. Exactly for this situation, the Dependency concept provides a very stable and fancy solution.

The second aspect of Dependencies is that they optimize the execution of tests. With the current means we have to start the SUT before each Test case and stop the SUT after each. That is no problem for a small application like the CarConfigurator, but for huge applications, like Eclipse/RCP application or any ERP system, this might become insufficient. That is why Dependencies call the Cleanup steps only if required.

Another advantage of the Dependencies concept are global Error handler and Catch for recovery management. This feature becomes quite important, if you run lots of test cases in a series and one is failing and preventing the other test cases from continuing execution because of modal error dialogs, e.g. 'OutOfMemoryException'.

In a nutshell Dependencies are

1. a place to define the prerequisites of a Test case.
2. very helpful to make test cases more independent from other ones.
3. a better approach of implementing Setup and Cleanup steps.
4. allows you to implement recovery steps in case of errors or exceptions.
5. optimizing test execution.
6. re-usable as they can be defined in the Procedures area.

The following sections demonstrate how to use Dependencies.

29.2

Ensuring prerequisites

Please copy the test suite qftest-9.0.0/doc/tutorial/ advanced-demos/en/dependencies_work.qft to a project-related folder first and open it there if you haven't done that so far.

This file contains a Test set 'Discount Tests' with three Test cases and the conventional implementation using the Setup and Cleanup. We will change this Test set to make use of a Dependency now.

First we have to insert a Dependency node. This can be achieved by a right click at the Test set and then selecting »Insert node«-»Dependencies«-»Dependency«. Specify a name for that Dependency, e.g. "SUT started".

The next step is to move the Setup and Cleanup nodes into that Dependency. Therefore open the Dependency node and move the mentioned nodes into it. You can do this via drag and drop or via right mouse click »Cut« and »Paste« or Ctrl⁠+⁠X and Ctrl⁠+⁠V.

The test suite should now look like this:

Let us test the Dependency now: Please stop all running clients before that, then select one Test case, e.g. 'Discount 10', and run it.

You should see that the Test case 'Discount 10' has been executed and the SUT has not been stopped at the end of the test run. Please open the run log to make a deeper analysis of what happened.

If you open the Test case in the run log, you can see a 'Resolve dependency' node. Open it and there you find two nodes. The last one shows that the Setup has been executed. The other one will be explained in following chapters.

So far we have determined that the Setup node will be automatically executed before a Test case, if the Dependency is defined at the Test set level, but the Cleanup has not been executed so far. If you start another Test case now, e.g. 'Discount 15', the already started SUT is used.

The Setup node of a Dependency will be executed before each Test case in any case. That is to ensure that the prerequisites are fulfilled for each single Test case. The Cleanup node of a Dependency will only be executed on demand, i.e. if the steps of the Setup of that Dependency are not required anymore. In our case the Cleanup steps did not get executed, because both test cases have the same Dependency. The test execution passed because the Procedure startStop.startApplication already checks whether it's necessary to launch the application or not.

The next step is to launch the complete Test set by clicking it and pressing 'Run test'.

All three tests should pass and the SUT should not be stopped in between. We have achieved an optimized test execution. The Cleanup will not run because all three test cases refer to the same Dependency. We save a lot of time in executing the test cases and we are closer to testing a real life behavior as users usually do not restart the SUT all the time.

The next goal is to make that Dependency available for other Test sets in our project. For this you need to move the Dependency node into the Procedures area. Then click at the Test set and insert a Dependency reference node via »Insert node«-»Dependencies«-»Dependency reference«. The dialog looks similar to the Procedure call dialog and you can select the Dependency on the same way. The test suite should now look like this:

We recommend to add all Dependencies to a separate Package called 'dependencies'.

If you run the Test set the first time after moving the Dependency, QF-Test stops the SUT before the first Test case and re-starts the SUT, because the Dependency from the Test set is different from the Dependency defined in the Procedures node. But this will be explained more detailed in following chapters.

Please take a closer look at the second Test set 'Discount Tests with SUT stop' of the demo test suite. The second Test case 'Dicount 10' stops the SUT. But the third Test case also relies on a running SUT. As we know the Dependencies concept already, we should see that the SUT will be started for the third Test case again. That is exactly the advantage of the Dependencies concept.

29.3

Combining dependencies

Managing prerequisites can be a more complex matter than just verifying whether the SUT is up and running. In many projects several groups of test cases with different prerequisites like others exist.

Let us assume you want to test a big ERP system with more views, like a 'vendor' view and an 'article' view. Then all tests for the 'vendor' view will rely on an opened 'vendor' view and all tests for the 'article' view rely on an opened 'article' view. The opening of the specific view again relies on a logged-in user and the login depends on a started SUT. So you can see something like a tree of prerequisites.

QF-Test allows the user to build such a structure of Dependency nodes via adding Dependency reference nodes to a Dependency. We will build up a small example with just two dependencies for the CarConfigurator now.

In the CarConfigurator you can open a 'vehicles' dialog via the menu 'Options' -> 'Vehicles'. We want to create certain tests for that dialog now. Later we want to create some tests on the 'accessory' dialog which can also be opened via the menu via 'Options' -> 'Accessories'.

First let us define the tests which we want to create.

Test Case 1: Create vehicle 'test1' with price '100'.

• Start SUT if necessary.
• Open vehicles dialog via the menu.
• Specify 'test1' as name and '100' for the price.
• Press 'New'.
• Close vehicles dialog via pressing 'Ok'.
• Open vehicles dialog via the menu.
• Select the newly created vehicle 'test1'.
• Close the vehicles dialog via pressing 'Cancel'.
• Stop SUT if necessary.

Test Case 2: Create vehicle 'test2' with price '99999'.

• Start SUT if necessary.
• Open vehicles dialog via the menu.
• Specify 'test2' as name and '99999' for the price.
• Press 'New'.
• Close vehicles dialog via pressing 'Ok'.
• Open vehicles dialog via the menu.
• Select the newly created vehicle 'test2'.
• Close the vehicles dialog via pressing 'Cancel'.
• Stop SUT if necessary.

Test Case 3: Create accessory 'testaccessory' with price '12'.

• Start SUT if necessary.
• Open accessories dialog via the menu.
• Specify 'testaccessory' as name and '12' for the price.
• Press 'New'.
• Press 'Ok'.
• Open accessories dialog via the menu.
• Select the newly created accessory 'testaccessory'.
• Close the accessories dialog via pressing 'Cancel'.
• Stop SUT if necessary.

Let us take a closer look at the test steps of the tests above. We see that each test case requires a running application, so we should implement the 'Start SUT' step as prerequisite. This has to be done in the Setup of a Dependency. The 'Stop SUT' step is an optional step which can be part of the Cleanup of the Dependency. This 'SUT started' Dependency has been implemented by us in the previous examples already, so we could re-use it.

The next issue is that test case 1 and test case 2 require an opened vehicle dialog. Because we plan more tests in that area, we can create a new Dependency 'vehicles dialog opened', which will contain the opening of the dialog as Setup and the closing of the dialog via 'Cancel' as Cleanup. We are able to open this dialog only if the SUT is up and running already, so this Dependency is dependent on the 'SUT started' Dependency. The implementation of that 'open vehicle dialog' Dependency looks like this:

Note In the Setup we have to check whether the dialog is already opened, because a previous test case could have forgotten to close the dialog. The 'Timeout' attribute of the Wait for component to appear node is set to '0' because the dialog is expected to be here. If the dialog is already opened, it is fine and we can continue, otherwise we have to open the dialog.

We have to create a Dependency 'accessories dialog opened' as well. This Dependency is similar to the 'vehicles dialog opened' Dependency, but it deals with the accessory dialog instead of the vehicles dialog.

After creating those dependencies we have to record the according test steps and create the test cases. The test steps have already been implemented by us and you can find them in the according Package structure in the test suite qftest-9.0.0/doc/tutorial/ advanced-demos/en/dependencies.qft.

The test cases should be organized in one Test set called 'Combined Dependencies Tests'. This Test set should contain two more Test sets. The first one is 'Tests for vehicles' and the second one is 'Tests for accessories'. The Test set 'Tests for vehicles' has to contain the implementation of test case 1 and test case 2 and a Dependency reference to the 'vehicles dialog opened' Dependency. The second Test set 'Tests for accessories' has to contain test case 3 and a Dependency reference to the Dependency 'accessories dialog opened'.

If you run that Test set now, you will see that QF-Test stops the SUT first, because that comes from a dependency of the previous examples. Then QF-Test starts the SUT, performs the steps of test case 1 and then the steps of test case 2. Then it will perform the steps of test case 3. If you take a closer look at the beginning of test case 3 in the run log, you will see that the Cleanup of the 'vehicles dialog opened' Dependency has been executed. That happened because the 'vehicles dialog opened' Dependency was not required anymore. The following test case test case 3 required the 'accessories dialog opened' Dependency. However, both Dependencies require the 'SUT started' Dependency, that is why that Cleanup of the 'SUT started' Dependency has not been executed.

The ability to combine dependencies or to call the Cleanup of dependencies on demand can enable you to put a lot of preparing and cleaning steps into a Dependency. Another use case for the CarConfigurator could be to create a 'Vehicle created' Dependency, which creates a vehicle, which is required for the test and removes it afterwards. That would be an approach to ensure that the test data has been established correctly.

29.4

Error and exception handling

29.4.1

Error handling

Please copy the test suite qftest-9.0.0/doc/tutorial/ advanced-demos/en/dependencies_work.qft to a project-related folder first and open it there if you haven't done that so far. You can find a Test set 'Tests with error'. The second Test case of that Test set fails.

Let us assume that we want to trigger a specific behavior after a test case, but only if it failed. In our case we could stop the SUT to guarantee that the following test case can rely on a clean environment again. We know that the Setup node is executed before each Test case and the Cleanup node will only run on demand. So how could we run specific steps only in case of an error?

The solution is the so called Error handler node for a Dependency. You have to click at the closed Cleanup node of that Dependency and insert the Error handler node via right mouse click and selecting »Insert node«-»Dependencies«-»Error handler«.

In the Error handler you can define the steps for stopping the SUT again. The Dependency SUT started should look like this now:

Please run the whole Test set 'Tests with error' and switch into the run log after the test run has finished.

In the run log you can see that the Error handler was executed after the second Test case only.

29.4.2

Exception handling

In the previous section we have learned that Error handler nodes contain steps which are executed if the Test case fails. Besides errors also exceptions can occur during a test run. An exception is an unexpected behavior during the test run, e.g. an error dialog appears suddenly or a component could not be found anymore. How should we handle such exceptions?

You can find an example Test set 'Tests with exception' in the test suite dependencies_work.qft.

Of course you could surround the test steps in each Test case by try-catch and implement a dedicated exception handling in each single Test case, like implemented in the demo Test set. But this approach could lead to a lot of redundancy and makes the Test cases a little bit more unreadable.

Our goal is to reduce the redundancy and to move the exception handling - which is always the same - to one global location. This location will be our Dependency.

The first step is adding the Catch node to the Dependency via clicking at the closed Error handler node and selecting »Insert node«-»Control structures«-»Catch« in the menu. Then we can copy the procedure-calls for stopping the SUT into that Catch. Now the try-catch nodes in the single test cases become unnecessary. Please move the procedure-calls out of the Try and remove the empty try-catch structure.

Your test suite should look like this now:

Now you can start the Test set 'Test with exception'. The second Test case throws an IndexNotFoundException, because the selected vehicle does not exist. It is intended to handle that exception in the Catch of the Dependency.

Note If the debugger is activated, QF-Test interrupts the test run at the place where the exception arises. In this case you might have to re-throw the exception via the 'Rethrow exception' button. An alternative is to deactivate the debugger via »Debugger«-»Enable debugger«.

After the test run you can open the run log and verify what happened.

In a normal project you should at least try to catch a 'TestException' and the Catch should contain a procedure call of either qfs.swing.cleanup.closeAllModalDialogs, qfs.fx.cleanup.closeAllModalDialogs or qfs.swt.cleanup.closeAllModalDialogsAndModalShells. Those procedures close all modal dialogs, e.g. dialog windows, which are blocking the execution.

29.4.3

Summary

You have seen that you can implement a very strong recovery system for your test cases using the Error handler and, even more important, the Catch of Dependencies.

In most projects the global Catch node becomes very important, especially if ComponentNotFoundExceptions and ModalDialogExceptions appear.

29.5

More about dependencies

In the previous sections we have learned that we can combine several dependencies and that the Cleanup of a Dependency is only executed, if the stack of combined dependencies changes. We could also instruct a Dependency to call its Cleanup every time after a Test case. This can be achieved by checking the attribute 'Forced cleanup' of the Dependency.

There are even more interesting features about dependencies, e.g. you can use a variable instructing QF-Test to run the Cleanup. This variable is called 'Characteristic variable'. You will find more details about that feature in the manual at chapter Dependencies. This approach could be used to create one 'user login' Dependency which will only run through its Cleanup, i.e. perform the logout, if another user is required by the next Test case.

A detailed description of dependencies can be found in the manual at chapter Dependencies.