The impact of geometry on component recognition

This is a short article about why QF-Test records geometry information for components, what impact these values have and how to change them either after or during recording .

For the interested, the whole component recognition algorithm is explained in the technical details of the QF-Test manual, section "Component recognition":

Here's a short summary:

Based on the component information stored in the test-suite QF-Test determines a match probability for each visible component in the SUT. The component with the highest probability wins, provided that probability is a above a configurable threshold. Otherwise a  ComponentNotFoundException is thrown. Though the algorithm has been fine-tuned many times, it's basic concept hasn't changed since QF-Test's early days and the default settings have proven useful in most cases.

Geometry is the least relevant part of component recognition, but it may have a significant impact in case the other characteristics are not conclusive.

The probability for a component starts with a 99.9% match. Then geometry is applied in one of three ways:

• Each of the values - if specified and ignoring X and Y for top-level windows - is compared to the actual values and a deviation reduces the probability.
• Consequently - and this is important - if geometry information in the test-suite is removed, all components start out equal, but with a perfect geometry match.
• If any of the geometry values is set to '-' the probability is immediately reduced to zero, so again all components start out equal but with zero probability so that a match of any of the other characteristics is required.

Hands-on example

Let's work on an example to see first-hand what this means:

Note: In case you have a commercial license that does not include the Swing engine the CarConfig demo will not connect. To work through the example you can start QF-Test in demo mode with the command qftest -license=

• Open the Help → "Explore sample test-suites" menu item and select the first example, "Swing CarConfig Suite".
• To start the SUT expand the Test-set, select the Dependency reference "dependencies.sutStartedAndReset" and press [Return] or click the Run toolbar button. The QF-Test CarConfig demo should start up and the record button should become enabled.
• Open a new, empty test-suite and record three mouse clicks on the value fields for "Base price", "Specials price" and "Discount".

The recorded components should look like in the following screenshot

• Naturally - this is a demo application after all - the recorded component information is totally overqualified. We have a perfect name, explicitly associated label as feature, implicitly determined qfs:label as extra feature plus structure and geometry information that is still fresh and valid.
• Select the mouse click on JCarConfigurator.DiscountValue, activate 'Replay as "hard" event' and run it. The click should be spot-on.

In a real-world situation, name and feature information may be missing, the label might have change and structure and geometry information deteriorate as the layout of the components in the SUT changes or components get added or removed.

Playing with component information to learn about the effects

Let's modify our recorded information in several steps to learn about their effect on component recognition:

1. No name

Remove the name "DiscountValue" from the component information, then replay the mouse click.

→ Still good. To see how good, open the options dialog via the Edit->Options menu-item, select Run-logs->Content in the tree and set the option "Log level for the SUT" to "All messages". Don't forget to reset this option at the end so that your run-log don't grow needlessly.

Then replay the mouse click and open the run-log via [Ctrl-L].

Expand the nodes via [Alt+RightArrow] and select "Looking for matching top-level components...". The information shown there is not user-friendly. Its main goal is to provide information for QFS support when helping to resolve component recognition issues.

Scroll down to the bottom, go back a little and you'll find something close to the following:

...
.2 Calculating probability for

javax.swing.JTextField[DiscountValue,239,0,48x30,layout=javax.swing.plaf.basic.BasicTextUI$UpdateHandler,alignmentX=0.0,alignmentY=0.0,border=,flags=296,maximumSize=,minimumSize=,preferredSize=,caretColor=javax.swing.plaf.ColorUIResource[r=0,g=0,b=0],disabledTextColor=javax.swing.plaf.ColorUIResource[r=170,g=170,b=170],editable=true,margin=javax.swing.plaf.InsetsUIResource[top=1,left=2,bottom=2,right=2],selectedTextColor=javax.swing.plaf.ColorUIResource[r=0,g=0,b=0],selectionColor=javax.swing.plaf.ColorUIResource[r=172,g=210,b=248],columns=4,columnWidth=12,command=,horizontalAlignment=RIGHT]

Geometry probability: 99.9%Structure matchStructure probability: 100%, penalty: 100%Extra feature matchExtra feature probability: 100%, penalty: 100%Feature matchFeature probability: 100%, penalty: 100%Combined probability: 100%, acceptableFinal probability: 99%

Calculation starts with the "Geometry probability" as base value, then structure, extra feature, feature and name are applied - if available - as explained in the manual leading to the final probabilty.

2. No feature

Remove the feature "Label: Discount" from the component information, then replay the mouse click and check the run-log as before.

→ Still good, almost unchanged, just no Feature match:

Geometry probability: 99.9%
Structure match
Structure probability: 100%, penalty: 100%
Extra feature match
Extra feature probability: 100%, penalty: 100%
Combined probability: 100%, acceptable
Final probability: 99%

3. Changed feature

Let's make it more interesting. Undo the last change to restore the label, then edit the value, e.g. by appending "bla", then run again and check the log.

→ The click is still on target, but the log shows a feature mismatch and the final probability is barely above the minimum probability of 50%:

Geometry probability: 99.9%
Structure match
Structure probability: 100%, penalty: 100%
Extra feature match
Extra feature probability: 100%, penalty: 100%
Feature mismatch
Feature probability: 100%, penalty: 55%
Combined probability: 55%, acceptable
Final probability: 54.4%

4. Changed feature, extra feature and structure

Let's give it all we've got: Change the class count to 5 to simulate that a previously existing TextField has been removed in the meantime and modify the extra feature qfs:label, e.g. by also appending "bla" to the value "Discount" as shown in the

→ Surprisingly the click is still on target and despite 3 mismatch warnings the resulting probability has barely changed:

Geometry probability: 99.9%
Structure mismatch index: 2, count: 4
Structure probability: 99.9%, penalty: 60%
Extra feature mismatch
Extra feature probability: 99.9%, penalty: 55%
Feature mismatch
Feature probability: 99.9%, penalty: 55%
Combined probability: 54.9%, acceptable
Final probability: 54.4%

So in this case geometry saves the day, keeps the test running and enables you to recover current, valid component information by right-clicking the component node and selecting "Update component(s)". Go ahead and try it, but then use "Undo" to restore the deliberately broken state from the last screenshot.

5. Empty geometry - match everything

Now lets finally start messing with the geometry values. As we have a perfect geometry match anyway, things shouldn't change if we remove the geometry values X, Y, Width and Height entirely (cf. summary at the beginning).

→ Oops! Now the click goes to the Base price value. Looking at the run-log we see that all four TextFields now end up with the same final probability of 54.4% which makes sense as all start with the same geometry probability and have the same mismatches. For lack of further options QF-Test simply chooses the first one.

This shows that removing geometry values entirely is inviting disaster in the form of false positive matches. The ability to leave individual geometry fields empty is still useful though. For example you can target the individual TextFields based on geometry only by setting the Y field to 30, 60, 90 or 120 respectively.

6. Geometry "-" - match nothing

False positive matches are difficult to analyze so we may want to avoid a geometry based match altogether. Set one or all of the X, Y, Width and Height attribute to just '-'.

→ OK, we've now broken things to the point where we get a ComponentNotFoundException.  The details in the run-log show that all candidates now have zero probability as expected.

7. Restore structure match

Let's restore the structure information by changing Class count back to 4.

→ No luck

The run-log shows that the structure match would have brought us up to an acceptable 70% match, but the feature and extra feature mismatch bring it down again to 38.5% which is below the threshold.

8. Restore feature

Undo to break structure again and fix the feature instead, setting it back to Label: Discount

→ The click is back on target

From the run-log we see that (in the absence of names) the feature has the highest relevance and is sufficient to raise the probability above the threshold in spite of the other mismatches.

9. Roll your own

Go ahead and experiment with restoring structure and/or extra feature information while also removing the higher relevance attributes feature and/or extra feature. You'll see that correct structure info alone is sufficient when
not hindered by some other mismatch.

Generic components

Note: The manual chapter "How to achieve robust component recognition" is generally worth reading. Section Avoiding recording every component or using generic components () explains the concept of generic components which we are going to need for the following.

So let's apply the knowledge gained to creating a generic component based on just the qfs:label extra feature:

• Select TextField JCarConfigurator.DiscountValue , copy, select Window JCarConfigurator and paste. Select "Make QF-Test IDs unique" to prevent ID conflicts, then change the QF-Test ID of the copied node to genericTextField.
• Set geometry to '-', remove feature, structure and all extra features except qfs:label and set the value of the qfs:label extra feature to $(label). The result should look as shown below:

• To make use of that component, create a Test-step under Extras with a variable named "label", value "Discount". Copy the mouse click node on JCarConfigurator.DiscountValue into the Test-step and change the QF-Test component ID to genericTextField. It should work right away.
• And as for why and where this can be useful: Add a data driver node to the Test-step and inside that a Data table, name "label", column "label", four rows with values "Base price", "Specials price", "Discount" and "Final price" as shown below, then run the whole test-step

Summary

• Geometry values in recorded components are useful in combination with other characteristics. If you tend to work mostly with recorded components, just keep it as is. In case most of the recorded components have no reliable names or other features you'll get best result by ensuring fixed window sizes upon SUT startup.
• Removing all geometry values is a bad idea because that can lead to false-positive matches when features change. To take geometry out of the picture, set the values to '-' instead.
• Generic components should always use geometry '-' unless they are explicitly based on coordinates.

If you feel that disabling geometry via '-' could improve your tests you may wonder how to implement that efficiently. So I'll finish this article by looking at how to mass-replace geometry values with '-' and provide an as-yet-undocumented tool to make sure new components get recorded that way.

Mass-changing recorded geometry values

Note: Before performing any kind of mass operations, always make sure that you test-suites are properly saved and backed up, ideally in a version control system like git!

Mass-replacement in a single test-suite is simple: Select the 'Windows and components' node, press [Ctrl-H] to open the replace dialog (menu Edit->Replace...), make sure to use the advanced dialog, clicking the '>>' icon if necessary and set the values as shown: