Documentation

Documentation

Making Tests Reliable

Tutorial on creating reliable tests.

There are many factors that can cause problems with your tests. For instance, changes in the application could cause tests to fail. Also, problems in the testing environment might cause tests to fail. Or it may be that tests are just not understood by other developers or testers and are disabled or broken.

You need to take these factors into account or you may have a test suite where a few tests always fail. A test suite with such failures is useless because developers become desensitized to a test suite which is always a bit red and will thereby learn to ignore the tests.

Tip
 You should make sure that your test suite is run on a regular basis. Having a manually triggered test suite which is run only after plenty of changes have been made to the application makes maintenance difficult. The best approach is to run the test suite on every change.

Creating Readable Tests

As with any code, it’s important to write tests so that the reader understands the intent. When each test contains high-level, meaningful calls, the reader immediately grasps what’s being tested. If they want to know more details about some part of the test, they can then dig into that part.

If the test is full of low-level details about how you locate the parts of the application with which you want to interact, it becomes overwhelming to try to decode what the test is actually trying to verify. By using page/view objects, you can abstract away the low-level details about how the view is built and what exact components are used. You can also use Behavior-Driven-Development (BDD) to describe your test scenarios using normal English sentences.

Guarding against Application Changes

If your application never changes, you can test it manually once and know that it works. However, applications are usually developed forward and you will need to maintain the tests as the application evolves.

As long as you abstract away the details from the tests to page/view objects, you only need to take care that your page/view objects are built in a robust way.

You should avoid depending on the HTML DOM structure. If you depend on finding a <div> inside a <span> or something similar, you have to update the page/view object for every small detail that changes in the application.

Similarly, you should avoid depending on strings targeted for people in your application. Although it’s often tempting to find the button with the text "Save", you will run into unnecessary problems when someone decides to change the text to "Store", or something to internationalize the application.

Define Ids for the Components

For most cases, it makes sense to define ids for all of the elements you want to interact with inside your page/view object. The ids are created to be able to identify a given element. There is typically no reason to change them when the application evolves.

When using templates, you also don’t need to worry about global ids and ids colliding with each other since the id of a given element only needs to be unique inside the shadow root (i.e., the template). For layouts and components outside templates — and inside a single template — you should take care that you don’t use the same id in multiple places.

Tip
 Use ids which describe the action that occurs when the button is pressed, not, for example, where in the hierarchy the button is situated. If your id is tied to the hierarchy, you indirectly depend on the hierarchy and lose many benefits of using ids.

Dealing with Test Environment Problems

When dealing with browser-based tests, you need to take into account that the environment isn’t always as stable as you would want it to be.

Ideally, the test would start the browser, execute the actions and close the browser nicely – always. In practice, there is potential for network problems, especially when using a cloud-based browser provider. There can also be browser problems causing randomness, or even browser crashes.

When the point of failure is outside of your control (e.g., a temporary network failure), your options are very limited. To deal with all kinds of unexpected actions — in the network or the browsers — TestBench offers a RetryRule. This is a way to run automatically the test again to see if the temporary problem has disappeared.

RetryRule is used as a JUnit 4 @Rule, with a parameter describing the maximum number of times the test should be run. Below is an example of this:

public class RandomFailureTest extends TestBenchTestCase {

    // Run the test max two times
    @Rule
    public RetryRule rule = new RetryRule(2);

    @Test
    public void doStuff() {
      ...
    }

}

If the test passes on the first try, it isn’t rerun. Only if the first try fails does it try again. It will keep trying until either the test passes or the maximum number of attempts has been reached.

RetryRule affects all of the test methods in the class and also the child classes.

The default value of maxAttempts is 1, meaning that the test is run only once. You can change the value of maxAttempts globally using the Java system property -Dcom.vaadin.testbench.Parameters.maxAttempts=2.

Note
 Use RetryRule when you’re sure that the test fails because of problems with the WebDriver, not your application. Using RetryRule without caution may hide random problems that are happening in your application.

Rerun Failed Tests using JUnit 5

@RetryRule is not available for JUnit 5 tests. Please use native test runners retry features:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-failsafe-plugin</artifactId>
    <version>3.0.0-M7</version>
    <configuration>
        <rerunFailingTestsCount>3</rerunFailingTestsCount>
    </configuration>
</plugin>
test {
    retry {
        maxRetries = 2
        maxFailures = 20
        failOnPassedAfterRetry = true
    }
}

872ED8E3-65E6-4926-A5AB-E1CEA77B9A69