Documentation

Documentation

Testing Vaadin Applications Using TestBench

TestBench allows you to create UI unit and end-to-end tests for your Vaadin applications.
Note
Commercial feature

A commercial Vaadin subscription is required to use TestBench in your project.

TestBench includes special support for other Vaadin products, making testing easy and robust compared to generic web testing solutions, such as Selenium.

TestBench provides two approaches – UI unit testing and end-to-end testing – to test your application, with different advantages. Although Selenium has certain limitations compared to TestBench end-to-end testing, you can also use it for testing your applications.

UI Unit Testing
Explains the value of UI unit testing, to test Vaadin applications faster.
End-to-End Testing
Simulating an application with end-to-end testing to verify expected actions took place.
Testing with Playwright
How to set up end-to-end testing of a Vaadin application with Playwright.
Testing with Selenium
Describes how to set up end-to-end testing for a Vaadin application using Selenium.

Comparison

UI unit tests allow you to write user interface tests that don’t need a browser or a servlet container to run. This testing approach speeds up your test runs, making the results less flaky compared to end-to-end tests. End-to-end tests are closer to a real-world user experience and can simulate user actions that are impossible with UI unit tests.

Because UI unit tests are faster and more stable, they are suitable for development workflows where you need to write many tests, perhaps with every commit, and you want to run those tests often. This is the case, for example, with the Test-Driven Development (TDD) approach.

End-to-end tests are more suitable for testing the critical parts of your application, such as the login and the checkout process, to simulate your users' real-life experience with your application.

Advantages of UI Unit Testing

Advantages of UI unit testing over end-to-end testing:

  • Fast: Browser-less tests are typically 100× faster than end-to-end tests and run in 5–60 milliseconds, depending on their complexity.

  • Reliable: No arbitrary sleeps are needed, as the test is executed on the server side and it can wait until the request is fully processed. No random failures because of incompatibility between the Selenium drivers used in end-to-end testing and the browser.

  • Headless: The tests run headless, as there is no browser. No need to set up the screen in your CI environment.

  • Robust: The test runs in the same JVM as the server-side components. If the server-side bootstrap fails and throws an exception, the test method fails with the same exception. No need to go hunting for exceptions in a log located somewhere on a CI server.

  • No need to write Page Objects: You are already on the server, and you have access to the actual Java components that are already providing you with high-level APIs, exactly as Page Objects do.

  • Full access to the database: You are on the server side, so you can access the database from your tests in the same way your business logic accesses the database. You can run a bunch of SQL statements to restore the database to a known state before every test. Even better, you can run the test in a transaction, then roll back after the test to perform a fast database revert to a known state.

With this technique, you can run 600 UI tests in 7 seconds, as opposed to 1 to 2 hours with the end-to-end testing approach. Because of the speed, you can let the UI tests run after every commit via your continuous integration server.

Bypassing the browser and talking to the Vaadin server API directly eliminates the need to start the servlet container. You can add the server JARs to the testing classpath and call the Vaadin server API, which, in turn, invokes your server logic.

Advantages of End-to-End Testing

Though UI unit testing is faster and less flaky, end-to-end testing is still sometimes needed.

Specifically, end-to-end testing is more suitable for testing the critical parts of your application, such as the login and the checkout process, to simulate your users' real-life experience with your application.

Moreover, end-to-end testing enables you to test the client-side functionality of your application, such as JavaScript code or the parts of the UI that are implemented using the Polymer or Lit libraries.

These tests are impossible with the UI unit testing approach because UI unit tests run on the server side without initializing the browser.

Differences between TestBench, Selenium and Playwright

TestBench, Selenium and Playwright are all end-to-end testing frameworks. TestBench is the only tool that also provides helpers for unit testing Vaadin UIs.

The end-to-end testing part in TestBench is based on Selenium. However, TestBench is specifically designed for testing Vaadin applications. Playwright is similar in nature to Selenium, with no special Vaadin support. Compared to Selenium and Playwright, TestBench is designed to:

  • ease the handling of Vaadin components. Unlike Selenium, which operates on the DOM of the web page and provides basic methods to interact with elements like click() or sendKeys(), TestBench provides helper methods to interact with Vaadin components, like select() or setValue(). Moreover, TestBench provides a more convenient way to interact with elements in the shadow DOM of Vaadin components.

  • take the nature of client-server communication in Vaadin applications into account. During the course of a test, TestBench automatically suspends client-side execution if the server is busy and resume it when the server is ready. This makes it possible to write tests in TestBench without the need to explicitly set wait timeouts for various events like page load or waiting for long-running server calls to finish.

  • an easy-to-use API for conducting visual regression tests (screenshot comparison).

  • enable parallel testing of Vaadin applications.