/ TESTING, END-TO-END TESTING, BEHAVIOR-DRIVEN DEVELOPMENT, SELENIUM, CUCUMBER, SPRING

Starting with Cucumber for end-to-end testing

This week, my team decided to create a smoke test harness around our web app to avoid the most stupid regressions. I was not in favor of that, because of my prior experience with the fragility of end-to-end testing. But since we don’t have enough testers on our team, that was the only sane thing to do. I stepped forward to develop that suite.

A simple TestNG MVP

At first, I wanted to make a simple working test harness, so I chose technologies I was familiar with:

  • Selenium to manage browser interactions.
  • TestNG for the testing framework.

TestNG is a much better choice that JUnit (even compared to the latest 5th version) for end-to-end testing because it lets you order test methods.

However, the problem with this approach is readability when failures happen. Stack traces are hardly understandable by team members that are not developers. After having developed something that worked, I thus wanted to add some degree of usability for all team members.

Migrating to Cucumber

Cucumber is a BDD tool available on the JVM. It can be integrated with Selenium to have a thin BDD layer on top of GUI testing.

Project structure

Cucumber is based on 3 main components :

  • A feature, as its name implies, is a high-level feature of the system. It contains different scenarios, smaller-grained features that realize a feature. Each scenario is made in turn by a combination of step_s: _Given, When and Then that are well-know to BDD practitioners.

    In Cucumber, a feature is written in its own file Gherkin

  • The step definition is a Java file that implements in code the steps described in the feature
  • Last but not least, the test class is a JUnit (or TestNG) test class that binds the 2 former components

For example, let’s analyze how to create a test harness for an e-commerce shop.

The following is an excerpt of the feature to handle the the checkout:

Feature: Checkout                                                (1)
  A customer should be able to browse the shop,                  (2)
                               put an item in the cart,
                               proceed to checkout,
                               login
                               and pay by credit card

  Scenario: Customer goes to the homepage and chooses a category (3)
    Given the homepage is displayed                              (4)
    Then the page should display 5 navigation categories         (5)
    And the page should display the search box                   (6)
    When the first displayed category is chosen                  (7)
    Then the customer is shown the category page                 (8)

  # Other scenarios follow
1 The name of the feature. The name should be short and descriptive.
2 A longer text that describes the feature in detail. It’s only meant for documentation.
3 The title of the scenario.
4 Initialization with the Given keyword
5 An assertion via the Then keyword
6 And could be replaced by Then but it feels more readable
7 an interaction, the When keyword is used
8 We assert the state of the app again

The corresponding step definition to the previous feature could look like that (in Kotlin):

class HomePageStepDef @Autowired constructor(page: HomePage) : En {

    init {
        Given("^the homepage is displayed$") { page.displayHomePage() }
        Then("^the page should display (\\d+) navigation categories$") { numberOfCategories: Int ->
            val displayedCategories = page.displayedCategories
            assertThat(displayedCategories).isNotNull().isNotEmpty().hasSize(numberOfCategories)
        }
        And("^the page should display the search box$") {
            val searchBox = page.searchBox
            assertThat(searchBox).isNotNull().isNotEmpty().hasSize(1)
        }
        When("^the first displayed category is chosen$") { page.chooseFirstDisplayedCategory() }
        Then("^the customer is shown the category page$") {
            assertThat(page.url).isNotNull().isNotEmpty().containsPattern("/c/")
        }
    }
}

Let’s forget for now about line 1 from the above snippet from above, included @Autowired and focus on the rest.

For each Given/When/Then line in the feature file, there’s a corresponding method with a matching regexp in the class file:

  • Cucumber matches the step defined in the feature with the method by using the first parameter - the regexp.
  • Parameters can be defined in the feature and used in the step. As an example, compare line 10 of the first snippet with line 8 of the second: the regexp will capture the number of categories so it can be easily changed in the feature without additional development.
  • The second method parameter is the lambda that will get executed by Cucumber.
  • Given I’m using a Java runtime v8, those methods are default methods implemented in the En interface. There’s one such interface for each available language, so that step definitions can be implemented in your own language.
  • The class has no direct dependency on the Selenium API, it’s wrapped behind the Page Object pattern (see below).

Finally, here’s the entry point test class:

@RunWith(Cucumber::class)
@CucumberOptions(
        features = arrayOf("classpath:feature/guest_checkout.feature"),
        glue = arrayOf("de.sansibar.glue"),
        format = arrayOf("pretty"))
class GuestCheckoutIT

As can be seen, it’s empty: it just provides the entry point and binds a feature to the step definitions package. At that point, running the test class in the IDE or through Maven will run the associated Cucumber feature.

Improving beyond the first draft

So far, so good. But the existing code deserves to be improved.

Coping with fragility

I cheated a little for this one as it was already implemented in the first TestNG MVP but let’s pretend otherwise.

If you’ve read the step definition class above, you might have noticed that there’s no Selenium dependency anywhere in the code. All of it has been hidden in a class that represents the page:

class HomePage(driver: WebDriver, private val contextConfigurator: ContextConfigurator): AbstractPage(driver) {

    val displayedCategories: List<WebElement> by lazy { driver.findElements(className("navigationBannerHome")) }
    val searchBox: List<WebElement> by lazy { driver.findElements(id("input_SearchBox")) }

    fun displayHomePage() {
        val url = contextConfigurator.url
        driver.get(url)
    }

    fun chooseFirstDisplayedCategory() {
        displayedCategories[0].click()
    }
}

This approach is known as the Page Object pattern.

Mixing selectors and tests into the same class makes tests brittle, especially in the early stage of the project when the GUI changes a lot. Isolating selectors into a dedicated class let us buffer changes into that class only.

There are a couple of good practices there - suggested by colleagues and from my personal experience:

  • Use id attributes on elements used for selection. This makes it less likely to break the test by changing the structure of the DOM.
  • Use coarse-grained methods mapped to a business case. For example, instead of having a whole bunch of selectTitle(), fillFirstName(), fillLastName(), submitRegistration(), etc. methods for each registration field, have a single register() method that inputs and submits the data. Again, this isolates possible breaking changes in the page class.

Improved design

The Page needs to select components through the Selenium API, thus it needs a reference to a WebDriver. This is a problem when a single feature contains several scenarios as this reference needs to be shared among all scenarios. Possible solutions to this include:

  1. A single scenario per feature. Every scenario will have to define its starting point. For our e-commerce checkout scenario, this defeats the purpose of testing itself.
  2. A single scenario containing steps of all scenarios. In this case, all scenarios will be merged into a very long one. That makes for a hard-to-read scenario and an even harder to read (and maintain) class.
  3. To be able to have multiple scenarios per feature while putting methods into their relevant step definitions, one needs to share the same driver instance among all step definitions. This can be achieved by applying the Singleton pattern to a dedicated class.
  4. The last alternative is to use…​ DI! Actually, Cucumber integrates quite nicely to integrate with some commons DI frameworks, including Weld and Spring.

This is great news, as it’s possible to use the libraries we already use in development in the tests. Regular readers know me as a Spring proponent, so I naturally used it as the DI framework. Those are dependencies that are required for that in the POM:

<!-- Cucumber Spring integration -->
<dependency>
  <groupId>info.cukes</groupId>
  <artifactId>cucumber-spring</artifactId>
  <version>${cucumber.version}</version>
  <scope>test</scope>
</dependency>
<!-- Spring -->
<dependency>
  <groupId>org.springframework</groupId>
  <artifactId>spring-core</artifactId>
  <version>${spring.version}</version>
  <scope>test</scope>
</dependency>
<dependency>
  <groupId>org.springframework</groupId>
  <artifactId>spring-beans</artifactId>
  <version>${spring.version}</version>
  <scope>test</scope>
</dependency>
<dependency>
  <groupId>org.springframework</groupId>
  <artifactId>spring-context</artifactId>
  <version>${spring.version}</version>
  <scope>test</scope>
</dependency>
<!-- Spring test - mandatory but not available via transitivity -->
<dependency>
  <groupId>org.springframework</groupId>
  <artifactId>spring-test</artifactId>
  <version>${spring.version}</version>
  <scope>test</scope>
</dependency>

At this point, it’s quite easy to create a standard Spring configuration file to generate the driver, as well as the page objects and their necessaries dependencies:

@Configuration
open class AppConfiguration {

    @Bean open fun driver() = ChromeDriver()
    @Bean open fun contextConfigurator() = ContextConfigurator(properties)
    @Bean open fun homePage(contextConfigurator: ContextConfigurator) = HomePage(driver(), contextConfigurator)
    // Other page beans
}

In thie configuration, the driver bean is a singleton managed by Spring and the single instance can be shared among all page beans.

Those are also singletons part of the Spring bean factory. That’s the reason of the @Autowired annotation in the step definition constructor. But why don’t step definitions get created in the Spring configuration class? Because they are to be created by the Cucumber framework itself using package scanning - yuck. Note that they don’t need to be self-annotated, it’s part of Cucumber’s magic but they still are part of the Spring context and can be injected.

Screen capture

A common mistake made when testing is to think nothing will break, ever. I fell into this trap when I was more junior, and now I try to prepare for failure.

In order to achieve that, I wanted to take a screenshot when a test fails so that it will be easier to fix the failure. Cucumber provides a dedicated lifecycle - before and after code run around each test through respective @Before and @After annotations. Note those are not the same as JUnit’s, and Cucumber doesn’t parse JUnit’s own annotations.

The easy way would be to create an @After-annotated method in each step definition. Yet, that would just be code duplication. Cucumber also offers hooks, classes which annotated methods are run around each step definition test method. The only constraint for hooks is to place them in the same package as the step definitions so they can be discovered and managed by Cucumber Spring package scanning as for step definitions.

private val LOGGER = LoggerFactory.getLogger(ScreenCaptureHook::class.java)
private var DATE_FORMAT = "yyyy-MM-dd-HH-mm-ss"

class ScreenCaptureHook @Autowired constructor(driver: WebDriver) {

    private val screenshotFolder = File(File("target", "e2e"), "screenshots")
    private val takesScreenshot = driver as TakesScreenshot

    @Before
    fun ensureScreenshotFolderExists() {
        if (!screenshotFolder.exists()) {
            val folderCreated = screenshotFolder.mkdirs()
            if (!folderCreated) {
                LOGGER.warn("Could not create takesScreenshot folder. Screen capture won't work")
            }
        }
    }

    @After
    fun takeScreenshot(scenario: Scenario) {
        if (scenario.isFailed) {
            LOGGER.warn(scenario.status)
            LOGGER.info("Test failed. Taking takesScreenshot")
            val screenshot = takesScreenshot.getScreenshotAs(FILE)
            val format = SimpleDateFormat(DATE_FORMAT)
            val imageFile = File(screenshotFolder, format.format(Date()) + "-" + scenario.id + ".png")
            FileUtils.copyFile(screenshot, imageFile)
        }
    }
}
  • Cucumber unfortunately doesn’t distinguish between a failed test and a skipped one. Thus, screenshots will be taken for the failed test as well as the skipped tests ran after it.
  • There’s no global hook in Cucumber. The ensureScreenshotFolderExists() will thus be ran before each step definition test method. This way requires to manage state, so as to initialize only once.

Conclusion

The final result is a working end-to-end testing harness, that non-technical people are able to understand. Even better, using Cucumber over TestNG let us improve the code with Dependency Injection. The setup of Cucumber over Selenium was not trivial, but quite achievable with a little effort.

Despite all the above good parts, the harness fails…​ sometimes, for reasons unrelated to code, as all end-to-end tests go. That’s the reason I’m extremely reluctant to set them up in the build process. If that would happen every so often, the team would lose confidence in the harness and all would have been for nothing. So far, we’ll use it manually before promoting the application to an environment.

Nicolas Fränkel

Nicolas Fränkel

Developer Advocate with 15+ years experience consulting for many different customers, in a wide range of contexts (such as telecoms, banking, insurances, large retail and public sector). Usually working on Java/Java EE and Spring technologies, but with focused interests like Rich Internet Applications, Testing, CI/CD and DevOps. Also double as a trainer and triples as a book author.

Read More
—
Starting with Cucumber for end-to-end testing
Share this