Archive

Posts Tagged ‘testing’
  • Why are you testing your software?

    Burned bus at a bus stop

    :page-liquid: :experimental:

    15 years ago, automated tests didn’t exist in the Java ecosystem. One had to build the application and painfully test it manually by using it. I was later introduced to the practice of adding a main method to every class and putting some testing code there. That was only marginally better, as it still required to manually run the methods. Then came https://junit.org[JUnit^], the reference unit testing framework in Java which brought test execution automation. At that point, I had to convince teams I was part of that we had to use it to create an automated tests harness to prevent regression bugs. Later, this became an expectation: no tests meant no changes in the code for fear of breaking something.

    More recently, however, it happened that I sometimes have to advocate for the opposite: not write too many tests. Yes, you read that well, and yet I’m no turncoat. The reason for this lies in the title of the post: why are you testing your software? It may sound like the answer to this question is pretty obvious - but it’s not, and the answer is tightly coupled to the concept of quality.

    [quote, Wikipedia] __ In the context of software engineering, software quality refers to two related but distinct notions that exist wherever quality is defined in a business context:

    • Software functional quality reflects how well it complies with or conforms to a given design, based on functional requirements or specifications. That attribute can also be described as the fitness for purpose of a piece of software or how it compares to competitors in the marketplace as a worthwhile product. It is the degree to which the correct software was produced.
    • Software structural quality refers to how it meets non-functional requirements that support the delivery of the functional requirements, such as robustness or maintainability. It is the degree to which the software was produced correctly. __

    Before answering the question of the why, let’s consider some reason why not I’ve already been confronted to:

    • Because everyone does it
    • Because the boss/the lead/colleagues/authority figures say so
    • To achieve link:{% post_url 2011-04-18-100-code-coverage %}[100% of code coverage^]
    • To achieve more code coverage than another team/colleague
    • And so on, and so forth

    All in all, those “reasons” boil down to either plain cargo culting or mistaking a metric for the goal. Which brings us back to the question, why do you test software?

    [IMPORTANT]

    The only valid reason for testing is that resources spent in making sure the software conforms to non-/functional requirements will be less over the course of time than the resources spent if not done. ====

    That’s pure and simple Return Over Investment. If ROI is positive, do test; if it’s negative, don’t. It’s as simple as that.

    [quote] __ Perfect is the enemy of good __

    The real difficulty lies in estimating the cost of testing vs the cost of not testing. The following is a non-exhaustive list of ROI-influencing parameters:

    Business:: No bugs are allowed in some industries e.g. medical, airplanes, banks without serious consequences to the business, while this is less critical for others such as mobile gaming. Estimated lifespan of the app:: the longer the lifespan of an app, the better the ROI because the same amount of testing code will yield more times e.g. nearly no tests for one-shot single-event apps vs. more traditional testing for tradition business apps running for a decade or so. Nature of the app:: some technologies are more mature than others, allowing for easier automated testing. The testing echosystem around webapps is richer than around native or mobile apps. Architecture of the app:: The more distributed the app, the harder it is to test. In particular, the migration from monoliths to microservices has some interesting side-effects on the testing side. It’s easier to test each component separately, but harder to test the whole system. Also, testing specific scenarios in clustered/distributed environments, such as node failure in a cluster, increase the overall cost. Nature and number of infrastructure dependencies:: The higher the number of dependencies, the more test doubles are required to test the app in isolation, which in turns drive up testing costs. Also, some dependencies are more widespread e.g. databases and web services, with many available tools while some are not e.g. FTP servers. Size of the app:: Of course, the bigger the size of the app, the bigger the number of possible combinations that needs to be tested. Maturity of the developers, and the size of the team(s):: Obviously, developers range from the ones who don’t care about testing to those who integrate testing requirements in their code from the start. Also, just for developers, adding more testers is subject to the https://en.wikipedia.org/wiki/Diminishing_returns[law of diminishing returns^]. Nature of the tests:: I don’t want to start a war, suffice to say there are many kinds of tests - unit, integration, end-to-end, performance, penetration, etc. Each one is good at one specific thing, and has pros and cons. Get to know them and use them wisely. Strength of the type system:: Developing in dynamically-typed languages require more tests to handle the job of the compiler in comparison to more statically-typed languages.

    While it’s good to listen to other’s advices - including well-established authority figures and this post, it’s up to every delivery team to draw the line between not enough testing and too much testing according to its own context.

    Categories: Java Tags: code qualitytestingcode coverage
  • Should tests be ordered or not?

    Everything ordered and tidy

    :imagesdir: /assets/resources/should-tests-be-ordered-or-not/

    // Image by Karsten Seiferlin licensed under CC https://www.flickr.com/photos/timecaptured/14471140911

    Most of our day-to-day job is learned through mentorship and experience and not based upon scientific research. Once a dogma has permeated a significant minority of practitioners, it becomes very hard to challenge it.

    Yet, in this post, I’ll attempt to not only challenge that sometimes tests must be ordered but prove that in different use-cases.

    == Your tests shall not be ordered (shall they?)

    Some of my conference talks are more or less related to testing, and I never fail to point out that TestNG is superior to JUnit if only because it allows for test method ordering. At that point, I’m regularly asked at the end of the talk why method ordering matters. It’s a widespread belief that tests shouldn’t be ordered. Here are some samples found here and there:

    [quote,JUnit Wiki - Test execution order,https://github.com/junit-team/junit4/wiki/test-execution-order] Of course, well-written test code would not assume any order, but some do.

    [quote,JUnit FAQ - How do I use a test fixture?,http://junit.org/junit4/faq.html=atests_2] __ Each test runs in its own test fixture to isolate tests from the changes made by other tests. That is, tests don’t share the state of objects in the test fixture. Because the tests are isolated, they can be run in any order. __

    [quote, Writing Great Unit Tests: Best and Worst Practices,http://blog.stevensanderson.com/2009/08/24/writing-great-unit-tests-best-and-worst-practises] You’ve definitely taken a wrong turn if you have to run your tests in a specific order […]

    [quote,Top 12 Selected Unit Testing Best Practices,http://www.sw-engineering-candies.com/blog-1/unit-testing-best-practices=TOC-Always-Write-Isolated-Test-Cases] __ Always Write Isolated Test Cases

    The order of execution has to be independent between test cases. This gives you the chance to rearrange the test cases in clusters (e.g. short-, long-running) and retest single test cases. __

    And this goes on ad nauseam

    In most cases, this makes perfect sense. If I’m testing an add(int, int) method, there’s no reason why one test case should run before another. However, this is hardly a one-size-fits-all rule. The following use-cases take advantage of test ordering.

    == Tests should fail for a single reason

    Let’s start with a simple example: the code consists of a controller that stores a list of x Foo entities in the HTTP request under the key bar.

    === The naive approach

    The first approach would be to create a test method that asserts the following:

    . a value is stored under the key bar in the request . the value is of type List . the list is not empty . the list has size x . the list contains no null entities . the list contains only Foo entities

    Using AssertJ, the code looks like the following:

    [source,java]

    // 1: asserts can be chained through the API // 2: AssertJ features can make the code less verbose @Test public void should_store_list_of_x_Foo_in_request_under_bar_key() { controller.doStuff(); Object key = request.getAttribute(“bar”); assertThat(key).isNotNull(); // <1> assertThat(key).isInstanceOf(List.class); // <2> List list = (List) key; assertThat(list).isNotEmpty(); // <3> assertThat(list).hasSize(x); // <4> list.stream().forEach((Object it) -> { assertThat(object).isNotNull(); // <5> assertThat(object).isInstanceOf(Foo.class); // <6> }); } —-

    If this test method fails, the reason can be found in any of the previous steps. A customary glance at the failure report is not enough to tell exactly which one.

    image::single-test-result.png[Single tests result,541,52,align=”center”]

    To know that, one has to analyze the stack trace then the source code.


    java.lang.AssertionError: Expecting actual not to be null

    at ControllerTest.should_store_list_of_x_Foo_in_request_under_bar_key(ControllerTest.java:31) ----
    

    === A test method per assertion

    An alternative could be to refactor each assertion into its own test method:

    [source,java]

    @Test public void bar_should_not_be_null() { controller.doStuff(); Object bar = request.getAttribute(“bar”); assertThat(bar).isNotNull(); }

    @Test public void bar_should_of_type_list() { controller.doStuff(); Object bar = request.getAttribute(“bar”); assertThat(bar).isInstanceOf(List.class); }

    @Test public void list_should_not_be_empty() { controller.doStuff(); Object bar = request.getAttribute(“bar”); List<?> list = (List) bar; assertThat(list).isNotEmpty(); }

    @Test public void list_should_be_of_size_x() { controller.doStuff(); Object bar = request.getAttribute(“bar”); List<?> list = (List) bar; assertThat(list).hasSize(x); }

    @Test public void instances_should_be_of_type_foo() { controller.doStuff(); Object bar = request.getAttribute(“bar”); List<?> list = (List) bar; list.stream().forEach((Object it) -> { assertThat(it).isNotNull(); assertThat(it).isInstanceOf(Foo.class); }); } —-

    Now, every failing test is correctly displayed. But if the bar attribute is not found in the request, every test will still run and still fail, whereas they should merely be skipped.

    image::unordered-test-results.png[Unordered tests results,385,136,align=”center”]

    Even if the waste is small, it still takes time to run unnecessary tests. Worse, it’s waste of time to analyze the cause of the failure.

    === A private method per assertion

    It seems ordering the tests makes sense. But ordering is bad, right? Let’s try to abide by the rule, by having a single test calling private methods:

    [source,java]

    public void should_store_list_of_x_Foo_in_request_under_bar_key() { controller.doStuff(); Object bar = request.getAttribute(“bar”); bar_should_not_be_null(bar); bar_should_of_type_list(bar); List<?> list = (List) bar; list_should_not_be_empty(list); list_should_be_of_size_x(list); instances_should_be_of_type_foo(list); }

    private void bar_should_not_be_null(Object bar) { assertThat(bar).isNotNull(); }

    private void bar_should_of_type_list(Object bar) { assertThat(bar).isInstanceOf(List.class); }

    private void list_should_not_be_empty(List<?> list) { assertThat(list).isNotEmpty(); }

    private void list_should_be_of_size_x(List<?> list) { assertThat(list).hasSize(x); }

    private void instances_should_be_of_type_foo(List<?> list) { list.stream().forEach((Object it) -> { assertThat(it).isNotNull(); assertThat(it).isInstanceOf(Foo.class); }); } —-

    Unfortunately, it’s back to square one: it’s not possible to just know in which step the test failed just at a glance.

    image::single-test-result.png[Single test result,541,52,align=”center”]

    At least the stack trace conveys a little more information:


    java.lang.AssertionError: Expecting actual not to be null

    at ControllerTest.bar_should_not_be_null(ControllerTest.java:40)
    at ControllerTest.should_store_list_of_x_Foo_in_request_under_bar_key(ControllerTest.java:31) ----
    

    How to skip unnecessary tests, and easily know the exact reason of the failure?

    === Ordering it is

    Like it or not, there’s no way to achieve skipping and easy analysis without ordering:

    [source,java]

    // Ordering is achieved using TestNG

    @Test public void bar_should_not_be_null() { controller.doStuff(); Object bar = request.getAttribute(“bar”); assertThat(bar).isNotNull(); }

    @Test(dependsOnMethods = “bar_should_not_be_null”) public void bar_should_of_type_list() { controller.doStuff(); Object bar = request.getAttribute(“bar”); assertThat(bar).isInstanceOf(List.class); }

    @Test(dependsOnMethods = “bar_should_of_type_list”) public void list_should_not_be_empty() { controller.doStuff(); Object bar = request.getAttribute(“bar”); List<?> list = (List) bar; assertThat(list).isNotEmpty(); }

    @Test(dependsOnMethods = “list_should_not_be_empty”) public void list_should_be_of_size_x() { controller.doStuff(); Object bar = request.getAttribute(“bar”); List<?> list = (List) bar; assertThat(list).hasSize(x); }

    @Test(dependsOnMethods = “list_should_be_of_size_x”) public void instances_should_be_of_type_foo() { controller.doStuff(); Object bar = request.getAttribute(“bar”); List<?> list = (List) bar; list.stream().forEach((Object it) -> { assertThat(it).isNotNull(); assertThat(it).isInstanceOf(Foo.class); }); } —-

    The result is the following:

    image::ordered-test-results.png[Ordered tests results,431,202,align=”center”]

    Of course, the same result is achieved when the test is run with Maven:


    Tests run: 5, Failures: 1, Errors: 0, Skipped: 4, Time elapsed: 0.52 sec «< FAILURE! bar_should_not_be_null(ControllerTest) Time elapsed: 0.037 sec «< FAILURE! java.lang.AssertionError: Expecting actual not to be null at ControllerTest.bar_should_not_be_null(ControllerTest.java:31)

    Results :

    Failed tests: ControllerTest.bar_should_not_be_null:31 Expecting actual not to be null

    Tests run: 5, Failures: 1, Errors: 0, Skipped: 4

    In this case, by ordering in unit test methods, one can achieve both optimization of testing time and fast failure analysis by skipping tests that are bound to fail anyway.

    == Unit testing and Integration testing

    In my talks about Integration testing, I usually use the example of a prototype car. Unit testing is akin to testing every nut and bolt of the car, while Integration testing is like taking the prototype on a test drive.

    No project manager would take the risk of sending the car on a test drive without having made sure its pieces are of good enough quality. It would be too expensive to fail just because of a faulty screw; test drives are supposed to validate higher-levels concerns, those that cannot be checked by Unit testing.

    Hence, unit tests should be ran first, and only integration tests only afterwards. In that case, one can rely on the https://maven.apache.org/surefire/maven-failsafe-plugin/[Maven Failsafe plugin^] to run Integration tests later in the Maven lifecycle.

    == Integration testing scenarios

    What might be seen as a corner-case in unit-testing is widespread in integration tests, and even more so in end-to-end tests. In the latest case, an example I regularly use is the e-commerce application. Steps of a typical scenario are as follow:

    . Browse the product catalog . Put one product in the cart . Display the summary page . Fill in the delivery address . Choose a payment type . Enter payment details . Get order confirmation

    In a context with no ordering, this has several consequences:

    • Step X+1 is dependent on step X e.g. to enter payment details, one must have chosen a payment type first, requiring that the latter works
    • Step X+2 and X+1 both need to set up step X. This leads either to code duplication - as setup code is copied-pasted in all required steps, or common setup code - which increases maintenance cost (yes, sharing is caring but it’s also more expensive).
    • The initial state of step X+1 is the final state of step X i.e. at the end of testing step X, the system is ready to start testing step X+1
    • Trying to test step X+n if step X failed already is time wasted, both in terms of server execution time and and of failure analysis time. Of course, the higher n, the more waste.

    This is very similar to the section above about unit tests order. Given this, it makes no doubt for me that ordering steps in an integration testing scenario is far from a bad practice but good judgement.

    == Conclusion

    As in many cases in software development, a rule has to be contextualized. While in general, it makes no sense to have ordering between tests, there are more than a few cases where it does.

    Software development is hard because the “real” stuff is not learned by sitting on universities benches but through repeated practice and experimenting under the tutorship of more senior developers. If enough more-senior-than-you devs tend to hold the same opinion on a subject, chances are you’ll take that for granted as well. At some point, one should single out of such opinion and challenge it to check whether it’s right or not in one’s own context.

  • Starting with Cucumber for end-to-end testing

    Cucumber logo

    :imagesdir: /assets/resources/starting-with-cucumber-for-end-to-end-testing/

    This week, my team decided to create a smoke test harness around our web app to avoid the most stupid regressions. I was not in favor of that, because of my prior experience with the fragility of end-to-end testing. But since we don’t have enough testers on our team, that was the only sane thing to do. I stepped forward to develop that suite.

    == A simple TestNG MVP

    At first, I wanted to make a simple working test harness, so I chose technologies I was familiar with:

    • http://www.seleniumhq.org/[Selenium^] to manage browser interactions.
    • http://testng.org/[TestNG^] for the testing framework.

    TestNG is a much better choice that JUnit (even compared to the latest 5^th^ version) for end-to-end testing because it lets you order test methods.

    However, the problem with this approach is readability when failures happen. Stack traces are hardly understandable by team members that are not developers. After having developed something that worked, I thus wanted to add some degree of usability for all team members.

    == Migrating to Cucumber

    https://cucumber.io/docs/reference/jvm[Cucumber^] is a +++BDD+++ tool available on the JVM. It can be integrated with Selenium to have a thin BDD layer on top of +++GUI+++ testing.

    image:structure.png[Project structure,337,600,float=”right”]

    Cucumber is based on 3 main components :

    • A feature, as its name implies, is a high-level feature of the system. It contains different scenarios, smaller-grained features that realize a feature. Each scenario is made in turn by a combination of step_s: _Given, When and Then that are well-know to BDD practitioners. + In Cucumber, a feature is written in its own file https://github.com/cucumber/cucumber/wiki/Gherkin[Gherkin^]
    • The step definition is a Java file that implements in code the steps described in the feature
    • Last but not least, the test class is a JUnit (or TestNG) test class that binds the 2 former components

    For example, let’s analyze how to create a test harness for an e-commerce shop.

    The following is an excerpt of the feature to handle the the checkout:

    [source]

    Feature: Checkout <1> A customer should be able to browse the shop, <2> put an item in the cart, proceed to checkout, login and pay by credit card

    Scenario: Customer goes to the homepage and chooses a category <3> Given the homepage is displayed <4> Then the page should display 5 navigation categories <5> And the page should display the search box <6> When the first displayed category is chosen <7> Then the customer is shown the category page <8>

    # Other scenarios follow

    <1> The name of the feature. The name should be short and descriptive. <2> A longer text that describes the feature in detail. It’s only meant for documentation. <3> The title of the scenario. <4> Initialization with the Given keyword <5> An assertion via the Then keyword <6> And could be replaced by Then but it feels more readable <7> an interaction, the When keyword is used <8> We assert the state of the app again

    The corresponding step definition to the previous feature could look like that (in Kotlin):

    [source,java]

    class HomePageStepDef @Autowired constructor(page: HomePage) : En {

    init {
        Given("^the homepage is displayed$") { page.displayHomePage() }
        Then("^the page should display (\\d+) navigation categories$") { numberOfCategories: Int ->
            val displayedCategories = page.displayedCategories
            assertThat(displayedCategories).isNotNull().isNotEmpty().hasSize(numberOfCategories)
        }
        And("^the page should display the search box$") {
            val searchBox = page.searchBox
            assertThat(searchBox).isNotNull().isNotEmpty().hasSize(1)
        }
        When("^the first displayed category is chosen$") { page.chooseFirstDisplayedCategory() }
        Then("^the customer is shown the category page$") {
            assertThat(page.url).isNotNull().isNotEmpty().containsPattern("/c/")
        }
    } } ----
    

    Let’s forget for now about line 1 from the above snippet from above, included @Autowired and focus on the rest.

    For each Given/When/Then line in the feature file, there’s a corresponding method with a matching regexp in the class file:

    • Cucumber matches the step defined in the feature with the method by using the first parameter - the regexp.
    • Parameters can be defined in the feature and used in the step. As an example, compare line 10 of the first snippet with line 8 of the second: the regexp will capture the number of categories so it can be easily changed in the feature without additional development.
    • The second method parameter is the lambda that will get executed by Cucumber.
    • Given I’m using a Java runtime v8, those methods are default methods implemented in the En interface. There’s one such interface for each available language, so that step definitions can be implemented in your own language.
    • The class has no direct dependency on the Selenium API, it’s wrapped behind the Page Object pattern (see link:#coping-with-fragility[below]).

    Finally, here’s the entry point test class:

    [source,java]

    @RunWith(Cucumber::class) @CucumberOptions( features = arrayOf(“classpath:feature/guest_checkout.feature”), glue = arrayOf(“de.sansibar.glue”), format = arrayOf(“pretty”)) class GuestCheckoutIT —-

    As can be seen, it’s empty: it just provides the entry point and binds a feature to the step definitions package. At that point, running the test class in the IDE or through Maven will run the associated Cucumber feature.

    == Improving beyond the first draft

    So far, so good. But the existing code deserves to be improved.

    === Coping with fragility

    I cheated a little for this one as it was already implemented in the first TestNG MVP but let’s pretend otherwise.

    If you’ve read the step definition class above, you might have noticed that there’s no Selenium dependency anywhere in the code. All of it has been hidden in a class that represents the page:

    [source,java]

    class HomePage(driver: WebDriver, private val contextConfigurator: ContextConfigurator): AbstractPage(driver) {

    val displayedCategories: List<WebElement> by lazy { driver.findElements(className("navigationBannerHome")) }
    val searchBox: List<WebElement> by lazy { driver.findElements(id("input_SearchBox")) }
    
    fun displayHomePage() {
        val url = contextConfigurator.url
        driver.get(url)
    }
    
    fun chooseFirstDisplayedCategory() {
        displayedCategories[0].click()
    } } ----
    

    This approach is known as the http://martinfowler.com/bliki/PageObject.html[Page Object^] pattern.

    Mixing selectors and tests into the same class makes tests brittle, especially in the early stage of the project when the GUI changes a lot. Isolating selectors into a dedicated class let us buffer changes into that class only.

    There are a couple of good practices there - suggested by colleagues and from my personal experience:

    • Use id attributes on elements used for selection. This makes it less likely to break the test by changing the structure of the DOM.
    • Use coarse-grained methods mapped to a business case. For example, instead of having a whole bunch of selectTitle(), fillFirstName(), fillLastName(), submitRegistration(), etc. methods for each registration field, have a single register() method that inputs and submits the data. Again, this isolates possible breaking changes in the page class.

    === Improved design

    The Page needs to select components through the Selenium API, thus it needs a reference to a WebDriver. This is a problem when a single feature contains several scenarios as this reference needs to be shared among all scenarios. Possible solutions to this include:

    . A single scenario per feature. Every scenario will have to define its starting point. For our e-commerce checkout scenario, this defeats the purpose of testing itself. . A single scenario containing steps of all scenarios. In this case, all scenarios will be merged into a very long one. That makes for a hard-to-read scenario and an even harder to read (and maintain) class. . To be able to have multiple scenarios per feature while putting methods into their relevant step definitions, one needs to share the same driver instance among all step definitions. This can be achieved by applying the Singleton pattern to a dedicated class. . The last alternative is to use… +++DI+++! Actually, Cucumber https://cucumber.io/docs/reference/java-di[integrates^] quite nicely to integrate with some commons DI frameworks, including Weld and Spring.

    This is great news, as it’s possible to use the libraries we already use in development in the tests. Regular readers know me as a Spring proponent, so I naturally used it as the DI framework. Those are dependencies that are required for that in the +++POM+++:

    [source,xml]

    info.cukes cucumber-spring ${cucumber.version} test org.springframework spring-core ${spring.version} test org.springframework spring-beans ${spring.version} test org.springframework spring-context ${spring.version} test org.springframework spring-test ${spring.version} test

    At this point, it’s quite easy to create a standard Spring configuration file to generate the driver, as well as the page objects and their necessaries dependencies:

    [source,java]

    @Configuration open class AppConfiguration {

    @Bean open fun driver() = ChromeDriver()
    @Bean open fun contextConfigurator() = ContextConfigurator(properties)
    @Bean open fun homePage(contextConfigurator: ContextConfigurator) = HomePage(driver(), contextConfigurator)
    // Other page beans } ----
    

    In thie configuration, the driver bean is a singleton managed by Spring and the single instance can be shared among all page beans.

    Those are also singletons part of the Spring bean factory. That’s the reason of the @Autowired annotation in the step definition constructor. But why don’t step definitions get created in the Spring configuration class? Because they are to be created by the Cucumber framework itself using package scanning - yuck. Note that they don’t need to be self-annotated, it’s part of Cucumber’s magic but they still are part of the Spring context and can be injected.

    === Screen capture

    A common mistake made when testing is to think nothing will break, ever. I fell into this trap when I was more junior, and now I try to prepare for failure.

    In order to achieve that, I wanted to take a screenshot when a test fails so that it will be easier to fix the failure. Cucumber provides a dedicated lifecycle - before and after code run around each test through respective @Before and @After annotations. Note those are not the same as JUnit’s, and Cucumber doesn’t parse JUnit’s own annotations.

    The easy way would be to create an @After-annotated method in each step definition. Yet, that would just be code duplication. Cucumber also offers hooks, classes which annotated methods are run around each step definition test method. The only constraint for hooks is to place them in the same package as the step definitions so they can be discovered and managed by Cucumber Spring package scanning as for step definitions.

    [source,java]

    private val LOGGER = LoggerFactory.getLogger(ScreenCaptureHook::class.java) private var DATE_FORMAT = “yyyy-MM-dd-HH-mm-ss”

    class ScreenCaptureHook @Autowired constructor(driver: WebDriver) {

    private val screenshotFolder = File(File("target", "e2e"), "screenshots")
    private val takesScreenshot = driver as TakesScreenshot
    
    @Before
    fun ensureScreenshotFolderExists() {
        if (!screenshotFolder.exists()) {
            val folderCreated = screenshotFolder.mkdirs()
            if (!folderCreated) {
                LOGGER.warn("Could not create takesScreenshot folder. Screen capture won't work")
            }
        }
    }
    
    @After
    fun takeScreenshot(scenario: Scenario) {
        if (scenario.isFailed) {
            LOGGER.warn(scenario.status)
            LOGGER.info("Test failed. Taking takesScreenshot")
            val screenshot = takesScreenshot.getScreenshotAs(FILE)
            val format = SimpleDateFormat(DATE_FORMAT)
            val imageFile = File(screenshotFolder, format.format(Date()) + "-" + scenario.id + ".png")
            FileUtils.copyFile(screenshot, imageFile)
        }
    } } ----
    
    • Cucumber unfortunately doesn’t distinguish between a failed test and a skipped one. Thus, screenshots will be taken for the failed test as well as the skipped tests ran after it.
    • There’s no global hook in Cucumber. The ensureScreenshotFolderExists() will thus be ran before each step definition test method. This way requires to manage state, so as to initialize only once.

    == Conclusion

    The final result is a working end-to-end testing harness, that non-technical people are able to understand. Even better, using Cucumber over TestNG let us improve the code with Dependency Injection. The setup of Cucumber over Selenium was not trivial, but quite achievable with a little effort.

    Despite all the above good parts, the harness fails… sometimes, for reasons unrelated to code, as all end-to-end tests go. That’s the reason I’m extremely reluctant to set them up in the build process. If that would happen every so often, the team would lose confidence in the harness and all would have been for nothing. So far, we’ll use it manually before promoting the application to an environment.

  • Refactoring code for testability: an example

    :page-liquid: :experimental:

    Working on a legacy project those last weeks gave me plenty of material to write about tests, Mockito and PowerMock. Last week, I wrote about link:{% post_url 2015-12-13-on-powermock-abuse %}[abusing PowerMock^]. However, this doesn’t mean that you should never use PowerMock; only that if its usage is commonplace, it’s a code smell. In this article, I’d like to show an example how one can refactor legacy code to a more testable design with the temporary help of PowerMock.

    Let’s check how we can do that using the following code as an example:

    [source,java]

    public class CustomersReader {

    public JSONObject read() throws IOException {
        String url = Configuration.getCustomersUrl();
        CloseableHttpClient client = HttpClients.createDefault();
        HttpGet get = new HttpGet(url);
    
        try (CloseableHttpResponse response = client.execute(get)) {
            HttpEntity entity = response.getEntity();
            String result = EntityUtils.toString(entity);
            return new JSONObject(result);
        }
    } } ----
    

    Note that the Configuration class is outside our reach, in a third-party library. Also, for brevity’s sake, I cared only about the happy path; real-world code would probably be much more complex with failure handling.

    Obviously, this code reads an HTTP URL from this configuration, browse the URL and return its output wrapped into a JSONObject. The problem with that it’s that it’s pretty hard to test, so we’d better refactor it to a more testable design. However, refactoring is a huge risk, so we have to first create tests to ensure non-regression. Worst, unit tests do not help in this case, as refactoring will change classes and break existing tests.

    Before anything, we need tests to verify the existing behavior - whatever we can hack together, even if they don’t adhere to good practives. Two alternatives are possible:

    • Fakes: set up an HTTP server to answer the HTTP client and a database/file for the configuration class to read (depending on the exact implementation)
    • Mocks: create mocks and stub their behavior as usual

    Though PowerMock is dangerous, it’s less fragile and easy to set up than Fakes. So let’s start with PowerMock but only as a temporary measure. The goal is to refine both design and tests in parallel to that at the end, PowerMock will be removed. This test is a good start:

    [source,java]

    @RunWith(PowerMockRunner.class) public class CustomersReaderTest {

    @Mock private CloseableHttpClient client;
    @Mock private CloseableHttpResponse response;
    @Mock private HttpEntity entity;
    private CustomersReader customersReader;
    
    @Before
    public void setUp() {
        customersReader = new CustomersReader();
    }
    
    @Test
    @PrepareForTest({Configuration.class, HttpClients.class})
    public void should_return_json() throws IOException {
    
        mockStatic(Configuration.class, HttpClients.class);
    
        when(Configuration.getCustomersUrl()).thenReturn("crap://test");
        when(HttpClients.createDefault()).thenReturn(client);
        when(client.execute(any(HttpUriRequest.class))).thenReturn(response);
        when(response.getEntity()).thenReturn(entity);
    
        InputStream stream = new ByteArrayInputStream("{ \"hello\" : \"world\" }".getBytes());
        when(entity.getContent()).thenReturn(stream);
        JSONObject json = customersReader.read();
    
        assertThat(json.has("hello"));
        assertThat(json.get("hello")).isEqualTo("world");
    } } ----
    

    At this point, the test harness is in place and the design can change bit by bit (to ensure non-regression).

    The first problem is calling Configuration.getCustomersUrl(). Let’s introduce a service ConfigurationService class as a simple broker between the CustomersReader class and the Configuration class.

    [source,java]

    public class ConfigurationService {

    public String getCustomersUrl() {
        return Configuration.getCustomersUrl();
    } } ----
    

    Now, let’s inject this service into our main class:

    [source,java]

    public class CustomersReader {

    private final ConfigurationService configurationService;
    
    public CustomersReader(ConfigurationService configurationService) {
        this.configurationService = configurationService;
    }
    
    public JSONObject read() throws IOException {
        String url = configurationService.getCustomersUrl();
        // Rest of code unchanged
    } } ----
    

    Finally, let’s change the test accordingly:

    [source,java]

    @RunWith(PowerMockRunner.class) public class CustomersReaderTest {

    @Mock private ConfigurationService configurationService;
    @Mock private CloseableHttpClient client;
    @Mock private CloseableHttpResponse response;
    @Mock private HttpEntity entity;
    private CustomersReader customersReader;
    
    @Before
    public void setUp() {
        customersReader = new CustomersReader(configurationService);
    }
    
    @Test
    @PrepareForTest(HttpClients.class)
    public void should_return_json() throws IOException {
    
        when(configurationService.getCustomersUrl()).thenReturn("crap://test");
        // Rest of code unchanged
    } } ----
    

    The next step is to cut the dependency to the static method call to HttpClients.createDefault(). In order to do that, let’s delegate this call to another class and inject the instance into ours.

    [source,java]

    public class CustomersReader {

    private final ConfigurationService configurationService;
    private final CloseableHttpClient client;
    
    public CustomersReader(ConfigurationService configurationService, CloseableHttpClient client) {
        this.configurationService = configurationService;
        this.client = client;
    }
    
    public JSONObject read() throws IOException {
    
        String url = configurationService.getCustomersUrl();
        HttpGet get = new HttpGet(url);
    
        try (CloseableHttpResponse response = client.execute(get)) {
            HttpEntity entity = response.getEntity();
            String result = EntityUtils.toString(entity);
            return new JSONObject(result);
        }
    } } ----
    

    The final step is to remove PowerMock altogether. Easy as pie:

    [source,java]

    @RunWith(MockitoJUnitRunner.class) public class CustomersReaderTest {

    @Mock private ConfigurationService configurationService;
    @Mock private CloseableHttpClient client;
    @Mock private CloseableHttpResponse response;
    @Mock private HttpEntity entity;
    private CustomersReader customersReader;
    
    @Before
    public void setUp() {
        customersReader = new CustomersReader(configurationService, client);
    }
    
    @Test
    public void should_return_json() throws IOException {
    
        when(configurationService.getCustomersUrl()).thenReturn("crap://test");
        when(client.execute(any(HttpUriRequest.class))).thenReturn(response);
        when(response.getEntity()).thenReturn(entity);
    
        InputStream stream = new ByteArrayInputStream("{ \"hello\" : \"world\" }".getBytes());
        when(entity.getContent()).thenReturn(stream);
    
        JSONObject json = customersReader.read();
    
        assertThat(json.has("hello"));
        assertThat(json.get("hello")).isEqualTo("world");
    } } ----
    

    No trace of PowerMock whatsoever, neither in mocking static methods nor in the runner. We achieved a 100% testing-friendly design, according to our initial goal. Of course, this is a very simple example, real-life code is much more intricate. However, by changing code little bit by little bit with the help of PowerMock, it’s possible to achieve a clean design in the end.