• Scala vs Kotlin: inline and infix

    This is the third post in the Scala vs Kotlin comparison serie:

    1. Pimp my library
    2. Operator overloading

    This week, I’d like to address two features: inline and infix - not because they’re related but because neither of them would be enough to fill a post.

    Inlining comes from C (and then C++). In those languages, a hint could be provided to the compiler through the inline keyword. By doing so, it may replace an inlined function call by the function body itself in order to skip the overhead of a function call.

    Infix notation is to be in line with prefix and postfix, it’s relative to the place of the operator compared its 2 operands. Hopefully, the following example is clear enough:

    • Prefix: + 2 2
    • Postfix: 2 2 +
    • Infix: 2 + 2

    Scala

    Scala offers inlining through the @inline annotation on a function. As for C/C++, this is a an hint to the compiler. As per the ScalaDocs:

    An annotation on methods that requests that the compiler should try especially hard to inline the annotated method.

    The compiler has the final say in whether the function will be inlined, or not. On the opposite site, a function can be annotated with @noinline to prevent inlining altogether:

    An annotation on methods that forbids the compiler to inline the method, no matter how safe the inlining appears to be.

    As for infix annotation, it’s interestingly quite different from the definition above. In this context, it means that dot and parentheses can be omitted while calling functions that have a single parameter. There are some additional constraints:

    • Either the function must have no side-effects - be pure
    • Or the parameter must be a function
    1
    2
    val isLess1 = 1.<(2)
    val isLess2 = 1 < 2
    

    Lines 1 and 2 are equivalent. Obviously, line 2 is much more readable. Thanks to infix annotation, Scala doesn’t need operators, as every function can not only look but be called like an operator.

    Kotlin

    In Kotlin, inlining is set with the inline keyword. However, it’s much more than just a compiler hint: it’s a requirement. Whenever inline is used, the compiler will inline the function, no matter what.

    As such, it’s very important to use inlining only on small functions. Other limitations might include keeping its use to code under our control, e.g. to use it only for application code or code that is not part of a library’s public API.

    Note that inlining affects both the function itself as well as arguments that are lambdas. To make lambda arguments not inlined, use the noinline keyword.

    Infix notation is not automatic in Kotlin as it requires the function to be marked with the infix keyword. Additionally, the function needs to be attached to a class, either because it’s a member or an extension. Of course, the single parameter still applies.

    // Defined in Kotlin's runtime
    infix fun and(other: kotlin.Int): kotlin.Int { /* compiled code */ }
    
    val bool1 = 1.and(2)
    val bool2 = 1 and 2
    

    Be aware that infix notation only looks similar to an operator, it’s still a regular method call underneath.

    // This is valid
    val bool3 = 1 < 2
    
    // This is not valid, because < is an operator
    val bool4 = 1.<(2)
    
    Categories: Development Tags: scalakotlin
  • Starting with Cucumber for end-to-end testing

    Cucumber logo

    This week, my team decided to create a smoke test harness around our web app to avoid the most stupid regressions. I was not in favor of that, because of my prior experience with the fragility of end-to-end testing. But since we don’t have enough testers on our team, that was the only sane thing to do. I stepped forward to develop that suite.

    A simple TestNG MVP

    At first, I wanted to make a simple working test harness, so I chose technologies I was familiar with:

    • Selenium to manage browser interactions.
    • TestNG for the testing framework.

    TestNG is a much better choice that JUnit (even compared to the latest 5th version) for end-to-end testing because it lets you order test methods.

    However, the problem with this approach is readability when failures happen. Stack traces are hardly understandable by team members that are not developers. After having developed something that worked, I thus wanted to add some degree of usability for all team members.

    Migrating to Cucumber

    Cucumber is a BDD tool available on the JVM. It can be integrated with Selenium to have a thin BDD layer on top of GUI testing.

    Project structure

    Cucumber is based on 3 main components :

    • A feature, as its name implies, is a high-level feature of the system. It contains different scenarios, smaller-grained features that realize a feature. Each scenario is made in turn by a combination of step_s: _Given, When and Then that are well-know to BDD practitioners.
      In Cucumber, a feature is written in its own file Gherkin
    • The step definition is a Java file that implements in code the steps described in the feature
    • Last but not least, the test class is a JUnit (or TestNG) test class that binds the 2 former components

    For example, let’s analyze how to create a test harness for an e-commerce shop.

    The following is an excerpt of the feature to handle the the checkout:

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    Feature: Checkout
      A customer should be able to browse the shop,
                                   put an item in the cart,
                                   proceed to checkout,
                                   login
                                   and pay by credit card
    
      Scenario: Customer goes to the homepage and chooses a category
        Given the homepage is displayed
        Then the page should display 5 navigation categories
        And the page should display the search box
        When the first displayed category is chosen
        Then the customer is shown the category page
    
      # Other scenarios follow
    
    • On line 1 stands the name of the feature. The name should be short and descriptive.
    • Follows on line 2 a longer text that describes the feature in detail. It’s only meant for documentation.
    • The title of the scenario is set on line 8.
    • Initialization is set on line 9 with the Given keyword
    • On line 10 & 11 are some assertions via the Then keyword. Note that And could be replaced by Then but it feels more readable.
    • On line 12 is an interaction, the When keyword is used
    • And on line 13, we assert the state of the app again

    The corresponding step definition to the previous feature could look like that (in Kotlin):

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    class HomePageStepDef @Autowired constructor(page: HomePage) : En {
    
        init {
            Given("^the homepage is displayed$") { page.displayHomePage() }
            Then("^the page should display (\\d+) navigation categories$") { numberOfCategories: Int ->
                val displayedCategories = page.displayedCategories
                assertThat(displayedCategories).isNotNull().isNotEmpty().hasSize(numberOfCategories)
            }
            And("^the page should display the search box$") {
                val searchBox = page.searchBox
                assertThat(searchBox).isNotNull().isNotEmpty().hasSize(1)
            }
            When("^the first displayed category is chosen$") { page.chooseFirstDisplayedCategory() }
            Then("^the customer is shown the category page$") {
                assertThat(page.url).isNotNull().isNotEmpty().containsPattern("/c/")
            }
        }
    }
    

    Let’s forget for now about line 1 from the above snippet from above, included @Autowired and focus on the rest.

    For each Given/When/Then line in the feature file, there’s a corresponding method with a matching regexp in the class file:

    • Cucumber matches the step defined in the feature with the method by using the first parameter - the regexp.
    • Parameters can be defined in the feature and used in the step. As an example, compare line 10 of the first snippet with line 8 of the second: the regexp will capture the number of categories so it can be easily changed in the feature without additional development.
    • The second method parameter is the lambda that will get executed by Cucumber.
    • Given I’m using a Java runtime v8, those methods are default methods implemented in the En interface. There’s one such interface for each available language, so that step definitions can be implemented in your own language.
    • The class has no direct dependency on the Selenium API, it’s wrapped behind the Page Object pattern (see below).

    Finally, here’s the entry point test class:

    @RunWith(Cucumber::class)
    @CucumberOptions(
            features = arrayOf("classpath:feature/guest_checkout.feature"),
            glue = arrayOf("de.sansibar.glue"),
            format = arrayOf("pretty"))
    class GuestCheckoutIT
    

    As can be seen, it’s empty: it just provides the entry point and binds a feature to the step definitions package. At that point, running the test class in the IDE or through Maven will run the associated Cucumber feature.

    Improving beyond the first draft

    So far, so good. But the existing code deserves to be improved.

    Coping with fragility

    I cheated a little for this one as it was already implemented in the first TestNG MVP but let’s pretend otherwise.

    If you’ve read the step definition class above, you might have noticed that there’s no Selenium dependency anywhere in the code. All of it has been hidden in a class that represents the page:

    class HomePage(driver: WebDriver, private val contextConfigurator: ContextConfigurator): AbstractPage(driver) {
    
        val displayedCategories: List<WebElement> by lazy { driver.findElements(className("navigationBannerHome")) }
        val searchBox: List<WebElement> by lazy { driver.findElements(id("input_SearchBox")) }
    
        fun displayHomePage() {
            val url = contextConfigurator.url
            driver.get(url)
        }
    
        fun chooseFirstDisplayedCategory() {
            displayedCategories[0].click()
        }
    }

    This approach is known as the Page Object pattern.

    Mixing selectors and tests into the same class makes tests brittle, especially in the early stage of the project when the GUI changes a lot. Isolating selectors into a dedicated class let us buffer changes into that class only.

    There are a couple of good practices there - suggested by colleagues and from my personal experience:

    • Use id attributes on elements used for selection. This makes it less likely to break the test by changing the structure of the DOM.
    • Use coarse-grained methods mapped to a business case. For example, instead of having a whole bunch of selectTitle(), fillFirstName(), fillLastName(), submitRegistration(), etc. methods for each registration field, have a single register() method that inputs and submits the data. Again, this isolates possible breaking changes in the page class.

    Improved design

    The Page needs to select components through the Selenium API, thus it needs a reference to a WebDriver. This is a problem when a single feature contains several scenarios as this reference needs to be shared among all scenarios. Possible solutions to this include:

    1. A single scenario per feature. Every scenario will have to define its starting point. For our e-commerce checkout scenario, this defeats the purpose of testing itself.
    2. A single scenario containing steps of all scenarios. In this case, all scenarios will be merged into a very long one. That makes for a hard-to-read scenario and an even harder to read (and maintain) class.
    3. To be able to have multiple scenarios per feature while putting methods into their relevant step definitions, one needs to share the same driver instance among all step definitions. This can be achieved by applying the Singleton pattern to a dedicated class.
    4. The last alternative is to use… DI! Actually, Cucumber integrates quite nicely to integrate with some commons DI frameworks, including Weld and Spring.

    This is great news, as it’s possible to use the libraries we already use in development in the tests. Regular readers know me as a Spring proponent, so I naturally used it as the DI framework. Those are dependencies that are required for that in the POM:

    <!-- Cucumber Spring integration -->
    <dependency>
      <groupId>info.cukes</groupId>
      <artifactId>cucumber-spring</artifactId>
      <version>${cucumber.version}</version>
      <scope>test</scope>
    </dependency>
    <!-- Spring -->
    <dependency>
      <groupId>org.springframework</groupId>
      <artifactId>spring-core</artifactId>
      <version>${spring.version}</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.springframework</groupId>
      <artifactId>spring-beans</artifactId>
      <version>${spring.version}</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.springframework</groupId>
      <artifactId>spring-context</artifactId>
      <version>${spring.version}</version>
      <scope>test</scope>
    </dependency>
    <!-- Spring test - mandatory but not available via transitivity -->
    <dependency>
      <groupId>org.springframework</groupId>
      <artifactId>spring-test</artifactId>
      <version>${spring.version}</version>
      <scope>test</scope>
    </dependency>
    

    At this point, it’s quite easy to create a standard Spring configuration file to generate the driver, as well as the page objects and their necessaries dependencies:

    @Configuration
    open class AppConfiguration {
    
        @Bean open fun driver() = ChromeDriver()
        @Bean open fun contextConfigurator() = ContextConfigurator(properties)
        @Bean open fun homePage(contextConfigurator: ContextConfigurator) = HomePage(driver(), contextConfigurator)
        // Other page beans
    }
    

    In thie configuration, the driver bean is a singleton managed by Spring and the single instance can be shared among all page beans.

    Those are also singletons part of the Spring bean factory. That’s the reason of the @Autowired annotation in the step definition constructor. But why don’t step definitions get created in the Spring configuration class? Because they are to be created by the Cucumber framework itself using package scanning - yuck. Note that they don’t need to be self-annotated, it’s part of Cucumber’s magic but they still are part of the Spring context and can be injected.

    Screen capture

    A common mistake made when testing is to think nothing will break, ever. I fell into this trap when I was more junior, and now I try to prepare for failure.

    In order to achieve that, I wanted to take a screenshot when a test fails so that it will be easier to fix the failure. Cucumber provides a dedicated lifecycle - before and after code run around each test through respective @Before and @After annotations. Note those are not the same as JUnit’s, and Cucumber doesn’t parse JUnit’s own annotations.

    The easy way would be to create an @After-annotated method in each step definition. Yet, that would just be code duplication. Cucumber also offers hooks, classes which annotated methods are run around each step definition test method. The only constraint for hooks is to place them in the same package as the step definitions so they can be discovered and managed by Cucumber Spring package scanning as for step definitions.

    private val LOGGER = LoggerFactory.getLogger(ScreenCaptureHook::class.java)
    private var DATE_FORMAT = "yyyy-MM-dd-HH-mm-ss"
    
    class ScreenCaptureHook @Autowired constructor(driver: WebDriver) {
    
        private val screenshotFolder = File(File("target", "e2e"), "screenshots")
        private val takesScreenshot = driver as TakesScreenshot
    
        @Before
        fun ensureScreenshotFolderExists() {
            if (!screenshotFolder.exists()) {
                val folderCreated = screenshotFolder.mkdirs()
                if (!folderCreated) {
                    LOGGER.warn("Could not create takesScreenshot folder. Screen capture won't work")
                }
            }
        }
    
        @After
        fun takeScreenshot(scenario: Scenario) {
            if (scenario.isFailed) {
                LOGGER.warn(scenario.status)
                LOGGER.info("Test failed. Taking takesScreenshot")
                val screenshot = takesScreenshot.getScreenshotAs(FILE)
                val format = SimpleDateFormat(DATE_FORMAT)
                val imageFile = File(screenshotFolder, format.format(Date()) + "-" + scenario.id + ".png")
                FileUtils.copyFile(screenshot, imageFile)
            }
        }
    }
    
    • Cucumber unfortunately doesn’t distinguish between a failed test and a skipped one. Thus, screenshots will be taken for the failed test as well as the skipped tests ran after it.
    • There’s no global hook in Cucumber. The ensureScreenshotFolderExists() will thus be ran before each step definition test method. This way requires to manage state, so as to initialize only once.

    Conclusion

    The final result is a working end-to-end testing harness, that non-technical people are able to understand. Even better, using Cucumber over TestNG let us improve the code with Dependency Injection. The setup of Cucumber over Selenium was not trivial, but quite achievable with a little effort.

    Despite all the above good parts, the harness fails… sometimes, for reasons unrelated to code, as all end-to-end tests go. That’s the reason I’m extremely reluctant to set them up in the build process. If that would happen every so often, the team would lose confidence in the harness and all would have been for nothing. So far, we’ll use it manually before promoting the application to an environment.

  • Performance cost of reflection

    In Java, it’s widely admitted that reflection - usage of the java.reflect API, comes at a high cost in terms of performance. Older Java versions had huge performance overhead, while it seems that newer versions bring it in the acceptable range. But what does “acceptable” really mean?

    This is the question I asked when commenting on a performance review that advised to replace code based on reflection by standard code. As many of our decisions are not based on facts but on beliefs, I decided to perform some tests to get metrics in Java 8.

    Testing protocol

    In order to get realistic metrics through an unchallenged protocol, I used the excellent JMH testing framework. JMH has several advantages, among others:

    • An existing Maven artifact is readily available
    • Methods to benchmark have only to be annotated with @Benchmark
    • It handles the warming up of the JVM
    • It also handles the writing the results on the console

    Here’s a JMH snippet:

    @Benchmark
    public void executePerformanceTest() {
    	// Code goes here
    }
    

    JMH will take care of executing the above executePerformanceTest() and taking care of measuring the time taken.

    The code

    To highlight the cost of reflection, let’s check the difference between the time needed to access attributes with reflection and to call simple getters without.

    // With reflection
    Field firstName = clazz.getDeclaredField("firstName");
    Field lastName = clazz.getDeclaredField("lastName");
    Field birthDate = clazz.getDeclaredField("birthDate");
    Field.setAccessible(new AccessibleObject[] { firstName, lastName, birthDate }, true);
    firstName.get(person);
    lastName.get(person);
    birthDate.get(person);
    
    // Without reflection
    person.getFirstName();
    person.getLastName();
    person.getBirthDate();
    

    Checking possible optimizations

    I was wondering if immutable data structures were compiled into optimized bytecode that could decrease the performance overhead of reflection.

    Thus I created the same basic data structure in two-different ways:

    • One mutable with a no-args constructor and setters
    • One immutable with final attributes and constructor initialization

    Results

    Running the tests on my machine yields the following results:

    # Run complete. Total time: 00:26:55
    
    Benchmark                                    Mode  Cnt         Score        Error  Units
    BenchmarkRun.runImmutableWithReflection     thrpt  200   2492673.501 ±  37994.941  ops/s
    BenchmarkRun.runImmutableWithoutReflection  thrpt  200  26499946.587 ± 242499.198  ops/s
    BenchmarkRun.runMutableWithReflection       thrpt  200   2505239.277 ±  27697.028  ops/s
    BenchmarkRun.runMutableWithoutReflection    thrpt  200  26635097.050 ± 150798.911  ops/s
    

    For table-minded readers:

      Non-reflection Reflection
    Mutable 26,635,097.050 ± 150,798.911 2,505,239.277 ± 27,697.028
    Immutable 26,499,946.587 ± 242,499.198 2,492,673.501 ± 37,994.941

    And for bar chart interested readers (note the scale is linear):

    Test results

    Conclusion

    Whatever the way results are displayed, it will take about 10 times more to access a field with reflection than without… on my machine. My guess is that this can be extrapolated to be true on any machine.

    There’s a corollary conclusion: whatever you think holds true at a given time, you should always do some fact-checking to ensure a solid foundation for your decisions.

    If you want to run the tests yourself, the project is available online in Maven format.

    Categories: Java Tags: performancesoftware engineering
  • You don't talk about refactoring club

    The first rule of Fight Club is: You do not talk about Fight Club. The second rule of Fight Club is: You do not talk about Fight Club.

    — Tyler Durden

    I guess the same could be said about refactoring. That would first requires to define what I mean by refactoring in the context of this post:

    Refactoring is any action on the codebase that improves quality.

    Which in turn requires to define what is quality. Everyone I’ve talked with agrees on this: it’s quite hard to do. Let’s settle for now for the following tentative explanation:

    Quality is a feature of the codebase (including and not limited to architecture, design, etc.) which the lack of stalls further meaningful changes in the codebase. At the limits:

    • 100% quality means changing the codebase to develop a new feature would require the smallest possible time;
    • 0% quality means the time to do it would be infinite.

    Given this definition, refactoring includes:

    • Improving the design of classes
    • Adding unit tests
    • Removing useless code
    • Following accepted good practices
    • Anything related to improve readability
    • etc.

    Now back to the subject of this post. Should we ask the customer/manager if a refactoring is necessary? Should we put a refactoring sprint in the backlog? I’ve witnessed first hand many cases where it was asked. As expected, in nearly all cases, the decision was not to perform the refactoring. Taking ages to implement some feature? No design change. Not enough test harness? No tests added. Why? Because the customer/manager has no clue what refactoring and quality means.

    Let’s use a simple analogy: when I take my car to the mechanic, do I get to choose whether he’ll check if the repairs have been correctly executed? Not at all. Checks are part of the overall package I get when I choose a professional mechanic. If choice was made possible, some people who probably opt not to do the checks - to pay less. So far, so good. But then if trouble happened, and probability is in favor of that, the mechanic would be in deep trouble. Because he’s the professional and didn’t do his job well.

    Developers would also get into trouble if they delivered applications with no tests or with a messy codebase; not their customer nor their manager - especially not their managing (depending on their kind of manager if you catch my drift). So I wonder why developers have to let people that don’t know about code taking such important decisions.

    As a professional developer, you and no one else are responsible for the quality of the application you deliver. Your name is in the source code and the commit history, not your manager’s. Stop searching for excuses not to refactor: don’t ask, do it. Refactoring is part of the software development package, period.

    That doesn’t mean that you have to hide the fact that you’re refactoring, only that it’s up to you to decide if the code is good enough or need to be improved.

    Categories: Development Tags: refactoringsoftware engineering
  • Scala vs Kotlin: Operator overloading

    Last week, I started my comparison of Scala and Kotlin with the Pimp my library pattern. In the second part of this serie, I’d like to address operator overloading.

    Overview

    Before to dive into the nitty-gritty details, let’s try first to tell what it’s all about.

    In every language where there are functions (or methods), a limited set of characters is allowed to define the name of said functions. Some languages are more lenient toward allowed characters: naming a function \O/ might be perfectly valid.

    Some others are much more strict about it. It’s interesting to note that Java eschewed the ability to use symbols in function names besides $ - probably in response to previous abuses in older languages. It definitely stands on the less lenient part of the spectrum and the Java compiler won’t compile the previous \O/ function.

    The name operator overloading is thus slightly misleading, even if widespread. IMHO, it’s semantically more correct to talk about operator characters in function names.

    Scala

    Scala stands on the far side of leniency spectrum, and allows characters such as + and £ to be used to name functions, alone or in combinations. Note I couldn’t find any official documentation regarding accepted characters (but some helpful discussion is available here).

    This enables libraries to offer operator-like functions to be part of their API. One example is the foldLeft function belonging to the TraversableOnce type, which is also made available as the /: function.

    This allows great flexibility, especially in defining DSLs. For example, mathematics: functions can be named π, or . On the flip side, this flexibility might be subject to abuse, as \O/, ^_^ or even |-O are perfectly valid function names. Anyone for an emoticon-based API?

    def ∑(i: Int*) = i.sum
    
    val s = ∑(1, 2, 3, 5) // = 11
    

    Kotlin

    Kotlin stands on the middle of the leniency scale, as it’s possible to define only a limited set of operators.

    Each such operator has a corresponding standard function signature. To define a specific operator on a type, the associated function should be implemented and prepended with the operator keyword. For example, the + operator is associated with the plus() method. The following shows how to define this operator for an arbitrary new type and how to use it:

    class Complex(val i: Int, val j: Int) {
        operator fun plus(c: Complex) = Complex(this.i + c.i, this.j + c.j)
    }
    
    val c = Complex(1, 0) + Complex(0, 1) // = Complex(1, 1)
    

    Conclusion

    Scala’s flexibility allows for an almost unlimited set of operator-looking functions. This makes it suited to design DSL with a near one-to-one mapping between domains names and function names. But it also relies on implicitness: every operator has to be known to every member of the team, present and future.

    Kotlin takes a much more secure path, as it allows to define only a limited set of operators. However, those operators are so ubiquitous that even beginning software developer know them and their meaning (and even more so experienced ones).

    Categories: Development Tags: scalakotlin
  • Scala vs Kotlin: Pimp my library

    I’ve been introduced to the world of immutable data structures with the Scala programming language - to write I’ve been introduced to the FP world would sound too presumptuous. Although I wouldn’t recommend its usage in my day-to-day projects, I’m still grateful to it for what I learned: my Java code is now definitely not the same because Scala made me aware of some failings in both the language and my coding practices.

    On the other hand, I became recently much interested in Kotlin, another language that tries to bridge between the Object-Oriented and Functional worlds. In this serie of articles, I’d like to compare some features of Scala and Kotlin and how each achieve it.

    In this article, I’ll be tackling how both offer a way to improve the usage of existing Java libraries.

    Scala

    Let’s start with Scala, as it coined the term Pimp My Library 10 years ago.

    Scala’s approach is based on conversion. Consider a base type lacking the desired behavior. For example, Java’s double primitive type - mapped to Scala’s scala.Double type, is pretty limited.

    The first step is to create a new type with said behavior. Therefore, Scala provides a RichDouble type to add some methods e.g. isWhole().

    The second step is to provide an implicit function that converts from the base type to the improved type. The signature of such a function must follow the following rules:

    • Have a single parameter of the base type
    • Return the improved type
    • Be tagged implicit

    Here’s how the Scala library declares the Double to RichDouble conversion function:

    private[scala] abstract class LowPriorityImplicits {
        ...
        implicit def doubleWrapper(x: Double) = new runtime.RichDouble(x)
        ...
    }
    

    An alternative is to create an implicit class, which among other requirements must have a constructor with a single parameter of base type.

    The final step step is to bring the conversion in scope. For conversion functions, it means importing the function in the class file where the conversion will be used. Note that in this particular case, the conversion function is part of the automatic imports (there’s no need to explicitly declare it).

    At this point, if a function is not defined for a type, the compiler will look for an imported conversion function that transforms this type to a new type that provides this function. In that case, the type will be replaced with the conversion function.

    val x = 45d
    val isWhole = x.isWhole // Double has no isWhole() function
    
    // But there's a conversion function in scope which transforms Double to RichDouble
    // And RichDouble has a isWhole() function
    val isWhole = doubleWrapper(x).isWhole
    

    Kotlin

    One of the main reasons I’m cautious about using Scala is indeed the implicit part: it makes it much harder to reason about the code - just like AOP. Homeopathic usage of AOP is a life saver, widespread usage is counter-productive.

    Kotlin eschews implicitness: instead of conversions, it provides extension methods (and properties).

    Let’s analyze how to add additional behavior to the java.lang.Double type.

    The first step is to provide an extension function: it’s a normal function, but grafted to an existing type. To add the same isWhole() function as above, the syntax is the following:

    fun Double.isWhole() = this == Math.floor(this) && !java.lang.Double.isInfinite(this)
    

    As for Scala, the second step is to bring this function in scope. As of Scala, it’s achieved through an import. If the previous function has been defined in any file of the ch.frankel.blog package:

    import ch.frankel.blog.isWhole
    
    val x = 45.0
    val isWhole = x.isWhole // Double has no isWhole() function
    
    // But there's an extension function in scope for isWhole()
    val isWhole = x == Math.floor(x) && !java.lang.Double.isInfinite(x)
    

    Note that extension methods are resolved statically.

    Extensions do not actually modify classes they extend. By defining an extension, you do not insert new members into a class, but merely make new functions callable with the dot-notation on instances of this class.

    We would like to emphasize that extension functions are dispatched statically, i.e. they are not virtual by receiver type. This means that the extension function being called is determined by the type of the expression on which the function is invoked, not by the type of the result of evaluating that expression at runtime.

    Conclusion

    Obviously, Scala has one more indirection level - the conversion. I let anyone decide whether this is a good or a bad thing. For me, it makes it harder to reason about the code.

    The other gap is the packaging of the additional functions. While in Scala those are all attached to the enriched type and can be imported as a whole, they have to be imported one by one in Kotlin.

    Categories: Development Tags: scalakotlin
  • Fixing floating-point arithmetics with Kotlin

    This week saw me finally taking time to analyze our code base with Sonar. In particular, I was made aware of plenty of issues regarding floating-point arithmetics.

    Fun with Java’s floating-point arithmetics

    Those of you who learned Java in an academic context probably remember something fishy around FP arithmetics. Then if you never used them, you probably forgot about them. Here’s a very quick example of interesting it turns out to be:

    double a = 5.8d;
    double b = 5.6d;
    double sub = a - b;
    assertThat(sub).isEqualTo(0.2d);
    

    Contrary to common sense, this snippet throws an AssertionError: sub is not equal to 0.2 but to 0.20000000000000018.

    BigDecimal as a crutch

    Of course, no language worthy of the name could let that stand. BigDecimal is Java’s answer:

    The BigDecimal class provides operations for arithmetic, scale manipulation, rounding, comparison, hashing, and format conversion.

    Let’s update the above snippet with BigDecimal:

    BigDecimal a = new BigDecimal(5.8d);
    BigDecimal b = new BigDecimal(5.6d);
    BigDecimal sub = a.subtract(b);
    assertThat(sub).isEqualTo(new BigDecimal(0.2d));
    

    And run the test again… Oooops, it still fails:

    java.lang.AssertionError: 
    Expecting:
     <0.20000000000000017763568394002504646778106689453125>
    to be equal to:
     <0.200000000000000011102230246251565404236316680908203125>
    but was not.
    

    Using constructors changes nothing, one has to use the static valueOf() method instead.

    BigDecimal a = BigDecimal.valueOf(5.8d);
    BigDecimal b = BigDecimal.valueOf(5.6d);
    BigDecimal sub = a.subtract(b);
    assertThat(sub).isEqualTo(BigDecimal.valueOf(0.2d));
    

    Finally it works, but as the cost of a lot of ceremony…

    Kotlin to the rescue

    Just porting the code to Kotlin only marginally improves the readability:

    val a = BigDecimal.valueOf(5.8)
    val b = BigDecimal.valueOf(5.6)
    val sub = a.subtract(b)
    assertThat(sub).isEqualTo(BigDecimal.valueOf(0.2))
    

    Note that in Kotlin, floating-points numbers are doubles by default.

    In order to make the API more fluent and thus the code more readable, two valuable Kotlin features can be applied.

    The first one is extension method (I’ve already showed their use in a former post to improve logging with SLF4J. Let’s use it here to easily create BigDecimal objects from Double:

    fun Double.toBigDecimal(): BigDecimal = BigDecimal.valueOf(this)
    
    val a = 5.8.toBigDecimal() // Now a is a BigDecimal
    

    The second feature - coupled with method extension, is operator overloading. Kotlin sits between Java where operator overloading is impossible, and Scala where every operator can be overloaded (I’m wondering why there aren’t already any emoticons library): only some operators can be overloaded, including those from arithmetics - +, -, * and /.

    They can be overriden quite easily, as shown here:

    operator fun BigDecimal.plus(a: BigDecimal) = this.add(a)
    operator fun BigDecimal.minus(a: BigDecimal) = this.subtract(a)
    operator fun BigDecimal.times(a: BigDecimal) = this.multiply(a)
    operator fun BigDecimal.div(a: BigDecimal) = this.divide(a)
    
    val sub = a - b
    

    Note this is already take care of in Kotlin’s stdlib/

    The original snippet can now be written like this:

    val a = 5.8.toBigDecimal()
    val b = 5.6.toBigDecimal()
    assertThat(a - b).isEqualTo(0.2.toBigDecimal())
    

    The assertion line can probably be improved further. Possible solutions includes AssertJ custom assertions or… extension method again, in order for the isEqualTo() method to accept Double parameters.

    Conclusion

    Any complex API or library can be made easier to read by using Kotlin extension methods. What are you waiting for?

    Categories: Technical Tags: kotlinarithmetics
  • Comparing Lombok and Kotlin

    I know about Lombok since a long time, and I even wrote on how to create a new (at the time) @Delegate annotation. Despite this and even though I think it’s a great library, I’ve never used it in my projects. The reason for this mostly because I consider setting up the Lombok agent across various IDEs and build tools too complex for my own taste in standard development teams.

    Comes Kotlin which has support for IDEs and build tools right out-of-the-box plus seamless Java interoperability. So, I was wondering whether Lombok would still be relevant. In this article, I’ll check if Kotlin offers the same feature as Lombok and how. My goal is not to downplay Lombok’s features in a any way, but to perform some fact-checking to let you decide what’s the right tool for you.

    The following assumes readers are familiar with Lombok’s features.

    LombokSnippetKotlin equivalent
    val
    val e = new ArrayList<String>();
    val e = ArrayList<String>()
    @NonNull
    public void f(@NonNull String b) {
        ....
    }
    fun f(b: String) {
      ...
    }
    Kotlin types can be either nullable i.e. String? or not i.e. String. Such types do no inherit from one another.
    @Cleanup
    @Cleanup
    InputStream in =
        new FileInputStream("foo.txt")
    FileInputStream("foo.txt").use {
      // Do stuff here using 'it'
      // to reference the stream
    }
    • use() is part of Kotlin stdlib
    • Referencing the stream is not even necessary
    @NoArgsConstructor, @RequiredArgsConstructor and @AllArgsConstructor No equivalent as class and constructor declaration are merged
    @Getter
    public class Foo {
        @Getter
        private String bar;
    }
    class Foo() {
      var bar:String? = null
        private set
    }
    • Generates a private setter instead of no setter
    • Not idiomatic Kotlin (see below)
    public class Foo {
        @Getter
        private String bar;
        public Foo(String bar) {
            this.bar = bar;
        }
    }
    class Foo(val bar:String)
    Takes advantage of merging class and constructor declaration with slightly different semantics
    @Setter
    public class Foo {
        @Setter
        private String bar;
    }
    No equivalent but pretty unusual to just have a setter (does not favor immutability)
    public class Foo {
        @Setter
        private String bar;
        public Foo(String bar) {
            this.bar = bar;
        }
    }
    class Foo(var bar: String)
    @ToString No direct equivalent but included in data classes, see below
    @EqualsAndHashCode
    @Data
    @Data
    public class DataExample {
        private String name;
        private int age;
    }
    data class DataExample(
      var name: String,
      var age: Int)
    @Value
    @Value
    public class ValueExample {
      String name;
      int age;
    }
    data class ValueExample(
      val name: String,
      val age: Int)
    Notice the difference between val and var with the above snippet
    lazy=true
    public class GetterLazyExample {
      @Getter(lazy=true)
      private Double cached = expensive();
    }
    class GetterLazyExample {
      val cached by lazy {
        expensive()
      }
    }
    @Log
    @Log
    public class LogExample {
      public static void main(String... args) {
        log.error("Something's wrong here");
      }
    }
    No direct equivalent, but possible in many different ways
    @SneakyThrows
    @SneakyThrows
    public void run() {
     throw new Throwable();
    }
    fun run() {
      throw Throwable()
    }
    There's no such thing as checked exceptions in Kotlin. Using Java checked exceptions requires no specific handling.
    @Builder
    @Builder
    public class BuilderExample {
      private String name;
      private int age;
    }
    
    BuilderExample b = new BuilderExampleBuilder()
      .name("Foo")
      .age(42)
      .build();
    class BuilderExample(
      val name: String,
      val age: Int)
    
    val b = BuilderExample(
      name = "Foo",
      age = 42
    )
    No direct equivalent, use named arguments instead.
    @Synchronized
    public class SynchronizedExample {
      private final Object readLock = new Object();
      @Synchronized("readLock")
      public void foo() {
        System.out.println("bar");
      }
    }
    It's possible to omit to reference an object in the annotation: Lombok will create one under the hood (and use it).
    class SynchronizedExample {
      private val readLock = Object()
      fun foo() {
        synchronized(readLock) {
          println("bar")
        }
      }
    }
    synchronized() is part of Kotlin stdlib
    Experimental features
    LombokSnippetKotlin equivalent
    @ExtensionMethod
    class Extensions {
      public static String extends(String in) {
        // do something with in and
        return it;
      }
    }
    
    @ExtensionMethod(
      {String.class, Extensions.class}
    )
    public class Foor {
      public String bar() {
        return "bar".toTitleCase();
      }
    }
    fun String.extends: String {
      // do something with this and
      return it;
    }
    
    class Foo {
      fun bar = "bar".toTitleCase()
    }
    @Wither No direct equivalent, available through data classes
    @FieldDefaults
    class Foo(val bar: String = "bar")
    @Delegate Available in many different flavors
    @UtilityClass
    @UtilityClass
    public class UtilityClassExample {
      private final int CONSTANT = 5;
      public void addSomething(int in) {
        return in + CONSTANT;
      }
    }
    val CONSTANT = 5
    fun addSomething(i: Int) = i + CONSTANT
    @Helper
    public class HelperExample {
      int someMethod(int arg1) {
        int localVar = 5;
        @Helper class Helpers {
          int helperMethod(int arg) {
            return arg + localVar;
          }
        }
        return helperMethod(10);
      }
    }
    class HelperExample {
      fun someMethod(arg1: Int): Int {
        val localVar = 5
        fun helperMethod(arg: Int): Int {
          return arg + localVar
        }
        return helperMethod(10)
      }
    }

    In conclusion:

    • Lombok offers some more features
    • Lombok offers some more configuration options, but most don’t make sense taken separately (e.g. data classes)
    • Lombok and Kotlin have slightly different semantics when a feature is common so double-check if you’re a Lombok user willing to migrate

    Of course, this post completely left out Kotlin’s original features. This is a subject left out for future articles.

    Categories: Java Tags: lombokkotlin
  • Faster Mutation Testing

    As an ardent promoter of Mutation Testing, I sometimes get comments that it’s too slow to be of real use. This is always very funny as it also applies to Integration Testing, or GUI. Yet, this argument is only used againt Mutation Testing, though it cost nothing to setup, as opposed to the former. This will be the subject of another post. In this one, I will provide proposals on how to speed up mutation testing, or more precisely PIT, the Java Mutation Testing reference.

    Setting the bar

    A project reference for this article is required. Let’s use the codec submodule of Netty 4.1.

    At first, let’s compile the project to measure only PIT-related time. The following should be launched at the root of the Maven project hierarchy:

    mvn -pl codec clean test-compile
    

    The -pl option let the command be applied to only specified sub-projects - the codec sub-project in this case. Now just run the tests:

    mvn -pl codec surefire:test
    
    ...
    Results :
    
    Tests run: 340, Failures: 0, Errors: 0, Skipped: 0
    
    [INFO] ------------------------------------------------------------------------
    [INFO] BUILD SUCCESS
    [INFO] ------------------------------------------------------------------------
    [INFO] Total time: 8.954 s
    [INFO] Finished at: 2016-06-13T21:41:10+02:00
    [INFO] Final Memory: 12M/309M
    [INFO] ------------------------------------------------------------------------
    

    For all it’s worth, 9 seconds will be the baseline. This is not really precise measurement, but good enough for the scope of this article.

    Now let’s run PIT:

    mvn -pl codec -DexcludedClasses=io.netty.handler.codec.compression.LzfDecoderTest \
     org.pitest:pitest-maven:mutationCoverage
    

    Note: PIT flags the above class as failing, even though Surefire has no problem about that

    [INFO] ------------------------------------------------------------------------
    [INFO] BUILD SUCCESS
    [INFO] ------------------------------------------------------------------------
    [INFO] Total time: 14:14 min
    [INFO] Finished at: 2016-06-13T22:02:48+02:00
    [INFO] Final Memory: 14M/329M
    

    A whooping 14 minutes! Let’s try to reduce the time taken.

    Speed vs. relevancy trade-off

    There is no such thing as a free lunch.

    PIT offers a lot of tweaks to improve testing speed. However, most of them imply a huge drop in relevancy: faster means less feedback on code quality less than complete. As it’s the ultimate goal of Mutation Testing, it’s IMHO meaningless.

    Those configuration options are:

    • Set a limited a set of mutators
    • Limit scope of target classes
    • Limit number of tests
    • Limit dependency distance

    All benefits those flags bring are negated by less information.

    Running on multiple cores

    By default, PIT uses a single core even though most personal computers, not to mention servers have many more available.

    The easiest way to run faster is to use more cores. If it takes X minutes to run PIT on a single core, and if Y additional cores are 100% available, then it should only take X / (Y + 1) minutes to run on the first and additional cores.

    The number of cores is governed by the thread Java system property when launching Maven.

    mvn -pl codec -DexcludedClasses=io.netty.handler.codec.compression.LzfDecoderTest \
     -Dthreads=4 org.pitest:pitest-maven:mutationCoverage
    

    This yields the following results, which probably means that those additional cores were already performing some tasks during the run.

    [INFO] ------------------------------------------------------------------------
    [INFO] BUILD SUCCESS
    [INFO] ------------------------------------------------------------------------
    [INFO] Total time: 07:30 min
    [INFO] Finished at: 2016-06-13T22:25:38+02:00
    [INFO] Final Memory: 15M/318M
    

    Still, 50% is good enough for such a small configuration effort.

    Incremental analysis

    The rationale between incremental analysis is that unchanged tests running unchanged classes will produce the same outcome as the last one.

    Hence, PIT will create a hash for every test and production class. If they are similar, the test won’t run and PIT will reuse the same result. This just requires to configure where to store those hashes.

    Let’s run PIT with incremental analysis. The historyInputFile system property is the file where PIT will read hashes, historyOutputFile the file where it will write them. Obviously, they should point to the same file; anyone care to enlighten me as why they can be different? Anyway:

    mvn -pl codec -DexcludedClasses=io.netty.handler.codec.compression.LzfDecoderTest \
     -DhistoryInputFile=~/.fastermutationtesting -DhistoryOutputFile=~/.fastermutationtesting \
     -Dthreads=4 org.pitest:pitest-maven:mutationCoverage
    

    That produces:

    [INFO] ------------------------------------------------------------------------
    [INFO] BUILD SUCCESS
    [INFO] ------------------------------------------------------------------------
    [INFO] Total time: 07:05 min
    [INFO] Finished at: 2016-06-13T23:04:59+02:00
    [INFO] Final Memory: 15M/325M
    

    That’s about the time of the previous run… It didn’t run any faster! What could have happened? Well, that’s the first run so that hashes were not created. If PIT is ran again a second time with the exact same command-line, the output is now:

    [INFO] ------------------------------------------------------------------------
    [INFO] BUILD SUCCESS
    [INFO] ------------------------------------------------------------------------
    [INFO] Total time: 17.806 s
    [INFO] Finished at: 2016-06-13T23:12:43+02:00
    [INFO] Final Memory: 20M/575M
    

    That’s way better! Just the time to startup the engine, create the hashes and compare them.

    Using SCM

    An alternative to the above incremental analysis is to delegate change checks to the VCS. However, that requires the scm section to be adequately configured in the POM and the usage of the scmMutationCoverage goal in place of the mutationCoverage one.

    Note this is the reason why Maven should be launched at the root of the project hierarchy, to benefit from the SCM configuration in the root POM.

    mvn -pl codec -DexcludedClasses=io.netty.handler.codec.compression.LzfDecoderTest \
     -Dthreads=4 org.pitest:pitest-maven:scmMutationCoverage
    

    The output is the following:

    [INFO] No locally modified files found - nothing to mutation test
    [INFO] ------------------------------------------------------------------------
    [INFO] BUILD SUCCESS
    [INFO] ------------------------------------------------------------------------
    [INFO] Total time: 7.587 s
    [INFO] Finished at: 2016-06-14T22:11:29+02:00
    [INFO] Final Memory: 14M/211M
    

    This is even faster than incremental analysis, probably since there’s no hash comparison involved.

    Changing files without committing will correctly send the related mutants to be tested. Hence, this goal should be used only be developers on their local repository to check their changes before commit.

    Conclusion

    Now that an authoritative figure has positively written about Mutation Testing, more and more people will be willing to use it. In that case, the right configuration might make a difference between wide adoption and complete rejection.

    Categories: Java Tags: mutation testingqualityperformance
  • Smart logging in Java 8 and Kotlin

    Logging is a not a sexy subject but it’s important nonetheless. In the Java world, logging frameworks range from Log4J to SLF4J via Commons Logging and JDK logging (let’s exclude Log4J 2 for the time being). Though different in architecture and features, all of their API look the same. The logger has a method for each log level e.g.:

    • debug(String message)
    • info(String message)
    • error(String message)
    • etc.

    Levels are organized into a hierarchy. Once the framework is configured at a certain level, only messages logged with the same or a higher priority will be written.

    So far, so good. The problem comes when messages contain variables so that they must be concatenated.

    LOGGER.debug("Customer + " customer.getId() + " has just ordered " + item.getName() + " (" + item.getId() + ")");
    

    String concatenation has a definite performance cost in Java, and whatever the configured log level, it will occur.

    For this reason, modern logging frameworks such as SLF4J provide improved signature accepting a message format typed as String and variables as varargs of Objects. In that case, concatenation occurs only when the logger effectively writes.

    LOGGER.debug("Customer {} has just ordered {} ({})", customer.getId(), item.getName(), item.getId());
    

    Sometimes, however, variables are not readily available but have to be computed explicitly solely for the purpose of logging.

    LOGGER.debug("Customer {} has just ordered {}", customer.getId(), order.expensiveComputation());
    

    SLF4J doesn’t help there, as the method is evaluated even if the logger decides latter not to write because the framework is configured with a higher priority. In that case, it’s therefore advised to wrap the logger method call inside a relevant priority check.

    if (LOGGER.isDebug()) {
        LOGGER.debug("Customer {} has just ordered {}", customer.getId(), order.expensiveComputation());
    }
    

    That has to be done for every expensive method call, so it requires a strong coding discipline and reviews to make sure the wrapping occurs when it’s relevant - but only then. Besides, it decreases readability and thus maintainability. To achieve the same result automatically without those cons, one could use AOP at the cost of extra-complexity.

    Comes Java 8 and the Supplier<T> interface which returns a String, so that a method can be created like so:

    public void debug(Supplier<String> s) {
       if (LOGGER.isDebugEnabled()) {
            LOGGER.debug(s.get());
       }
    }
    

    In that case, the get() method is called only when the wrapping condition evaluates to true.

    Using this method is as simple as that:

    debug(() -> ("Customer + " customer.getId() + " has just ordered " + order.expensiveComputation());
    

    Great! But where to put this improved debug() method?

    • In the class itself: and duplicate for every class. Really?
    • In an utility class: one should add the LOGGER as the first parameter. Did you say cumbersome?
    • In the logger: one can create a wrapper and keep a standard logger as a delegate but the factory is final (at least in SLF4J) and has a lot of private methods.
    • In an aspect: back to square one…

    That’s the step where things start not being so nice in the Java realm.

    What about Kotlin? It comes with extension functions (and properties). This will probably be the subject of a future post since you should adopt Kotlin if only for this feature. Suffice to say that Kotlin can make it look like one can add state and behavior for already defined types.

    So debug() can be defined in a aptly named file:

    fun Logger.debug(s: () -> String) {
        if (isDebugEnabled) debug(s.invoke())
    }
    

    And calling it really feels like calling a standard logger, only a lambda is passed:

    LOGGER.debug { "Customer + " customer.getId() + " has just ordered " + order.expensiveComputation() }
    

    As a finishing touch, let’s make the defined function inlined. By doing that, the compiler will effectively replace every place the method is called with the method in the bytecode. We get the readable syntax benefit and avoid the overhead of un-wrapping the lambda at runtime:

    inline fun Logger.debug(s: () -> String) {
        if (isDebugEnabled) debug(s.invoke())
    }
    

    Note that it can still be called in Java, albeit with a not-so-nice syntax:

    Slf4KUtilsKt.debug(LOGGER, () -> "Customer + " + customer.getId() + " has just ordered " + order.expensiveComputation());
    

    Also note that Log4J 2 already implements this feature out-of-the-box.

    At this point, it’s simple enough to copy-code the above snippet when required. But developers are lazy by nature so I created a full-fledged wrapper around SLF4J methods. The source code is available on Github and the binary artifact on Bintray so that you only need the following dependency to use it:

    <dependency>
      <groupId>ch.frankel.log4k</groupId>
      <artifactId>slf4k-api</artifactId>
      <version>1.0.0</version>
    </dependency>
    
    Categories: Development Tags: kotlinloggingperformance