• A SonarQube plugin for Kotlin - Analyzing with ANTLR

    SonarQube Continuous Inspection logo

    Last week, we used ANTLR to generate a library to be able to analyze Kotlin code. It’s time to use the generated API to check for specific patterns.

    API overview

    Let’s start by having a look at the generated API:

    • KotlinLexer: Executes lexical analysis.
    • KotlinParser: Wraps classes representing all Kotlin tokens, and handles parsing errors.
    • KotlinParserVisitor: Contract for implementing the Visitor pattern on Kotlin code. KotlinParserBaseVisitor is its empty implementation, to ease the creation of subclasses.
    • KotlinParserListener: Contract for callback-related code when visiting Kotlin code, with KotlinParserBaseListener its empty implementation.

    Class diagrams are not the greatest diagrams to ease the writing of code. The following snippet is a very crude analysis implementation. I’ll be using Kotlin, but any JVM language interoperable with Java could be used:

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    val stream = CharStreams.fromString("fun main(args : Array<String>) {}")
    val lexer = KotlinLexer(stream)
    val tokens = CommonTokenStream(lexer)
    val parser = KotlinParser(tokens)
    val context = parser.kotlinFile()
    ParseTreeWalker().apply {
        walk(object : KotlinParserBaseListener() {
            override fun enterFunctionDeclaration(ctx: KotlinParser.FunctionDeclarationContext) {
                println(ctx.SimpleName().text)
            }
        }, context)
    }
    

    Here’s the explanation:

    1. Create a CharStream to feed the lexer on the next line. The CharStreams offers plenty of static fromXXX() methods, each accepting a different type (String, InputStream, etc.)
    2. Instantiate the lexer, with the stream
    3. Instantiate a token stream over the lexer. The class provides streaming capabilities over the lexer.
    4. Instantiate the parser, with the token stream
    5. Define the entry point into the code. In that case, it’s a Kotlin file - and probably will be for the plugin.
    6. Create the overall walker that will visit each node in turn
    7. Start the visiting process by calling walk and passing the desired behavior as an object
    8. Override the desired function. Here, it will be invoked every time a function node is entered
    9. Do whatever is desired e.g. print the function name

    Obviously, lines 1 to 7 are just boilerplate to wire all components together. The behavior that need to be implemented should replace lines 8 and 9.

    First simple check

    In Kotlin, if a function returns Unit - nothing, then explicitly declaring its return type is optional. It would be a great rule to check that there’s no such explicit return. The following snippets, both valid Kotlin code, are equivalent - one with an explicit return type and the other without:

    fun hello1(): Unit {
        println("Hello")
    }
    
    fun hello2() {
        println("Hello")
    }
    

    Let’s use grun to graphically display the parse tree (grun was explained in the previous post). It yields the following:

    As can be seen, the snippet with an explicit return type has a type branch under functionDeclaration. This is confirmed by the snippet from the KotlinParser ANTLR grammar file:

    functionDeclaration
      : modifiers 'fun' typeParameters?
          (type '.' | annotations)?
          SimpleName
          typeParameters? valueParameters (':' type)?
          typeConstraints
          functionBody?
          SEMI*
      ;
    

    The rule should check that if such a return type exists, then it shouldn’t be Unit. Let’s update the above code with the desired effect:

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    ParseTreeWalker().apply {
        walk(object : KotlinParserBaseListener() {
            override fun enterFunctionDeclaration(ctx: KotlinParser.FunctionDeclarationContext) {
                if (ctx.type().isNotEmpty()) {
                    val typeContext = ctx.type(0)
                    with(typeContext.typeDescriptor().userType().simpleUserType()) {
                        val typeName = this[0].SimpleName()
                        if (typeName.symbol.text == "Unit") {
                            println("Found Unit as explicit return type " +
                            	"in function ${ctx.SimpleName()} at line ${typeName.symbol.line}")
                        }
                    }
                }
            }
        }, context)
    }
    

    Here’s the explanation:

    • Line 4: Check there’s an explicit return type, whatever it is
    • Line 5: Strangely enough, the grammar allows for a multi-valued return type. Just take the first one.
    • Line 6: Follow the parse tree up to the final type name - refer to the above parse tree screenshot for a graphical representation of the path.
    • Line 8: Check that the return type is Unit
    • Line 9: Prints a message in the console. In the next step, we will call the SonarQube API there.

    Running the above code correctly yields the following output:

    Found Unit as explicit return type in function hello1 at line 1
    

    A more advanced check

    In Kotlin, the following snippets are all equivalent:

    fun hello1(name: String): String {
        return "Hello $name"
    }
    
    fun hello2(name: String): String = "Hello $name"
    
    fun hello3(name: String) = "Hello $name"
    

    Note that in the last case, the return type can be inferred by the compiler and omitted by the developer. That would make a good check: in the case of a expression body, the return type should be omitted. The same technique as above can be used:

    1. Display the parse tree from the snippet using grun:

    2. Check for differences. Obviously:
      • Functions that do not have an explicit return type miss a type node in the functionDeclaration tree, as above
      • Functions with an expression body have a functionBody whose first child is = and whose second child is an expression
    3. Refer to the initial grammar, to make sure all cases are covered.
      functionBody
        : block
        | '=' expression
        ;
      
    4. Code!
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    ParseTreeWalker().apply {
        walk(object : KotlinParserBaseListener() {
            override fun enterFunctionDeclaration(ctx: KotlinParser.FunctionDeclarationContext) {
                val bodyChildren = ctx.functionBody().children
                if (bodyChildren.size > 1
                        && bodyChildren[0] is TerminalNode && bodyChildren[0].text == "="
                        && ctx.type().isNotEmpty()) {
                    val firstChild = bodyChildren[0] as TerminalNode
                    println("Found explicit return type for expression body " +
                            "in function ${ctx.SimpleName()} at line ${firstChild.symbol.line}")
                }
            }
        }, context)
    }
    

    The code is pretty self-explanatory and yields the following:

    Found explicit return type for expression body in function hello2 at line 5
    
    Categories: Technical Tags: code qualitySonarQubeKotlinplugin
  • A SonarQube plugin for Kotlin - Paving the way

    SonarQube Continuous Inspection logo

    Since I started my journey into Kotlin, I wanted to use the same libraries and tools I use in Java. For libraries - Spring Boot, Mockito, etc., it’s straightforward as Kotlin is 100% interoperable with Java. For tools, well, it depends. For example, Jenkins works flawlessly, while SonarQube lacks a dedicated plugin. The SonarSource team has limited resources: Kotlin, though on the rise - and even more so since Google I/O 17, is not in their pipe. This post serie is about creating such a plugin, and this first post is about parsing Kotlin code.

    ANTLR

    In the realm of code parsing, ANTLR is a clear leader in the JVM world.

    ANTLR (ANother Tool for Language Recognition) is a powerful parser generator for reading, processing, executing, or translating structured text or binary files. It’s widely used to build languages, tools, and frameworks. From a grammar, ANTLR generates a parser that can build and walk parse trees.

    Designing the grammar

    ANTLR is able to generate parsing code for any language thanks to a dedicated grammar file. However, creating such a grammar from scratch for regular languages is not trivial. Fortunately, thanks to the power of the community, a grammar for Kotlin already exists on Github.

    With this existing grammar, ANTLR is able to generate Java parsing code to be used by the SonarQube plugin. The steps are the following:

    • Clone the Github repository
      git clone [email protected]:antlr/grammars-v4.git
      
    • By default, classes will be generated under the root package, which is discouraged. To change that default:
      • Create a src/main/antlr4/<fully>/<qualified>/<package> folder such as src/main/antlr4/ch/frankel/sonarqube/kotlin/api
      • Move the g4 files there
      • In the POM, remove the sourceDirectory and includes bits from the antlr4-maven-plugin configuration to use the default
    • Build and install the JAR in the local Maven repo
      cd grammars-v4/kotlin
      mvn install
      

    This should generate a KotlinLexer and a KotlinParser class, as well as several related classes in target/classes. As Maven goes, it also packages them in a JAR named kotlin-1.0-SNAPSHOT.jar in the target folder - and in the local Maven repo as well.

    Testing the parsing code

    To test the parsing code, one can use the grun command. It’s an alias for the following:

    java -Xmx500M -cp "<path/to/antlr/complete/>.jar:$CLASSPATH" org.antlr.v4.Tool
    

    Create the alias manually or install the antlr package via Homebrew on OSX.

    With grun, Kotlin code can parsed then displayed in different ways, textual and graphical. The following expects an input in the console:

    cd target/classes
    grun Kotlin kotlinFile -tree
    

    After having typed valid Kotlin code, it yields its parse tree in text. By replacing the -tree option with the -gui option, it displays the tree graphically instead. For example, the following tree comes from this snippet:

    fun main(args : Array<String>) { 
        val firstName : String = "Adam"
        val name : String? = firstName 
        print("$name") 
    }
    

    In order for the JAR to be used later in the SonarQube plugin, it has been deployed on Bintray. In the next post, we will be doing proper code analysis to check for violations.

    Categories: Technical Tags: code qualitySonarQubeKotlinpluginANTLR
  • It depends

    Light bulb

    In the industry, there’s this widespread joke. Whatever the question you ask a consultant, the answer will be:

    It depends.

    This joke is meant to highlight that consultants never give a straight answer to a simple question, because they don’t want to take any responsibility. While I understand the frustration of the business when faced with this situation, I’d like to write about the other side of the fence.

    Let’s enlarge the scope: in the IT industry, it’s probably more about every developer rather than only consultants. But why is this answer so common? For one single reason: if the question - deemed simple by the person asking, doesn’t provide enough context - and they rarely do, then the answer cannot be any other.

    Let’s highlight that with a simple question:

    What’s the best transportation?

    Guess what? There’s no right answer to that, because no context has been provided. For example, context parameters that could help refine the answer include:

    • the distance, do you expect to move 200 meters or 200 kilometers?
    • the nature of the terrain, road vs. land vs. water
    • the time constraint
    • the expected comfort level
    • the cost
    • the physical condition - it’s hard to consider running/biking when in bad shape
    • for some, the pollution
    • etc.

    I hope this pretty down-to-earth example makes those who ask questions realize that the quality of the answer is heavily constrained by the quality of the question - garbage in, garbage out.

    I could stop there and everyone would ponder how stupid the business is, but every coin has 2 sides. In general, developers don’t react in a constructive way to context-less questions: "It depends", "I don’t know", "I have no clue", "Nobody can answer that", or even "That’s a stupid question" are comments I’ve heard already. If you’re a developer and reading this post, know it’s your job to guide the business. What do they know about scalability, databases, and so on? Probably nothing because it’s not their job. So, pro-actively guide them through questions. With the example above, the following could be an actual dialog:

    "What’s the best transportation?
    - Where do you want to go?
    - Munich
    - Ok, what’s your starting point?
    - Home
    - Ah…​ Where’s you home located?
    - Germany
    - Which city in Germany?
    - etc."

    I’m very aware that most developers - including myself, don’t like to talk to the business so much. I mean, that’s the reason most of us chose to work in IT, because we are more comfortable talking to computers than people. But in essence, it changes nothing.

    Be professional, ask for context instead of ditching the question.

    Categories: Miscellaneous Tags: communication
  • What archive format should you use, WAR or JAR?

    Female hitting a male with boxing gloves on

    Some time ago, RAM and disk space were scarce resources. At that time, the widespread strategy was to host different applications onto the same platform. That was the golden age of the application server. I wrote in an earlier post that the current tendency toward cheaper resources will make it obsolete, in the short or long term. However, a technology trend might bring it back in favor.

    Having an application server is good when infrastructure resources are expensive, and sharing them across apps brings a significant cost decrease. On the down side, it requires a deep insight into the load of each application sharing the same resources, as well as skilled sysadmins that can deploy on the same app server applications that are compatible. For old-timers, does an application requiring to be run alone because it mismanages resources ring a bell? When infrastructure costs decrease, laziness and aversion to risk take precedence over them, and hosting a single app on an application server become the norm. At that point, the next logical step is to consider why application servers as dedicated components are still required. It seems the Spring guys came to the same conclusion, for Spring Boot applications' default mode is to package executable JARs - also known as Fat JARs. Those apps can be run as java -jar fat.jar. Hence the famous:

    Make JAR, not WAR

    — Josh Long

    I’m still not completely sold on that, as I believe it too easily discards the expertise of most Ops teams regarding application servers' management. However, one compelling argument about Fat JARs is that since the booting technology is in charge of app management from the start, it can handle load classes in any way it wants. For example, with Dev Tools, Spring Boot provides a mechanism based on two classloaders, one for libraries and one for classes, so that classes can be changed and reloaded without restarting the whole JVM - a neat trick that gives a very fast feedback loop at each code change.

    It wrongly thought that application server providers were still stuck with the legacy way - thanks to Ivar Grimstad for making me aware of this option (a good reason to visit talks that do not necessarily target your interest at conferences). Wildlfy, TomEE and other app server implementers can be configured to package Fat JARs as well, albeit with one huge difference: there’s nothing like Spring Dev Tools, so the restart of the whole app server is still required when code changes. The only alternative for faster feedback regarding those changes is to work at a lower level e.g. JRebel licenses for the whole team. However, there’s still one reason to use WAR archives, and that reason is Docker. By providing a common app server Docker image as a base image, one just needs to add one’s WAR on top of it, thus making the WAR image quite lightweight. And this cannot be achieved (yet?) with the JAR approach.

    Note that it’s not Spring Boot vs JavaEE but mostly JAR vs WAR, as Spring Boot is perfectly able to package either format, while many app server providers as well. As I pointed out above, the only missing piece is for the later to reload classes instead of restarting the whole JVM when a change occurs - but I believe it will happen at some point.

    Choosing between the WAR and the JAR approaches is highly dependent whether the company values more fast feedback cycles during development or more optimized and manageable Docker images.

    Categories: Java Tags: WARJARJavaEESpring Bootarchive
  • Why are you testing your software?

    Burned bus at a bus stop

    15 years ago, automated tests didn’t exist in the Java ecosystem. One had to build the application and painfully test it manually by using it. I was later introduced to the practice of adding a main method to every class and putting some testing code there. That was only marginally better, as it still required to manually run the methods. Then came JUnit, the reference unit testing framework in Java which brought test execution automation. At that point, I had to convince teams I was part of that we had to use it to create an automated tests harness to prevent regression bugs. Later, this became an expectation: no tests meant no changes in the code for fear of breaking something.

    More recently, however, it happened that I sometimes have to advocate for the opposite: not write too many tests. Yes, you read that well, and yet I’m no turncoat. The reason for this lies in the title of the post: why are you testing your software? It may sound like the answer to this question is pretty obvious - but it’s not, and the answer is tightly coupled to the concept of quality.

    In the context of software engineering, software quality refers to two related but distinct notions that exist wherever quality is defined in a business context:

    • Software functional quality reflects how well it complies with or conforms to a given design, based on functional requirements or specifications. That attribute can also be described as the fitness for purpose of a piece of software or how it compares to competitors in the marketplace as a worthwhile product. It is the degree to which the correct software was produced.
    • Software structural quality refers to how it meets non-functional requirements that support the delivery of the functional requirements, such as robustness or maintainability. It is the degree to which the software was produced correctly.
    — Wikipedia

    Before answering the question of the why, let’s consider some reason why not I’ve already been confronted to:

    • Because everyone does it
    • Because the boss/the lead/colleagues/authority figures say so
    • To achieve 100% of code coverage
    • To achieve more code coverage than another team/colleague
    • And so on, and so forth

    All in all, those "reasons" boil down to either plain cargo culting or mistaking a metric for the goal. Which brings us back to the question, why do you test software?

    The only valid reason for testing is that resources spent in making sure the software conforms to non-/functional requirements will be less over the course of time than the resources spent if not done.

    That’s pure and simple Return Over Investment. If ROI is positive, do test; if it’s negative, don’t. It’s as simple as that.

    Perfect is the enemy of good

    The real difficulty lies in estimating the cost of testing vs the cost of not testing. The following is a non-exhaustive list of ROI-influencing parameters:

    Business

    No bugs are allowed in some industries e.g. medical, airplanes, banks without serious consequences to the business, while this is less critical for others such as mobile gaming.

    Estimated lifespan of the app

    the longer the lifespan of an app, the better the ROI because the same amount of testing code will yield more times e.g. nearly no tests for one-shot single-event apps vs. more traditional testing for tradition business apps running for a decade or so.

    Nature of the app

    some technologies are more mature than others, allowing for easier automated testing. The testing echosystem around webapps is richer than around native or mobile apps.

    Architecture of the app

    The more distributed the app, the harder it is to test. In particular, the migration from monoliths to microservices has some interesting side-effects on the testing side. It’s easier to test each component separately, but harder to test the whole system. Also, testing specific scenarios in clustered/distributed environments, such as node failure in a cluster, increase the overall cost.

    Nature and number of infrastructure dependencies

    The higher the number of dependencies, the more test doubles are required to test the app in isolation, which in turns drive up testing costs. Also, some dependencies are more widespread e.g. databases and web services, with many available tools while some are not e.g. FTP servers.

    Size of the app

    Of course, the bigger the size of the app, the bigger the number of possible combinations that needs to be tested.

    Maturity of the developers, and the size of the team(s)

    Obviously, developers range from the ones who don’t care about testing to those who integrate testing requirements in their code from the start. Also, just for developers, adding more testers is subject to the law of diminishing returns.

    Nature of the tests

    I don’t want to start a war, suffice to say there are many kinds of tests - unit, integration, end-to-end, performance, penetration, etc. Each one is good at one specific thing, and has pros and cons. Get to know them and use them wisely.

    Strength of the type system

    Developing in dynamically-typed languages require more tests to handle the job of the compiler in comparison to more statically-typed languages.

    While it’s good to listen to other’s advices - including well-established authority figures and this post, it’s up to every delivery team to draw the line between not enough testing and too much testing according to its own context.

    Categories: Java Tags: code qualitytestingcode coverage
  • Concurrency: Java Futures and Kotlin Coroutines

    A long time ago, one had to manually start new threads when wanting to run code concurrently in Java. Not only was this hard to write, it was easy to introduce bugs that were hard to find. Testing, reading and maintaining such code was no walk in the park either. Since that time - and with a little incentive coming from multi-core machines, the Java API has evolved to make developing concurrent code easier. Meanwhile, alternative JVM languages also have their opinion about helping developers write such code. In this post, I’ll compare how it’s implemented in Java and Kotlin.

    To keep the post focused, I deliberately left out performance to write about code readability.

    About the use-case

    The use-case is not very original. We need to call different web services. The naive solution would be to call them sequentially, one after the other, and collect the result of each of them. In that case, the overall call time would be the sum of the call time of each service. An easy improvement is to call them in parallel, and wait for the last one to finish. Thus, performance improves from linear to constant - or for the more mathematically inclined, from o(n) to o(1).

    To simulate calling a web service with a delay, let’s use the following code (in Kotlin because this is so much less verbose):

    class DummyService(private val name: String) {
    
        private val random = SecureRandom()
    
        val content: ContentDuration
            get() {
                val duration = random.nextInt(5000)
                Thread.sleep(duration.toLong())
                return ContentDuration(name, duration)
            }
    }
    
    data class ContentDuration(val content: String, val duration: Int)
    

    The Java Future API

    Java offers a whole class hierarchy to handle concurrent calls. It’s based on the following classes:

    Callable
    A Callable is a "task that returns a result". From another view point, it's similar to a function that takes no parameter and returns this result.
    Future
    A Future is "the result of an asynchronous computation". Also, "The result can only be retrieved using method get when the computation has completed, blocking if necessary until it is ready". In other words, it represents a wrapper around a value, where this value is the outcome of a calculation.
    Executor Service
    An ExecutorService "provides methods to manage termination and methods that can produce a Future for tracking progress of one or more asynchronous tasks". It is the entry point into concurrent handling code in Java. Implementations of this interface - as well are more specialized ones, can be obtained through static methods in the Executors class.

    This is summarized in the following class diagram:

    Calling our services using the concurrent package is a 2-steps process.

    Creating a collection of callables

    First, there need to be a collection of Callable to pass to the executor service. This is how it might go:

    1. From a stream of service names
    2. For each service name, create a new dummy service initialized with the string
    3. For every service, return the service’s getContent() method reference as a Callable. This works because the method signature, matches Callable.call() and Callable is a functional interface.

    This is the preparation phase. It translates into the following code:

    1
    2
    3
    4
    List<Callable<ContentDuration>> callables = Stream.of("Service A", "Service B", "Service C")
          .map(DummyService::new)
          .map(service -> (Callable<ContentDuration>) service::getContent)
          .collect(Collectors.toList());
    

    Processing the callables

    Once the list has been prepared, it’s time for the ExecutorService to process it aka the “real work”.

    1. Create a new executor service - any will do
    2. Pass the list of Callable to the executor service, and stream the resulting list of Future
    3. For every future,
    4. Either return the result
    5. Or handle the exception

    The following snippet is a possible implementation:

    1
    2
    3
    4
    5
    6
    ExecutorService executor = Executors.newWorkStealingPool();
    List<ContentDuration> results = executor.invokeAll(callables).stream()
        .map(future -> {
             try { return future.get(); } 
             catch (InterruptedException | ExecutionException e) { throw new RuntimeException(e); }
         }).collect(Collectors.toList());
    

    The Future API, but in Kotlin

    Let’s face it, while Java makes it possible to write concurrent code, reading and maintaining it is not that easy, mainly due to:

    • Going back and forth between collections and streams
    • Handling checked exception in lambdas
    • Casting explicitly

    Just porting the above code to Kotlin removes those limitations and makes it more straightforward:

    var callables: List<Callable<ContentDuration>> = arrayOf("Service A", "Service B", "Service C")
        .map { DummyService(it) }
        .map { Callable<ContentDuration> { it.content } }
    
    val executor = Executors.newWorkStealingPool()
    val results = executor.invokeAll(callables).map { it.get() }
    

    Kotlin Coroutines

    With version 1.1 of Kotlin comes a new experimental feature called coroutines.

    Basically, coroutines are computations that can be suspended without blocking a thread. Blocking threads is often expensive, especially under high load […]. Coroutine suspension is almost free, on the other hand. No context switch or any other involvement of the OS is required.

    The leading design principle behind coroutines is that they must feel like sequential code but run like concurrent code. They are based on the following (simplified) class diagram:

    Nothing beats the code itself though. Let’s implement the same as above, but with coroutines in Kotlin instead of Java futures.

    As a pre-step, let’s just extend the service to ease further processing by adding a new computed property wrapped around content, of type Deferred:

    val DummyService.asyncContent: Deferred<ContentDuration>
        get() = async(CommonPool) { content }
    

    This is standard Kotlin extension property code, but notice the CommonPool parameter. This is the magic that makes the code run concurrent. It’s a companion object (i.e. a singleton) that uses a multi-fallback algorithm to get an ExecutorService instance.

    Now, on to the code flow proper:

    1. Coroutines are handled inside a block. Declare a variable list outside the block to be assigned inside it
    2. Open the synchronization block
    3. Create the array of service names
    4. For each name, create a service and return it
    5. For each service, get its async content (declared above) and return it
    6. For each deferred, get the result and return it
    1
    2
    3
    4
    5
    6
    7
    8
    // Variable must be initialized or the compiler complains
    // And the variable cannot be used afterwards
    var results = runBlocking {
        arrayOf("Service A", "Service B", "Service C")
            .map { DummyService(it) }
            .map { it.asyncContent }
            .map { it.await() }
    }
    

    Takeaways

    The Future API is not so much a problem than the Java language itself. As soon as the code is translated into Kotlin, readability improves a lot. Yet, having to create a collection to pass to the executor service breaks the nice functional pipeline.

    On the coroutines side, remember that they are still experimental. Despite that, code does look sequential - and is thus more readable, and behaves parallel.

    Categories: Java Tags: futurekotlincoroutine
  • Java Service Loader vs Spring Factories Loader

    Coloured factory

    Inversion of Control is not only possible at the class level, but at the module level. OSGi has been doing it for a long time. However, there are IoC approaches directly available in Java, as well as in Spring.

    Java Service Loader

    Out-of-the-box, the Java API offers a specific form of Inversion of Control. It’s implemented by the Service Loader class. It is designed to locate implementation classes of an interface on the classpath. This way allows to discover which available implementations of an interface are available on the classpath at runtime, and thus paves the way for modules designed around a clean separation between an API module - i.e. JAR, and multiple implementation modules.

    This is the path chosen by the logging framework, SLF4J. SLF4J itself is just the API, while different implementations are available (_e.g. Logback, Log4J, etc.). SLF4J clients only interact with the SLF4J API, while the implementation available on the classpath takes care of the nitty-gritty details at runtime.

    It’s implemented as a file located in the META-INF/services folder of a JAR. The name of the file is the fully qualified name of the interface, while its content is a list of qualified names of available implementations. For example, for an interface ch.frankel.blog.serviceloader.Foo, there must be a file named META-INF/services/ch.frankel.blog.serviceloader.Foo which content might look like this:

    ch.frankel.blog.serviceloader.FooImpl1
    ch.frankel.blog.serviceloader.FooImpl2
    

    Note classes listed above must implement the ch.frankel.blog.serviceloader.Foo interface.

    From a code perspective, it’s very straightforward:

    ServiceLoader<Foo> loader = ServiceLoader.load(Foo.class);
    loader.iterator();
    

    Service Loader Spring integration

    Core Spring offers an integration with the above Service Loader via the factory beans. For example, the following code assumes there will be a least of candidate implementations:

    @Configuration
    public class ServiceConfiguration {
    
        @Bean
        public ServiceListFactoryBean serviceListFactoryBean() {
            ServiceListFactoryBean serviceListFactoryBean = new ServiceListFactoryBean();
            serviceListFactoryBean.setServiceType(Foo.class);
            return serviceListFactoryBean;
        }
    }
    
    Object object = serviceListFactoryBean.getObject();
    

    Obviously, this requires further operations to get the data in the right form (hint: it’s a linked list).

    Spring Factories Loader

    In parallel to the Java Service Loader, Spring offers another Inversion of Control implementation. There’s only a single property file involved, it must be named spring.factories and located under META-INF. From a code perspective, the file is read through the SpringFactoriesLoader.loadFactories() static method - yes, for Spring, it’s quite a shock.

    Client code couldn’t get any simpler:

    List<Foo> foos = SpringFactoriesLoader.loadFactories(Foo.class, null);
    

    Note the 2nd argument is the optional class loader.

    Compared to the Java Service Loader, the differences are two-fold:

    1. Whether one file format is better .i.e. more readable or more maintainable, than the other is a matter of personal taste.
    2. There’s no requirement in spring.factories for the key to be an interface and for the values to implement it. For example, Spring Boot uses this approach to handle auto-configuration beans: the key is an annotation i.e. org.springframework.boot.autoconfigure.EnableAutoConfiguration while values are classes annotated with @Configuration. This allows for a much more flexible design, if used wisely.
  • Fixing my own Spring Boot starter demo

    Spring Boot logo

    Since one year or so, I try to show the developer community that there’s no magic involved in Spring Boot but rather just straightforward software engineering. This is achieved with blog posts and conference talks. At jDays, Stéphane Nicoll was nice enough to attend my talk and pointed out an issue in the code. I didn’t fix it then, and it came back to bite me last week during a Pivotal webinar. Since a lesson learned is only as useful as its audience, I’d like to share my mistake with the world, or at least with you, dear readers.

    The context is that of a Spring Boot starter for the XStream library:

    XStream is a simple library to serialize objects to XML and back again.

    1. (De)serialization capabilities are implemented as instance methods of the XStream class
    2. Customization of the (de)serialization process is implemented through converters registered in the XStream instance

    The goal of the starter is to ease the usage of the library. For example, it creates an XStream instance in the context if there’s none already:

    @Configuration
    public class XStreamAutoConfiguration {
    
        @Bean
        @ConditionalOnMissingBean(XStream.class)
        public XStream xStream() {
            return new XStream();
        }
    }
    

    Also, it will collect all custom converters from the context and register them in the existing instance:

    @Configuration
    public class XStreamAutoConfiguration {
    
        @Bean
        @ConditionalOnMissingBean(XStream.class)
        public XStream xStream() {
            return new XStream();
        }
    
        @Bean
        public Collection<Converter> converters(XStream xstream, Collection<Converter> converters) {
            converters.forEach(xstream::registerConverter);
            return converters;
        }
    }
    

    The previous snippet achieves the objective, as Spring obediently inject both xstream and converters dependency beans in the method. Yet, a problem happened during the webinar demo: can you spot it?

    If you answered about an extra converters bean registered into the context, you are right. But the issue occurs if there’s another converters bean registered by the client code, and it’s of different type i.e. not a Collection. Hence, registration of converters must not happen in beans methods, but only in a @PostConstruct one.

    • Option 1

      The first option is to convert the @Bean to @PostConstruct.

      @Configuration
      public class XStreamAutoConfiguration {
      
        @Bean
        @ConditionalOnMissingBean(XStream.class)
        public XStream xStream() {
            return new XStream();
        }
      
        @PostConstruct
        public void register(XStream xstream, Collection<Converter> converters) {
            converters.forEach(xstream::registerConverter);
        }
      }
      

      Unfortunately, @PostConstruct doesn’t allow for methods to have arguments. The code doesn’t compile.

    • Option 2

      The alternative is to inject both beans into attributes of the configuration class, and use them in the @PostConstruct-annotated method.

      @Configuration
      public class XStreamAutoConfiguration {
      
        @Autowired
        private XStream xstream;
      
        @Autowired
        private Collection<Converter> converters;
      
        @Bean
        @ConditionalOnMissingBean(XStream.class)
        public XStream xStream() {
            return new XStream();
        }
      
        @PostConstruct
        public void register() {
            converters.forEach(xstream::registerConverter);
        }
      }    
      

    This compiles fine, but Spring enters a cycle trying both to inject the XStream instance into the configuration and to create it as a bean at the same time.

    • Option 3

      The final (and only valid) option is to learn from the master and use another configuration class - a nested one. Looking at Spring Boot code source, it’s obviously a pattern.

      The final code looks like this:

      @Configuration
      public class XStreamAutoConfiguration {
      
        @Bean
        @ConditionalOnMissingBean(XStream.class)
        public XStream xStream() {
            return new XStream();
        }
      
        @Configuration
        public static class XStreamConverterAutoConfiguration {
      
            @Autowired
            private XStream xstream;
      
            @Autowired
            private Collection<Converter> converters;
      
            @PostConstruct
            public void registerConverters() {
                converters.forEach(converter -> xstream.registerConverter(converter));
            }
        }
      }
      

    The fixed code is available on Github. And grab the webinar while it’s hot.

    Categories: Java Tags: Spring Bootdesign
  • Custom collectors in Java 8

    Among the many features available in Java 8, streams seem to be one of the biggest game changers regarding the way to write Java code. Usage is quite straightforward: the stream is created from a collection (or from a static method of an utility class), it’s processed using one or many of the available stream methods, and the collected back into a collection or an object. One generally uses one of the static method that the Collectors utility class offers:

    • Collectors.toList()
    • Collectors.toSet()
    • Collectors.toMap()
    • etc.

    Sometimes, however, there’s a need for more. The goal of this post is to describe how to achieve that.

    The Collector interface

    Every one of the above static methods returns a Collector. But what is a Collector? The following is a simplified diagram:

    Interface From the JavaDocs
    Supplier Represents a supplier of results. There is no requirement that a new or distinct result be returned each time the supplier is invoked.
    BiConsumer Represents an operation that accepts two input arguments and returns no result. Unlike most other functional interfaces, BiConsumer is expected to operate via side-effects.
    Function Represents a function that accepts one argument and produces a result.
    BinaryOperator Represents an operation upon two operands of the same type, producing a result of the same type as the operands. This is a specialization of BiFunction for the case where the operands and the result are all of the same type.

    The documentation of each dependent interface doesn’t tell much, apart from the obvious. Looking at the Collector documentation yields a little more:

    A Collector is specified by four functions that work together to accumulate entries into a mutable result container, and optionally perform a final transform on the result. They are:

    • creation of a new result container (supplier())
    • incorporating a new data element into a result container (accumulator())
    • combining two result containers into one (combiner())
    • performing an optional final transform on the container (finisher())

    The Stream.collect() method

    The real insight comes from the Stream.collect() method documentation:

    Performs a mutable reduction operation on the elements of this stream. A mutable reduction is one in which the reduced value is a mutable result container, such as an ArrayList, and elements are incorporated by updating the state of the result rather than by replacing the result. This produces a result equivalent to:

        R result = supplier.get();
        for (T element : this stream)
            accumulator.accept(result, element);
        return result;
    

    Note the combiner() method is not used - it is only used within parallel streams, and for simplification purpose, will be set aside for the rest of this post.

    Examples

    Let’s have some examples to demo the development of custom collectors.

    Single-value example

    To start, let’s compute the size of a collection using a collector. Though not very useful, it’s a good introduction. Here are the requirements for the 4 interfaces:

    1. Since the end result should be an integer, the supplier should probably also return some kind of integer. The problem is that neither int nor Integer are mutable, and this is required for the next step. A good candidate type would be MutableInt from Apache Commons Lang.
    2. The accumulator should only increment the MutableInt, whatever the element in the collection is.
    3. Finally, the finisher just returns the int value wrapped by the MutableInt.

    Source is available on Github.

    Grouping example

    The second example shall be more useful. From a collection of strings, let’s create a Apache Commons Lang multi-valued map:

    • The key should be a char
    • The corresponding values should be the strings that start with this char
    1. The supplier is pretty straightforward, it returns a MultiValuedMap instance
    2. The accumulator just calls the put method from the multi-valued map, using the above “specs”
    3. The finisher returns the map itself

    Source is available on Github.

    Partitioning example

    The third example matches a use-case I encountered this week: given a collection and a predicate, dispatch elements that match into a collection and elements that do not into another.

    1. As the supplier returns a single instance, a new data structure e.g. DoubleList should first be designed
    2. The accumulator must be initialized with the predicate, so that the accept() contract method signature is the same.
    3. As for the above example, the finisher should return the DoubleList itself

    Source is available on Github.

    Final consideration

    Developing a custom collector is not that hard, provided one understands the basic concepts behind it.

    The real issue behind collectors is the whole Stream API. Streams need to be created first and then collected afterwards. Newer languages, with Functional Programming paradigm designed from the start - such as Scala or Kotlin, provide collections with such capabilities directly backed-in.

    For example, to filter out something from a map in Java:

    map.entrySet().stream()
            .filter( entry -> entry.getKey().length() == 4)
            .collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue));
    

    That would translate as the following in Kotlin:

    map.entries.filter { it.key.length == 4 }
    
    Categories: Java Tags: java 8lambdacollector
  • Kotlin for front-end developers

    Kotlin logo

    I assume that readers are already familiar with the Kotlin language. Advertised from the site:

    Statically typed programming language for the JVM, Android and the browser 100% interoperable with Java™

    — Headline from the Kotlin website

    Kotlin first penetrated the Android ecosystem and has seen a huge adoption there. There’s also a growing trend on the JVM, via Spring Boot. Since its latest 1.1 release, Kotlin also offers a production-grade Kotlin to JavaScript transpiler. This post is dedicated to the latter.

    IMHO, the biggest issue regarding the process of transpiling Kotlin to JavaScript is that documentation is only aimed at server-side build tools just as Maven and Gradle - or worse, the IntelliJ IDEA IDE. Those work very well for backend-driven projects or prototypes, but is not an incentive to front-end developers whose bread and butter are npm, Grunt/Gulp, yarn, etc.

    Let’s fix that by creating a simple Google Maps using Kotlin.

    Managing the build file

    This section assumes:

    • The Grunt command-line has already been installed, via npm install -g grunt-cli
    • The kotlin package has been installed and the kotlin compiler is on the path. I personally use Homebrew - brew install kotlin

    The build file looks like the following:

    Gruntfile.js
    module.exports = function (grunt) {
    
        grunt.initConfig({
            clean: 'dist/**',
            exec: {
                cmd: 'kotlinc-js src -output dist/script.js' (1)
            },
            copy: { (2)
                files: {
                    expand: true,
                    flatten: true,
                    src: ['src/**/*.html', 'src/**/*.css'],
                    dest: 'dist'
                }
            }
        });
    
        grunt.loadNpmTasks('grunt-contrib-clean');
        grunt.loadNpmTasks('grunt-contrib-copy');
        grunt.loadNpmTasks('grunt-exec');
    
        grunt.registerTask('default', ['exec', 'copy']);
    };
    1 Transpiles all Kotlin files found in the src folder to a single JavaScript file
    2 Copies CSS and HTML files to the dist folder

    Bridging Google Maps API in Kotlin

    The following snippet creates a Map object using Google Maps API which will be displayed on a specific div element:

    var element = document.getElementById('map');
    new google.maps.Map(element, {
    	center: { lat: 46.2050836, lng: 6.1090691 },
        zoom: 8
    });

    Like in TypeScript, there must be a thin Kotlin adapter to bridge the original JavaScript API. Unlike in TypeScript, there’s no existing repository of such adapters. The following is a naive first draft:

    external class Map(element: Element?, options: Json?)
    The external keyword is used to tell the transpiler the function body is implemented in another file - or library in that case.

    The first issue comes from a name collision: Map is an existing Kotlin class and is imported by default. Fortunately, the @JsName annotation allows to translate the name at transpile time.

    gmaps.kt
    @JsName("Map")
    external class GoogleMap(element: Element?, options: Json?)

    The second issue occurs because the original API is in a namespace: the object is not Map, but google.maps.Map. The previous annotation doesn’t allow for dots, but a combination of other annotations can do the trick:

    /google/maps/gmaps.kt
    @JsModule("google")
    @JsQualifier("maps")
    @JsName("Map")
    external class GoogleMap(element: Element?, options: Json?)

    This still doesn’t work - it doesn’t even compile, as @JsQualifier cannot be applied to a class. The final working code is:

    /google/maps/gmaps.kt
    @file:JsModule("google")
    @file:JsQualifier("maps")
    @file:JsNonModule
    
    package google.maps
    
    @JsName("Map")
    external class GoogleMap(element: Element?, options: Json?)

    Calling Google Maps

    Calling the above code is quite straightforward, but note the second parameter of the constructor is of type Json. That for sure is quite different from the strongly-typed code which was the goal of using Kotlin. To address that, let’s create real types:

    internal class MapOptions(val center: LatitudeLongitude, val zoom: Byte)
    internal class LatitudeLongitude(val latitude: Double, val longitude: Double)

    And with Kotlin’s extension function - and an out-of-the-box json() function, let’s make them able to serialize themselves to JSON:

    internal fun LatitudeLongitude.toJson() = json("lat" to latitude, "lng" to longitude)
    internal fun MapOptions.toJson() = json("center" to center.toJson(), "zoom" to zoom)

    This makes it possible to write the following:

    fun initMap() {
        val div = document.getElementById("map")
        val latLng = LatitudeLongitude(latitude = -34.397, longitude = 150.644)
        val options = MapOptions(center = latLng, zoom = 8)
        GoogleMap(element = div, options = options.toJson())
    }

    Refinements

    We could stop at this point, with the feeling to have achieved something. But Kotlin allows much more.

    The low hanging fruit would be to move the JSON serialization to the Map constructor:

    internal class KotlinGoogleMap(element: Element?, options: MapOptions) : GoogleMap(element, options.toJson())
    
    KotlinGoogleMap(element = div, options = options)

    Further refinements

    The "domain" is quite suited to be written using a DSL. Let’s update the "API":

    external open class GoogleMap(element: Element?) {
        fun setOptions(options: Json)
    }
    
    internal class MapOptions {
        lateinit var center: LatitudeLongitude
        var zoom: Byte = 1
        fun center(init: LatitudeLongitude.() -> Unit) {
            center = LatitudeLongitude().apply(init)
        }
        fun toJson() = json("center" to center.toJson(), "zoom" to zoom)
    }
    
    internal class LatitudeLongitude() {
        var latitude: Double = 0.0
        var longitude: Double = 0.0
        fun toJson() = json("lat" to latitude, "lng" to longitude)
    }
    
    internal class KotlinGoogleMap(element: Element?) : GoogleMap(element) {
        fun options(init: MapOptions.() -> Unit) {
            val options = MapOptions().apply(init)
            setOptions(options = options.toJson())
        }
    }
    
    internal fun kotlinGoogleMap(element: Element?, init: KotlinGoogleMap.() -> Unit) = KotlinGoogleMap(element).apply(init)

    The client code now can be written as:

    fun initMap() {
        val div = document.getElementById("map")
        kotlinGoogleMap(div) {
            options {
                zoom = 6
                center {
                    latitude = 46.2050836
                    longitude = 6.1090691
                }
            }
        }
    }

    Conclusion

    Though the documentation is rather terse, it’s possible to only use the JavaScript ecosystem to transpile Kotlin code to JavaScript. Granted, the bridging of existing libraries is a chore, but this is only a one-time effort as the community starts sharing their efforts. On the other hand, the same features that make Kotlin a great language to use server-side - e.g. writing a DSL, also benefit on the front-end.

    Categories: Development Tags: JavaScriptKotlinfront-end