• Fully configurable mappings for Spring MVC

    Spring Boot logog

    As I wrote some weeks earlier, I’m trying to implement features of the Spring Boot actuator in a non-Boot Spring MVC applications. Developing the endpoints themselves is quite straightforward. Much more challenging, however, is to be able to configure the mapping in a properties file, like in the actuator. This got me to check more closely at how it was done in the current code. This post sums up my “reverse-engineering” attempt around the subject.

    Standard MVC

    Usage

    In Spring MVC, in a class annotated with the @Controller annotation, methods can be in turn annotated with @RequestMapping. This annotation accepts a value attribute (or alternatively a path one) to define from which path it can be called.

    The path can be fixed e.g. /user but can also accepts variables e.g. /user/{id} filled at runtime. In that case, parameters should can be mapped to method parameters via @PathVariable:

    @RequestMapping(path = "/user/{id}", method = arrayOf(RequestMethod.GET))
    fun getUser(@PathVariable("id") id:String) = repository.findUserById(id)
    

    While adapted to REST API, this has to important limitations regarding configuration:

    • The pattern is set during development time and does not change afterwards
    • The filling of parameters occurs at runtime

    Implementation

    With the above, mappings in @Controller-annotated classes will get registered during context startup through the DefaultAnnotationHandlerMapping class. Note there’s a default bean of this type registered in the context. This is summed up in the following diagram:

    DefaultAnnotationHandlerMapping sequence diagram

    In essence, the magic applies only to @Controller-annotated classes. Or, to be more strict, quoting the DefaultAnnotationHandlerMapping’s Javadoc:

    Annotated controllers are usually marked with the Controller stereotype at the type level. This is not strictly necessary when RequestMapping is applied at the type level (since such a handler usually implements the org.springframework.web.servlet.mvc.Controller interface). However, Controller is required for detecting RequestMapping annotations at the method level if RequestMapping is not present at the type level.

    Actuator

    Usage

    Spring Boot actuator allows for configuring the path associated with each endpoint in the application.properties file (or using alternative methods for Boot configuration).

    For example, the metrics endpoint is available by default via the metrics path. But’s it possible to configure a completely different path:

    endpoints.metrics.id=mymetrics
    

    Also, actuator endpoints are by default accessible directly under the root, but it’s possible to group them under a dedicated sub-context:

    management.context-path=/manage
    

    With the above configuration, the metrics endpoint is now available under the /manage/mymetrics.

    Implementation

    Additional actuator endpoints should implements the MvcEndpoint interface. Methods annotated with @RequestMapping will work in the exact same way as for standard controllers above. This is achieved via a dedicated handler mapping, EndpointHandlerMapping in the Spring context.

    HandlerMapping to map Endpoints to URLs via Endpoint.getId(). The semantics of @RequestMapping should be identical to a normal @Controller, but the endpoints should not be annotated as @Controller (otherwise they will be mapped by the normal MVC mechanisms).

    The class hierarchy is the following:

    MvcEndpoint class diagram

    This diagram shows what’s part of Spring Boot and what’s not.

    Conclusion

    Actuator endpoints reuse some from the existing Spring MVC code to handle @RequestMapping. It’s done in a dedicated mapping class so as to separate standard MVC controllers and Spring Boot’s actuator endpoint class hierarchy. In order to achieve fully configurable mappings in Spring MVC, this is the part of the code study, to duplicate and to adapt should one wants fully configurable mappings.

    Categories: JavaEE Tags: Spring MVCactuatorSpring Boot
  • Coping with stringly-typed

    UPDATED on March 13, 2017: Add Builder pattern section

    Most developers have strong opinions regarding whether a language should be strongly-typed or weakly-typed, whatever notions they put behind those terms. Some also actively practice stringly-typed programming - mostly without even being aware of it. It happens when most of attributes and parameters of a codebase are String. In this post, I will make use of the following simple snippet as an example:

    public class Person {
    
        private final String title;
        private final String givenName;
        private final String familyName;
        private final String email;
      
        public Person(String title, String givenName, String familyName, String email) {
            this.title = title;
            this.givenName = givenName;
            this.familyName = familyName;
            this.email = email;
        }
        ...
    }
    

    The original sin

    The problem with that code is that it’s hard to remember which parameter represents what and in which order they should be passed to the constructor.

    Person person = new Person("[email protected]", "John", "Doe", "Sir");
    

    In the previous call, the email and the title parameter values were switched. Ooops.

    This is even worse if more than one constructor is available, offering optional parameters:

    public Person(String givenName, String familyName, String email) {
        this(null, givenName, familyName, email);
    }
    
    Person another = new Person("Sir", "John", "Doe");
    

    In that case, title was the optional parameter, not email. My bad.

    Solving the problem the OOP way

    Object-Oriented Programming and its advocates have a strong aversion to stringly-typed code for good reasons. Since everything in the world has a specific type, so must it be in the system.

    Let’s rewrite the previous code à la OOP:

    public class Title {
        private final String value;
        public Title(String value) {
        	this.value = value;
        }
    }
    
    public class GivenName {
        private final String value;
        public FirstName(String value) {
        	this.value = value;
        }
    }
    
    public class FamilyName {
        private final String value;
        public LastName(String value) {
        	this.value = value;
        }
    }
    
    public class Email {
        private final String value;
        public Email(String value) {
        	this.value = value;
        }
    }
    
    public class Person {
    
        private final Title title;
        private final GivenName givenName;
        private final FamilyName familyName;
        private final Email email;
      
        public Person(Title title, GivenName givenName, FamilyName familyName, Email email) {
            this.title = title;
            this.givenName = givenName;
            this.familyName = familyName;
            this.email = email;
        }
        ...
    }
    
    
    Person person = new Person(new Title(null), new FirstName("John"), new LastName("Doe"), new Email("[email protected]"));
    

    That way drastically limits the possibility of mistakes. The drawback is a large increase in verbosity - which might lead to other bugs.

    Pattern to the rescue

    A common way to tackle this issue in Java is to use the Builder pattern. Let’s introduce a new builder class and rework the code:

    public class Person {
    
        private String title;
        private String givenName;
        private String familyName;
        private String email;
    
        private Person() {}
    
        private void setTitle(String title) {
            this.title = title;
        }
    
        private void setGivenName(String givenName) {
            this.givenName = givenName;
        }
    
        private void setFamilyName(String familyName) {
            this.familyName = familyName;
        }
    
        private void setEmail(String email) {
            this.email = email;
        }
    
        public static class Builder {
    
            private Person person;
    
            public Builder() {
                person = new Person();
            }
    
            public Builder title(String title) {
                person.setTitle(title);
                return this;
            }
    
            public Builder givenName(String givenName) {
                person.setGivenName(givenName);
                return this;
            }
    
            public Builder familyName(String familyName) {
                person.setFamilyName(familyName);
                return this;
            }
    
            public Builder email(String email) {
                person.setEmail(email);
                return this;
            }
    
            public Person build() {
                return person;
            }
        }
    }
    

    Note that in addition to the new builder class, the constructor of the Person class has been set to private. Using the Java language features, this allows only the Builder to create new Person instances. The same is used for the different setters.

    Using this pattern is quite straightforward:

    Person person = new Builder()
                   .title("Sir")
                   .givenName("John")
                   .familyName("Doe")
                   .email("[email protected]")
                   .build();
    

    The builder patterns shifts the verbosity from the calling part to the design part. Not a bad trade-off.

    Languages to the rescue

    Verbosity is unfortunately the mark of Java. Some other languages (Kotlin, Scala, etc.) would be much more friendly to this approach, not only for class declarations, but also for object creation.

    Let’s port class declarations to Kotlin:

    class Title(val value: String?)
    class GivenName(val value: String)
    class FamilyName(val value: String)
    class Email(val value: String)
    
    class Person(val title: Title, val givenName: GivenName, val familyName: FamilyName, val email: Email)
    

    This is much better, thanks to Kotlin! And now object creation:

    val person = Person(Title(null), GivenName("John"), FamilyName("Doe"), Email("[email protected]"))
    

    For this, verbosity is only marginally decreased compared to Java.

    Named parameters to the rescue

    OOP fanatics may stop reading there, for their way is not the only one to cope with stringly-typed.

    One alternative is about named parameters, and is incidentally also found in Kotlin. Let’s get back to the original stringly-typed code, port it to Kotlin and use named parameters:

    class Person(val title: String?, val givenName: String, val familyName: String, val email: String)
    
    val person = Person(title = null, givenName = "John", familyName = "Doe", email = "[email protected]")
    
    val another = Person(email = "[email protected]", title = "Sir", givenName = "John", familyName = "Doe")
    

    A benefit of named parameters besides coping with stringly-typed code is that they are order-agnostic when invoking the constructor. Plus, they also play nice with default values:

    class Person(val title: String? = null, val givenName: String, val familyName: String, val email: String? = null)
    
    val person = Person(givenName = "John", familyName = "Doe")
    val another = Person(title = "Sir", givenName = "John", familyName = "Doe")
    

    Type aliases to the rescue

    While looking at Kotlin, let’s describe a feature released with 1.1 that might help.

    A type alias is as its name implies a name for an existing type; the type can be a simple type, a collection, a lambda - whatever exists within the type system.

    Let’s create some type aliases in the stringly-typed world:

    typealias Title = String
    typelias GivenName = String
    typealias FamilyName = String
    typealias Email = String
    
    class Person(val title: Title, val givenName: GivenName, val familyName: FamilyName, val email: Email)
    
    val person = Person(null, "John", "Doe", "[email protected]")
    

    The declaration seems more typed. Unfortunately object creation doesn’t bring any betterment.

    Note the main problem of type aliases is that they are just that - aliases: no new type is created so if 2 aliases point to the same type, all 3 are interchangeable with one another.

    Libraries to the rescue

    For the rest of this post, let’s go back to the Java language.

    Twisting the logic a bit, parameters can be validated at runtime instead of compile-time with the help of specific libraries. In particular, the Bean validation library does the job:

    public Person(@Title String title, @GivenName String givenName, @FamilyName String familyName, @Email String email) {
        this.title = title;
        this.givenName = givenName;
        this.familyName = familyName;
        this.email = email;
    }
    

    Admittedly, it’s not the best solution… but it works.

    Tooling to the rescue

    I have already written about tooling and that it’s as important (if not more) as the language itself.

    Tools fill gaps in languages, while being non-intrusive. The downside is that everyone has to use it (or find a tool with the same feature).

    For example, when I started my career, coding guidelines mandated developers to order methods by alphabetical order in the class file. Nowadays, that would be senseless, as every IDE worth its salt can display the methods of a class in order.

    Likewise, named parameters can be a feature of the IDE, for languages that lack it. In particular, latest versions of IntelliJ IDEA emulates named parameters for the Java language for types that are deemed to generic. The following shows the Person class inside the IDE:

    Conclusion

    While proper OOP design is the historical way to cope with stringly-typed code, it also makes it quite verbose and unwieldy in Java. This post describes alternatives, with their specific pros and cons. Each needs to be evaluated in the context of one’s own specific context to decide which one is the best fit.

  • A use-case for Spring component scan

    Regular readers of this blog know I’m a big proponent of the Spring framework, but I’m quite opinionated in the way it should be used. For example, I favor explicit object instantiation and explicit component wiring over self-annotated classes, component scanning and autowiring.

    Concepts

    Though those concepts are used by many Spring developers, my experience has taught me they are not always fully understood. Some explanation is in order.

    Self-annotated classes

    Self-annotated classes are classes which define how they will be instantiated by Spring via annotations on the classes themselves. @Component, @Controller, @Service and @Repository are the usual self-annotated classes found in most Spring projects.

    @Component
    class MySelfAnnotatedClass
    

    The main disadvantages of self-annotated classes is the hard coupling between the class and the bean. For example, it’s not possible to instantiate 2 singleton beans of the same class, as in explicit creation. Worse, it also couples the class to the Spring framework itself.

    Note that @Configuration classes are also considered to be self-annotated.

    Component scanning

    Self-annotated classes need to be listed and registered in the context. This can be done explicitly:

    @Configuration
    class MyConfig {
    
        @Bean
        fun myClass() =  MySelfAnnotatedClass()
    }
    

    However, the most widespread option is to let the Spring framework search for every self-annotated class on the project classpath and register them according to the annotations.

    @Configuration @ComponentScan
    class MyConfig
    

    Autowiring

    Some beans require dependencies in order to be initialized. Wiring the dependency into the dependent bean can be either:

    • explicit: it’s the developer’s responsibility to tell which beans will fulfill the dependency.
    • implicit (or auto): the Spring framework is responsible to provide the dependency. In order to do that, a single bean must be eligible.

    Regarding the second option, please re-read an old post of mine for its related problems.

    Sometimes, however, there’s no avoiding autowiring. When bean Bean1 defined in configuration fragment Config1 depends on bean Bean2 defined in fragment Config2, the only possible injection option is autowiring.

    @Configuration
    class Config2 {
    
        @Bean
        fun bean2() = Bean2()
    }
    
    @Configuration
    class Config1 {
    
        @Bean @Autowired
        fun bean1(bean2: Bean2) = Bean1(bean2)
    }
    

    In the above snippet, autowiring is used without self-annotated classes.

    Reinventing the wheel

    This week, I found myself re-implementing Spring Boot’s actuator in a legacy non-Spring Boot application.

    The architecture is dead simple: the HTTP endpoint returns a Java object (or a list of them) serialized through the Jackson library. Every endpoint might return a different object, and each can be serialized using a custom serializer.

    I’ve organized the project in a package-per-endpoint way (as opposed to package-per-layer), and have already provided several endpoints. I’d like people to contribute other endpoints, and I want it to be as simple as possible. In particular, they should only:

    1. Declare controllers
    2. Declare configuration classes
    3. Instantiate Jackson serializers

    The rest should taken care of by generic code written by me.

    The right usage of autowire

    Controllers and configuration classes are easily taken care of by using @ComponentScan on the main configuration class located in the project’s main package. But what about serializers?

    Spring is able to collect all beans of a specific class registed into a context into a list. That means that every package will declare its serializers independently, and common code can take care of the registration:

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    @Configuration @EnableWebMvc @ComponentScan
    class WebConfiguration : WebMvcConfigurerAdapter() {
    
        @Autowired
        private lateinit var serializers: List<StdSerializer<*>>
    
        override fun configureMessageConverters(converters: MutableList<HttpMessageConverter<*>>) {
            converters.add(MappingJackson2HttpMessageConverter().apply {
                objectMapper.registerModule(SimpleModule().apply {
                    serializers.forEach { addSerializer(it) }
                })
            })
        }
    }
    

    Magic happens on lines 6 and 7 above. This configuration class has already been written, and new packages don’t need to do anything, serializers will be part of the list.

    Here’s an example of such a configuration class, declaring a serializer bean:

    @Configuration
    class FooConfiguration {
    
        @Bean
        fun fooSerializer()  = FooSerializer()
    }
    
    class FooSerializer : StdSerializer<Foo>(Foo::class.java) {
        ...
    }
    

    Even better, if packages need to be modularized further into full-fledged JARs, this setup will work in the exact same way without any change.

    Conclusion

    A better understanding of self-annotated classes, component-scanning and autowiring is beneficial to all Spring developers.

    Moreover, while they have a lot of drawbacks in “standard” beans classe, it’s not only perfectly acceptable but even an advantage to do so within the scope of configuration classes. In projects designed in a package-by-feature way, it improves modularization and decoupling.

    Categories: Development Tags: Springautowiringcomponent scangood practice
  • A use-case for local class declaration

    Polar bear in a bubble

    One of the first things one learns when starting with Java development is how to declare a class into its own file. Potential later stages include:

    But this doesn’t stop there: the JLS is a trove full of surprises. I recently learned classes can be declared inside any block, including methods. This is called local class declarations (§14.3).

    A local class is a nested class (§8) that is not a member of any class and that has a name. All local classes are inner classes (§8.1.3). Every local class declaration statement is immediately contained by a block. Local class declaration statements may be intermixed freely with other kinds of statements in the block.

    The scope of a local class immediately enclosed by a block (§14.2) is the rest of the immediately enclosing block, including its own class declaration. The scope of a local class immediately enclosed by in a switch block statement group (§14.11)is the rest of the immediately enclosing switch block statement group, including its own class declaration.

    Cool, isn’t it? But use it just for the sake of it is not reason enough… until this week: I started to implement something like the Spring Boot Actuator in a non-Boot application, using Jackson to serialize the results.

    Jackson offers several ways to customize the serialization process. For objects that require only to hide fields or change their names and which classes stand outside one’s reach, it offers mixins. As an example, let’s tweak serialization of the following class:

    public class Person {
    
        private final String firstName;
        private final String lastName;
        
        public Person(String firstName, String lastName) {
            this.firstName = firstName;
            this.lastName = lastName;
        }
        
        public String getFirstName() {
            return firstName;
        }
        
        public String getLastName() {
            return lastName;
        }
    }
    

    Suppose the requirement is to have givenName and familyName attributes. In a regular Spring application, the mixin class should be registered during the configuration of message converters:

    public class WebConfiguration extends WebMvcConfigurerAdapter {
    
        @Override
        public void configureMessageConverters(List<HttpMessageConverter<?>> converters) {
            MappingJackson2HttpMessageConverter jackson2HttpMessageConverter = new MappingJackson2HttpMessageConverter();
            jackson2HttpMessageConverter.getObjectMapper().addMixIn(Person.class, PersonMixin.class);
            converters.add(jackson2HttpMessageConverter);
        }
    }
    

    Now, where does it make the most sense to declare this mixin class? The principle to declare something in the smallest possible scope applies: having it in a dedicated file is obviously wrong, but even a private nested class is overkill. Hence, the most restricted scope is the method itself:

    public class WebConfiguration extends WebMvcConfigurerAdapter {
    
        @Override
        public void configureMessageConverters(List<HttpMessageConverter<?>> converters) {
            MappingJackson2HttpMessageConverter jackson2HttpMessageConverter = new MappingJackson2HttpMessageConverter();
            abstract class PersonMixin {
                @JsonProperty("givenName") abstract String getFirstName();
                @JsonProperty("familyName") abstract String getLastName();
            }
            jackson2HttpMessageConverter.getObjectMapper().addMixIn(Person.class, PersonMixin.class);
            converters.add(jackson2HttpMessageConverter);
        }
    }
    

    While this way makes sense from a pure software engineering point of view, there is a reason not to design code like this: the principle of least surprise. Unless every member of the team is aware and comfortable with local classes, this feature shouldn’t be used.

    Categories: Java Tags: classmethoddesign
  • ElasticSearch API cheatsheet

    ElasticSearch documentation is exhaustive, but the way it’s structured has some room for improvement. This post is meant as a cheat-sheet entry point into ElasticSearch APIs.

    Category Description Call examples

    Document API

    Single Document API

    Adds a new document

    PUT /my_index/my_type/1
    {
      "my_field" : "my_value"
    }
    
    POST /my_index/my_type
    {
      …​
    }
    
    PUT /my_index/my_type/1/_create
    {
      …​
    }

    Gets an existing document

    GET /my_index/my_type/0

    Deletes a document

    DELETE /my_index/my_type/0

    Updates a document

    PUT /my_index/my_type/1
    {
      …​
    }

    Multi-Document API

    Multi-get

    GET /_mget
    {
      "docs" : [
        {
          "_index" : "my_index",
          "_type" : "my_type",
          "_id" : "1"
        }
      ]
    }
    
    GET /my_index/_mget
    {
      "docs" : [
        {
          "_type" : "my_type",
          "_id" : "1"
        }
      ]
    }
    
    
    GET /my_index/my_type/_mget
    {
      "docs" : [
        {
          "_id" : "1"
        }
      ]
    }

    Performs many index/delete operations in one call

    Deletes by query

    POST /my_index/_delete_by_query
    {
      "query": {
        "match": {
          …​
        }
      }
    }

    Updates by query

    POST /my_index/_update_by_query?conflicts=proceed
    POST /my_index/_update_by_query?conflicts=proceed
    {
      "query": {
        "term": {
          "my_field": "my_value"
        }
      }
    }
    
    POST /my_index1,my_index2/my_type1,my_type2/_update_by_query

    Reindexes

    POST /_reindex
    {
      "source": {
        "index": "old_index"
      },
      "dest": {
        "index": "new_index"
      }
    }

    Search API

    URI Search

    Executes a search with query parameters on the URL

    GET /my_index/my_type/_search?q=my_field:my_value
    GET /my_index/my_type/_search
    {
      "query" : {
        "term" : { "my_field" : "my_value" }
      }
    }

    Search Shards API

    Gets indices/shards of a search would be executed against

    GET /my_index/_search_shards

    Count API

    Executes a count query

    GET /my_index/my_type/_count?q=my_field:my_value
    GET /my_index/my_type/_count
    {
      "query" : {
        "term" : { "my_field" : "my_value" }
      }
    }

    Validate API

    Validates a search query

    GET /my_index/my_type/_validate?q=my_field:my_value
    GET /my_index/my_type/_validate
    {
      "query" : {
        "term" : { "my_field" : "my_value" }
      }
    }

    Explain API

    Provides feedback on computation of a search

    GET /my_index/my_type/0/_explain
    {
      "query" : {
        "match" : { "message" : "elasticsearch" }
      }
    }
    GET /my_index/my_type/0/_explain?q=message:elasticsearch

    Profile API

    Provides timing information on individual components during a search

    GET /_search
    {
      "profile": true,
      "query" : {
        …​
      }
    }

    Field Stats API

    Finds statistical properties of fields without executing a search

    GET /_field_stats?fields=my_field
    GET /my_index/_field_stats?fields=my_field
    GET /my_index1,my_index2/_field_stats?fields=my_field

    Indices API

    Index management

    Instantiates a new index

    PUT /my_index
    {
      "settings" : {
        …​
      }
    }

    Deletes existing indices

    DELETE /my_index
    DELETE /my_index1,my_index2
    DELETE /my_index*
    DELETE /_all

    Retrieves information about indices

    GET /my_index
    GET /my_index*
    GET my_index/_settings,_mappings

    Checks whether an index exists

    HEAD /my_index

    Closes/opens an index

    POST /my_index/_close
    POST /my_index/_open

    Shrinks an index to a new index with fewer primary shards

    Rolls over an alias to a new index if conditions are met

    POST /my_index/_rollover
    {
      "conditions": {
        …​
      }
    }

    Mapping management

    Adds a new type to an existing index

    PUT /my_index/_mapping/new_type
    {
      "properties": {
        "my_field": {
          "type": "text"
        }
      }
    }

    Retrieves mapping definition for fields

    GET /my_index/_mapping/my_type/field/my_field
    GET /my_index1,my_index2/_mapping/my_type/field/my_field
    GET /_all/_mapping/my_type1,my_type2/field/my_field1,my_field2
    GET /_all/_mapping/my_type1*/field/my_field*

    Checks whether a type exists

    HEAD /my_index/_mapping/my_type

    Alias management

    Creates an alias over an index

    POST /_aliases
    {
      "actions" : [
        { "add" :
          { "index" : "my_index", "alias" : "my_alias" }
        }
      ]
    }
    
    POST /_aliases
    {
      "actions" : [
        { "add" :
          { "index" : ["index1", "index2"] , "alias" : "another_alias" }
        }
      ]
    }

    Removes an alias

    POST /_aliases
    {
      "actions" : [
        { "remove" :
          { "index" : "my_index", "alias" : "my_old_alias" }
        }
      ]
    }

    Index settings

    Updates settings of indices

    PUT /my_index/_settings
    {
      …​
    }

    Retrieves settings of indices

    GET /my_index/_settings

    Performs an analysis process of a text and return the tokens

    GET /_analyze
    {
      "analyzer" : "standard",
      "text" : "this is a test"
    }

    Creates a new template

    PUT /_template/my_template
    {
      …​
    }

    Deletes an existing template

    DELETE /_template/my_template

    Gets info about an existing template

    GET /_template/my_template

    Checks whether a template exists

    HEAD /_template/my_template

    Replica configuration

    Sets index data location on a disk

    Monitoring

    Provides statistics on indices

    GET /_stats
    GET /my_index1/_stats
    GET /my_index1,my_index2/_stats
    GET /my_index1/_stats/flush,merge

    Provides info on Lucene segments

    GET /_segments
    GET /my_index1/_segments
    GET /my_index1,my_index2/_segments

    Provide recovery info on indices

    GET /_recovery
    GET /my_index1/_recovery
    GET /my_index1,my_index2/_recovery

    Provide store info on shard copies of indices

    GET /_shard_stores
    GET /my_index1/_shard_stores
    GET /my_index1,my_index2/_shard_stores

    Status management

    Clears the cache of indices

    POST /_cache/clear
    POST /my_index/_cache/clear
    POST /my_index1,my_index2/_cache/clear

    Explicitly refreshes indices

    POST /_refresh
    POST /my_index/_refresh
    POST /my_index1,my_index2/_refresh

    Flushes in-memory transaction log on disk

    POST /_flush
    POST /my_index/_flush
    POST /my_index1,my_index2/_flush

    Merge Lucene segments

    POST /_forcemerge?max_num_segments=1
    POST /my_index/_forcemerge?max_num_segments=1
    POST /my_index1,my_index2/_forcemerge?max_num_segments=1

    cat API

    cat aliases

    Shows information about aliases, including filter and routing infos

    GET /_cat/aliases?v
    GET /_cat/aliases/my_alias?v

    cat allocations

    Provides a snapshot on how many shards are allocated and how much disk space is used for each node

    GET /_cat/allocation?v

    cat count

    Provides quick access to the document count

    GET /_cat/count?v
    GET /_cat/count/my_index?v

    cat fielddata

    Shows heap memory currently being used by fielddata

    GET /_cat/fielddata?v
    GET /_cat/fielddata/my_field1,my_field2?v

    cat health

    One-line representation of the same information from /_cluster/health

    GET /_cat/health?v
    GET /_cat/health?v&ts=0

    cat indices

    Provides a node-spanning cross-section of each index

    GET /_cat/indices?v
    GET /_cat/indices?v&s=index
    GET /_cat/indices?v&health=yellow
    GET /_cat/indices/my_index*?v&health=yellow

    cat master

    Displays the master’s node ID, bound IP address, and node name

    GET /_cat/master?v

    cat nodeattrs

    Shows custom node attributes

    GET /_cat/nodeattrs?v
    GET /_cat/nodeattrs?v&h=name,id,pid,ip

    cat nodes

    Shows cluster topology

    GET /_cat/nodes?v
    GET /_cat/nodes?v&h=name,id,pid,ip

    cat pending tasks

    Provides the same information as /_cluster/pending_tasks

    GET /_cat/pending_tasks?v

    cat plugins

    Provides a node-spanning view of running plugins per node

    GET /_cat/plugins?v
    GET /_cat/plugins?v&h=name,id,pid,ip

    cat recovery

    Shows on-going and completed index shard recoveries

    GET /_cat/recovery?v
    GET /_cat/recovery?v&h=name,id,pid,ip

    cat repositories

    Shows snapshot repositories registered in the cluster

    GET /_cat/repositories?v

    cat thread pool

    Shows cluster-wide thread pool statistics per node

    GET /_cat/thread_pool?v
    GET /_cat/thread_pool?v&h=id,pid,ip

    cat shards

    Displays shards to nodes relationships

    GET /_cat/shards?v
    GET /_cat/shards/my_index?v
    GET /_cat/shards/my_ind*?v

    cat segments

    Provides information similar to _segments

    GET /_cat/segments?v
    GET /_cat/segments/my_index?v
    GET /_cat/segments/my_index1,my_index2?v

    cat snapshots

    Shows snapshots belonging to a repository

    /_cat/snapshots/my_repo?v

    cat templates

    Provides information about existing templates

    GET /_cat/templates?v
    GET /_cat/templates/my_template
    GET /_cat/templates/my_template*

    Cluster API

    Cluster Health

    Gets the status of a cluster’s health

    GET /_cluster/health
    GET /_cluster/health?wait_for_status=yellow&timeout=50s
    GET /_cluster/health/my_index1
    GET /_cluster/health/my_index1,my_index2

    Cluster State

    Gets state information about a cluster

    GET /_cluster/state
    GET /_cluster/state/version,nodes/my_index1
    GET /_cluster/state/version,nodes/my_index1,my_index2
    GET /_cluster/state/version,nodes/_all

    Cluster Stats

    Retrieves statistics from a cluster

    GET /_cluster/stats
    GET /_cluster/stats?human&pretty

    Pending cluster tasks

    Returns a list of any cluster-level changes

    GET /_cluster/pending_tasks

    Cluster Reroute

    Executes a cluster reroute allocation

    GET /_cluster/reroute {
      …​
    }

    Cluster Update Settings

    Update cluster-wide specific settings

    GET /_cluster/settings
    {
      "persistent" : {
        …​
      },
      "transient" : {
        …​
      }
    }

    Node Stats

    Retrieves cluster nodes statistics

    GET /_nodes/stats
    GET /_nodes/my_node1,my_node2/stats
    GET /_nodes/127.0.0.1/stats
    GET /_nodes/stats/indices,os,process

    Node Info

    Retrieves cluster nodes information

    GET /_nodes
    GET /_nodes/my_node1,my_node2
    GET /_nodes/_all/indices,os,process
    GET /_nodes/indices,os,process
    GET /_nodes/my_node1,my_node2/_all

    Task Management API

    Retrieve information about tasks currently executing on nodes in the cluster

    GET /_tasks
    GET /_tasks?nodes=my_node1,my_node2
    GET /_tasks?nodes=my_node1,my_node2&actions=cluster:*

    Nodes Hot Threads

    Gets current hot threads on nodes in the cluster

    GET /_nodes/hot_threads
    GET /_nodes/hot_threads/my_node
    GET /_nodes/my_node1,my_node2/hot_threads

    Cluster Allocation Explain API

    Answers the question "why is this shard unnassigned?"

    GET /_cluster/allocation/explain
    GET /_cluster/allocation/explain
    {
      "index": "myindex",
      "shard": 0,
      "primary": false
    }

    marks an experimental (respectively new) API that is subject to removal (resp. change) in future versions

    Last updated on Feb. 22th
    Categories: Development Tags: ElasticElasticSearchAPI
  • Signing and verifying a standalone JAR

    Seal

    Last week, I wrote about the JVM policy file that explicitly lists allowed sensitive API calls when running the JVM in sandboxed mode. This week, I’d like to improve the security by signing the JAR.

    The nominal way

    This way doesn’t work. Readers more interested in the solution than the process should skip it.

    Create a keystore

    The initial step is to create a keystore if none is already available. There are plenty of online tutorials showing how to do that.

    keytool -genkey -keyalg RSA -alias selfsigned -keystore /path/to/keystore.jks -storepass password -validity 360

    Fill in information accordingly.

    Sign the application JAR

    Signing the application JAR must be part of the build process. With Maven, the JAR signer plugin is dedicated to that. Its usage is quite straightforward:

    <plugin>
      <artifactId>maven-jarsigner-plugin</artifactId>
      <version>1.4</version>
      <executions>
      <execution>
        <id>sign</id>
        <goals>
        <goal>sign</goal>
        </goals>
      </execution>
      </executions>
      <configuration>
      <keystore>/path/to/keystore.jks</keystore>
      <alias>selfsigned</alias>
      <storepass>${store.password}</storepass>
      <keypass>${key.password}</keypass>
      </configuration>
    </plugin>

    To create the JAR, invoke the usual command-line and pass both passwords as system properties:

    mvn package -Dstore.password=password -Dkey.password=password

    Alternatively, Maven’s encryption capabilities can be used to store passwords in a dedicated settings-security.xml to further improve security.

    Configure the policy file

    Once the JAR is signed, the policy file can be updated to make use of it. This requires only the following configuration steps:

    1. Point to the keystore
    2. Configure the allowed alias
    keystore "keystore.jks";
    
    grant signedBy "selfsigned" codeBase "file:target/spring-petclinic-1.4.2.jar" {
      ...
    }

    Notice the signedBy keyword followed by the alias name - the same one as in the keystore above.

    Launching the JAR with the policy file

    The same launch command can be used without any change:

    java -Djava.security.manager -Djava.security.policy=jvm.policy -jar target/spring-petclinic-1.4.2.jar

    Unfortunately, it doesn’t work though this particular permission had already been configured!

    Caused by: java.security.AccessControlException: access denied ("java.lang.reflect.ReflectPermission" "suppressAccessChecks")
      at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
      at java.security.AccessController.checkPermission(AccessController.java:884)
      at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
      at java.lang.reflect.AccessibleObject.setAccessible(AccessibleObject.java:128)
      at org.springframework.util.ReflectionUtils.makeAccessible(ReflectionUtils.java:475)
      at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:141)
      at org.springframework.boot.SpringApplication.createSpringFactoriesInstances(SpringApplication.java:420)

    The strangest part is that permissions requested before this one work all right. The reason is to be found in the particular structure of the JAR created by the Spring Boot plugin: JAR dependencies are packaged untouched in a BOOT-INF/lib folder in the executable JAR. Then Spring Boot code uses custom class-loading magic to load required classes from there.

    JAR signing works by creating a specific hash for each class, and by writing them into the JAR manifest file. During the verification phase, the hash of a class is computed and compared to the hash of the manifest. Hence, permissions related to classes located in the BOOT-INF/classes folder work as expected.

    However, the org.springframework.boot.SpringApplication class mentioned in the stack trace above is part of the spring-boot.jar located under BOOT-INF/lib: verification fails as there’s no hash available for the class in the manifest.

    Thus, usage of the Spring Boot plugin for JAR creation/launch is not compatible with JAR signing.

    The workaround

    Aside from Spring Boot, there’s a legacy way to create standalone JARs: the Maven Shade plugin. This will extract every class of every dependency in the final JAR. This is possible with Spring Boot apps, but it requires some slight changes to the POM:

    1. In the POM, remove the Spring Boot Maven plugin
    2. Configure the main class in the Maven JAR plugin:
      <plugin>
        <artifactId>maven-jar-plugin</artifactId>
        <version>3.0.2</version>
        <configuration>
          <archive>
            <manifest>
              <mainClass>org.springframework.samples.petclinic.PetClinicApplication</mainClass>
            </manifest>
          </archive>
        </configuration>
      </plugin>
    3. Finally, add the Maven Shade plugin to work its magic:
      <plugin>
        <artifactId>maven-shade-plugin</artifactId>
        <version>2.4.3</version>
        <configuration>
          <minimizeJar>true</minimizeJar>
        </configuration>
        <executions>
          <execution>
            <phase>package</phase>
            <goals>
              <goal>shade</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
    The command-line to launch the JAR doesn’t change but permissions depend on the executed code, coupled to the JAR structure. Hence, the policy file should be slightly modified.

    Lessons learned

    While it requires to be a little creative, it’s entirely possible to sign Spring Boot JARs by using the same techniques as for any other JARs.

    To go further:

    Categories: Java Tags: JVMsecurityJARSpring Bootpolicy
  • Proposal for a Java policy files crafting process

    Security guy on an escalator

    I’ve already written about the JVM security manager, and why it should be used - despite it being rarely the case, if ever. However, just advocating for it won’t change the harsh reality unless some guidelines are provided to do so. This post has the ambition to be the basis of such guidelines.

    As a reminder, the JVM can run in two different modes, standard and sandboxed. In the former, all API are available with no restriction; in the later, some API calls deemed sensitive are forbidden. In that case, explicit permissions to allow some of those calls can be configured in a dedicated policy file.

    Though running the JVM in sandbox mode is important, it doesn’t stop there .e.g. executing only digitally-signed code is also part of securing the JVM. This post is the first in a 2 parts-serie regarding JVM security.

    Description

    The process is based on the principle of least privilege. That directly translates into the following process:

    1. Start with a blank policy file
    2. Run the application
    3. Check the thrown security exception
    4. Add the smallest-grained permission possible in the policy file that allows to pass step 2
    5. Return to step 2 until the application can be run normally

    Relevant system properties include:

    • java.security.manager: activates the Java Security manager
    • java.security.policy: points to the desired policy file
    • java.security.debug: last but not least, activates debugging information when an absent privilege is required. There are a ton of options.

    That sounds easy enough but let’s go detail how it works with an example.

    A case study

    As a sample application, we will be using the Spring Pet Clinic, a typical albeit small-sized Spring Boot application.

    First steps

    Once the application has been built, launch it with the security manager:

    java -Djava.security.manager -Djava.security.policy=jvm.policy -jar target/spring-petclinic-1.4.2.jar

    This, of course, fails. The output is the following:

    Exception in thread "main" java.lang.IllegalStateException:
        java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "getProtectionDomain")
      at org.springframework.boot.loader.ExecutableArchiveLauncher.<init>(ExecutableArchiveLauncher.java:43)
      at org.springframework.boot.loader.JarLauncher.<init>(JarLauncher.java:37)
      at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:58)
    Caused by: java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "getProtectionDomain")
      at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
      at java.security.AccessController.checkPermission(AccessController.java:884)
      at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
      at java.lang.Class.getProtectionDomain(Class.java:2299)
      at org.springframework.boot.loader.Launcher.createArchive(Launcher.java:117)
      at org.springframework.boot.loader.ExecutableArchiveLauncher.<init>(ExecutableArchiveLauncher.java:40)
      ... 2 more

    Let’s add the permission relevant to the above "access denied" exception to the policy file:

    grant codeBase "file:target/spring-petclinic-1.4.2.jar" {
      permission java.lang.RuntimePermission "getProtectionDomain";
    };

    Notice the path pointing to the JAR. It prevents other potentially malicious archives to execute critical code. Onto the next blocker.

    Exception in thread "main"
        java.security.AccessControlException: access denied ("java.util.PropertyPermission" "java.protocol.handler.pkgs" "read")

    This can be fixed by adding the below line to the policy file:

    grant codeBase "file:target/spring-petclinic-1.4.2.jar" {
      permission java.lang.RuntimePermission "getProtectionDomain";
      permission java.util.PropertyPermission "java.protocol.handler.pkgs", "read";
    };

    Next please.

    Exception in thread "main"
        java.security.AccessControlException: access denied ("java.util.PropertyPermission" "java.protocol.handler.pkgs" "read")

    Looks quite similar, but it needs a write permission in addition to the read one. Sure it can be fixed by adding one more line, but there’s a shortcut available. Just specify all necessary attributes of the permission on the same line:

    grant codeBase "file:target/spring-petclinic-1.4.2.jar" {
      permission java.lang.RuntimePermission "getProtectionDomain";
      permission java.util.PropertyPermission "java.protocol.handler.pkgs", "read,write";
    };

    Rinse and repeat. Without further ado, the (nearly) final policy can be found online: a whooping ~1800 lines of configuration for the Spring Boot Pet Clinic as an executable JAR.

    Now that the general approach has been explained, it just needs to be followed until the application functions properly. The next section describe some specific glitches along the way.

    Securing Java logging

    At some point, nothing gets printed in the console anymore. The command-line just returns, that’s it. Comes the java.security.debug system property - described above, that helps resolve the issue:

    java -Djava.security.manager -Djava.security.policy=jvm.policy
         -Djava.security.debug=access,stacktrace -jar target/spring-petclinic-1.4.2.jar

    That yields the following stack:

    java.lang.Exception: Stack trace
      at java.lang.Thread.dumpStack(Thread.java:1329)
      at java.security.AccessControlContext.checkPermission(AccessControlContext.java:419)
      at java.security.AccessController.checkPermission(AccessController.java:884)
      at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
      at java.util.logging.LogManager.checkPermission(LogManager.java:1586)
      at java.util.logging.Logger.checkPermission(Logger.java:422)
      at java.util.logging.Logger.setLevel(Logger.java:1688)
      at java.util.logging.LogManager.resetLogger(LogManager.java:1354)
      at java.util.logging.LogManager.reset(LogManager.java:1332)
      at java.util.logging.LogManager$Cleaner.run(LogManager.java:239)

    It’s time for some real software engineering (also known as Google Search). The LogManager’s Javadoc tells about the LoggingPermission that needs to be added to the existing list of permissions:

    grant codeBase "file:target/spring-petclinic-1.4.2.jar" {
      permission java.lang.RuntimePermission "getProtectionDomain";
      ...
      permission java.util.PropertyPermission "PID", "read,write";
      permission java.util.logging.LoggingPermission "control";
    };

    That makes it possible to go further.

    Securing the reading of system properties and environment variables

    It’s even possible to watch Spring Boot log…​ until one realizes it’s made entirely of error messages about not being able to read a bunchload of system properties and environment variables. Here’s an excerpt:

    2017-01-22 00:30:17.118  INFO 46549 --- [           main] o.s.w.c.s.StandardServletEnvironment     :
      Caught AccessControlException when accessing system environment variable [logging.register_shutdown_hook];
      its value will be returned [null].
      Reason: access denied ("java.lang.RuntimePermission" "getenv.logging.register_shutdown_hook")
    2017-01-22 00:30:17.118  INFO 46549 --- [           main] o.s.w.c.s.StandardServletEnvironment     :
      Caught AccessControlException when accessing system property [logging_register-shutdown-hook];
      its value will be returned [null].
      Reason: access denied ("java.util.PropertyPermission" "logging_register-shutdown-hook" "read")

    I will spare you dear readers a lot of trouble: there’s no sense in configuring every property one by one as JCache requires read and write permissions on all properties. So just remove every fine-grained PropertyPermission so far and replace it with a catch-all coarse-grained one:

    permission java.util.PropertyPermission "*", "read,write";

    Seems like security was not one of JCache developers first priority. The following snippet is the code excerpt for javax.cache.Caching.CachingProviderRegistry.getCachingProviders():

    if (System.getProperties().containsKey(JAVAX_CACHE_CACHING_PROVIDER)) {
        String className = System.getProperty(JAVAX_CACHE_CACHING_PROVIDER);
        ...
    }

    Wow, it reads all properties! Plus the next line makes it a little redundant, no?

    As for environment variables, the Spring team seem to try to avoid developers configuration issues related to case and check every possible case combination, so there is a lot of different options.

    Variables and subdirectories

    At one point, Spring’s embedded Tomcat attempts - and fails, to create a subfolder into the java.io.tmpdir folder.

    java.lang.SecurityException: Unable to create temporary file
      at java.io.File.createTempFile(File.java:2018) ~[na:1.8.0_92]
      at java.io.File.createTempFile(File.java:2070) ~[na:1.8.0_92]
      at org.springframework.boot.context.embedded.AbstractEmbeddedServletContainerFactory.createTempDir(...)
      at org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedServletContainerFactory.getEmbeddedServletContainer(...)
      at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.createEmbeddedServletContainer(...)
      at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.onRefresh(...)
      ... 16 common frames omitted

    One could get away with that by "hard-configuring" the path, but that would just be a major portability issue. Permissions are able to use System properties.

    The second issue is the subfolder: there’s no way of knowing the folder name, hence it’s not possible to configure it beforehand. However, file permissions accept any direct children or any descendant in the hierarachy; the former is set with jokers, and the second with dashes. The final configuration looks like this:

    permission java.io.FilePermission "${java.io.tmpdir}/-", "read,write,delete";

    CGLIB issues

    CGLIB is used heavily in the Spring framework to extend classes at compile-time. By default, the name of a generated class:

    […​] is composed of a prefix based on the name of the superclass, a fixed string incorporating the CGLIB class responsible for generation, and a hashcode derived from the parameters used to create the object.

    Consequently, one if faced with the following exception:

    java.security.AccessControlException: access denied ("java.io.FilePermission"
        "[...]/boot/autoconfigure/web/MultipartAutoConfiguration$$EnhancerBySpringCGLIB$$cb1b157aCustomizer.class"
        "read")
      at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) ~[na:1.8.0_92]
      at java.security.AccessController.checkPermission(AccessController.java:884) [na:1.8.0_92]
      at java.lang.SecurityManager.checkPermission(SecurityManager.java:549) ~[na:1.8.0_92]
      at java.lang.SecurityManager.checkRead(SecurityManager.java:888) ~[na:1.8.0_92]
      at java.io.File.exists(File.java:814) ~[na:1.8.0_92]
      at org.apache.catalina.webresources.DirResourceSet.getResource(...)
      at org.apache.catalina.webresources.StandardRoot.getResourceInternal(...)
      at org.apache.catalina.webresources.Cache.getResource(Cache.java:62) ~[tomcat-embed-core-8.5.6.jar!/:8.5.6]
      at org.apache.catalina.webresources.StandardRoot.getResource(...)
      at org.apache.catalina.webresources.StandardRoot.getClassLoaderResource(...)
      at org.apache.catalina.loader.WebappClassLoaderBase.findClassInternal(...)
      at org.apache.catalina.loader.WebappClassLoaderBase$PrivilegedFindClassByName.run(...)
      at org.apache.catalina.loader.WebappClassLoaderBase$PrivilegedFindClassByName.run(...)
      at java.security.AccessController.doPrivileged(Native Method) [na:1.8.0_92]
      at org.apache.catalina.loader.WebappClassLoaderBase.findClass()
      at org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader.findClassIgnoringNotFound()
      at org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader.loadClass()
      at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(...)
      at java.lang.Class.forName0(Native Method) [na:1.8.0_92]
      at java.lang.Class.forName(Class.java:348) [na:1.8.0_92]

    It looks quite an easy file permission fix, but it isn’t: for whatever reason, the hashcode used by CGLIB to extend MultipartAutoConfiguration changes at every compilation. Hence, a more lenient generic permission is required:

    permission java.io.FilePermission "src/main/webapp/WEB-INF/classes/org/springframework/boot/autoconfigure/web/*", "read";

    Launching is not the end

    Unfortunately, once the application has been successfully launched doesn’t mean it stops there. Browsing the home page yields a new bunch of security exceptions.

    For example, Tomcat needs to bind to port 8080, but this is a potential insecure action:

    java.security.AccessControlException: access denied ("java.net.SocketPermission" "localhost:8080" "listen,resolve")

    The permission to fix it is pretty straightforward:

    permission java.net.SocketPermission "localhost:8080", "listen,resolve";

    However, actually browsing the app brings a new exception:

    java.security.AccessControlException: access denied ("java.net.SocketPermission" "[0:0:0:0:0:0:0:1]:56733" "accept,resolve")

    That wouldn’t be bad if the port number didn’t change with every launch. A few attempts reveal that it seems to start from around 55400. Good thing that the socket permission allows for a port range:

    permission java.net.SocketPermission "[0:0:0:0:0:0:0:1]:55400-", "accept,resolve";

    Lessons learned

    Though it was very fulfilling to have created the policy file, the true value lies in the lessons learned.

    • The crafting of a custom policy file for a specific application is quite trivial, but very time-consuming. I didn’t finish completely and spent around one day for a small-sized application. Time might be a valid reason why policy files are never in use.
    • For large applications, I believe it’s not only possible but desirable to automate the crafting process: run the app, read the exception, create the associated permission, and update the policy file accordingly.
    • Patterns are recognizable in the policy file: sets of permissions are dedicated to a specific library, such as Spring Boot’s actuator. If each framework/library would provide the minimum associated policy file that allows it to work correctly, crafting a policy file for an app would just mean aggregating all files for every library.
    • Randomness (such as random port number) and bad coding practices (such as JCache’s) require more coarse-grained permissions. On one hand, it speeds up the crafting process; on the other hand, it increases the potential attack surface.

    In all cases, running the JVM in sandbox mode is not an option in security-aware environments.

    To go further:

    Categories: Java Tags: JVMsecuritySpring Bootpolicy
  • Compilation of Java code on the fly

    Rusty machine

    Java makes it possible to compile Java code at runtime… any Java code.

    The entry-point to the compilation is the ToolProvider class. From its Javadoc:

    Provides methods for locating tool providers, for example, providers of compilers. This class complements the functionality of ServiceLoader.

    This class is available in Java since version 1.6 - released 10 years ago, but seems to have been largely ignored.

    The code

    Here’s a snippet that allows that:

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    public class EvilExecutor {
    
        private String readCode(String sourcePath) throws FileNotFoundException {
            InputStream stream = new FileInputStream(sourcePath);
            String separator = System.getProperty("line.separator");
            BufferedReader reader = new BufferedReader(new InputStreamReader(stream));
            return reader.lines().collect(Collectors.joining(separator));
        }
    
        private Path saveSource(String source) throws IOException {
            String tmpProperty = System.getProperty("java.io.tmpdir");
            Path sourcePath = Paths.get(tmpProperty, "Harmless.java");
            Files.write(sourcePath, source.getBytes(UTF_8));
            return sourcePath;
        }
    
        private Path compileSource(Path javaFile) {
            JavaCompiler compiler = ToolProvider.getSystemJavaCompiler();
            compiler.run(null, null, null, javaFile.toFile().getAbsolutePath());
            return javaFile.getParent().resolve("Harmless.class");
        }
    
        private void runClass(Path javaClass)
                throws MalformedURLException, ClassNotFoundException, IllegalAccessException, InstantiationException {
            URL classUrl = javaClass.getParent().toFile().toURI().toURL();
            URLClassLoader classLoader = URLClassLoader.newInstance(new URL[]{classUrl});
            Class<?> clazz = Class.forName("Harmless", true, classLoader);
            clazz.newInstance();
        }
    
        public void doEvil(String sourcePath) throws Exception {
            String source = readCode(sourcePath);
            Path javaFile = saveSource(source);
            Path classFile = compileSource(javaFile);
            runClass(classFile);
        }
    
        public static void main(String... args) throws Exception {
            new EvilExecutor().doEvil(args[0]);
        }
    }
    

    Some explanations are in order:

    • readCode(): reads the source code from an arbitrary file on the file system, and returns it as a string. An alternative implementation would get the source from across the network.
    • saveSource(): creates a new file from the source code in a read-enabled directory. The file name is hard-coded, more refined versions would parse the code parameter to create a file named according to the class name it contains.
    • compileSource(): compiles the class file out of the java file.
    • runClass: loads the compiled class and instantiates a new object. To be independent from any cast, the to-be-executed code should be set in the constructor of the external source code class.

    The issue

    From a feature point of view, compiling code on the fly boosts the value of the Java language compared to others that don’t provide this feature. From a security point of view, this is a nightmare. The thought of being able to execute arbitrary code in production should send shivers down anyone’s spine who is part of any IT organization, developers included, if not mostly.

    Seasoned developers/ops or regular readers probably remember about the Java security manager and how to activate it:

    java -Djava.security.manager -cp target/classes ch.frankel.blog.runtimecompile.EvilExecutor harmless.txt
    

    Executing the above command-line will yield the following result:

    [email protected]
    Exception in thread "main" java.security.AccessControlException:
        access denied ("java.io.FilePermission" "harmless.txt" "read")
      at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
      at java.security.AccessController.checkPermission(AccessController.java:884)
      at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
      at java.lang.SecurityManager.checkRead(SecurityManager.java:888)
      at java.io.FileInputStream.<init>(FileInputStream.java:127)
      at java.io.FileInputStream.<init>(FileInputStream.java:93)
      at ch.frankel.blog.runtimecompile.EvilExecutor.readCode(EvilExecutor.java:19)
      at ch.frankel.blog.runtimecompile.EvilExecutor.doEvil(EvilExecutor.java:47)
      at ch.frankel.blog.runtimecompile.EvilExecutor.main(EvilExecutor.java:56)
    

    Conclusion

    The JVM offers plenty of features. A with any tool, they can be used for good or bad. It’s up to everyone to feel responsible about properly securing one’s JVM, doubly so in sensitive fields - banking, military, etc.

    Categories: Java Tags: security
  • Exploring data sets with Kibana

    In this post, I’d like to explore a sample data set using Kibana.

    This requires some data to start with: let’s index some tweets. It’s quite straightforward to achieve that by following explanations found in my good friend David’s blog post and wait for some time to fill the index with data.

    Basic metric

    Let’s start with something basic, the number of tweets indexed so far.

    In Kibana, go to Visualize ▸ Metric, then choose the twitter index. For the Aggregation field, choose "Count"; then click on Save and name the visualization accordingly e.g. "Number of tweets".

    Create a basic metric Display the number of tweets

    Geo-map

    Another simple visualization is to display the tweets based on their location on a world map.

    In Kibana, go to Visualize ▸ Tile map, then choose the twitter index.

    Select Geo Coordinates for the bucket type and keep default values,Geohash for Aggregation and coordinates.coordinates for Field.

    Localized map of tweets

    Bucket metric

    For this kind of metric, suppose a business requirement is to display the top 5 users. Unfortunately, as some (most?) business requirements go, this is not deterministic enough. It misses both the range and the aggregation period. Let’s agree for range time to be a sliding window over the last day, and the period to be an hour.

    In Kibana, go to Visualize ▸ Vertical bar chart, then choose the twitter index. Then:

    • For the Y-Axis, keep Count for the Aggregation field
    • Choose X-Axis for the buckets type
      • Select Date histogram for the Aggregation field
      • Keep the value @timestamp for the Field field
      • Set the Interval field to Hourly
    • Click on Add sub-buckets
    • Choose Split bars for the buckets type
      • Select Terms for the Sub Aggregation field
      • user.screen.name for the Field field
      • Keep the other fields default value
    • Don’t forget to click on the Apply changes
    • Click on Save and name the visualization accordingly e.g. "Top 5 users hourly".

    Create a bucket metric Display the top 5 users hourly

    Equivalent visualisations

    Other visualizations can be used with the exact same configuration: Area chart and Data table.

    The output of the Area chart is not as readable, regarding the explored data set, but the Data table offers interesting options.

    From a visualization, click on the bottom right arrow icon to display a table view of the data instead of a graphic.

    Alternative tabular metric display

    Visualizations make use of Elasticsearch public API. From the tabular view, the JSON request can also be displayed by clicking on the Request button (oh, surprise…​). This way, Kibana can be used as a playground to quickly prototype requests before using them in one’s own applications.

    Executed API request

    Changing requirements a bit

    The above visualization picks out the 5 top users having the most tweeted during each hour and display them during the last day. That’s the reason why there are more than 5 users displayed. But the above requirement can be interpreted in another way: take the top 5 users over the course of the last day, and break their number of tweets by hour.

    To do that, just move the X-Axis bucket below the Split bars bucket. This will change the output accordingly.

    Display the top 5 users over the last day

    Filtering irrelevant data

    As can be seen in the above histogram, top users mostly are about recruiting and/or job offers. This is not really what is wanted in the first place. It’s possible to remove this noise by adding a filter: in the Split bars section, click on Advanced to display additional parameters and type the desired regex in the Exclude field.

    Filter out a bucket metric

    The new visualization is quite different:

    Display the top 5 users hourly without any recruitment-related user

    Putting it all together

    With the above visualizations available and configured, it’s time to put them together on a dedicated dashboard. Go to Dashboard ▸ Add to list all available visualizations.

    Add visualizations to a dashboard

    It’s as simple as clicking on the desired one, laying it out on the board and resetting its size. Rinse and repeat until happy with the result and then click on Save.

    A configured dashboard

    Icing on the cake, using the Rectangle tool on the map visualization will automatically add a filter that only displays data bound by the rectangle coordinates for all visualizations found on the dashboard.

    A filtered dashboard

    That trick is not limited to the map visualization (try playing with other ones) but filtering on location quickly gives insights when exploring data sets.

    Conclusion

    While this post only brushes off the surface of what Kibana has to offer, there are more visualizations available as well as Timelion, the new powerful (but sadly under-documented) the "time series expression interface". In all cases, even basic features as shown above already provide plenty of different options to make sense of one’s data sets.

    Categories: Technical Tags: elasticsearchelastickibanabig data
  • Open your classes and methods in Kotlin

    Kotlin icon

    Though Kotlin and Spring Boot play well together, there are some friction areas between the two. IMHO, chief among them is the fact that Kotlin classes and methods are final by default.

    The Kotlin docs cite the following reason:

    The open annotation on a class is the opposite of Java’s final: it allows others to inherit from this class. By default, all classes in Kotlin are final, which corresponds to Effective Java, Item 17: Design and document for inheritance or else prohibit it.

    Kotlin designers took this advice very seriously by making all classes final by default. In order to make a class (respectively a method) inheritable (resp. overridable), it has to be annotated with the open keyword. That’s unfortunate because a lot of existing Java libraries require classes and methods to be non-final. In the Spring framework, those mainly include @Configuration classes and @Bean methods but there also Mockito and countless other frameworks and libs. All have in common to use the cglib library to generate a child class of the referenced class. Ergo: final implies no cglib implies no Spring, Mockito, etc.

    That means one has to remember to annotate every required class/method. This is not only rather tedious, but also quite error-prone. As an example, the following is the message received when one forgets about the annotation on a Spring configuration class:

    org.springframework.beans.factory.parsing.BeanDefinitionParsingException: Configuration problem:
    @Configuration class 'KotlindemoApplication' may not be final. Remove the final modifier to continue.
    

    Here’ what happens when the @Bean method is not made open:

    org.springframework.beans.factory.parsing.BeanDefinitionParsingException: Configuration problem:
    @Bean method 'controller' must not be private or final; change the method's modifiers to continue
    

    The good thing is that the rule are quite straightforward: if a class is annotated with @Configuration or a method with @Bean, they should be marked open as well. From Kotlin 1.0.6, there’s a compiler plugin to automate this process, available in Maven (and in Gradle as well) through a compiler plugin’s dependency. Here’s the full configuration snippet:

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    <plugin>
      <artifactId>kotlin-maven-plugin</artifactId>
      <groupId>org.jetbrains.kotlin</groupId>
      <version>${kotlin.version}</version>
      <configuration>
        <compilerPlugins>
          <plugin>all-open</plugin>
        </compilerPlugins>
        <pluginOptions>
          <option>all-open:annotation=org.springframework.boot.autoconfigure.SpringBootApplication</option>
          <option>all-open:annotation=org.springframework.context.annotation.Bean</option>
        </pluginOptions>
      </configuration>
      <dependencies>
        <dependency>
          <groupId>org.jetbrains.kotlin</groupId>
          <artifactId>kotlin-maven-allopen</artifactId>
          <version>${kotlin.version}</version>
        </dependency>
      </dependencies>
      <executions>
        <execution>
          <id>compile</id>
          <phase>compile</phase>
          <goals>
            <goal>compile</goal>
          </goals>
        </execution>
        <execution>
          <id>test-compile</id>
          <phase>test-compile</phase>
          <goals>
            <goal>test-compile</goal>
          </goals>
        </execution>
      </executions>
    </plugin>
    

    Note on lines 10-11 the list of all annotations for which the open keyword is now not mandatory anymore.

    Even better, there’s an alternative plugin dedicated to Spring projects, that makes listing Spring-specific annotations not necessary. Lines 6-13 above can be replaced with the following for a shorter configuration:

    <configuration>
      <compilerPlugins>
        <plugin>spring</plugin>
      </compilerPlugins>
    </configuration>
    

    Whether using Spring, Mockito or any cglib-based framework/lib, the all-open plugin is a great way to streamline the development with the Kotlin language.

    Categories: Technical Tags: kotlinspring frameworkfinal