• ElasticSearch API cheatsheet

    ElasticSearch documentation is exhaustive, but the way it’s structured has some room for improvement. This post is meant as a cheat-sheet entry point into ElasticSearch APIs.

    Category Description Call examples

    Document API

    Single Document API

    Adds a new document

    PUT /my_index/my_type/1
    {
      "my_field" : "my_value"
    }
    
    POST /my_index/my_type
    {
      …​
    }
    
    PUT /my_index/my_type/1/_create
    {
      …​
    }

    Gets an existing document

    GET /my_index/my_type/0

    Deletes a document

    DELETE /my_index/my_type/0

    Updates a document

    PUT /my_index/my_type/1
    {
      …​
    }

    Multi-Document API

    Multi-get

    GET /_mget
    {
      "docs" : [
        {
          "_index" : "my_index",
          "_type" : "my_type",
          "_id" : "1"
        }
      ]
    }
    
    GET /my_index/_mget
    {
      "docs" : [
        {
          "_type" : "my_type",
          "_id" : "1"
        }
      ]
    }
    
    
    GET /my_index/my_type/_mget
    {
      "docs" : [
        {
          "_id" : "1"
        }
      ]
    }

    Performs many index/delete operations in one call

    Deletes by query

    POST /my_index/_delete_by_query
    {
      "query": {
        "match": {
          …​
        }
      }
    }

    Updates by query

    POST /my_index/_update_by_query?conflicts=proceed
    POST /my_index/_update_by_query?conflicts=proceed
    {
      "query": {
        "term": {
          "my_field": "my_value"
        }
      }
    }
    
    POST /my_index1,my_index2/my_type1,my_type2/_update_by_query

    Reindexes

    POST /_reindex
    {
      "source": {
        "index": "old_index"
      },
      "dest": {
        "index": "new_index"
      }
    }

    Search API

    URI Search

    Executes a search with query parameters on the URL

    GET /my_index/my_type/_search?q=my_field:my_value
    GET /my_index/my_type/_search
    {
      "query" : {
        "term" : { "my_field" : "my_value" }
      }
    }

    Search Shards API

    Gets indices/shards of a search would be executed against

    GET /my_index/_search_shards

    Count API

    Executes a count query

    GET /my_index/my_type/_count?q=my_field:my_value
    GET /my_index/my_type/_count
    {
      "query" : {
        "term" : { "my_field" : "my_value" }
      }
    }

    Validate API

    Validates a search query

    GET /my_index/my_type/_validate?q=my_field:my_value
    GET /my_index/my_type/_validate
    {
      "query" : {
        "term" : { "my_field" : "my_value" }
      }
    }

    Explain API

    Provides feedback on computation of a search

    GET /my_index/my_type/0/_explain
    {
      "query" : {
        "match" : { "message" : "elasticsearch" }
      }
    }
    GET /my_index/my_type/0/_explain?q=message:elasticsearch

    Profile API

    Provides timing information on individual components during a search

    GET /_search
    {
      "profile": true,
      "query" : {
        …​
      }
    }

    Field Stats API

    Finds statistical properties of fields without executing a search

    GET /_field_stats?fields=my_field
    GET /my_index/_field_stats?fields=my_field
    GET /my_index1,my_index2/_field_stats?fields=my_field

    Indices API

    Index management

    Instantiates a new index

    PUT /my_index
    {
      "settings" : {
        …​
      }
    }

    Deletes existing indices

    DELETE /my_index
    DELETE /my_index1,my_index2
    DELETE /my_index*
    DELETE /_all

    Retrieves information about indices

    GET /my_index
    GET /my_index*
    GET my_index/_settings,_mappings

    Checks whether an index exists

    HEAD /my_index

    Closes/opens an index

    POST /my_index/_close
    POST /my_index/_open

    Shrinks an index to a new index with fewer primary shards

    Rolls over an alias to a new index if conditions are met

    POST /my_index/_rollover
    {
      "conditions": {
        …​
      }
    }

    Mapping management

    Adds a new type to an existing index

    PUT /my_index/_mapping/new_type
    {
      "properties": {
        "my_field": {
          "type": "text"
        }
      }
    }

    Retrieves mapping definition for fields

    GET /my_index/_mapping/my_type/field/my_field
    GET /my_index1,my_index2/_mapping/my_type/field/my_field
    GET /_all/_mapping/my_type1,my_type2/field/my_field1,my_field2
    GET /_all/_mapping/my_type1*/field/my_field*

    Checks whether a type exists

    HEAD /my_index/_mapping/my_type

    Alias management

    Creates an alias over an index

    POST /_aliases
    {
      "actions" : [
        { "add" :
          { "index" : "my_index", "alias" : "my_alias" }
        }
      ]
    }
    
    POST /_aliases
    {
      "actions" : [
        { "add" :
          { "index" : ["index1", "index2"] , "alias" : "another_alias" }
        }
      ]
    }

    Removes an alias

    POST /_aliases
    {
      "actions" : [
        { "remove" :
          { "index" : "my_index", "alias" : "my_old_alias" }
        }
      ]
    }

    Index settings

    Updates settings of indices

    PUT /my_index/_settings
    {
      …​
    }

    Retrieves settings of indices

    GET /my_index/_settings

    Performs an analysis process of a text and return the tokens

    GET /_analyze
    {
      "analyzer" : "standard",
      "text" : "this is a test"
    }

    Creates a new template

    PUT /_template/my_template
    {
      …​
    }

    Deletes an existing template

    DELETE /_template/my_template

    Gets info about an existing template

    GET /_template/my_template

    Checks whether a template exists

    HEAD /_template/my_template

    Replica configuration

    Sets index data location on a disk

    Monitoring

    Provides statistics on indices

    GET /_stats
    GET /my_index1/_stats
    GET /my_index1,my_index2/_stats
    GET /my_index1/_stats/flush,merge

    Provides info on Lucene segments

    GET /_segments
    GET /my_index1/_segments
    GET /my_index1,my_index2/_segments

    Provide recovery info on indices

    GET /_recovery
    GET /my_index1/_recovery
    GET /my_index1,my_index2/_recovery

    Provide store info on shard copies of indices

    GET /_shard_stores
    GET /my_index1/_shard_stores
    GET /my_index1,my_index2/_shard_stores

    Status management

    Clears the cache of indices

    POST /_cache/clear
    POST /my_index/_cache/clear
    POST /my_index1,my_index2/_cache/clear

    Explicitly refreshes indices

    POST /_refresh
    POST /my_index/_refresh
    POST /my_index1,my_index2/_refresh

    Flushes in-memory transaction log on disk

    POST /_flush
    POST /my_index/_flush
    POST /my_index1,my_index2/_flush

    Merge Lucene segments

    POST /_forcemerge?max_num_segments=1
    POST /my_index/_forcemerge?max_num_segments=1
    POST /my_index1,my_index2/_forcemerge?max_num_segments=1

    cat API

    cat aliases

    Shows information about aliases, including filter and routing infos

    GET /_cat/aliases?v
    GET /_cat/aliases/my_alias?v

    cat allocations

    Provides a snapshot on how many shards are allocated and how much disk space is used for each node

    GET /_cat/allocation?v

    cat count

    Provides quick access to the document count

    GET /_cat/count?v
    GET /_cat/count/my_index?v

    cat fielddata

    Shows heap memory currently being used by fielddata

    GET /_cat/fielddata?v
    GET /_cat/fielddata/my_field1,my_field2?v

    cat health

    One-line representation of the same information from /_cluster/health

    GET /_cat/health?v
    GET /_cat/health?v&ts=0

    cat indices

    Provides a node-spanning cross-section of each index

    GET /_cat/indices?v
    GET /_cat/indices?v&s=index
    GET /_cat/indices?v&health=yellow
    GET /_cat/indices/my_index*?v&health=yellow

    cat master

    Displays the master’s node ID, bound IP address, and node name

    GET /_cat/master?v

    cat nodeattrs

    Shows custom node attributes

    GET /_cat/nodeattrs?v
    GET /_cat/nodeattrs?v&h=name,id,pid,ip

    cat nodes

    Shows cluster topology

    GET /_cat/nodes?v
    GET /_cat/nodes?v&h=name,id,pid,ip

    cat pending tasks

    Provides the same information as /_cluster/pending_tasks

    GET /_cat/pending_tasks?v

    cat plugins

    Provides a node-spanning view of running plugins per node

    GET /_cat/plugins?v
    GET /_cat/plugins?v&h=name,id,pid,ip

    cat recovery

    Shows on-going and completed index shard recoveries

    GET /_cat/recovery?v
    GET /_cat/recovery?v&h=name,id,pid,ip

    cat repositories

    Shows snapshot repositories registered in the cluster

    GET /_cat/repositories?v

    cat thread pool

    Shows cluster-wide thread pool statistics per node

    GET /_cat/thread_pool?v
    GET /_cat/thread_pool?v&h=id,pid,ip

    cat shards

    Displays shards to nodes relationships

    GET /_cat/shards?v
    GET /_cat/shards/my_index?v
    GET /_cat/shards/my_ind*?v

    cat segments

    Provides information similar to _segments

    GET /_cat/segments?v
    GET /_cat/segments/my_index?v
    GET /_cat/segments/my_index1,my_index2?v

    cat snapshots

    Shows snapshots belonging to a repository

    /_cat/snapshots/my_repo?v

    cat templates

    Provides information about existing templates

    GET /_cat/templates?v
    GET /_cat/templates/my_template
    GET /_cat/templates/my_template*

    Cluster API

    Cluster Health

    Gets the status of a cluster’s health

    GET /_cluster/health
    GET /_cluster/health?wait_for_status=yellow&timeout=50s
    GET /_cluster/health/my_index1
    GET /_cluster/health/my_index1,my_index2

    Cluster State

    Gets state information about a cluster

    GET /_cluster/state
    GET /_cluster/state/version,nodes/my_index1
    GET /_cluster/state/version,nodes/my_index1,my_index2
    GET /_cluster/state/version,nodes/_all

    Cluster Stats

    Retrieves statistics from a cluster

    GET /_cluster/stats
    GET /_cluster/stats?human&pretty

    Pending cluster tasks

    Returns a list of any cluster-level changes

    GET /_cluster/pending_tasks

    Cluster Reroute

    Executes a cluster reroute allocation

    GET /_cluster/reroute {
      …​
    }

    Cluster Update Settings

    Update cluster-wide specific settings

    GET /_cluster/settings
    {
      "persistent" : {
        …​
      },
      "transient" : {
        …​
      }
    }

    Node Stats

    Retrieves cluster nodes statistics

    GET /_nodes/stats
    GET /_nodes/my_node1,my_node2/stats
    GET /_nodes/127.0.0.1/stats
    GET /_nodes/stats/indices,os,process

    Node Info

    Retrieves cluster nodes information

    GET /_nodes
    GET /_nodes/my_node1,my_node2
    GET /_nodes/_all/indices,os,process
    GET /_nodes/indices,os,process
    GET /_nodes/my_node1,my_node2/_all

    Task Management API

    Retrieve information about tasks currently executing on nodes in the cluster

    GET /_tasks
    GET /_tasks?nodes=my_node1,my_node2
    GET /_tasks?nodes=my_node1,my_node2&actions=cluster:*

    Nodes Hot Threads

    Gets current hot threads on nodes in the cluster

    GET /_nodes/hot_threads
    GET /_nodes/hot_threads/my_node
    GET /_nodes/my_node1,my_node2/hot_threads

    Cluster Allocation Explain API

    Answers the question "why is this shard unnassigned?"

    GET /_cluster/allocation/explain
    GET /_cluster/allocation/explain
    {
      "index": "myindex",
      "shard": 0,
      "primary": false
    }

    marks an experimental (respectively new) API that is subject to removal (resp. change) in future versions

    Last updated on Feb. 22th
    Categories: Development Tags: ElasticElasticSearchAPI
  • Signing and verifying a standalone JAR

    Seal

    Last week, I wrote about the JVM policy file that explicitly lists allowed sensitive API calls when running the JVM in sandboxed mode. This week, I’d like to improve the security by signing the JAR.

    The nominal way

    This way doesn’t work. Readers more interested in the solution than the process should skip it.

    Create a keystore

    The initial step is to create a keystore if none is already available. There are plenty of online tutorials showing how to do that.

    keytool -genkey -keyalg RSA -alias selfsigned -keystore /path/to/keystore.jks -storepass password -validity 360

    Fill in information accordingly.

    Sign the application JAR

    Signing the application JAR must be part of the build process. With Maven, the JAR signer plugin is dedicated to that. Its usage is quite straightforward:

    <plugin>
      <artifactId>maven-jarsigner-plugin</artifactId>
      <version>1.4</version>
      <executions>
      <execution>
        <id>sign</id>
        <goals>
        <goal>sign</goal>
        </goals>
      </execution>
      </executions>
      <configuration>
      <keystore>/path/to/keystore.jks</keystore>
      <alias>selfsigned</alias>
      <storepass>${store.password}</storepass>
      <keypass>${key.password}</keypass>
      </configuration>
    </plugin>

    To create the JAR, invoke the usual command-line and pass both passwords as system properties:

    mvn package -Dstore.password=password -Dkey.password=password

    Alternatively, Maven’s encryption capabilities can be used to store passwords in a dedicated settings-security.xml to further improve security.

    Configure the policy file

    Once the JAR is signed, the policy file can be updated to make use of it. This requires only the following configuration steps:

    1. Point to the keystore
    2. Configure the allowed alias
    keystore "keystore.jks";
    
    grant signedBy "selfsigned" codeBase "file:target/spring-petclinic-1.4.2.jar" {
      ...
    }

    Notice the signedBy keyword followed by the alias name - the same one as in the keystore above.

    Launching the JAR with the policy file

    The same launch command can be used without any change:

    java -Djava.security.manager -Djava.security.policy=jvm.policy -jar target/spring-petclinic-1.4.2.jar

    Unfortunately, it doesn’t work though this particular permission had already been configured!

    Caused by: java.security.AccessControlException: access denied ("java.lang.reflect.ReflectPermission" "suppressAccessChecks")
      at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
      at java.security.AccessController.checkPermission(AccessController.java:884)
      at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
      at java.lang.reflect.AccessibleObject.setAccessible(AccessibleObject.java:128)
      at org.springframework.util.ReflectionUtils.makeAccessible(ReflectionUtils.java:475)
      at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:141)
      at org.springframework.boot.SpringApplication.createSpringFactoriesInstances(SpringApplication.java:420)

    The strangest part is that permissions requested before this one work all right. The reason is to be found in the particular structure of the JAR created by the Spring Boot plugin: JAR dependencies are packaged untouched in a BOOT-INF/lib folder in the executable JAR. Then Spring Boot code uses custom class-loading magic to load required classes from there.

    JAR signing works by creating a specific hash for each class, and by writing them into the JAR manifest file. During the verification phase, the hash of a class is computed and compared to the hash of the manifest. Hence, permissions related to classes located in the BOOT-INF/classes folder work as expected.

    However, the org.springframework.boot.SpringApplication class mentioned in the stack trace above is part of the spring-boot.jar located under BOOT-INF/lib: verification fails as there’s no hash available for the class in the manifest.

    Thus, usage of the Spring Boot plugin for JAR creation/launch is not compatible with JAR signing.

    The workaround

    Aside from Spring Boot, there’s a legacy way to create standalone JARs: the Maven Shade plugin. This will extract every class of every dependency in the final JAR. This is possible with Spring Boot apps, but it requires some slight changes to the POM:

    1. In the POM, remove the Spring Boot Maven plugin
    2. Configure the main class in the Maven JAR plugin:
      <plugin>
        <artifactId>maven-jar-plugin</artifactId>
        <version>3.0.2</version>
        <configuration>
          <archive>
            <manifest>
              <mainClass>org.springframework.samples.petclinic.PetClinicApplication</mainClass>
            </manifest>
          </archive>
        </configuration>
      </plugin>
    3. Finally, add the Maven Shade plugin to work its magic:
      <plugin>
        <artifactId>maven-shade-plugin</artifactId>
        <version>2.4.3</version>
        <configuration>
          <minimizeJar>true</minimizeJar>
        </configuration>
        <executions>
          <execution>
            <phase>package</phase>
            <goals>
              <goal>shade</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
    The command-line to launch the JAR doesn’t change but permissions depend on the executed code, coupled to the JAR structure. Hence, the policy file should be slightly modified.

    Lessons learned

    While it requires to be a little creative, it’s entirely possible to sign Spring Boot JARs by using the same techniques as for any other JARs.

    To go further:

    Categories: Java Tags: JVMsecurityJARSpring Bootpolicy
  • Proposal for a Java policy files crafting process

    Security guy on an escalator

    I’ve already written about the JVM security manager, and why it should be used - despite it being rarely the case, if ever. However, just advocating for it won’t change the harsh reality unless some guidelines are provided to do so. This post has the ambition to be the basis of such guidelines.

    As a reminder, the JVM can run in two different modes, standard and sandboxed. In the former, all API are available with no restriction; in the later, some API calls deemed sensitive are forbidden. In that case, explicit permissions to allow some of those calls can be configured in a dedicated policy file.

    Though running the JVM in sandbox mode is important, it doesn’t stop there .e.g. executing only digitally-signed code is also part of securing the JVM. This post is the first in a 2 parts-serie regarding JVM security.

    Description

    The process is based on the principle of least privilege. That directly translates into the following process:

    1. Start with a blank policy file
    2. Run the application
    3. Check the thrown security exception
    4. Add the smallest-grained permission possible in the policy file that allows to pass step 2
    5. Return to step 2 until the application can be run normally

    Relevant system properties include:

    • java.security.manager: activates the Java Security manager
    • java.security.policy: points to the desired policy file
    • java.security.debug: last but not least, activates debugging information when an absent privilege is required. There are a ton of options.

    That sounds easy enough but let’s go detail how it works with an example.

    A case study

    As a sample application, we will be using the Spring Pet Clinic, a typical albeit small-sized Spring Boot application.

    First steps

    Once the application has been built, launch it with the security manager:

    java -Djava.security.manager -Djava.security.policy=jvm.policy -jar target/spring-petclinic-1.4.2.jar

    This, of course, fails. The output is the following:

    Exception in thread "main" java.lang.IllegalStateException:
        java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "getProtectionDomain")
      at org.springframework.boot.loader.ExecutableArchiveLauncher.<init>(ExecutableArchiveLauncher.java:43)
      at org.springframework.boot.loader.JarLauncher.<init>(JarLauncher.java:37)
      at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:58)
    Caused by: java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "getProtectionDomain")
      at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
      at java.security.AccessController.checkPermission(AccessController.java:884)
      at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
      at java.lang.Class.getProtectionDomain(Class.java:2299)
      at org.springframework.boot.loader.Launcher.createArchive(Launcher.java:117)
      at org.springframework.boot.loader.ExecutableArchiveLauncher.<init>(ExecutableArchiveLauncher.java:40)
      ... 2 more

    Let’s add the permission relevant to the above "access denied" exception to the policy file:

    grant codeBase "file:target/spring-petclinic-1.4.2.jar" {
      permission java.lang.RuntimePermission "getProtectionDomain";
    };

    Notice the path pointing to the JAR. It prevents other potentially malicious archives to execute critical code. Onto the next blocker.

    Exception in thread "main"
        java.security.AccessControlException: access denied ("java.util.PropertyPermission" "java.protocol.handler.pkgs" "read")

    This can be fixed by adding the below line to the policy file:

    grant codeBase "file:target/spring-petclinic-1.4.2.jar" {
      permission java.lang.RuntimePermission "getProtectionDomain";
      permission java.util.PropertyPermission "java.protocol.handler.pkgs", "read";
    };

    Next please.

    Exception in thread "main"
        java.security.AccessControlException: access denied ("java.util.PropertyPermission" "java.protocol.handler.pkgs" "read")

    Looks quite similar, but it needs a write permission in addition to the read one. Sure it can be fixed by adding one more line, but there’s a shortcut available. Just specify all necessary attributes of the permission on the same line:

    grant codeBase "file:target/spring-petclinic-1.4.2.jar" {
      permission java.lang.RuntimePermission "getProtectionDomain";
      permission java.util.PropertyPermission "java.protocol.handler.pkgs", "read,write";
    };

    Rinse and repeat. Without further ado, the (nearly) final policy can be found online: a whooping ~1800 lines of configuration for the Spring Boot Pet Clinic as an executable JAR.

    Now that the general approach has been explained, it just needs to be followed until the application functions properly. The next section describe some specific glitches along the way.

    Securing Java logging

    At some point, nothing gets printed in the console anymore. The command-line just returns, that’s it. Comes the java.security.debug system property - described above, that helps resolve the issue:

    java -Djava.security.manager -Djava.security.policy=jvm.policy
         -Djava.security.debug=access,stacktrace -jar target/spring-petclinic-1.4.2.jar

    That yields the following stack:

    java.lang.Exception: Stack trace
      at java.lang.Thread.dumpStack(Thread.java:1329)
      at java.security.AccessControlContext.checkPermission(AccessControlContext.java:419)
      at java.security.AccessController.checkPermission(AccessController.java:884)
      at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
      at java.util.logging.LogManager.checkPermission(LogManager.java:1586)
      at java.util.logging.Logger.checkPermission(Logger.java:422)
      at java.util.logging.Logger.setLevel(Logger.java:1688)
      at java.util.logging.LogManager.resetLogger(LogManager.java:1354)
      at java.util.logging.LogManager.reset(LogManager.java:1332)
      at java.util.logging.LogManager$Cleaner.run(LogManager.java:239)

    It’s time for some real software engineering (also known as Google Search). The LogManager’s Javadoc tells about the LoggingPermission that needs to be added to the existing list of permissions:

    grant codeBase "file:target/spring-petclinic-1.4.2.jar" {
      permission java.lang.RuntimePermission "getProtectionDomain";
      ...
      permission java.util.PropertyPermission "PID", "read,write";
      permission java.util.logging.LoggingPermission "control";
    };

    That makes it possible to go further.

    Securing the reading of system properties and environment variables

    It’s even possible to watch Spring Boot log…​ until one realizes it’s made entirely of error messages about not being able to read a bunchload of system properties and environment variables. Here’s an excerpt:

    2017-01-22 00:30:17.118  INFO 46549 --- [           main] o.s.w.c.s.StandardServletEnvironment     :
      Caught AccessControlException when accessing system environment variable [logging.register_shutdown_hook];
      its value will be returned [null].
      Reason: access denied ("java.lang.RuntimePermission" "getenv.logging.register_shutdown_hook")
    2017-01-22 00:30:17.118  INFO 46549 --- [           main] o.s.w.c.s.StandardServletEnvironment     :
      Caught AccessControlException when accessing system property [logging_register-shutdown-hook];
      its value will be returned [null].
      Reason: access denied ("java.util.PropertyPermission" "logging_register-shutdown-hook" "read")

    I will spare you dear readers a lot of trouble: there’s no sense in configuring every property one by one as JCache requires read and write permissions on all properties. So just remove every fine-grained PropertyPermission so far and replace it with a catch-all coarse-grained one:

    permission java.util.PropertyPermission "*", "read,write";

    Seems like security was not one of JCache developers first priority. The following snippet is the code excerpt for javax.cache.Caching.CachingProviderRegistry.getCachingProviders():

    if (System.getProperties().containsKey(JAVAX_CACHE_CACHING_PROVIDER)) {
        String className = System.getProperty(JAVAX_CACHE_CACHING_PROVIDER);
        ...
    }

    Wow, it reads all properties! Plus the next line makes it a little redundant, no?

    As for environment variables, the Spring team seem to try to avoid developers configuration issues related to case and check every possible case combination, so there is a lot of different options.

    Variables and subdirectories

    At one point, Spring’s embedded Tomcat attempts - and fails, to create a subfolder into the java.io.tmpdir folder.

    java.lang.SecurityException: Unable to create temporary file
      at java.io.File.createTempFile(File.java:2018) ~[na:1.8.0_92]
      at java.io.File.createTempFile(File.java:2070) ~[na:1.8.0_92]
      at org.springframework.boot.context.embedded.AbstractEmbeddedServletContainerFactory.createTempDir(...)
      at org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedServletContainerFactory.getEmbeddedServletContainer(...)
      at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.createEmbeddedServletContainer(...)
      at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.onRefresh(...)
      ... 16 common frames omitted

    One could get away with that by "hard-configuring" the path, but that would just be a major portability issue. Permissions are able to use System properties.

    The second issue is the subfolder: there’s no way of knowing the folder name, hence it’s not possible to configure it beforehand. However, file permissions accept any direct children or any descendant in the hierarachy; the former is set with jokers, and the second with dashes. The final configuration looks like this:

    permission java.io.FilePermission "${java.io.tmpdir}/-", "read,write,delete";

    CGLIB issues

    CGLIB is used heavily in the Spring framework to extend classes at compile-time. By default, the name of a generated class:

    […​] is composed of a prefix based on the name of the superclass, a fixed string incorporating the CGLIB class responsible for generation, and a hashcode derived from the parameters used to create the object.

    Consequently, one if faced with the following exception:

    java.security.AccessControlException: access denied ("java.io.FilePermission"
        "[...]/boot/autoconfigure/web/MultipartAutoConfiguration$$EnhancerBySpringCGLIB$$cb1b157aCustomizer.class"
        "read")
      at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) ~[na:1.8.0_92]
      at java.security.AccessController.checkPermission(AccessController.java:884) [na:1.8.0_92]
      at java.lang.SecurityManager.checkPermission(SecurityManager.java:549) ~[na:1.8.0_92]
      at java.lang.SecurityManager.checkRead(SecurityManager.java:888) ~[na:1.8.0_92]
      at java.io.File.exists(File.java:814) ~[na:1.8.0_92]
      at org.apache.catalina.webresources.DirResourceSet.getResource(...)
      at org.apache.catalina.webresources.StandardRoot.getResourceInternal(...)
      at org.apache.catalina.webresources.Cache.getResource(Cache.java:62) ~[tomcat-embed-core-8.5.6.jar!/:8.5.6]
      at org.apache.catalina.webresources.StandardRoot.getResource(...)
      at org.apache.catalina.webresources.StandardRoot.getClassLoaderResource(...)
      at org.apache.catalina.loader.WebappClassLoaderBase.findClassInternal(...)
      at org.apache.catalina.loader.WebappClassLoaderBase$PrivilegedFindClassByName.run(...)
      at org.apache.catalina.loader.WebappClassLoaderBase$PrivilegedFindClassByName.run(...)
      at java.security.AccessController.doPrivileged(Native Method) [na:1.8.0_92]
      at org.apache.catalina.loader.WebappClassLoaderBase.findClass()
      at org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader.findClassIgnoringNotFound()
      at org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader.loadClass()
      at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(...)
      at java.lang.Class.forName0(Native Method) [na:1.8.0_92]
      at java.lang.Class.forName(Class.java:348) [na:1.8.0_92]

    It looks quite an easy file permission fix, but it isn’t: for whatever reason, the hashcode used by CGLIB to extend MultipartAutoConfiguration changes at every compilation. Hence, a more lenient generic permission is required:

    permission java.io.FilePermission "src/main/webapp/WEB-INF/classes/org/springframework/boot/autoconfigure/web/*", "read";

    Launching is not the end

    Unfortunately, once the application has been successfully launched doesn’t mean it stops there. Browsing the home page yields a new bunch of security exceptions.

    For example, Tomcat needs to bind to port 8080, but this is a potential insecure action:

    java.security.AccessControlException: access denied ("java.net.SocketPermission" "localhost:8080" "listen,resolve")

    The permission to fix it is pretty straightforward:

    permission java.net.SocketPermission "localhost:8080", "listen,resolve";

    However, actually browsing the app brings a new exception:

    java.security.AccessControlException: access denied ("java.net.SocketPermission" "[0:0:0:0:0:0:0:1]:56733" "accept,resolve")

    That wouldn’t be bad if the port number didn’t change with every launch. A few attempts reveal that it seems to start from around 55400. Good thing that the socket permission allows for a port range:

    permission java.net.SocketPermission "[0:0:0:0:0:0:0:1]:55400-", "accept,resolve";

    Lessons learned

    Though it was very fulfilling to have created the policy file, the true value lies in the lessons learned.

    • The crafting of a custom policy file for a specific application is quite trivial, but very time-consuming. I didn’t finish completely and spent around one day for a small-sized application. Time might be a valid reason why policy files are never in use.
    • For large applications, I believe it’s not only possible but desirable to automate the crafting process: run the app, read the exception, create the associated permission, and update the policy file accordingly.
    • Patterns are recognizable in the policy file: sets of permissions are dedicated to a specific library, such as Spring Boot’s actuator. If each framework/library would provide the minimum associated policy file that allows it to work correctly, crafting a policy file for an app would just mean aggregating all files for every library.
    • Randomness (such as random port number) and bad coding practices (such as JCache’s) require more coarse-grained permissions. On one hand, it speeds up the crafting process; on the other hand, it increases the potential attack surface.

    In all cases, running the JVM in sandbox mode is not an option in security-aware environments.

    To go further:

    Categories: Java Tags: JVMsecuritySpring Bootpolicy
  • Compilation of Java code on the fly

    Rusty machine

    Java makes it possible to compile Java code at runtime… any Java code.

    The entry-point to the compilation is the ToolProvider class. From its Javadoc:

    Provides methods for locating tool providers, for example, providers of compilers. This class complements the functionality of ServiceLoader.

    This class is available in Java since version 1.6 - released 10 years ago, but seems to have been largely ignored.

    The code

    Here’s a snippet that allows that:

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    public class EvilExecutor {
    
        private String readCode(String sourcePath) throws FileNotFoundException {
            InputStream stream = new FileInputStream(sourcePath);
            String separator = System.getProperty("line.separator");
            BufferedReader reader = new BufferedReader(new InputStreamReader(stream));
            return reader.lines().collect(Collectors.joining(separator));
        }
    
        private Path saveSource(String source) throws IOException {
            String tmpProperty = System.getProperty("java.io.tmpdir");
            Path sourcePath = Paths.get(tmpProperty, "Harmless.java");
            Files.write(sourcePath, source.getBytes(UTF_8));
            return sourcePath;
        }
    
        private Path compileSource(Path javaFile) {
            JavaCompiler compiler = ToolProvider.getSystemJavaCompiler();
            compiler.run(null, null, null, javaFile.toFile().getAbsolutePath());
            return javaFile.getParent().resolve("Harmless.class");
        }
    
        private void runClass(Path javaClass)
                throws MalformedURLException, ClassNotFoundException, IllegalAccessException, InstantiationException {
            URL classUrl = javaClass.getParent().toFile().toURI().toURL();
            URLClassLoader classLoader = URLClassLoader.newInstance(new URL[]{classUrl});
            Class<?> clazz = Class.forName("Harmless", true, classLoader);
            clazz.newInstance();
        }
    
        public void doEvil(String sourcePath) throws Exception {
            String source = readCode(sourcePath);
            Path javaFile = saveSource(source);
            Path classFile = compileSource(javaFile);
            runClass(classFile);
        }
    
        public static void main(String... args) throws Exception {
            new EvilExecutor().doEvil(args[0]);
        }
    }
    

    Some explanations are in order:

    • readCode(): reads the source code from an arbitrary file on the file system, and returns it as a string. An alternative implementation would get the source from across the network.
    • saveSource(): creates a new file from the source code in a read-enabled directory. The file name is hard-coded, more refined versions would parse the code parameter to create a file named according to the class name it contains.
    • compileSource(): compiles the class file out of the java file.
    • runClass: loads the compiled class and instantiates a new object. To be independent from any cast, the to-be-executed code should be set in the constructor of the external source code class.

    The issue

    From a feature point of view, compiling code on the fly boosts the value of the Java language compared to others that don’t provide this feature. From a security point of view, this is a nightmare. The thought of being able to execute arbitrary code in production should send shivers down anyone’s spine who is part of any IT organization, developers included, if not mostly.

    Seasoned developers/ops or regular readers probably remember about the Java security manager and how to activate it:

    java -Djava.security.manager -cp target/classes ch.frankel.blog.runtimecompile.EvilExecutor harmless.txt
    

    Executing the above command-line will yield the following result:

    [email protected]
    Exception in thread "main" java.security.AccessControlException:
        access denied ("java.io.FilePermission" "harmless.txt" "read")
      at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
      at java.security.AccessController.checkPermission(AccessController.java:884)
      at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
      at java.lang.SecurityManager.checkRead(SecurityManager.java:888)
      at java.io.FileInputStream.<init>(FileInputStream.java:127)
      at java.io.FileInputStream.<init>(FileInputStream.java:93)
      at ch.frankel.blog.runtimecompile.EvilExecutor.readCode(EvilExecutor.java:19)
      at ch.frankel.blog.runtimecompile.EvilExecutor.doEvil(EvilExecutor.java:47)
      at ch.frankel.blog.runtimecompile.EvilExecutor.main(EvilExecutor.java:56)
    

    Conclusion

    The JVM offers plenty of features. A with any tool, they can be used for good or bad. It’s up to everyone to feel responsible about properly securing one’s JVM, doubly so in sensitive fields - banking, military, etc.

    Categories: Java Tags: security
  • Exploring data sets with Kibana

    In this post, I’d like to explore a sample data set using Kibana.

    This requires some data to start with: let’s index some tweets. It’s quite straightforward to achieve that by following explanations found in my good friend David’s blog post and wait for some time to fill the index with data.

    Basic metric

    Let’s start with something basic, the number of tweets indexed so far.

    In Kibana, go to Visualize ▸ Metric, then choose the twitter index. For the Aggregation field, choose "Count"; then click on Save and name the visualization accordingly e.g. "Number of tweets".

    Create a basic metric Display the number of tweets

    Geo-map

    Another simple visualization is to display the tweets based on their location on a world map.

    In Kibana, go to Visualize ▸ Tile map, then choose the twitter index.

    Select Geo Coordinates for the bucket type and keep default values,Geohash for Aggregation and coordinates.coordinates for Field.

    Localized map of tweets

    Bucket metric

    For this kind of metric, suppose a business requirement is to display the top 5 users. Unfortunately, as some (most?) business requirements go, this is not deterministic enough. It misses both the range and the aggregation period. Let’s agree for range time to be a sliding window over the last day, and the period to be an hour.

    In Kibana, go to Visualize ▸ Vertical bar chart, then choose the twitter index. Then:

    • For the Y-Axis, keep Count for the Aggregation field
    • Choose X-Axis for the buckets type
      • Select Date histogram for the Aggregation field
      • Keep the value @timestamp for the Field field
      • Set the Interval field to Hourly
    • Click on Add sub-buckets
    • Choose Split bars for the buckets type
      • Select Terms for the Sub Aggregation field
      • user.screen.name for the Field field
      • Keep the other fields default value
    • Don’t forget to click on the Apply changes
    • Click on Save and name the visualization accordingly e.g. "Top 5 users hourly".

    Create a bucket metric Display the top 5 users hourly

    Equivalent visualisations

    Other visualizations can be used with the exact same configuration: Area chart and Data table.

    The output of the Area chart is not as readable, regarding the explored data set, but the Data table offers interesting options.

    From a visualization, click on the bottom right arrow icon to display a table view of the data instead of a graphic.

    Alternative tabular metric display

    Visualizations make use of Elasticsearch public API. From the tabular view, the JSON request can also be displayed by clicking on the Request button (oh, surprise…​). This way, Kibana can be used as a playground to quickly prototype requests before using them in one’s own applications.

    Executed API request

    Changing requirements a bit

    The above visualization picks out the 5 top users having the most tweeted during each hour and display them during the last day. That’s the reason why there are more than 5 users displayed. But the above requirement can be interpreted in another way: take the top 5 users over the course of the last day, and break their number of tweets by hour.

    To do that, just move the X-Axis bucket below the Split bars bucket. This will change the output accordingly.

    Display the top 5 users over the last day

    Filtering irrelevant data

    As can be seen in the above histogram, top users mostly are about recruiting and/or job offers. This is not really what is wanted in the first place. It’s possible to remove this noise by adding a filter: in the Split bars section, click on Advanced to display additional parameters and type the desired regex in the Exclude field.

    Filter out a bucket metric

    The new visualization is quite different:

    Display the top 5 users hourly without any recruitment-related user

    Putting it all together

    With the above visualizations available and configured, it’s time to put them together on a dedicated dashboard. Go to Dashboard ▸ Add to list all available visualizations.

    Add visualizations to a dashboard

    It’s as simple as clicking on the desired one, laying it out on the board and resetting its size. Rinse and repeat until happy with the result and then click on Save.

    A configured dashboard

    Icing on the cake, using the Rectangle tool on the map visualization will automatically add a filter that only displays data bound by the rectangle coordinates for all visualizations found on the dashboard.

    A filtered dashboard

    That trick is not limited to the map visualization (try playing with other ones) but filtering on location quickly gives insights when exploring data sets.

    Conclusion

    While this post only brushes off the surface of what Kibana has to offer, there are more visualizations available as well as Timelion, the new powerful (but sadly under-documented) the "time series expression interface". In all cases, even basic features as shown above already provide plenty of different options to make sense of one’s data sets.

    Categories: Technical Tags: elasticsearchelastickibanabig data
  • Open your classes and methods in Kotlin

    Kotlin icon

    Though Kotlin and Spring Boot play well together, there are some friction areas between the two. IMHO, chief among them is the fact that Kotlin classes and methods are final by default.

    The Kotlin docs cite the following reason:

    The open annotation on a class is the opposite of Java’s final: it allows others to inherit from this class. By default, all classes in Kotlin are final, which corresponds to Effective Java, Item 17: Design and document for inheritance or else prohibit it.

    Kotlin designers took this advice very seriously by making all classes final by default. In order to make a class (respectively a method) inheritable (resp. overridable), it has to be annotated with the open keyword. That’s unfortunate because a lot of existing Java libraries require classes and methods to be non-final. In the Spring framework, those mainly include @Configuration classes and @Bean methods but there also Mockito and countless other frameworks and libs. All have in common to use the cglib library to generate a child class of the referenced class. Ergo: final implies no cglib implies no Spring, Mockito, etc.

    That means one has to remember to annotate every required class/method. This is not only rather tedious, but also quite error-prone. As an example, the following is the message received when one forgets about the annotation on a Spring configuration class:

    org.springframework.beans.factory.parsing.BeanDefinitionParsingException: Configuration problem:
    @Configuration class 'KotlindemoApplication' may not be final. Remove the final modifier to continue.
    

    Here’ what happens when the @Bean method is not made open:

    org.springframework.beans.factory.parsing.BeanDefinitionParsingException: Configuration problem:
    @Bean method 'controller' must not be private or final; change the method's modifiers to continue
    

    The good thing is that the rule are quite straightforward: if a class is annotated with @Configuration or a method with @Bean, they should be marked open as well. From Kotlin 1.0.6, there’s a compiler plugin to automate this process, available in Maven (and in Gradle as well) through a compiler plugin’s dependency. Here’s the full configuration snippet:

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    <plugin>
      <artifactId>kotlin-maven-plugin</artifactId>
      <groupId>org.jetbrains.kotlin</groupId>
      <version>${kotlin.version}</version>
      <configuration>
        <compilerPlugins>
          <plugin>all-open</plugin>
        </compilerPlugins>
        <pluginOptions>
          <option>all-open:annotation=org.springframework.boot.autoconfigure.SpringBootApplication</option>
          <option>all-open:annotation=org.springframework.context.annotation.Bean</option>
        </pluginOptions>
      </configuration>
      <dependencies>
        <dependency>
          <groupId>org.jetbrains.kotlin</groupId>
          <artifactId>kotlin-maven-allopen</artifactId>
          <version>${kotlin.version}</version>
        </dependency>
      </dependencies>
      <executions>
        <execution>
          <id>compile</id>
          <phase>compile</phase>
          <goals>
            <goal>compile</goal>
          </goals>
        </execution>
        <execution>
          <id>test-compile</id>
          <phase>test-compile</phase>
          <goals>
            <goal>test-compile</goal>
          </goals>
        </execution>
      </executions>
    </plugin>
    

    Note on lines 10-11 the list of all annotations for which the open keyword is now not mandatory anymore.

    Even better, there’s an alternative plugin dedicated to Spring projects, that makes listing Spring-specific annotations not necessary. Lines 6-13 above can be replaced with the following for a shorter configuration:

    <configuration>
      <compilerPlugins>
        <plugin>spring</plugin>
      </compilerPlugins>
    </configuration>
    

    Whether using Spring, Mockito or any cglib-based framework/lib, the all-open plugin is a great way to streamline the development with the Kotlin language.

    Categories: Technical Tags: kotlinspring frameworkfinal
  • Feedback on Feeding Spring Boot metrics to Elasticsearch

    Logstash log

    Some weeks ago, I wrote a post detailing how to send JMX metrics from a Spring Boot app to Elasticsearch by developing another Spring Boot app.

    Getting to create such an app is not always the right idea but developers are makers - software makers, and developing new apps is quite alluring to them. However, in the overall scheme of things, this means time is not only spent in development, but also for maintenance during the entire lifetime of the app. Before going the development path, one should thoroughly check whether out-of-the-box alternatives exist.

    Back to JMX metrics: only the straightforward Logstash jmx plugin was tried before calling it quits because of an incompatibility with Elasticsearch 5. But an alternative exists with the Logstash http_poller plugin.

    This Logstash input plugin allows you to call an HTTP API, decode the output of it into event(s), and send them on their merry way. The idea behind this plugins came from a need to read springboot metrics endpoint, instead of configuring jmx to monitor my java application memory/gc/ etc.

    Jolokia is already in place, offering HTTP access to JMX. The last and only step is to configure HTTP poller plugin, which is fairly straighforward. The URL is composed of the standard actuator URL appended with /jolokia/read/ and the JMX’s ObjectName of the desired object. Here’s a sample configuration snippet, with URLs configured for:

    1. Response site for the root page
    2. HTTP 200 status code counter for the root page
    3. Operating system metrics
    4. Garbage collector metrics

    With the help of the jconsole, adding more metrics is a no-brainer.

    input {
      http_poller {
        urls => {
          "200.root" =>
            "http://localhost:8080/manage/jolokia/read/org.springframework.metrics:name=status,type=counter,value=200.root"
          "root" =>
            "http://localhost:8080/manage/jolokia/read/org.springframework.metrics:name=response,type=gauge,value=root"
          "OperatingSystem" => "http://localhost:8080/manage/jolokia/read/java.lang:type=OperatingSystem"
          "PS Scavenge" => "http://localhost:8080/manage/jolokia/read/java.lang:type=GarbageCollector,name=PS%20Scavenge"
        }
        request_timeout => 60
        schedule => { every => "10s"}
        codec => "json"
      }
    }
    

    This should output something akin to:

    {
              "request" => {
                "mbean" => "org.springframework.metrics:name=response,type=gauge,value=root",
                 "type" => "read"
              },
           "@timestamp" => 2016-01-01T10:19:45.914Z,
             "@version" => "1",
                "value" => {
                "Value" => 4163.0,
          "LastUpdated" => "2016-12-30T16:29:28+01:00"
                },
            "timestamp" => 1483121985,
               "status" => 200
    }
    

    Beyond the raw output, there are a couple of possible improvements:

    • value and request fields are inner objects, for no added value. Flattening the structure can go a long way toward making writing queries easier.
    • Adding tags depending on the type improve categorization. A possible alternative would be to parse the JMX compound name into dedicated fields with grok or dissect.
    • The @timestamp field can be replaced with the LastUpdated value and interpreted as a date.

    Filters to the rescue:

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    filter {
      mutate { add_field => { "[mbean][objectName]" => "%{request[mbean]}" }}
      mutate { remove_field => "request" }
    }
    
    filter {
      if [mbean][objectName] == "java.lang:type=OperatingSystem" {
        dissect { 
          mapping => {
            "mbean[objectName]" => "%{[mbean][prefix]}:type=%{[mbean][type]}"
          }
        }
        mutate { remove_field => "value[ObjectName]" }
      } else if [mbean][objectName] == "java.lang:name=PS Scavenge,type=GarbageCollector" {
        dissect { 
          mapping => {
            "mbean[objectName]" => "%{[mbean][prefix]}:name=%{[mbean][name]},type=%{[mbean][type]}"
          }
        }
        mutate { remove_field => "value[ObjectName]" }
      } else if [mbean][objectName] =~ "^.*,type=gauge,.*$" or [mbean][objectName] =~ "^.*,type=counter,.*$" {
        date { match => [ "%{value[lastUpdated]}", "ISO8601" ] }
        mutate { replace => { "value" => "%{value[Value]}" }}
        mutate { convert => { "value" => "float" }}
        if [mbean][objectName] =~ "^.*,type=gauge,.*$" {
          dissect { 
            mapping => {
              "mbean[objectName]" =>
                "%{[mbean][prefix]}:name=%{[mbean][name]},type=%{[mbean][type]},value=%{[mbean][page]}"
            }
          }
        } else if [mbean][objectName] =~ "^.*,type=counter,.*$" {
          dissect { 
            mapping => {
              "mbean[objectName]" => 
                "%{[mbean][prefix]}:name=%{[mbean][name]},type=%{[mbean][type]},value=%{[mbean][status]}.%{[mbean][page]}"
            }
          }
        }
      }
    }
    

    A little explanation might be in order.

    Lines 1-4
    Move the initial request->mbean field to the mbean->objectName field.
    Lines 7-13
    For OS metrics, create mbean nested fields out of the objectName nested field and remove it from the value field.
    Lines 14-20
    For GC metrics, create mbean nested fields out of the objectName nested field using a slightly different pattern and remove it from the value field.
    Lines 21-24
    For gauge or counter metrics, interpret the value->lastUpdated nested field as a date, move the nested value->Value field to the root and interpret as a float value.
    Lines 25-38
    For gauge or counter metrics, create mbean nested fields using a pattern specific for each metric.

    Coupled with the initial configuration, this outputs to the following (for the gauge):

    {
        "@timestamp" => 2016-01-01T10:54:15.359Z,
             "mbean" => {
            "prefix" => "org.springframework.metrics",
              "name" => "response",
        "objectName" => "org.springframework.metrics:name=response,type=gauge,value=root",
              "type" => "gauge",
             "value" => "root"
             },
          "@version" => "1",
             "value" => 4163.0,
         "timestamp" => 1483206855,
            "status" => 200,
              "tags" => []
    }
    

    Given the current complexity of the configuration, remember the next logical step is to decouple the single snippet into multiple files. The order used by Logstash is by lexicographical file name. Name files accordingly:

    • 00-app-input.conf
    • 10-global-filter.conf
    • 20-specific-filter.conf
    • 99-console-output.conf

    Note: gaps in the number scheme allow to add intermediary filters in the future, with no renaming

    All in all, everything required is available through configuration, without any coding. Always check that it’s not the case before reinventing the wheel.

    Categories: Technical Tags: elasticsearchlogstashelastic
  • Exploratory Infrastructure projects

    Exploring the jungle

    Nowadays, most companies use one or another Agile methodology for their software development projects. That makes people involved in software development projects at least aware of agile principles - whether they truly try to follow agile practices or just pay lip service to them for a variety of reasons remains debatable. To avoid any association with tainted practices, I’d rather use the name "Exploratory Development". As with software development, exploration has a vague feeling of the final target destination, and a more or less detailed understanding on how to get there.[1] Plus on a map, the plotted path is generally not a straight line.

    However, even with the rise of the DevOps movement, the operational side of projects still seems to remain oblivious to what happens on the other side of the curtain. This post aims to provide food for thoughts to think about how true Agile can be applied to Ops.

    The legacy approach

    As an example, let’s have an e-commerce application. Sometimes, bad stuff happens and the development team needs to access logs to analyze what happened. In general, due to "security" reasons, developers cannot directly access the system, and needs to ask the operations team to send the log file(s). This too common process results in frustration, time wasting, and contributes to build even taller walls between Dev and Ops. To improve the situation, an option could be to setup a datastore to store the logs into, and a webapp to make them available to developers.

    Here are some hypotheses regarding the source architecture components:

    • Load-balancer
    • Apache Web server
    • Apache Tomcat servlet container, where the e-commerce application is deployed
    • Solr server, that provide search and faceting features to the app

    In turn, relevant data/logs that needs to be stored include:

    • Web server logs
    • Application logs proper
    • Tomcat technical logs
    • JMX data
    • Solr logs

    Let’s implement those requirements with the Elastic stack. The target infrastructure could look like this:

    Target architecture

    Defining the architecture is generally not enough. Unless one work for a dream company that empowers employees to improve the situation on their own initiative (please send it my resume), chances are there’s a need for some estimates regarding the setup of this architecture. And that goes double in case it’s done for a customer company. You might push back, stating the evidence:

    We will ask for estimates and then treat them as deadlines

    Engineers will probably think that managers are asking them to stick their neck out to produce estimates for no real reasons, and push back. But the later will kindly remind the former to "just" base estimates on assumptions, as if it was a real solution instead of plausible deniability. In other words, an assumption is a way to escape blame for a wrong estimate, and yes, it sounds much closer to contract law than to engineering concepts. That means that if any of the listed assumption is not fulfilled, then it’s acceptable for estimates to be wrong. It also implies estimates are then not considered a deadline anymore - which they never were supposed to be in the first place, if only from a semantical viewpoint.[2]

    Notwithstanding all that, and for the sake of the argument, let’s try to review possible assumptions pertaining to the proposed architecture that might impact implementation time:

    • Hardware location: on-premise, cloud-based or a mix of both?
    • Underlying operating system(s): Nix, Windows, something else, or a mix?
    • Infrastructure virtualization degree: is the infrastructure physical, virtualized, both?
    • Co-location of the OS: are there requirements regarding the location of components on physical systems?
    • Hardware availability: should there be a need to purchase physical machines?
    • Automation readiness: is there any solution already in place to automate infrastructure management? If not, in how many environments will the implementation need to be setup and if more than 2, will replication be handled manually?
    • Clustering: is any component clustered? Which one(s)? For the application, is there a session replication solution in place? Which one?
    • Infrastructure access: needs to be on-site? Needs to get a security token hardware? From software?

    And those are quite basic items regarding hardware only. Other wide areas include software (volume of logs, criticality, hardware/software mis/match, versions, etc.), people (in-house support, etc.), planning (vacation seasons, etc.), and I’m probably forgetting some important ones too. Given the sheer number of available items - and assuming they all have been listed, it stands to reason that at least one assumption would prove wrong, hence making final estimates dead wrong. In that case, playing the estimate game is just another way to provide plausible deniability. A much more useful alternative would be to create a n-dimension matrix of all items, and estimate all possible combinations. But as in software projects, the event space has just too many parameters to do that in an acceptable timeframe.

    Proposal for an alternative

    That said, what about a real working alternative that might not be to satisfy dashboard managers but the underlying business? It would start by implementing the most basic requirement, and to add more features until it’s good enough, or enough budget has been spent. Here are some possible steps from the above example:

    Foundation setup

    The initial goal of the setup is to enable log access and the most important logs are applications logs. Hence, the first setup is the following:

    Foundation architecture
    More JVM logs

    From this point on, a near-zero effort is to add scraping Tomcat’s log, to help with incident analysis by adding correlation.

    More JVM logs
    Machine decoupling

    The next logical step is to move the Elasticsearch instance to its own dedicated machine, to add an extra level of modularity to the overall architecture.

    Machine decoupling
    Even more logs

    At this point, additional logs from other components - load balancer, Solr server, etc. can be sent to Elasticsearch to improve issue-solving involving different components.

    Performance improvement

    Given that Logstash is written in Ruby, there might be some performance issues on running Logstash directly along the component, depending on each machine specific load and performances. Elastic realized it some time ago and now propose better performance via dedicated Beat. Every Logstash instance can be replaced by Filebeats.

    Not only logs

    With the Jolokia library, it’s possible to expose JMX beans through an HTTP interface. Unfortunately, there are only a few available Beats and none of them handle HTTP. However, Logstash with the http-poller plugin gets the job done.

    Reliability

    In order to improve reliability, Elasticsearch can be cluster-ized.

    The good thing about those steps is that they can implemented in (nearly) any order. This means that after laying out the base foundation - the first step, the stakeholder can decide which on makes sense for its specific context, or to stop because it’s enough regarding the added value.

    At this point, estimates still might make sense regarding the first step. But after eliminating most complexity (and its related uncertainty), it feels much more comfortable estimating the setup of an Elastic stack in a specific context.

    Conclusion

    As stated above, whether Agile principles are implemented in software development projects can be subject to debate. However, my feeling is that they have not reached the Ops sphere yet. That’s a shame, because as in Development, projects can truly benefit from real Agile practices. However, to prevent association with Agile cargo cult, I proposed the use of the term "Exploratory Infrastructure". This post described a proposal to apply such an exploratory approach to a sample infrastructure project. The main drawback of such approach is that it will cost more, as the path straight; the main benefit is that at every step, the stakeholder can choose to pursue or stop, taking into account the law of diminishing returns.


    1. This an HTML joke, because "emphasis on less"
    2. If an estimate needs to be really taken as an estimate, the compound word guesstimate is available. Aren’t enterprise semantics wonderful?
    Categories: Technical Tags: agileinfrastructureopsdevops
  • Starting Beats for Java developers

    Beats logo

    Last week, I wrote about how one could start developing one’s Logstash plugin coming from a Java developer background. However, with the acquisition of Packetbeat, Logstash now has help from Beats to push data to Elasticsearch. Beats are developed in Go, another challenge for traditional Java developers. This week, I tried porting my Logstash Reddit plugin to a dedicated Beat. This post documents my findings (spoiler: I found it much easier than with Ruby).

    Setting up the environment

    On OSX, installing the go executable is easy as pie:

    brew install go
    

    While Java or Ruby (or any language I know for that matter) can reside anywhere on the local filesystem, Go projects must all be situated in one single dedicated location, available under the $GOPATH environment variable.

    Creating the project

    As for Logstash plugins, Beats projects can be created from a template. The documentation to do that is quite straightforward. Given Go’s rigid requirements regarding location on the filesystem, just following instructions yields a new ready-to-use Go project.

    The default template code will repeatedly send an event with an incremented counter in the console:

    ./redditbeat -e -d "*"
    2016/12/13 22:55:56.013362 beat.go:267: INFO
      Home path: [/Users/i303869/projects/private/go/src/github.com/nfrankel/redditbeat]
      Config path: [/Users/i303869/projects/private/go/src/github.com/nfrankel/redditbeat]
      Data path: [/Users/i303869/projects/private/go/src/github.com/nfrankel/redditbeat/data]
      Logs path: [/Users/i303869/projects/private/go/src/github.com/nfrankel/redditbeat/logs]
    2016/12/13 22:55:56.013390 beat.go:177: INFO Setup Beat: redditbeat; Version: 6.0.0-alpha1
    2016/12/13 22:55:56.013402 processor.go:43: DBG  Processors: 
    2016/12/13 22:55:56.013413 beat.go:183: DBG  Initializing output plugins
    2016/12/13 22:55:56.013417 logp.go:219: INFO Metrics logging every 30s
    2016/12/13 22:55:56.013518 output.go:167: INFO Loading template enabled. Reading template file:
      /Users/i303869/projects/private/go/src/github.com/nfrankel/redditbeat/redditbeat.template.json
    2016/12/13 22:55:56.013888 output.go:178: INFO Loading template enabled for Elasticsearch 2.x. Reading template file:
      /Users/i303869/projects/private/go/src/github.com/nfrankel/redditbeat/redditbeat.template-es2x.json
    2016/12/13 22:55:56.014229 client.go:120: INFO Elasticsearch url: http://localhost:9200
    2016/12/13 22:55:56.014272 outputs.go:106: INFO Activated elasticsearch as output plugin.
    2016/12/13 22:55:56.014279 publish.go:234: DBG  Create output worker
    2016/12/13 22:55:56.014312 publish.go:276: DBG  No output is defined to store the topology.
      The server fields might not be filled.
    2016/12/13 22:55:56.014326 publish.go:291: INFO Publisher name: LSNM33795267A
    2016/12/13 22:55:56.014386 async.go:63: INFO Flush Interval set to: 1s
    2016/12/13 22:55:56.014391 async.go:64: INFO Max Bulk Size set to: 50
    2016/12/13 22:55:56.014395 async.go:72: DBG  create bulk processing worker (interval=1s, bulk size=50)
    2016/12/13 22:55:56.014449 beat.go:207: INFO redditbeat start running.
    2016/12/13 22:55:56.014459 redditbeat.go:38: INFO redditbeat is running! Hit CTRL-C to stop it.
    2016/12/13 22:55:57.370781 client.go:184: DBG  Publish: {
      "@timestamp": "2016-12-13T22:54:47.252Z",
      "beat": {
        "hostname": "LSNM33795267A",
        "name": "LSNM33795267A",
        "version": "6.0.0-alpha1"
      },
      "counter": 1,
      "type": "redditbeat"
    }
    

    Regarding command-line parameters: -e logs to the standard err, while -d "*" enables all debugging selectors. For the full list of parameters, type ./redditbeat --help.

    The code

    Go code is located in .go files (what a surprise…) in the project sub-folder of the $GOPATH/src folder.

    Configuration type

    The first interesting file is config/config.go which defines a struct to declare possible parameters for the Beat. As for the former Logstash plugin, let’s add a subreddit parameter, and sets it default value:

    type Config struct {
    	Period time.Duration `config:"period"`
    	Subreddit string `config:"subreddit"`
    }
    
    var DefaultConfig = Config {
    	Period: 15 * time.Second,
    	Subreddit: "elastic",
    }
    

    Beater type

    Code for the Beat itself is found in beater/redditbean.go. The default template creates a struct for the Beat and three functions:

    1. The Beat constructor - it reads the configuration:
      func New(b *beat.Beat, cfg *common.Config) (beat.Beater, error) { ... }
      
    2. The Run function - should loop over the main feature of the Beat:
      func (bt *Redditbeat) Run(b *beat.Beat) error { ... }
      
    3. The Stop function handles graceful shutdown:
      func (bt *Redditbeat) Stop() { ... }
      

    Note 1: there’s no explicit interface implementation in Go. Just implementing all methods from an interface creates an implicit inheritance relationship. For documentation purposes, here’s the Beater interface:

    type Beater interface {
    	Run(b *Beat) error
    	Stop()
    }
    

    Hence, since the Beat struct implements both Run and Stop, it is a Beater.

    Note 2: there’s no concept of class in Go, so methods cannot be declared on a concrete type. However, it exists the concept of extension functions: functions that can add behavior to a type (inside a single package). It needs to declare the receiver type: this is done between the fun keyword and the function name - here, it’s the Redditbeat type (or more correctly, a pointer to the Redditbeat type, but there’s an implicit conversion).

    The constructor and the Stop function can stay as they are, whatever feature should be developed must be in the Run function. In this case, the feature is to call the Reddit REST API and send a message for every Reddit post.

    The final code looks like the following:

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    func (bt *Redditbeat) Run(b *beat.Beat) error {
    	bt.client = b.Publisher.Connect()
    	ticker := time.NewTicker(bt.config.Period)
    	reddit := "https://www.reddit.com/r/" + bt.config.Subreddit + "/.json"
    	client := &http.Client {}
    	for {
    		select {
    		case <-bt.done:
    			return nil
    		case <-ticker.C:
    		}
    		req, reqErr := http.NewRequest("GET", reddit, nil)
    		req.Header.Add("User-Agent", "Some existing header to bypass 429 HTTP")
    		if (reqErr != nil) {
    			panic(reqErr)
    		}
    		resp, getErr := client.Do(req)
    		if (getErr != nil) {
    			panic(getErr)
    		}
    		body, readErr := ioutil.ReadAll(resp.Body)
    		defer resp.Body.Close()
    		if (readErr != nil) {
    			panic(readErr)
    		}
    		trimmedBody := body[len(prefix):len(body) - len(suffix)]
    		messages := strings.Split(string(trimmedBody), separator)
    		for i := 0; i < len(messages); i ++ {
    			event := common.MapStr{
    				"@timestamp": common.Time(time.Now()),
    				"type":       b.Name,
    				"message":    "{" + messages[i] + "}",
    			}
    			bt.client.PublishEvent(event)
    		}
    	}
    }
    

    Here’s an explanation of the most important pieces:

    • line 4: creates the Reddit REST URL by concatenating Strings, including the configuration Subreddit parameter. Remember that its default value has been defined in the config.go file.
    • line 5: get a reference on a new HTTP client type
    • line 12: create a new HTTP request. Note that Go allows for multiple return values.
    • line 13: if no standard request header is set, Reddit’s API will return a 429 status code
    • line 14: Go standard errors are not handled through exceptions but are returned along regular returned values. According to the Golang wiki:

      Indicating error conditions to callers should be done by returning error value

    • line 15: the panic() function is similar to throwing an exception in Java, climbing up the stack until it’s handled. For more information, check the relevant documentation.
    • line 17: executes the HTTP request
    • line 21: reads the response body into a byte array
    • line 22: closes the body stream. Note the defer keyword:

      A defer statement defers the execution of a function until the surrounding function returns.

    • line 26: creates a slice - a reference to a part of an array, of the whole response body byte array. In essence, it removes the prefix and the suffix to keep the relevant JSON value. It would be overkill to parse the byte array into JSON.
    • line 27: splits the slice to get each JSON fragment separately
    • line 29: create the message as a simple dictionary structure
    • line 34: send it

    Configure, Build, and launch

    Default configuration parameters can be found in the redditbeat.yml file at the root of the project. Note that additional common Beat parameters are listed in the redditbeat.full.yml, along with relevant comments.

    The interesting thing about Beats is that their messages can be sent directly to Elasticsearch or to Logstash for further processing. This is configured in the aforementioned configuration file.

    redditbeat:
      period: 10s
    output.elasticsearch:
      hosts: ["localhost:9200"]
    output.logstash:
      hosts: ["localhost:5044"]
      enabled: true
    

    This configuration snippet will loop over the Run method every 10 seconds and send messages to the Logstash instance running on localhost on port 5044. This can be overridden when running the Beat (see below).

    Note: for Logstash to accept messages from Beats, the Logstash Beat plugin must be installed and Logstash input must be configured for Beats:

    input {
      beats {
        port => 5044
      }
    }
    

    To build the project, type make at the project’s root. It will create an executable that can be run.

    ./redditbeat -e -E redditbeat.subreddit=java
    

    The -E flag may override parameters found in the embedded redditbeat.yml configuration file (see above). Here, it sets the subreddit to be read to “java” instead of the default “elastic”.

    The output looks like the following:

    2016/12/17 14:51:19.748329 client.go:184: DBG  Publish: {
      "@timestamp": "2016-12-17T14:51:19.748Z",
      "beat": {
        "hostname": "LSNM33795267A",
        "name": "LSNM33795267A",
        "version": "6.0.0-alpha1"
      },
      "message": "{
        \"kind\": \"t3\", \"data\": {
          \"contest_mode\": false, \"banned_by\": null, 
          \"domain\": \"blogs.oracle.com\", \"subreddit\": \"java\", \"selftext_html\": null, 
          \"selftext\": \"\", \"likes\": null, \"suggested_sort\": null, \"user_reports\": [], 
          \"secure_media\": null, \"saved\": false, \"id\": \"5ipzgq\", \"gilded\": 0, 
          \"secure_media_embed\": {}, \"clicked\": false, \"report_reasons\": null, 
          \"author\": \"pushthestack\", \"media\": null, \"name\": \"t3_5ipzgq\", \"score\": 11, 
          \"approved_by\": null, \"over_18\": false, \"removal_reason\": null, \"hidden\": false, 
          \"thumbnail\": \"\", \"subreddit_id\": \"t5_2qhd7\", \"edited\": false, 
          \"link_flair_css_class\": null, \"author_flair_css_class\": null, \"downs\": 0, 
          \"mod_reports\": [], \"archived\": false, \"media_embed\": {}, \"is_self\": false, 
          \"hide_score\": false, \"spoiler\": false, 
          \"permalink\": \"/r/java/comments/5ipzgq/jdk_9_will_no_longer_bundle_javadb/\", 
          \"locked\": false, \"stickied\": false, \"created\": 1481943248.0, 
          \"url\": \"https://blogs.oracle.com/java-platform-group/entry/deferring_to_derby_in_jdk\", 
          \"author_flair_text\": null, \"quarantine\": false, 
          \"title\": \"JDK 9 will no longer bundle JavaDB\", \"created_utc\": 1481914448.0, 
          \"link_flair_text\": null, \"distinguished\": null, \"num_comments\": 4, 
          \"visited\": false, \"num_reports\": null, \"ups\": 11
        }
      }",
      "type": "redditbeat"
    }
    

    Conclusion

    Strangely enough, I found developing a Beat easier than a Logstash plugin. Go is more low-level and some concepts feel really foreign (like implicit interface implementation), but the ecosystem is much simpler - as the language is more recent. Also, Beats are more versatile, in that they can send to Elasticsearch and/or Logstash.

    Categories: Development Tags: LogstashElasticsearchBeatGo
  • Starting Logstash plugin development for Java developers

    Logstash old logo

    I recently became interested in Logstash, and after playing with it for a while, I decided to create my own custom plugin for learning purpose. I chose to pull data from Reddit because a) I use it often and b) there’s no existing plugin that offers that.

    The Elasticsearch site offers quite an exhaustive documentation to create one’s own Logstash plugin. Such endeavour requires Ruby skills - not only the language syntax but also the ecosystem. Expectedly, the site assumes the reader is familiar with both. Unfortunately, that’s not my case. I’ve been developing in Java a lot, I’ve dabbled somewhat in Scala, I’m quite interested in Kotlin - in the end, I’m just a JMV developer (plus some Javascript here and there). Long talk short, I start from scratch in Ruby.

    At this stage, there are two possible approaches:

    1. Read documentation and tutorials about Ruby, Gems, bundler, the whole nine yard and come back in a few months (or more)
    2. Or learn on the spot by diving right into development

    Given that I don’t have months, and that whatever I learned is good enough, I opted for the 2nd option. This post is a sum-up of the steps I went through, in the hopes it might benefit others who find themselves in the same situation.

    The first step is not the hardest

    Though new Logstash plugins can be started from scratch, the documentation advise to start from a template. This is explained in the online procedure. The generation yields the following structure:

    $ tree logstash-input-reddit
    ├── Gemfile
    ├── LICENSE
    ├── README.md
    ├── Rakefile
    ├── lib
    │   └── logstash
    │       └── inputs
    │           └── reddit.rb
    ├── logstash-input-reddit.gemspec
    └── spec
        └── inputs
            └── reddit_spec.rb
    

    Not so obviously for a Ruby newbie, this structure is one of a Ruby Gem. In general, dependencies are declared in the associated Gemfile:

    source 'https://rubygems.org'
    gemspec
    

    However, in this case, the gemspec directive adds one additional indirection level. Not only dependencies, but also meta-data, are declared in the associated gemspec file. This is a feature of the Bundler utility gem.

    To install dependencies, the bundler gem first needs to be installed. Aye, there’s the rub…

    Ruby is the limit

    Trying to install the gem yields the following:

    gem install bundler
    Fetching: bundler-1.13.6.gem (100%)
    ERROR:  While executing gem ... (TypeError)
        no implicit conversion of nil into String
    

    The first realization - and it took a lot of time (browsing and reading), is that there are different flavours of Ruby runtimes. Simple Ruby is not enough for Logstash plugin development: it requires a dedicated runtime that runs on the JVM aka JRuby.

    The second realization is that while it’s easy to install multiple Ruby runtimes on a machine, it’s impossible to have them run at the same time. While Homebrew makes the jruby package available, it seems there’s only one single gem repository per system and it reacts very poorly to being managed by different runtimes.

    After some more browsing, I found the solution: rbenv. It not only mangages ruby itself, but also all associated executables (gem, irb, rake, etc.) by isolating every runtime. This makes possible to run my Jekyll site with the latest 2.2.3 Ruby runtime and build the plugin with JRuby on my machine. rbenv is available via Homebrew:

    This is how it goes:

    Install rbenv
    brew install rbenv
    Configure the PATH
    echo 'eval "$(rbenv init -)"' >> ~/.bash_profile
    Source the bash profile script
    . ~/.bash_profile
    List all available runtimes
    rbenv install -l
    Available versions:
      1.8.5-p113
      1.8.5-p114
      ...
      ...
      ...
      ree-1.8.7-2012.02
      topaz-dev
    
    Install the desired runtime
    rbenv install jruby-9.1.6.0
    Configure the project to use the desired runtime
    cd logstash-input-reddit
    rbenv local jruby-9.1.6.0
    
    Check it's configured
    ruby --version
    jruby-9.1.6.0
    

    Finally, bundler can be installed:

    gem install bundler
    Successfully installed bundler-1.13.6
    1 gem installed
    

    And from this point on, all required gems can be installed as well:

    bundle install
    Fetching gem metadata from https://rubygems.org/.........
    Fetching version metadata from https://rubygems.org/..
    Fetching dependency metadata from https://rubygems.org/.
    Resolving dependencies...
    Installing rake 12.0.0
    Installing public_suffix 2.0.4
    ...
    ...
    ...
    Installing rspec-wait 0.0.9
    Installing logstash-core-plugin-api 2.1.17
    Installing logstash-codec-plain 3.0.2
    Installing logstash-devutils 1.1.0
    Using logstash-input-reddit 0.1.0 from source at `.`
    Bundle complete! 2 Gemfile dependencies, 57 gems now installed.
    Use `bundle show [gemname]` to see where a bundled gem is installed.
    Post-install message from jar-dependencies:
    
    if you want to use the executable lock_jars then install ruby-maven gem before using lock_jars 
    
       $ gem install ruby-maven -v '~> 3.3.11'
    
    or add it as a development dependency to your Gemfile
    
       gem 'ruby-maven', '~> 3.3.11'
    

    Plugin development proper

    With those requirements finally addressed, proper plugin development can start. Let’s skip finding the right API to use to make an HTTP request in Ruby or addressing Bundler warnings when installing dependencies, the final code is quite terse:

    class LogStash::Inputs::Reddit < LogStash::Inputs::Base
    
      config_name 'reddit'
      default :codec, 'plain'
      config :subreddit, :validate => :string, :default => 'elastic'
      config :interval, :validate => :number, :default => 10
    
      public
      def register
        @host = Socket.gethostname
        @http = Net::HTTP.new('www.reddit.com', 443)
        @get = Net::HTTP::Get.new("/r/#{@subreddit}/.json")
        @http.use_ssl = true
      end
    
      def run(queue)
        # we can abort the loop if stop? becomes true
        while !stop?
          response = @http.request(@get)
          json = JSON.parse(response.body)
          json['data']['children'].each do |child|
            event = LogStash::Event.new('message' => child, 'host' => @host)
            decorate(event)
            queue << event
          end
          Stud.stoppable_sleep(@interval) { stop? }
        end
      end
    end
    

    The plugin defines two configuration parameters, which subrredit will be parsed for data and the interval between 2 calls (in seconds).

    The register method initializes the class attributes, while the run method loops over:

    • Making the HTTP call to Reddit
    • Parsing the response body as JSON
    • Making dedicated fragments from the JSON, one for each post. This is particularly important because we want to index each post separately.
    • Sending each fragment as a Logstash event for indexing

    Of course, it’s very crude, there’s no error handling, it doesn’t save the timestamp of the last read post to prevent indexing duplicates, etc. In its current state, the plugin offers a lot of room for improvement, but at least it works from a MVP</a> point-of-view.

    Building and installing

    As written above, the plugin is a Ruby gem. It can be built as any other gem:

    gem build logstash-input-reddit
    

    This creates a binary file named logstash-input-reddit-0.1.0.gem - name and version both come from the Bundler’s gemspec. It can be installed using the standard Logtstash plugin installation procedure:

    bin/logstash-plugin install logstash-input-reddit-0.1.0.gem
    

    Downstream processing

    One huge benefit of Logstash is the power of its processing pipeline. The plugin is designed to produce raw data, but the indexing should handle each field separately. Extracting fields from another field can be achieved with the mutate filter.

    Here’s one Logstash configuration snippet example, to fill some relevant fields (and to remove message):

    filter{
      mutate {
        add_field => {
          "kind" => "%{message[kind]}"
          "subreddit" => "%{message[data][subreddit]}"
          "domain" => "%{message[data][domain]}"
          "selftext" => "%{message[data][selftext]}"
          "url" => "%{message[data][url]}"
          "title" => "%{message[data][title]}"
          "id" => "%{message[data][id]}"
          "author" => "%{message[data][author]}"
          "score" => "%{message[data][score]}"
          "created_utc" => "%{message[data][created_utc]}"
        }
        remove_field => [ "message" ]
      }
    }
    

    Once the plugin has been built and installed, Logstash can be run with a config file that includes the previous snippet. It should yield something akin to the following - when used in conjunction with the rubydebug codec:

    {
           "selftext" => "",
               "kind" => "t3",
             "author" => "nfrankel",
              "title" => "Structuring data with Logstash",
          "subreddit" => "elastic",
                "url" => "https://blog.frankel.ch/structuring-data-with-logstash/",
               "tags" => [],
              "score" => "9",
         "@timestamp" => 2016-12-07T22:32:03.320Z,
             "domain" => "blog.frankel.ch",
               "host" => "LSNM33795267A",
           "@version" => "1",
                 "id" => "5f66bk",
        "created_utc" => "1.473948927E9"
    }
    

    Conclusion

    Starting from near-zero kwnowledge about the Ruby ecosystem, I’m quite happy of the result.

    The only thing I couldn’t achieve was to add 3rd libraries (like rest-client), Logstash kept complaining about not being able to install the plugin because of a missing dependency. Falling back to standard HTTP calls solved the issue.

    Also, note that the default template has some warnings on install, but they can be fixed quite easily:

    • The license should read Apache-2.0 instead of Apache License (2.0)
    • Dependencies version are open-ended ('>= 0') whereas they should be more limited i.e. '~> 2'
    • Some meta-data is missing, like the homepage

    I hope this post will be useful to other Java developers wanting to develop their own Logstash plugin.

    Categories: Development Tags: Logstashpluginruby