Posts Tagged ‘elasticsearch’
  • Exploring data sets with Kibana

    :imagesdir: /assets/resources/exploring-data-sets-with-kibana/ :icons: font :experimental:

    In this post, I’d like to explore a sample data set using Kibana.

    This requires some data to start with: let’s index some tweets. It’s quite straightforward to achieve that by following explanations found in my good friend David’s[blog post^] and wait for some time to fill the index with data.

    == Basic metric

    Let’s start with something basic, the number of tweets indexed so far.

    In Kibana, go to menu:Visualize[Metric], then choose the twitter index. For the Aggregation field, choose “Count”; then click on btn:[Save] and name the visualization accordingly e.g. “Number of tweets”.

    image:basic-metric-create.png[Create a basic metric,351,146] image:basic-metric-display.png[Display the number of tweets,320,202]

    == Geo-map

    Another simple visualization is to display the tweets based on their location on a world map.

    In Kibana, go to menu:Visualize[Tile map], then choose the twitter index.

    Select Geo Coordinates for the bucket type and keep default values,Geohash for Aggregation and coordinates.coordinates for Field.

    image::geo-map-display.png[Localized map of tweets,637,402,align=center]

    == Bucket metric

    For this kind of metric, suppose a business requirement is to display the top 5 users. Unfortunately, as some (most?) business requirements go, this is not deterministic enough. It misses both the range and the aggregation period. Let’s agree for range time to be a sliding window over the last day, and the period to be an hour.

    In Kibana, go to menu:Visualize[Vertical bar chart], then choose the twitter index. Then:

    • For the Y-Axis, keep Count for the Aggregation field
    • Choose X-Axis for the buckets type ** Select Date histogram for the Aggregation field ** Keep the value @timestamp for the Field field ** Set the Interval field to Hourly
    • Click on btn:[Add sub-buckets]
    • Choose Split bars for the buckets type ** Select Terms for the Sub Aggregation field ** for the Field field ** Keep the other fields default value
    • Don’t forget to click on the btn:[Apply changes]
    • Click on btn:[Save] and name the visualization accordingly e.g. “Top 5 users hourly”.

    image:bucket-metric-create.png[Create a bucket metric,234,472] image:bucket-metric-display.png[Display the top 5 users hourly,637,202]

    === Equivalent visualisations

    Other visualizations can be used with the exact same configuration: Area chart and Data table.

    The output of the Area chart is not as readable, regarding the explored data set, but the Data table offers interesting options.

    From a visualization, click on the bottom right arrow icon to display a table view of the data instead of a graphic.

    image::tabular-metric.png[Alternative tabular metric display,635,200,align=center]

    Visualizations make use of Elasticsearch public API. From the tabular view, the JSON request can also be displayed by clicking on the btn:[Request] button (oh, surprise…). This way, Kibana can be used as a playground to quickly prototype requests before using them in one’s own applications.

    image::request-metric.png[Executed API request,635,200,align=center]

    === Changing requirements a bit

    The above visualization picks out the 5 top users having the most tweeted during each hour and display them during the last day. That’s the reason why there are more than 5 users displayed. But the above requirement can be interpreted in another way: take the top 5 users over the course of the last day, and break their number of tweets by hour.

    To do that, just move the X-Axis bucket below the Split bars bucket. This will change the output accordingly.

    image::another-bucket-metric-display.png[Display the top 5 users over the last day,637,203,align=center]

    === Filtering irrelevant data

    As can be seen in the above histogram, top users mostly are about recruiting and/or job offers. This is not really what is wanted in the first place. It’s possible to remove this noise by adding a filter: in the Split bars section, click on btn:[Advanced] to display additional parameters and type the desired regex in the Exclude field.

    image::filtered-bucket-metric-create.png[Filter out a bucket metric,342,69,align=center]

    The new visualization is quite different:

    image::filtered-bucket-metric-display.png[Display the top 5 users hourly without any recruitment-related user,637,202,align=center]

    == Putting it all together

    With the above visualizations available and configured, it’s time to put them together on a dedicated dashboard. Go to menu:Dashboard[Add] to list all available visualizations.

    image::add-visualization-dashboard.png[Add visualizations to a dashboard,732,391,align=center]

    It’s as simple as clicking on the desired one, laying it out on the board and resetting its size. Rinse and repeat until happy with the result and then click on btn:[Save].

    image::configured-dashboard.png[A configured dashboard,910,429,align=center]

    Icing on the cake, using the btn:[Rectangle] tool on the map visualization will automatically add a filter that only displays data bound by the rectangle coordinates for all visualizations found on the dashboard.

    image::filtered-dashboard.png[A filtered dashboard,910,453,align=center]

    That trick is not limited to the map visualization (try playing with other ones) but filtering on location quickly gives insights when exploring data sets.

    == Conclusion

    While this post only brushes off the surface of what Kibana has to offer, there are more visualizations available as well as Timelion, the new powerful (but sadly under-documented) the “time series expression interface”. In all cases, even basic features as shown above already provide plenty of different options to make sense of one’s data sets.

    Categories: Technical Tags: elasticsearchelastickibanabig data
  • Feedback on Feeding Spring Boot metrics to Elasticsearch

    Logstash log

    Some weeks ago, I wrote a post detailing how to send JMX metrics from a Spring Boot app to Elasticsearch by developing another Spring Boot app.

    Getting to create such an app is not always the right idea but developers are makers - software makers, and developing new apps is quite alluring to them. However, in the overall scheme of things, this means time is not only spent in development, but also for maintenance during the entire lifetime of the app. Before going the development path, one should thoroughly check whether out-of-the-box alternatives exist.

    Back to JMX metrics: only the straightforward Logstash jmx plugin was tried before calling it quits because of an incompatibility with Elasticsearch 5. But an alternative exists with the Logstash http_poller plugin.

    This Logstash input plugin allows you to call an HTTP API, decode the output of it into event(s), and send them on their merry way. The idea behind this plugins came from a need to read springboot metrics endpoint, instead of configuring jmx to monitor my java application memory/gc/ etc.

    Jolokia is already in place, offering HTTP access to JMX. The last and only step is to configure HTTP poller plugin, which is fairly straighforward. The URL is composed of the standard actuator URL appended with /jolokia/read/ and the JMX’s ObjectName of the desired object. Here’s a sample configuration snippet, with URLs configured for:

    1. Response site for the root page
    2. HTTP 200 status code counter for the root page
    3. Operating system metrics
    4. Garbage collector metrics

    With the help of the jconsole, adding more metrics is a no-brainer.

    input {
      http_poller {
        urls => {
          "200.root" =>
          "root" =>
          "OperatingSystem" => "http://localhost:8080/manage/jolokia/read/java.lang:type=OperatingSystem"
          "PS Scavenge" => "http://localhost:8080/manage/jolokia/read/java.lang:type=GarbageCollector,name=PS%20Scavenge"
        request_timeout => 60
        schedule => { every => "10s"}
        codec => "json"

    This should output something akin to:

              "request" => {
                "mbean" => "org.springframework.metrics:name=response,type=gauge,value=root",
                 "type" => "read"
           "@timestamp" => 2016-01-01T10:19:45.914Z,
             "@version" => "1",
                "value" => {
                "Value" => 4163.0,
          "LastUpdated" => "2016-12-30T16:29:28+01:00"
            "timestamp" => 1483121985,
               "status" => 200

    Beyond the raw output, there are a couple of possible improvements:

    • value and request fields are inner objects, for no added value. Flattening the structure can go a long way toward making writing queries easier.
    • Adding tags depending on the type improve categorization. A possible alternative would be to parse the JMX compound name into dedicated fields with grok or dissect.
    • The @timestamp field can be replaced with the LastUpdated value and interpreted as a date.

    Filters to the rescue:

    {% highlight raw linenos %} filter { mutate { add_field => { “[mbean][objectName]” => “%{request[mbean]}” }} mutate { remove_field => “request” } }

    filter { if [mbean][objectName] == “java.lang:type=OperatingSystem” { dissect { mapping => { “mbean[objectName]” => “%{[mbean][prefix]}:type=%{[mbean][type]}” } } mutate { remove_field => “value[ObjectName]” } } else if [mbean][objectName] == “java.lang:name=PS Scavenge,type=GarbageCollector” { dissect { mapping => { “mbean[objectName]” => “%{[mbean][prefix]}:name=%{[mbean][name]},type=%{[mbean][type]}” } } mutate { remove_field => “value[ObjectName]” } } else if [mbean][objectName] =~ “^.,type=gauge,.$” or [mbean][objectName] =~ “^.,type=counter,.$” { date { match => [ “%{value[lastUpdated]}”, “ISO8601” ] } mutate { replace => { “value” => “%{value[Value]}” }} mutate { convert => { “value” => “float” }} if [mbean][objectName] =~ “^.,type=gauge,.$” { dissect { mapping => { “mbean[objectName]” => “%{[mbean][prefix]}:name=%{[mbean][name]},type=%{[mbean][type]},value=%{[mbean][page]}” } } } else if [mbean][objectName] =~ “^.,type=counter,.$” { dissect { mapping => { “mbean[objectName]” => “%{[mbean][prefix]}:name=%{[mbean][name]},type=%{[mbean][type]},value=%{[mbean][status]}.%{[mbean][page]}” } } } } } {% endhighlight %}

    A little explanation might be in order.

    Lines 1-4
    Move the initial request->mbean field to the mbean->objectName field.
    Lines 7-13
    For OS metrics, create mbean nested fields out of the objectName nested field and remove it from the value field.
    Lines 14-20
    For GC metrics, create mbean nested fields out of the objectName nested field using a slightly different pattern and remove it from the value field.
    Lines 21-24
    For gauge or counter metrics, interpret the value->lastUpdated nested field as a date, move the nested value->Value field to the root and interpret as a float value.
    Lines 25-38
    For gauge or counter metrics, create mbean nested fields using a pattern specific for each metric.

    Coupled with the initial configuration, this outputs to the following (for the gauge):

        "@timestamp" => 2016-01-01T10:54:15.359Z,
             "mbean" => {
            "prefix" => "org.springframework.metrics",
              "name" => "response",
        "objectName" => "org.springframework.metrics:name=response,type=gauge,value=root",
              "type" => "gauge",
             "value" => "root"
          "@version" => "1",
             "value" => 4163.0,
         "timestamp" => 1483206855,
            "status" => 200,
              "tags" => []

    Given the current complexity of the configuration, remember the next logical step is to decouple the single snippet into multiple files. The order used by Logstash is by lexicographical file name. Name files accordingly:

    • 00-app-input.conf
    • 10-global-filter.conf
    • 20-specific-filter.conf
    • 99-console-output.conf

    Note: gaps in the number scheme allow to add intermediary filters in the future, with no renaming

    All in all, everything required is available through configuration, without any coding. Always check that it’s not the case before reinventing the wheel.

    Categories: Technical Tags: elasticsearchlogstashelastic