Archive

Posts Tagged ‘logstash’

Rspec tests on LogStash v1.5 and v1.5+

December 13, 2015 1 comment

Heimdall gazes east
A sail has caught his eye
He lifts his hand and sounds the horn
The undead army has arrived
(Amon Amarth – As Loke Falls)

Logstash 1.4 had an avalanche of bugs. I really wasn’t satisfied with the behavior of agent, it was constantly crashing, missing log lines, etc… Hence, I was waiting for new version hoping it will fix things. Testing 1.5.0 release candidates I did notice one thing that get me angry – logstash-test / rspec was removed from production RPM.. No way to run rspec tests any more with release version. Writing logstash conf and patterns is difficult without unit testing, and ends up with sysadmin usually introducing regressions and ending up with lots of _grokparsefailure entries. So, not having unit tests is simply not acceptable. In my previous article I explained in detail how to write tests. But, how to actually run tests with Logstash 1.5 or Logstash 2.0 or any newer vesion higher then 1.4?

Solution is to build the rspec yourself… First you’ll need some prerequisites, and this is an example from CentOS 7:

# yum install java-1.8.0-openjdk-devel rubygem-rake

After setting up all the requirements, you can grab logstash from GitHub, switch to version branch and prepare rspec:

$ git clone https://github.com/elastic/logstash
$ cd logstash
$ git checkout 2.1
$ rake bootstrap
$ rake test:install-core

After the build is over, you have your own, freshly baked rspec. Unfortunately, you’ll also need to modify your tests 😦 There are two changes needed for tests to work with new rspec:

  • replace require ‘test_utils’ with require ‘spec_helper’
  • remove extend LogStash::RSpec

This is example of new test:

require 'spec_helper'
require 'logstash/filters/grok'

describe LogStash::Filters::Grok do
  describe "my-super-logstash-test" do
    config <<-CONFIG
.... paste logstash config here ...
    CONFIG

    sample 'This is sample log entry' do
      insist { subject['message'] } == 'This is sample log entry'
    end
  end
end

And that’s it, you can test with simple bin/rspec /path/to/my/test.rb.

Configuration and patterns rspec tests on LogStash v1.4

April 6, 2015 Leave a comment

We were the warriors of the north
Notorious and brave
We’d never lost a fight in war
We feared not the grave
(Amon Amarth – Warriors of the North)

If you have a large farm of web servers you often end up depending on some kind of log aggregation stack. Most popular one today is ELK (ElasticSearch, Logstash, Kibana). From operations perspective, changing any system daemon or component that generates logs, often means changing Logstash patterns and filters, while maintaining backwards compatibility (because – all the systems won’t get updated at once). Solution is to write patterns which will work for both the old and new log entries.

To help us with testing, it’s always a good practice to write unit tests. In ruby, rspec is a great tool to accomplish testing, and LogStash is extended with rspec support. But, the documentation is very poor, and all the online examples show how to do tests for version 1.1. Those tests won’t work with 1.4. So, after couple of hours of trial and error, I’ve come up with a test that actually works:

# encoding: utf-8

require "test_utils"
require "logstash/filters/grok"

describe LogStash::Filters::Grok do
  extend LogStash::RSpec

  describe "simple syslog line" do
    # The logstash config goes here.
    # At this time, only filters are supported.
    config <<-CONFIG
    filter {
      grok {
        match => { "message" => "%{WORD:foo}" }
      }
    }
    CONFIG

    sample 'thisisnotspace' do
      insist { subject['foo'] } == "thisisnotspace"
    end
  end
end

Save this text to a file called /tmp/test.rb and run it:

# /opt/logstash/bin/logstash-test /root/logstash_test_off/test2.rb
Using Accessor#strict_set for specs
Run options: exclude {:redis=>true, :socket=>true, :performance=>true, :elasticsearch=>true, :broken=>true, :export_cypher=>true}
.

Finished in 0.173 seconds
1 example, 0 failures

Wonderful!

Now, a little bit of explaining. This is the anchor for our tests:

# encoding: utf-8

require "test_utils"
require "logstash/filters/grok"

describe LogStash::Filters::Grok do
  extend LogStash::RSpec

end

and this part should be the same in every .rb LogStash spec test. Inner “describe” section is what interests us the most. Name of the section is irrelevant, so you can put your description of the test:

describe "your test description goes here" do
end

Next part is the actual LogStash configuration file. The important section for our tests is filter, while we don’t need input and output sections:

    config <<-CONFIG
    filter {
      grok {
        match => { "message" => "%{WORD:foo}" }
      }
    }
    CONFIG

Now, the only thing we still need are actual log lines and expected values. Input (log line) example comes between sample and do keywords, while expected values can be defined within the block using the insist:

    sample 'thisisnotspace' do
      insist { subject['foo'] } == "thisisnotspace"
    end

We can add as many sample sections as we want, with different log lines and different expected values.

Now go and write your own tests for patterns and filters!

Using ElasticSearch templates with Logstash

October 18, 2014 1 comment

I cast aside my chains
Fall from reality
Suicidal disease
The face of all your fears
(At The Gates – Blinded by fear)

Logstash is great tool for acquiring logs and turning them from txt files into JSON documents. When using ElasticSearch as backend for Logstash, Logstash auto-creates indexes. This can be a bit of a problem if you have fields with dots in its contents, like “host”. If host name is recorded as FQDN (as it usually is), then ElasticSearch will analyse that field and split it at dots. Now, node04.example.com gets dissected into node04 and example.com – which can pose a problem in Kibana interface when you try to figure out where the logs came from. Example I got in my case was that 3 nodes in one cluster had approximately 1/3 of requests each, and there was also “example.com” showed as combined graph with those 3 values added, as you can see on the graph below:

blog_kibana

Green bar equals the addition of yellow, blue and orange. Now, this could make sense if you had different domains and wanted to see comparison between them, but since I did not want that I had to find a way to remove it. Enter Elasticsearch templates. Templates will automatically be applied to new indices, when they are created. Since Logstash automatically creates indices, this is ideal to set things up as we want.

Since I wanted to change mappings, first thing I have to do is get current mappings to see what am I up against:

% wget http://elasticsearch:9200/<index_name>/_mapping

That page will give you JSON document, which is in oneliner style. To correct the coding style to classic JSON, to make it more readable, just copy/paste it to http://jsonlint.com/ and click Validate. Now, remove everything except the fields you want to modify. My example mapping with everything except relevant lines removed looks something like this:

{
    "<index_name>": {
        "mappings": {
            "apache_error": {
                "properties": {"host":{"type":"string"},}
            },
            "apache_access":{
                "properties": {"host":{"type":"string"},}
            }
        },
    }
}

Now, we have to add  “index”: “not_analyzed” param for every host key in the upper json. Also, template regex should be added so that ElasticSearch knows indices that this template applies to:

After his manipulation, we can either wrap it into new <template_name> tag, or upload it directly to Elasticsearch with curl:

curl -XPUT localhost:9200/_template/template_developer -d '
{
    "template": "developer*",
    "mappings": {
        "apache_error": {
            "properties": {"host":{"type":"string"}, "index":"not_analyzed"}
        },
        "apache_access":{
            "properties":{"host":{"type":"string"}, "index":"not_analyzed"}
        }
    }
}
'

Now, some of us engineers don’t like to upload configurations to services, but like to keep them in text files. That is also possible, by creating a directory called templates under path.conf – which is usually /etc/elasticsearch. In the latter case, template should look a liitle bit different:

# cat /etc/elasticsearch/templates/template_developer.json 
{
    "template_developer": {
        "template": "developer*",
        "mappings": {
            "apache_error": {
                "properties": {
                    "host": {"type": "string", "index": "not_analyzed"}
                }
            },
            "apache_access": {
                "properties": {
                    "host": {"type": "string", "index": "not_analyzed"}
                }
            }
        }
    }
}

After this, you need to restart Elasticsearch, and recreate indice, and that’s it! New graph looks like:

blog_kibana_post

%d bloggers like this: