Archive

Archive for April, 2015

Configuration and patterns rspec tests on LogStash v1.4

April 6, 2015 Leave a comment

We were the warriors of the north
Notorious and brave
We’d never lost a fight in war
We feared not the grave
(Amon Amarth – Warriors of the North)

If you have a large farm of web servers you often end up depending on some kind of log aggregation stack. Most popular one today is ELK (ElasticSearch, Logstash, Kibana). From operations perspective, changing any system daemon or component that generates logs, often means changing Logstash patterns and filters, while maintaining backwards compatibility (because – all the systems won’t get updated at once). Solution is to write patterns which will work for both the old and new log entries.

To help us with testing, it’s always a good practice to write unit tests. In ruby, rspec is a great tool to accomplish testing, and LogStash is extended with rspec support. But, the documentation is very poor, and all the online examples show how to do tests for version 1.1. Those tests won’t work with 1.4. So, after couple of hours of trial and error, I’ve come up with a test that actually works:

# encoding: utf-8

require "test_utils"
require "logstash/filters/grok"

describe LogStash::Filters::Grok do
  extend LogStash::RSpec

  describe "simple syslog line" do
    # The logstash config goes here.
    # At this time, only filters are supported.
    config <<-CONFIG
    filter {
      grok {
        match => { "message" => "%{WORD:foo}" }
      }
    }
    CONFIG

    sample 'thisisnotspace' do
      insist { subject['foo'] } == "thisisnotspace"
    end
  end
end

Save this text to a file called /tmp/test.rb and run it:

# /opt/logstash/bin/logstash-test /root/logstash_test_off/test2.rb
Using Accessor#strict_set for specs
Run options: exclude {:redis=>true, :socket=>true, :performance=>true, :elasticsearch=>true, :broken=>true, :export_cypher=>true}
.

Finished in 0.173 seconds
1 example, 0 failures

Wonderful!

Now, a little bit of explaining. This is the anchor for our tests:

# encoding: utf-8

require "test_utils"
require "logstash/filters/grok"

describe LogStash::Filters::Grok do
  extend LogStash::RSpec

end

and this part should be the same in every .rb LogStash spec test. Inner “describe” section is what interests us the most. Name of the section is irrelevant, so you can put your description of the test:

describe "your test description goes here" do
end

Next part is the actual LogStash configuration file. The important section for our tests is filter, while we don’t need input and output sections:

    config <<-CONFIG
    filter {
      grok {
        match => { "message" => "%{WORD:foo}" }
      }
    }
    CONFIG

Now, the only thing we still need are actual log lines and expected values. Input (log line) example comes between sample and do keywords, while expected values can be defined within the block using the insist:

    sample 'thisisnotspace' do
      insist { subject['foo'] } == "thisisnotspace"
    end

We can add as many sample sections as we want, with different log lines and different expected values.

Now go and write your own tests for patterns and filters!

Advertisements
%d bloggers like this: