Archive

Posts Tagged ‘logstash rspec’

Rspec tests on LogStash v1.5 and v1.5+

December 13, 2015 1 comment

Heimdall gazes east
A sail has caught his eye
He lifts his hand and sounds the horn
The undead army has arrived
(Amon Amarth – As Loke Falls)

Logstash 1.4 had an avalanche of bugs. I really wasn’t satisfied with the behavior of agent, it was constantly crashing, missing log lines, etc… Hence, I was waiting for new version hoping it will fix things. Testing 1.5.0 release candidates I did notice one thing that get me angry – logstash-test / rspec was removed from production RPM.. No way to run rspec tests any more with release version. Writing logstash conf and patterns is difficult without unit testing, and ends up with sysadmin usually introducing regressions and ending up with lots of _grokparsefailure entries. So, not having unit tests is simply not acceptable. In my previous article I explained in detail how to write tests. But, how to actually run tests with Logstash 1.5 or Logstash 2.0 or any newer vesion higher then 1.4?

Solution is to build the rspec yourself… First you’ll need some prerequisites, and this is an example from CentOS 7:

# yum install java-1.8.0-openjdk-devel rubygem-rake

After setting up all the requirements, you can grab logstash from GitHub, switch to version branch and prepare rspec:

$ git clone https://github.com/elastic/logstash
$ cd logstash
$ git checkout 2.1
$ rake bootstrap
$ rake test:install-core

After the build is over, you have your own, freshly baked rspec. Unfortunately, you’ll also need to modify your tests 😦 There are two changes needed for tests to work with new rspec:

  • replace require ‘test_utils’ with require ‘spec_helper’
  • remove extend LogStash::RSpec

This is example of new test:

require 'spec_helper'
require 'logstash/filters/grok'

describe LogStash::Filters::Grok do
  describe "my-super-logstash-test" do
    config <<-CONFIG
.... paste logstash config here ...
    CONFIG

    sample 'This is sample log entry' do
      insist { subject['message'] } == 'This is sample log entry'
    end
  end
end

And that’s it, you can test with simple bin/rspec /path/to/my/test.rb.

Configuration and patterns rspec tests on LogStash v1.4

April 6, 2015 Leave a comment

We were the warriors of the north
Notorious and brave
We’d never lost a fight in war
We feared not the grave
(Amon Amarth – Warriors of the North)

If you have a large farm of web servers you often end up depending on some kind of log aggregation stack. Most popular one today is ELK (ElasticSearch, Logstash, Kibana). From operations perspective, changing any system daemon or component that generates logs, often means changing Logstash patterns and filters, while maintaining backwards compatibility (because – all the systems won’t get updated at once). Solution is to write patterns which will work for both the old and new log entries.

To help us with testing, it’s always a good practice to write unit tests. In ruby, rspec is a great tool to accomplish testing, and LogStash is extended with rspec support. But, the documentation is very poor, and all the online examples show how to do tests for version 1.1. Those tests won’t work with 1.4. So, after couple of hours of trial and error, I’ve come up with a test that actually works:

# encoding: utf-8

require "test_utils"
require "logstash/filters/grok"

describe LogStash::Filters::Grok do
  extend LogStash::RSpec

  describe "simple syslog line" do
    # The logstash config goes here.
    # At this time, only filters are supported.
    config <<-CONFIG
    filter {
      grok {
        match => { "message" => "%{WORD:foo}" }
      }
    }
    CONFIG

    sample 'thisisnotspace' do
      insist { subject['foo'] } == "thisisnotspace"
    end
  end
end

Save this text to a file called /tmp/test.rb and run it:

# /opt/logstash/bin/logstash-test /root/logstash_test_off/test2.rb
Using Accessor#strict_set for specs
Run options: exclude {:redis=>true, :socket=>true, :performance=>true, :elasticsearch=>true, :broken=>true, :export_cypher=>true}
.

Finished in 0.173 seconds
1 example, 0 failures

Wonderful!

Now, a little bit of explaining. This is the anchor for our tests:

# encoding: utf-8

require "test_utils"
require "logstash/filters/grok"

describe LogStash::Filters::Grok do
  extend LogStash::RSpec

end

and this part should be the same in every .rb LogStash spec test. Inner “describe” section is what interests us the most. Name of the section is irrelevant, so you can put your description of the test:

describe "your test description goes here" do
end

Next part is the actual LogStash configuration file. The important section for our tests is filter, while we don’t need input and output sections:

    config <<-CONFIG
    filter {
      grok {
        match => { "message" => "%{WORD:foo}" }
      }
    }
    CONFIG

Now, the only thing we still need are actual log lines and expected values. Input (log line) example comes between sample and do keywords, while expected values can be defined within the block using the insist:

    sample 'thisisnotspace' do
      insist { subject['foo'] } == "thisisnotspace"
    end

We can add as many sample sections as we want, with different log lines and different expected values.

Now go and write your own tests for patterns and filters!

%d bloggers like this: