Home > Development, Linux > Rspec tests on LogStash v1.5 and v1.5+

Rspec tests on LogStash v1.5 and v1.5+

Heimdall gazes east
A sail has caught his eye
He lifts his hand and sounds the horn
The undead army has arrived
(Amon Amarth – As Loke Falls)

Logstash 1.4 had an avalanche of bugs. I really wasn’t satisfied with the behavior of agent, it was constantly crashing, missing log lines, etc… Hence, I was waiting for new version hoping it will fix things. Testing 1.5.0 release candidates I did notice one thing that get me angry – logstash-test / rspec was removed from production RPM.. No way to run rspec tests any more with release version. Writing logstash conf and patterns is difficult without unit testing, and ends up with sysadmin usually introducing regressions and ending up with lots of _grokparsefailure entries. So, not having unit tests is simply not acceptable. In my previous article I explained in detail how to write tests. But, how to actually run tests with Logstash 1.5 or Logstash 2.0 or any newer vesion higher then 1.4?

Solution is to build the rspec yourself… First you’ll need some prerequisites, and this is an example from CentOS 7:

# yum install java-1.8.0-openjdk-devel rubygem-rake

After setting up all the requirements, you can grab logstash from GitHub, switch to version branch and prepare rspec:

$ git clone https://github.com/elastic/logstash
$ cd logstash
$ git checkout 2.1
$ rake bootstrap
$ rake test:install-core

After the build is over, you have your own, freshly baked rspec. Unfortunately, you’ll also need to modify your tests 😦 There are two changes needed for tests to work with new rspec:

  • replace require ‘test_utils’ with require ‘spec_helper’
  • remove extend LogStash::RSpec

This is example of new test:

require 'spec_helper'
require 'logstash/filters/grok'

describe LogStash::Filters::Grok do
  describe "my-super-logstash-test" do
    config <<-CONFIG
.... paste logstash config here ...
    CONFIG

    sample 'This is sample log entry' do
      insist { subject['message'] } == 'This is sample log entry'
    end
  end
end

And that’s it, you can test with simple bin/rspec /path/to/my/test.rb.

Advertisements
  1. April 25, 2016 at 8:24 am

    I while back I was confronted to the same problem, which i called the “logstash-induced-hair-loss” problem. I ended up building a tool for streamlining the process of unittesting filters and patterns : https://github.com/gaspaio/logstash-tester
    I’ve been using in production for a few months now and my team is happy again :-).

    Some other tools of the same sort exist : https://github.com/tcnksm/rspec-logstash-filter and https://github.com/magnusbaeck/logstash-filter-verifier

  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: