Archive

Archive for the ‘Development’ Category

Rspec tests on LogStash v1.5 and v1.5+

December 13, 2015 1 comment

Heimdall gazes east
A sail has caught his eye
He lifts his hand and sounds the horn
The undead army has arrived
(Amon Amarth – As Loke Falls)

Logstash 1.4 had an avalanche of bugs. I really wasn’t satisfied with the behavior of agent, it was constantly crashing, missing log lines, etc… Hence, I was waiting for new version hoping it will fix things. Testing 1.5.0 release candidates I did notice one thing that get me angry – logstash-test / rspec was removed from production RPM.. No way to run rspec tests any more with release version. Writing logstash conf and patterns is difficult without unit testing, and ends up with sysadmin usually introducing regressions and ending up with lots of _grokparsefailure entries. So, not having unit tests is simply not acceptable. In my previous article I explained in detail how to write tests. But, how to actually run tests with Logstash 1.5 or Logstash 2.0 or any newer vesion higher then 1.4?

Solution is to build the rspec yourself… First you’ll need some prerequisites, and this is an example from CentOS 7:

# yum install java-1.8.0-openjdk-devel rubygem-rake

After setting up all the requirements, you can grab logstash from GitHub, switch to version branch and prepare rspec:

$ git clone https://github.com/elastic/logstash
$ cd logstash
$ git checkout 2.1
$ rake bootstrap
$ rake test:install-core

After the build is over, you have your own, freshly baked rspec. Unfortunately, you’ll also need to modify your tests ūüė¶ There are two changes needed for tests to work with new rspec:

  • replace require ‘test_utils’ with require ‘spec_helper’
  • remove extend LogStash::RSpec

This is example of new test:

require 'spec_helper'
require 'logstash/filters/grok'

describe LogStash::Filters::Grok do
  describe "my-super-logstash-test" do
    config <<-CONFIG
.... paste logstash config here ...
    CONFIG

    sample 'This is sample log entry' do
      insist { subject['message'] } == 'This is sample log entry'
    end
  end
end

And that’s it, you can test with simple bin/rspec /path/to/my/test.rb.

Configuration and patterns rspec tests on LogStash v1.4

April 6, 2015 Leave a comment

We were the warriors of the north
Notorious and brave
We’d never lost a fight in war
We feared not the grave
(Amon Amarth – Warriors of the North)

If you have a large farm of web servers you often end up depending on some kind of log aggregation stack. Most popular one today is ELK (ElasticSearch, Logstash, Kibana). From operations perspective, changing any system daemon or component that generates logs, often means changing Logstash patterns and filters, while maintaining backwards compatibility (because – all the systems won’t get updated at once). Solution is to write patterns which will work for both the old and new log entries.

To help us with testing, it’s always a good practice to write unit tests. In ruby, rspec is a great tool to accomplish testing, and LogStash is extended with rspec support. But, the documentation is very poor, and all the online examples show how to do tests for version 1.1. Those tests won’t work with 1.4. So, after couple of hours of trial and error, I’ve come up with a test that actually works:

# encoding: utf-8

require "test_utils"
require "logstash/filters/grok"

describe LogStash::Filters::Grok do
  extend LogStash::RSpec

  describe "simple syslog line" do
    # The logstash config goes here.
    # At this time, only filters are supported.
    config <<-CONFIG
    filter {
      grok {
        match => { "message" => "%{WORD:foo}" }
      }
    }
    CONFIG

    sample 'thisisnotspace' do
      insist { subject['foo'] } == "thisisnotspace"
    end
  end
end

Save this text to a file called /tmp/test.rb and run it:

# /opt/logstash/bin/logstash-test /root/logstash_test_off/test2.rb
Using Accessor#strict_set for specs
Run options: exclude {:redis=>true, :socket=>true, :performance=>true, :elasticsearch=>true, :broken=>true, :export_cypher=>true}
.

Finished in 0.173 seconds
1 example, 0 failures

Wonderful!

Now, a little bit of explaining. This is the anchor for our tests:

# encoding: utf-8

require "test_utils"
require "logstash/filters/grok"

describe LogStash::Filters::Grok do
  extend LogStash::RSpec

end

and this part should be the same in every .rb LogStash spec test. Inner “describe” section is what interests us the most. Name of the section is irrelevant, so you can put your description of the test:

describe "your test description goes here" do
end

Next part is the actual LogStash configuration file. The important section for our tests is filter, while we don’t need input and output sections:

    config <<-CONFIG
    filter {
      grok {
        match => { "message" => "%{WORD:foo}" }
      }
    }
    CONFIG

Now, the only thing we still need are actual log lines and expected values. Input (log line) example comes between sample and do keywords, while expected values can be defined within the block using the insist:

    sample 'thisisnotspace' do
      insist { subject['foo'] } == "thisisnotspace"
    end

We can add as many sample sections as we want, with different log lines and different expected values.

Now go and write your own tests for patterns and filters!

Using ElasticSearch templates with Logstash

October 18, 2014 1 comment

I cast aside my chains
Fall from reality
Suicidal disease
The face of all your fears
(At The Gates – Blinded by fear)

Logstash is great tool for acquiring logs and turning them from txt files into JSON documents. When using ElasticSearch as backend for Logstash, Logstash auto-creates indexes. This can be a bit of a problem if you have fields with dots in its contents, like “host”. If host name is recorded as FQDN (as it usually is), then ElasticSearch will analyse that field and split it at dots. Now, node04.example.com gets dissected into node04 and example.com – which can pose a problem in Kibana interface when you try to figure out where the logs came from. Example I got in my case was that 3 nodes in one cluster had approximately 1/3 of requests each, and there was also “example.com” showed as combined graph with those 3 values added, as you can see on the graph below:

blog_kibana

Green bar equals the addition of yellow, blue and orange. Now, this could make sense if you had different domains and wanted to see comparison between them, but since I did not want that I had to find a way to remove it. Enter Elasticsearch templates. Templates will automatically be applied to new indices, when they are created. Since Logstash automatically creates indices, this is ideal to set things up as we want.

Since I wanted to change mappings, first thing I have to do is get current mappings to see what am I up against:

% wget http://elasticsearch:9200/<index_name>/_mapping

That page will give you JSON document, which is in oneliner style. To correct the coding style to classic JSON, to make it more readable, just copy/paste it to http://jsonlint.com/ and click Validate. Now, remove everything except the fields you want to modify. My example mapping with everything except relevant lines removed looks something like this:

{
    "<index_name>": {
        "mappings": {
            "apache_error": {
                "properties": {"host":{"type":"string"},}
            },
            "apache_access":{
                "properties": {"host":{"type":"string"},}
            }
        },
    }
}

Now, we have to add ¬†“index”: “not_analyzed” param for every¬†host key in the upper json. Also, template regex should be added so that ElasticSearch knows indices that this template applies to:

After his manipulation, we can either wrap it into new <template_name> tag, or upload it directly to Elasticsearch with curl:

curl -XPUT localhost:9200/_template/template_developer -d '
{
    "template": "developer*",
    "mappings": {
        "apache_error": {
            "properties": {"host":{"type":"string"}, "index":"not_analyzed"}
        },
        "apache_access":{
            "properties":{"host":{"type":"string"}, "index":"not_analyzed"}
        }
    }
}
'

Now, some of us engineers don’t like to upload configurations to services, but like to keep them in text files. That is also possible, by creating a directory called templates under path.conf – which is usually /etc/elasticsearch. In the latter case, template should look a liitle bit different:

# cat /etc/elasticsearch/templates/template_developer.json 
{
    "template_developer": {
        "template": "developer*",
        "mappings": {
            "apache_error": {
                "properties": {
                    "host": {"type": "string", "index": "not_analyzed"}
                }
            },
            "apache_access": {
                "properties": {
                    "host": {"type": "string", "index": "not_analyzed"}
                }
            }
        }
    }
}

After this, you need to restart Elasticsearch, and recreate indice, and that’s it! New graph looks like:

blog_kibana_post

Git over HTTP on CentOS 6

May 23, 2014 4 comments

Total war is here
Face it without fear
Age of sword, age of spear
Fight for honor, glory, death in fire!
(Amon Amarth – Death in Fire)

There is already a bunch of posts about setting up Git over HTTP(S), but this one is specificaly targeted at setting it up under CentOS as cleanly as possible. There was bunch of errors that I saw along the way, so I will try to explain the process step by step.

First, you have to install Apache and Git.:

# yum -y install httpd git
# /etc/init.d/httpd start

Now, lets create directories for git and create our first repo:

# mkdir /var/www/gitrepos
# cd /var/www/gitrepos
# mkdir repo01 && cd repo01
# git --bare init
# git update-server-info
# cd /var/www
# chown -R apache: gitrepos

We are using ‘git –bare’, so that online repository doesn’t have files but only git metadata. That will enable users to push directly to online repository, otherwise they wouldn’t be able to push thier changes. This was the first error I did, I created repo with ‘git init’ and was not able to push later. After the repo is set up and chowned, lets set up apache. This is my configuration for vhost:

#
# vhost for git repositories (http)
#
<VirtualHost *:80>
    ServerName     git
    DocumentRoot    /var/www/gitrepos

    <Location />
        DAV on

        # general auth settings
        AuthName "Git login:"
        AuthType Basic

        # file authentication
        AuthUserFile  /var/www/htpasswd
        AuthGroupFile /var/www/htgroup

        <LimitExcept PROPFIND>
            Require valid-user
        </LimitExcept> 
    </Location>

    <Location /repo01>
        <LimitExcept PROPFIND>
            Require group adminlinux
        </LimitExcept> 
    </Location>

    LogLevel warn
    ErrorLog  /var/log/httpd/git_error.log
    CustomLog /var/log/httpd/git_access.log combined
</VirtualHost>

If you wonder why is PROPFIND method treated differently from all other http/dav methods – it’s because webserver runs PROPFIND without user authentication, so if it’s not excluded from limit, it will get rejected and you will see a message similar to this one when trying to push from the client:

error: Cannot access URL https://git/puppet-adriatic/, return code 22
fatal: git-http-push failed

We can fill up htpasswd file with – tadddaaa htpasswd command ūüôā

# htpasswd -c /var/www/htpasswd user1
# htpasswd -c /var/www/htpasswd user2
# htpasswd -c /var/www/htpasswd user3

And htgroup with:

# echo "adminlinux: user1 user2" >> /var/www/htgroup

Now, on the client side, do a:

% git clone http://user1@git/repo01

And that’s it! After the first change/commit you do, be careful when you push those changes for the first time. This is the command I used for the first push:

% git push --set-upstream origin master

You may also encounter a 22/502 error on a MOVE command, like:

MOVE 12486a9c101c613c075d59b5cf61329f96f9ae12 failed, aborting (22/502)
MOVE 0c306c54862ae8c21226281e6e4f47c8339ed132 failed, aborting (22/502)
MOVE ce4c4fc9d1e4daf3a59516829a0e1bd6c66d4066 failed, aborting (22/502

This happened to me because I used http to https forwarding in apache, and I had a http specified in my .git/config on a client machine. After changing the destination to https, MOVE command did it’s magic. It seems that this error is result of server name/location being different in client repo and on a server side.

Note: I recommend using SSL and not plain text http, even with self-signed certificates. In that scenario you’ll probably want to use env variable GIT_SSL_NO_VERIFY=true.

Note2: CentOS ships with old version of git, 1.7.x so I recommend either using git from IUS repo (git2x packages) or backporting git from newer Fedora releases.

Categories: Development, Linux, RedHat Tags: , , ,

Finding dependencies for PHP applications

April 27, 2014 1 comment

Blame and lies, contradictions arise
Blame and lies, contradictions arise
Nobody will change my way
Life betrays, but I keep on going
(Sepultura – Inner Self)

I’m a big fan of packaging systems and I pretty much tend to hate deploying unpackaged software. So, I tend to build native OS pacakges as often as I can. Packaging PHP applicatoins (like WordPress for example) is pretty much straightforward job. Only problem one may encounter is listing all software dependencies correctly. Here comes pci to the rescue!

So what is a ‘pci‘ – it’s part of the phpcompatinfo PHP module. The purpose of the module, and of ‘pci’ executable is to find out version and the extensions required for a piece of (PHP) code to run. Yeah, it’s that simple! So, lets try it, on CentOS 6 + EPEL offcourse:

# yum -y install php-pear-PHP-CompatInfo
# mkdir /tmp/pcitest
# echo "<?php phpinfo(); ?>" >> /tmp/pcitest/phpinfo.ph
# pci -d /tmp/pcitest
+-----------------------------+---------+---+------------+--------------------+
| Files                       | Version | C | Extensions | Constants/Tokens   |
+-----------------------------+---------+---+------------+--------------------+
| /tmp/pcitest/*              | 4.0.0   | 0 |            |                    |
+-----------------------------+---------+---+------------+--------------------+

As you can see, phpinfo() requires at least versoin 4.0.0 of PHP and doesn’t require any extension. Let’s run it now on some real project, like TeamPass:

# pci -d teampass > pci.log
# cat pci.log | cut -d'|' -f 5 | grep -v ^+ | sort | uniq | \
    sed 's/^ /Requires: php-/g'
Requires: php-           
Requires: php-bcmath     
Requires: php-ctype      
Requires: php-date       
Requires: php-Extensions 
Requires: php-filter     
Requires: php-gd         
Requires: php-hash       
Requires: php-json       
Requires: php-mbstring   
Requires: php-mysql      
Requires: php-mysqli     
Requires: php-openssl    
Requires: php-pcre       
Requires: php-pgsql      
Requires: php-session    
Requires: php-SimpleXML  
Requires: php-SPL        
Requires: php-xml

With two simple commands we can get output that can easily be copy/pasted and used used in rpm spec file. Enjoy building packages! INNER SELF!

Check if JCE is installed

February 1, 2014 Leave a comment

Asg√•rd’s always been my home
But I’m of different blood
I will overthrow the throne
Deceiver! Deceiver of the gods!
(Amon Amarth – Deceiver of the Gods)

JCE stands for Java Cryptography Extension. Unlimited Strength Jurisdiction Policy Files are not distributed with standard Java distributions due to export laws limitations. But, some Java-powered software does need it to function properly. Often someone thinks JCE is installed but software still refuses to run properly. These are some of the possible errors:

org.apache.xml.security.encryption.XMLEncryptionException: 
Illegal key size or default parameters

So, how to check if unlimited JCE is really installed? After consulting google and stackoverflow answers I’ve written a small Java program that checks it very accurately. This is the source code:

import java.security.NoSuchAlgorithmException;
import javax.crypto.Cipher;

public class UnlimitedSupportJCE
{
  public static void main(final String[] args)
  {
    int strength = 0;
    try {
      strength = Cipher.getMaxAllowedKeyLength("AES");
    } catch (NoSuchAlgorithmException e) {
      System.out.println("isUnlimitedSupported=FALSE");
    }
    if ( strength > 128 ){
      System.out.printf("isUnlimitedSupported=TRUE,strength: %d%n",strength);
    } else {
      System.out.printf("isUnlimitedSupported=FALSE,strength: %d%n",strength);
    }
  }
}

You can save this code in a file named UnlimitedSupportJCE.java. You will also need JDK installed (with javac compiler). Now you can compile the code:

$ javac UnlimitedSupportJCE.java

This creates file called UnlimitedSupportJCE.class, which can now be run by JVM:

$ java UnlimitedSupportJCE
isUnlimitedSupported=TRUE, strength: 2147483647

We can see that on this particular JVM, Ulimited JCE is installed correctly. How about after removing the JARs?

$ java UnlimitedSupportJCE
isUnlimitedSupported=FALSE, strength: 128

We can see that strength of cyphers is dramaticaly lower with vanilla JCE files that are distributed with standard JRE/JDKs.

Using mercurial as SVN client

August 17, 2012 2 comments

Somewhere over the rainbow
Skies are blue,
And the dreams that you dare to dream
Really do come true.
(Harold Arlen and E.Y. Harburg)

There are two things I can’t stand with SVN. The whole unnecessary forest of .svn sub-directories scattered all around my working copy. Second is the propset way of ignoring files. Can I add third? ūüôā It’s the manual svn add/remove for every file I create/delete…¬† Ay my current company (srce.hr) we use SVN as central VCS. I use it for RPM packages and puppet modules/manifests.
Now, if you want to distribute same working copy on more than one location, and be able to pull and push changes, all those .svn directories are really clumsy. So I decided to try HGsubversion extension. I already use Mercurial a lot (for example with etckeeper) and love its simplicity. So, on my workstation running CentOS 6, I first fetched hgsubversion extension:

% hg clone http://bitbucket.org/durin42/hgsubversion/ ~/.hgsubversion
 % echo "[extensions]" > ~/.hgrc
 % echo "hgsubversion = ~/.hgsubversion/hgsubversion" >> ~/.hgrc

How many of you actually read README’s? ūüôā I surely don’t until things break out. And so they broke…

% hg clone svn+https://svnserver/svn/repo
 destination directory: repo
 Auth realm:  SVN login
 Password for jsosic:
 [r1] jsosic: import modula
 [r2] jsosic: Redesigned.
 [r3] jsosic: something
 [r4] jsosic: something else
 pulled 4 revisions
 updating to branch default
 151 files updated, 0 files merged, 0 files removed, 0 files unresolved
 ** unknown exception encountered, details follow
 ** report bug details to http://mercurial.selenic.com/bts/
 ** or mercurial@selenic.com
 ** Mercurial Distributed SCM (version 1.4)
 ** Extensions loaded: hgsubversion
 Traceback (most recent call last):
 File "/usr/bin/hg", line 27, in
 mercurial.dispatch.run()
 File "/usr/lib64/python2.6/site-packages/mercurial/dispatch.py", line 16, in run
 sys.exit(dispatch(sys.argv[1:]))
 File "/usr/lib64/python2.6/site-packages/mercurial/dispatch.py", line 30, in dispatch
 return _runcatch(u, args)
 File "/usr/lib64/python2.6/site-packages/mercurial/dispatch.py", line 46, in _runcatch
 return _dispatch(ui, args)
 File "/usr/lib64/python2.6/site-packages/mercurial/dispatch.py", line 449, in _dispatch
 return runcommand(lui, repo, cmd, fullargs, ui, options, d)
 File "/usr/lib64/python2.6/site-packages/mercurial/dispatch.py", line 319, in runcommand
 ret = _runcommand(ui, options, cmd, d)
 File "/usr/lib64/python2.6/site-packages/mercurial/dispatch.py", line 500, in _runcommand
 return checkargs()
 File "/usr/lib64/python2.6/site-packages/mercurial/dispatch.py", line 454, in checkargs
 return cmdfunc()
 File "/usr/lib64/python2.6/site-packages/mercurial/dispatch.py", line 448, in
 d = lambda: util.checksignature(func)(ui, *args, **cmdoptions)
 File "/usr/lib64/python2.6/site-packages/mercurial/util.py", line 386, in check
 return func(*args, **kwargs)
 File "/usr/lib64/python2.6/site-packages/mercurial/extensions.py", line 116, in wrap
 util.checksignature(origfn), *args, **kwargs)
 File "/usr/lib64/python2.6/site-packages/mercurial/util.py", line 386, in check
 return func(*args, **kwargs)
 File "/home/jsosic/.hgsubversion/hgsubversion/wrappers.py", line 540, in clone
 fd = dst.opener("hgrc", "a", text=True)
 AttributeError: 'bool' object has no attribute 'opener'
 [1]    15118 exit 1     hg clone svn+https://svnserver/svn/repo

It seems that I didn’t have subvertpy installed, which is availabe in EPEL for RHEL/CentOS so…

# yum install python-subvertpy

After that – another error…

% hg clone svn+https://svnserver/svn/repo
 destination directory: repo
 ** unknown exception encountered, details follow
 ** report bug details to http://mercurial.selenic.com/bts/
 ** or mercurial@selenic.com
 ** Mercurial Distributed SCM (version 1.4)
 ** Extensions loaded: hgsubversion
 Traceback (most recent call last):
 File "/usr/bin/hg", line 27, in
 mercurial.dispatch.run()
 File "/usr/lib64/python2.6/site-packages/mercurial/dispatch.py", line 16, in run
 sys.exit(dispatch(sys.argv[1:]))
 File "/usr/lib64/python2.6/site-packages/mercurial/dispatch.py", line 30, in dispatch
 return _runcatch(u, args)
 File "/usr/lib64/python2.6/site-packages/mercurial/dispatch.py", line 46, in _runcatch
 return _dispatch(ui, args)
 File "/usr/lib64/python2.6/site-packages/mercurial/dispatch.py", line 449, in _dispatch
 return runcommand(lui, repo, cmd, fullargs, ui, options, d)
 File "/usr/lib64/python2.6/site-packages/mercurial/dispatch.py", line 319, in runcommand
 ret = _runcommand(ui, options, cmd, d)
 File "/usr/lib64/python2.6/site-packages/mercurial/dispatch.py", line 500, in _runcommand
 return checkargs()
 File "/usr/lib64/python2.6/site-packages/mercurial/dispatch.py", line 454, in checkargs
 return cmdfunc()
 File "/usr/lib64/python2.6/site-packages/mercurial/dispatch.py", line 448, in
 d = lambda: util.checksignature(func)(ui, *args, **cmdoptions)
 File "/usr/lib64/python2.6/site-packages/mercurial/util.py", line 386, in check
 return func(*args, **kwargs)
 File "/usr/lib64/python2.6/site-packages/mercurial/extensions.py", line 116, in wrap
 util.checksignature(origfn), *args, **kwargs)
 File "/usr/lib64/python2.6/site-packages/mercurial/util.py", line 386, in check
 return func(*args, **kwargs)
 File "/home/jsosic/.hgsubversion/hgsubversion/wrappers.py", line 528, in clone
 orig(ui, source, dest, **opts)
 File "/usr/lib64/python2.6/site-packages/mercurial/util.py", line 386, in check
 return func(*args, **kwargs)
 File "/usr/lib64/python2.6/site-packages/mercurial/commands.py", line 649, in clone
 update=opts.get('updaterev') or not opts.get('noupdate'))
 File "/usr/lib64/python2.6/site-packages/mercurial/extensions.py", line 128, in wrap
 return wrapper(origfn, *args, **kwargs)
 File "/home/jsosic/.hgsubversion/hgsubversion/wrappers.py", line 517, in hgclonewrapper
 data['srcrepo'], data['dstrepo'] = orig(ui, *args, **opts)
 File "/usr/lib64/python2.6/site-packages/mercurial/hg.py", line 292, in clone
 dest_repo.clone(src_repo, heads=revs, stream=stream)
 File "/usr/lib64/python2.6/site-packages/mercurial/localrepo.py", line 2144, in clone
 return self.pull(remote, heads)
 File "/home/jsosic/.hgsubversion/hgsubversion/svnrepo.py", line 81, in wrapper
 return fn(self, *args, **opts)
 File "/home/jsosic/.hgsubversion/hgsubversion/svnrepo.py", line 104, in pull
 return wrappers.pull(self, remote, heads, force)
 File "/home/jsosic/.hgsubversion/hgsubversion/wrappers.py", line 271, in pull
 svn_url = source.svnurl
 File "/home/jsosic/.hgsubversion/hgsubversion/svnrepo.py", line 154, in svnurl
 return self.svn.svn_url
 File "/usr/lib64/python2.6/site-packages/mercurial/util.py", line 150, in __get__
 result = self.func(obj)
 File "/home/jsosic/.hgsubversion/hgsubversion/svnrepo.py", line 159, in svn
 return svnwrap.SubversionRepo(*self.svnauth, password_stores=self.password_stores)
 File "/home/jsosic/.hgsubversion/hgsubversion/svnwrap/subvertpy_wrapper.py", line 179, in __init__
 self.init_ra_and_client()
 File "/home/jsosic/.hgsubversion/hgsubversion/svnwrap/subvertpy_wrapper.py", line 231, in init_ra_and_client
 auth=auth)
 File "/usr/lib64/python2.6/site-packages/subvertpy/ra.py", line 47, in RemoteAccess
 return url_handlers[type](url, *args, **kwargs)
 subvertpy.SubversionException: ("OPTIONS of 'https://svnserver/svn/repo': authorization failed:
 Could not authenticate to server: rejected Basic challenge (https://svnserver)", 170001)
 [1]    15247 exit 1     hg clone svn+https://svnserver/svn/repo

Now this second error is due to subvertpy not asking for authentication credentials… So what you have to do to fix this one is to store SVN password in either gnome-keyring, kwallet or simply plaintext… After that, HG works finaly ūüôā

Edit: I’ve built hgsubversion RPM package and put it into SRCE rpm repositories.

Categories: Development Tags: , , ,
%d bloggers like this: