Configuration and patterns rspec tests on LogStash v1.4

April 6, 2015 Leave a comment

We were the warriors of the north
Notorious and brave
We’d never lost a fight in war
We feared not the grave
(Amon Amarth – Warriors of the North)

If you have a large farm of web servers you often end up depending on some kind of log aggregation stack. Most popular one today is ELK (ElasticSearch, Logstash, Kibana). From operations perspective, changing any system daemon or component that generates logs, often means changing Logstash patterns and filters, while maintaining backwards compatibility (because – all the systems won’t get updated at once). Solution is to write patterns which will work for both the old and new log entries.

To help us with testing, it’s always a good practice to write unit tests. In ruby, rspec is a great tool to accomplish testing, and LogStash is extended with rspec support. But, the documentation is very poor, and all the online examples show how to do tests for version 1.1. Those tests won’t work with 1.4. So, after couple of hours of trial and error, I’ve come up with a test that actually works:

# encoding: utf-8

require "test_utils"
require "logstash/filters/grok"

describe LogStash::Filters::Grok do
  extend LogStash::RSpec

  describe "simple syslog line" do
    # The logstash config goes here.
    # At this time, only filters are supported.
    config <<-CONFIG
    filter {
      grok {
        match => { "message" => "%{WORD:foo}" }
      }
    }
    CONFIG

    sample 'thisisnotspace' do
      insist { subject['foo'] } == "thisisnotspace"
    end
  end
end

Save this text to a file called /tmp/test.rb and run it:

# /opt/logstash/bin/logstash-test /root/logstash_test_off/test2.rb
Using Accessor#strict_set for specs
Run options: exclude {:redis=>true, :socket=>true, :performance=>true, :elasticsearch=>true, :broken=>true, :export_cypher=>true}
.

Finished in 0.173 seconds
1 example, 0 failures

Wonderful!

Now, a little bit of explaining. This is the anchor for our tests:

# encoding: utf-8

require "test_utils"
require "logstash/filters/grok"

describe LogStash::Filters::Grok do
  extend LogStash::RSpec

end

and this part should be the same in every .rb LogStash spec test. Inner “describe” section is what interests us the most. Name of the section is irrelevant, so you can put your description of the test:

describe "your test description goes here" do
end

Next part is the actual LogStash configuration file. The important section for our tests is filter, while we don’t need input and output sections:

    config <<-CONFIG
    filter {
      grok {
        match => { "message" => "%{WORD:foo}" }
      }
    }
    CONFIG

Now, the only thing we still need are actual log lines and expected values. Input (log line) example comes between sample and do keywords, while expected values can be defined within the block using the insist:

    sample 'thisisnotspace' do
      insist { subject['foo'] } == "thisisnotspace"
    end

We can add as many sample sections as we want, with different log lines and different expected values.

Now go and write your own tests for patterns and filters!

Deploying custom SELinux modules

March 11, 2015 Leave a comment

Asgård’s always been my home
But I’m of different blood
I will overthrow the throne
Deceiver! Deceiver of the gods!
(Amon Amarth – Deceiver of the Gods)

If you decide to run OS with SELinux enabled, sooner or later you’ll bump into roadblock. So, the question is how to write your own SELinux rules and deploy them. For managing rules, I’m using Puppet module spiette/selinux.

Lets review the practical example from one of my hosts. Lets say you want to pipe all your emails to a script which will save them to a certain directory as HTML files. /etc/aliases has line like this one:

mailcatcher:    "|/usr/local/bin/decode.pl"

After sending email to mailcatcher user, this is what we can see in maillog:

Mar  3 09:59:58 mailcatcher postfix/local[16030]: 589207055A: 
   to=<mailcatcher@devel.localdomain>, orig_to=<jsosic@example.com>,
   relay=local, delay=0.1, delays=0/0/0/0.1, dsn=5.3.0,
   status=bounced
   (Command died with status 13: "/usr/local/bin/decode.pl".
   Command output: Could not open file
   '/var/www/mailcatcher/1425598_2015-03-03_09-59-58.461450099.html'
   Permission denied at /usr/local/bin/decode.pl line 50. )

After checking up directory permissions, it’s obvious there’s something else blocking access, and if you’re running RedHat derivative (CentOS/Scientific/Fedora) that something is usually SELinux. To confirm that, take a look at /var/log/audit/auditd.log:

type=SYSCALL msg=audit(1425644753.803:20374): arch=c000003e
   syscall=83 success=no exit=-13 a0=16c6580 a1=1ff a2=0
   a3=786966682d726865 items=0 ppid=26763 pid=26764 auid=1505 uid=498
   gid=498 euid=498 suid=498 fsuid=498 egid=498 sgid=498 fsgid=498
   tty=(none) ses=1355 comm="decode.pl" exe="/usr/bin/perl"
   subj=unconfined_u:system_r:postfix_local_t:s0 key=(null)
type=AVC msg=audit(1425644759.713:20375): avc:  denied  { create }
  for  pid=26777
  comm="decode.pl" name="example-name"
  scontext=unconfined_u:system_r:postfix_local_t:s0
  tcontext=unconfined_u:object_r:httpd_sys_content_t:s0 tclass=dir

Now we have a reason why our script can’t write a file. Next step is to pipe this long to audit2allow -m mailcatcher, which will generate the following output:

module mailcatcher 1.0;

require {
    type httpd_sys_content_t;
    type postfix_local_t;
    class dir create;
}

#============= postfix_local_t ==============
allow postfix_local_t httpd_sys_content_t:dir create;

Next step is to compile this module and load it into SELinux, and retry if our script now works as designed.

To compile it, save it in a file called mailcatcher.te, and run the following commands:

# checkmodule -M -m -o mailcatcher.mod mailcatcher.te
# semodule_package -o mailcatcher.pp -m mailcatcher.mod
# semodule -r mailcatcher.pp
# semodule -i mailcatcher.pp

After runnning the last command, you can recheck if your script works. If it has permission problems again, just repeat the process until you create a module that works. In my case, final module looks like this:

# Allows postfix (running under type context postfix_local_t)
# to write to web directories (type httpd_sys_content_t).
module mailcatcher 1.1;

require {
    type httpd_sys_content_t;
    type postfix_local_t;
    class dir { write search getattr add_name };
    class file { write ioctl create open getattr };
}

#============= postfix_local_t ==============
allow postfix_local_t httpd_sys_content_t:dir { write search getattr add_name };
allow postfix_local_t httpd_sys_content_t:file { write ioctl create open getattr };

Now, to automate it via puppet, save the file under yourmodule/files/selinux/mailcatcher.te. You can later use it in (any) manifest with the following code snippet:

include ::selinux
::selinux::module { 'mailcatcher':
  ensure => present,
  source => 'puppet:///modules/yourmodule/selinux/',
}

Puppet will transfer .te file to destination host, compile it and load it for you.

Categories: Linux, RedHat, Security Tags: , ,

htop atop tmux on XenServer 6.x

December 13, 2014 Leave a comment

Save me from myself if you ever really cared
Save me from myself, tell me you’re not scared
(Damage Plan – Save me)

If you want to run diagnostic commands on XenServer, you’re pretty much limited with available options. Since XenServer is based on RHEL 5.x series, with mostly 32bit libraries installed, we can use packages for RHEL/CentOS 5.x series i386 arch. Three tools that I use most often for initial screening of the servers are htop, atop and tmux. Doing something like ‘xe vm-export‘ and backing up VMs to external USB can be like reading a book in a dark room. There’s no progress bar, there’s no info – nothing. Calculating speed looks something along these lines:

# ls -al pbx.xva; sleep 60; ls -al pbx.xva 
-rw------- 1 root root 13799469056 Dec 13 23:00 pbx.xva 
-rw------- 1 root root 14982000640 Dec 13 23:01 pbx.xva

And after that: (14982000640 – 13799469056 ) / 60 / 1024 / 1024 = 18.79 MB/s. 🙂

Attaching/detaching or sharing sessions is a wet dream… only way to do it is to run tmux on a machine from which you are connecting to SSH of XenServer.

So, after I really got annoyed, I tried to install tmux. Initial tries with 64bit package for CentOS 6 were complaining about missing x86_64 libraries, so I switched to 32bit packages. That didn’t work also, complaining about too new version of rpmlib 🙂 So solution was obvious – use 32bit EPEL packages! These are the packages that I use:

# rpm -qa | egrep '(top|tmux)'
tmux-1.4-3.el5.1
atop-1.27-2.el5
htop-0.8.3-1.el5

Now we’re talking business!

HaProxy error 503 if dependencies are down

November 16, 2014 Leave a comment

Revenge
I’m screaming revenge again
Wrong
I’ve been wrong for far too long
(Pantera – Mouth for War)

Every http app usually has a number of dependencies – like database (PostgreSQL or MySQL), caching layers (memcache, redis) or NoSQL backends like MongoDB. In case any of those dependencies are down, it’s good to inform public about it in a neat way. If you fail to do that, search engines will probably down rank your site. General advice from SEO sites is to avoid returning 200 OK or 404 Not Found when you want to inform users that site is down. In case of downtime, proper way to inform users (which include crawlers too) is to retutrn 503 Service Unavailable.

If your site depends heavily on some service like MySQL and won’t work without it, this check can be done at load balancer (HaProxy) level. Configuration is simple – first step is to create a user in MySQL database for MySQL check:

$ mysql -u root -e "INSERT INTO mysql.user (Host,User) \
  values ('127.0.0.1','haproxy_check'); FLUSH PRIVILEGES;" -p

HaProxy uses this database user for functional checks (runs SELECT(1)). HaProxy config file should look like this:

backend myhttpdfarm
        option httpchk HEAD /check.txt HTTP/1.0
        server web01 127.0.0.1:80 weight 1 web01

backend mysql
        option mysql-check user haproxy_check
        server mysql_master 127.0.0.1:3306 check

listen main
        bind    127.0.0.1:80
        errorfile 503 /etc/haproxy/503.html
        use_backend myhttpdfarm if { srv_is_up(mysql/mysql_master) eq 1 }

Crucial part is “us_backend … if” directive. You can even chain conditionals, and add additional checks for memcache, Mongo, etc…

Using ElasticSearch templates with Logstash

October 18, 2014 1 comment

I cast aside my chains
Fall from reality
Suicidal disease
The face of all your fears
(At The Gates – Blinded by fear)

Logstash is great tool for acquiring logs and turning them from txt files into JSON documents. When using ElasticSearch as backend for Logstash, Logstash auto-creates indexes. This can be a bit of a problem if you have fields with dots in its contents, like “host”. If host name is recorded as FQDN (as it usually is), then ElasticSearch will analyse that field and split it at dots. Now, node04.example.com gets dissected into node04 and example.com – which can pose a problem in Kibana interface when you try to figure out where the logs came from. Example I got in my case was that 3 nodes in one cluster had approximately 1/3 of requests each, and there was also “example.com” showed as combined graph with those 3 values added, as you can see on the graph below:

blog_kibana

Green bar equals the addition of yellow, blue and orange. Now, this could make sense if you had different domains and wanted to see comparison between them, but since I did not want that I had to find a way to remove it. Enter Elasticsearch templates. Templates will automatically be applied to new indices, when they are created. Since Logstash automatically creates indices, this is ideal to set things up as we want.

Since I wanted to change mappings, first thing I have to do is get current mappings to see what am I up against:

% wget http://elasticsearch:9200/<index_name>/_mapping

That page will give you JSON document, which is in oneliner style. To correct the coding style to classic JSON, to make it more readable, just copy/paste it to http://jsonlint.com/ and click Validate. Now, remove everything except the fields you want to modify. My example mapping with everything except relevant lines removed looks something like this:

{
    "<index_name>": {
        "mappings": {
            "apache_error": {
                "properties": {"host":{"type":"string"},}
            },
            "apache_access":{
                "properties": {"host":{"type":"string"},}
            }
        },
    }
}

Now, we have to add  “index”: “not_analyzed” param for every host key in the upper json. Also, template regex should be added so that ElasticSearch knows indices that this template applies to:

After his manipulation, we can either wrap it into new <template_name> tag, or upload it directly to Elasticsearch with curl:

curl -XPUT localhost:9200/_template/template_developer -d '
{
    "template": "developer*",
    "mappings": {
        "apache_error": {
            "properties": {"host":{"type":"string"}, "index":"not_analyzed"}
        },
        "apache_access":{
            "properties":{"host":{"type":"string"}, "index":"not_analyzed"}
        }
    }
}
'

Now, some of us engineers don’t like to upload configurations to services, but like to keep them in text files. That is also possible, by creating a directory called templates under path.conf – which is usually /etc/elasticsearch. In the latter case, template should look a liitle bit different:

# cat /etc/elasticsearch/templates/template_developer.json 
{
    "template_developer": {
        "template": "developer*",
        "mappings": {
            "apache_error": {
                "properties": {
                    "host": {"type": "string", "index": "not_analyzed"}
                }
            },
            "apache_access": {
                "properties": {
                    "host": {"type": "string", "index": "not_analyzed"}
                }
            }
        }
    }
}

After this, you need to restart Elasticsearch, and recreate indice, and that’s it! New graph looks like:

blog_kibana_post

Increase RabbitMQ file descriptor limit and memory watermark without restart

September 10, 2014 3 comments

Our time here
Is now at end
Can’t help but reminisce
A cold spring day
So long ago
When we set out to sea
(Amon Amarth – Varyags Of Miklagaard)

In this all-demanding world, there is often a need to change some security (or other) limits of running processes without restarting them. One situation I found my self in was need to increase RabbitMQ’s file descriptor limit. By default, for non-root processes limit is 1024. Lets describe the process when you want to change limits with restart first.

When you want to increase that limit, first thing to change would be the settings of security limits. We can do that by creating new file in /etc/security/limits.d/. Lets name it rabbitmq.conf:

# cat >> /etc/security/limits.d/rabbitmq.conf <<EOF
# rabbitmq
# Increase maximum number of open files from 1024 to 4096 for RabbitMQ

#<domain>    <type>    <item>    <value>
rabbitmq    soft    nofile    4096
EOF

To check if this setting is correct, run:This is to ensure Now, lets create directories for git and create our first repo:

# su - rabbitmq -s /bin/sh -c 'ulimit -n'
4096

Now, it would be enough to restart RabbitMQ and new settings would get detected. Unfortunately if you don’t want to or can’t restart Rabbit, we have to do some more mingling.

First, we need to find out all processes running as user RabbitMQ:

# ps auxw | grep ^rabbit | cut -d' ' -f 3
2103
6300
7204
7334
7335

Now we can check for each of those PIDs what are their current limits, for example for PID 2103:

# cat /proc/2103/limits  | grep open
Max open files            4096                 4096                 files

Now for each of these processes we need to increase ulimit. We can do that by running:

# echo -n "Max open files=4096:4096" > /proc/2103/limits

After this, limits will be increased and processes should know that. But looking at RabbitMQ status output, we can see that it’s not the case:

# rabbitmqctl status | grep -A 4 file_descriptors
 {file_descriptors,
     [{total_limit,924},
      {total_used,291},
      {sockets_limit,829},
      {sockets_used,195}]},

Now, to increase the value, just run the following command. Note that we increase it to ulimit minus 100, so it’s not 4096 but 3996. That’s the way RabbitMQ does it by default so we will stick to it:

# rabbitmqctl eval 'file_handle_cache:set_limit(3996)'

If you, on the other hand, just want to increase memory usage without restarting RabbitMQ, that can be done with simple:

# rabbitmqctl set_vm_memory_high_watermark 0.8

Also, don’t forget to save vm_memory_high_watermark settings in rabbitmq.config file, so that they are persistent across (eventual) restarts.

Note: If you run CentOS 7 or Fedora which use systemd, it’s not enough to change ulimits via limits.conf, you need to actually set “LimitNOFILE=32768” in .service file. Hip hip hurray for systemd (sigh).

Note2: Thanx to Simon MacMullen for his help on the issue.

Git over HTTP on CentOS 6

May 23, 2014 7 comments

Total war is here
Face it without fear
Age of sword, age of spear
Fight for honor, glory, death in fire!
(Amon Amarth – Death in Fire)

There is already a bunch of posts about setting up Git over HTTP(S), but this one is specificaly targeted at setting it up under CentOS as cleanly as possible. There was bunch of errors that I saw along the way, so I will try to explain the process step by step.

First, you have to install Apache and Git.:

# yum -y install httpd git
# /etc/init.d/httpd start

Now, lets create directories for git and create our first repo:

# mkdir /var/www/gitrepos
# cd /var/www/gitrepos
# mkdir repo01 && cd repo01
# git --bare init
# git update-server-info
# cd /var/www
# chown -R apache: gitrepos

We are using ‘git –bare’, so that online repository doesn’t have files but only git metadata. That will enable users to push directly to online repository, otherwise they wouldn’t be able to push thier changes. This was the first error I did, I created repo with ‘git init’ and was not able to push later. After the repo is set up and chowned, lets set up apache. This is my configuration for vhost:

#
# vhost for git repositories (http)
#
<VirtualHost *:80>
    ServerName     git
    DocumentRoot    /var/www/gitrepos

    <Location />
        DAV on

        # general auth settings
        AuthName "Git login:"
        AuthType Basic

        # file authentication
        AuthUserFile  /var/www/htpasswd
        AuthGroupFile /var/www/htgroup

        <LimitExcept PROPFIND>
            Require valid-user
        </LimitExcept> 
    </Location>

    <Location /repo01>
        <LimitExcept PROPFIND>
            Require group adminlinux
        </LimitExcept> 
    </Location>

    LogLevel warn
    ErrorLog  /var/log/httpd/git_error.log
    CustomLog /var/log/httpd/git_access.log combined
</VirtualHost>

If you wonder why is PROPFIND method treated differently from all other http/dav methods – it’s because webserver runs PROPFIND without user authentication, so if it’s not excluded from limit, it will get rejected and you will see a message similar to this one when trying to push from the client:

error: Cannot access URL https://git/puppet-adriatic/, return code 22
fatal: git-http-push failed

We can fill up htpasswd file with – tadddaaa htpasswd command 🙂

# htpasswd -c /var/www/htpasswd user1
# htpasswd -c /var/www/htpasswd user2
# htpasswd -c /var/www/htpasswd user3

And htgroup with:

# echo "adminlinux: user1 user2" >> /var/www/htgroup

Now, on the client side, do a:

% git clone http://user1@git/repo01

And that’s it! After the first change/commit you do, be careful when you push those changes for the first time. This is the command I used for the first push:

% git push --set-upstream origin master

You may also encounter a 22/502 error on a MOVE command, like:

MOVE 12486a9c101c613c075d59b5cf61329f96f9ae12 failed, aborting (22/502)
MOVE 0c306c54862ae8c21226281e6e4f47c8339ed132 failed, aborting (22/502)
MOVE ce4c4fc9d1e4daf3a59516829a0e1bd6c66d4066 failed, aborting (22/502

This happened to me because I used http to https forwarding in apache, and I had a http specified in my .git/config on a client machine. After changing the destination to https, MOVE command did it’s magic. It seems that this error is result of server name/location being different in client repo and on a server side.

Note: I recommend using SSL and not plain text http, even with self-signed certificates. In that scenario you’ll probably want to use env variable GIT_SSL_NO_VERIFY=true.

Note2: CentOS ships with old version of git, 1.7.x so I recommend either using git from IUS repo (git2x packages) or backporting git from newer Fedora releases.

Categories: Development, Linux, RedHat Tags: , , ,
%d bloggers like this: