Archive

Archive for October, 2014

Using ElasticSearch templates with Logstash

October 18, 2014 1 comment

I cast aside my chains
Fall from reality
Suicidal disease
The face of all your fears
(At The Gates – Blinded by fear)

Logstash is great tool for acquiring logs and turning them from txt files into JSON documents. When using ElasticSearch as backend for Logstash, Logstash auto-creates indexes. This can be a bit of a problem if you have fields with dots in its contents, like “host”. If host name is recorded as FQDN (as it usually is), then ElasticSearch will analyse that field and split it at dots. Now, node04.example.com gets dissected into node04 and example.com – which can pose a problem in Kibana interface when you try to figure out where the logs came from. Example I got in my case was that 3 nodes in one cluster had approximately 1/3 of requests each, and there was also “example.com” showed as combined graph with those 3 values added, as you can see on the graph below:

blog_kibana

Green bar equals the addition of yellow, blue and orange. Now, this could make sense if you had different domains and wanted to see comparison between them, but since I did not want that I had to find a way to remove it. Enter Elasticsearch templates. Templates will automatically be applied to new indices, when they are created. Since Logstash automatically creates indices, this is ideal to set things up as we want.

Since I wanted to change mappings, first thing I have to do is get current mappings to see what am I up against:

% wget http://elasticsearch:9200/<index_name>/_mapping

That page will give you JSON document, which is in oneliner style. To correct the coding style to classic JSON, to make it more readable, just copy/paste it to http://jsonlint.com/ and click Validate. Now, remove everything except the fields you want to modify. My example mapping with everything except relevant lines removed looks something like this:

{
    "<index_name>": {
        "mappings": {
            "apache_error": {
                "properties": {"host":{"type":"string"},}
            },
            "apache_access":{
                "properties": {"host":{"type":"string"},}
            }
        },
    }
}

Now, we have to add  “index”: “not_analyzed” param for every host key in the upper json. Also, template regex should be added so that ElasticSearch knows indices that this template applies to:

After his manipulation, we can either wrap it into new <template_name> tag, or upload it directly to Elasticsearch with curl:

curl -XPUT localhost:9200/_template/template_developer -d '
{
    "template": "developer*",
    "mappings": {
        "apache_error": {
            "properties": {"host":{"type":"string"}, "index":"not_analyzed"}
        },
        "apache_access":{
            "properties":{"host":{"type":"string"}, "index":"not_analyzed"}
        }
    }
}
'

Now, some of us engineers don’t like to upload configurations to services, but like to keep them in text files. That is also possible, by creating a directory called templates under path.conf – which is usually /etc/elasticsearch. In the latter case, template should look a liitle bit different:

# cat /etc/elasticsearch/templates/template_developer.json 
{
    "template_developer": {
        "template": "developer*",
        "mappings": {
            "apache_error": {
                "properties": {
                    "host": {"type": "string", "index": "not_analyzed"}
                }
            },
            "apache_access": {
                "properties": {
                    "host": {"type": "string", "index": "not_analyzed"}
                }
            }
        }
    }
}

After this, you need to restart Elasticsearch, and recreate indice, and that’s it! New graph looks like:

blog_kibana_post

Advertisements
%d bloggers like this: