Archive

Archive for the ‘Puppet’ Category

Puppet module for managing Cobbler

December 29, 2012 3 comments

I was working on a Puppet module for managing Cobbler last few months. It’s my first module with custom types, so I was on a steep learning curve. We’ve been using it in production for few weeks, but I felt it wasn’t as good as it should have been. Puppet agent runs were very slow (longer than a minute) because custom providers did several command line queries to get the current state of the system from Cobbler. These were all ordinary strings and providers had to cut out relevant information and store it into organized data formats like hashes. So, I’ve decided to rewrite the providers to use XMLRPC interface Cobbler offers and to fetch the state of the whole system with one single XMLRPC query. See the results of rewrite yourself:

Site1 (22 x cobblersystem)
- agent run before: 68.1s
- agent run after:   9.7s

Site2 (19 x cobblersystem)
- agent run before: 54.0s
- agent run after:  10.0s

Site3: (11 x cobblersystem)
- agent run before: 40.9s
- agent run after:   9.7s

Site4: (22 x cobblersystem)
- agent run before: 80.1s
- agent run after:  10.6s

As you see, cobbler custom providers did really big impact on agent run times before I rewrote them to use ‘self.instances’ function. So finally I was happy with the module. Now I can proudly publish it 🙂 Puppet module is available at Puppet Forge:

http://forge.puppetlabs.com/jsosic/cobbler

Code is available at BitBucket, as is the issue tracker:

https://bitbucket.org/jsosic/puppet-cobbler

Hope you like it!

Advertisements
Categories: Cobbler, Puppet

Precommit hook for Puppet code

December 13, 2012 1 comment

Please God, give me the strength to lead
It’s my charge to keep
Make the sorrow the fuel I need
It’s my charge to keep

(Iced Earth – A charge to keep)

As my colleagues slowly pick up the pace with Puppet modules development I’ve charged, we stumbled upon a problem experienced developers see so often. Code that was pushed to our repository started to get messy. Syntax errors were rare, but the coding style standard went berzerk with the first devop that joined the puppet “team”. My initial reaction was to re-factor the code every now and then, and by re-factor I mean running checks on my own working copy. And that started to consume too much of my time… rewriting modules, committing, checking if they they work as intended after my changes… and finally I was pissed… Being obsessive compulsive may seem counter-productive at times but some rules just have to be obeyed.
So, instead of continuing to fix other’s mistakes, I’ve decided to put a stop on them. This is what I was doing on our modules repository:

for i in $(find ~/puppet -regex ".*pp$"); do
  echo $i;
  puppet parser validate $i;
  if [[ ! $? -eq 0 ]]; then
    echo"* puppet syntax errors found!"
    break
  fi
  puppet-lint --no-80chars-check --fail-on-warnings $i
  if [[ ! $? -eq 0 ]]; then
    echo "* puppet code style broken!"
    break
  fi
done

This would spit out any syntax error or code style divergence. So, my idea now was to offload that to people writing code. So, I decided to make a precommit hook. We use mercurial, so my .hg/hgrc looks like this:

[paths]
default = https://code.google.com/p/lutak/

[hooks]
pretxncommit.puppet = .hg/check_puppet.rb

And finally, check_puppet.rb script (which has to be set executable):

#!/usr/bin/ruby

def puppet_parser_validate(file)
    if !system('puppet parser validate ' + file + ' > /dev/null 2>&1')
    print('!!! Syntax error in file: ' + file + "\n")
        system('puppet parser validate ' + file)
    exit(1)
    end
end

def puppet_lint(file)
    if !system('puppet-lint --no-80chars-check --fail-on-warnings ' + file + ' > /dev/null 2>&1')
    print('!!! Coding style error in file: ' + file + "\n")
        system('puppet-lint --no-80chars-check --fail-on-warnings ' + file)
    exit(1)
    end
end

def puppet_erb_check(file)
    if !system('erb -x -T \'-\' ' + file + ' | ruby -c > /dev/null 2>&1')
    print('!!! Syntax error in erb template: ' + file + "\n")
        system('erb -x -T \'-\' ' + file + ' | ruby -c')
    exit(1)
    end
end

def puppet_ruby_check(file)
    if !system('ruby -c ' + file + ' > /dev/null 2>&1')
    print('!!! Syntax error in erb template: ' + file + "\n")
        system('ruby -c ' + file)
    exit(1)
    end
end

# go through list of files, and call adequate checks
IO.popen('hg status | grep -v ^R').readlines.each { |file| 
    file.sub!(/^[\w|\?] (.*)\n/,'\1')
    if file.match('.pp$')
        puppet_parser_validate file
        puppet_lint file
    elsif file.match('.erb$')
    puppet_erb_check file
    elsif file.match('.rb$')
    puppet_ruby_check file
    end
}

Script checks every file that occurs in the output of hg status. Some may not be happy with that solution, so if you want to skip files that are not yet added to your repository, you could change grep -v ^R to egrep -v “^(p|\?)”. If you want to run this script, the machine has to have puppet and rubygem puppet-lint installed. You can fetch the latter from SRCE repositories if you use RHEL/CentOS:

ftp://ftp.srce.hr/redhat/base/el6/x86_64/rubygem-puppet-lint-0.3.2-1.el6.srce.noarch.rpm
ftp://ftp.srce.hr/redhat/base/el5/x86_64/rubygem-puppet-lint-0.3.2-1.el5.srce.noarch.rpm

Now, next problem is how to force every member of a team to adhere to precommit and not turn it off? Well, I haven’t come to that part yet 🙂 But social engineering seems to work so far. So, if the people understand the benefit of such approach, they will in most cases respect it and use it. Otherwise I’ll be hanging at their necks 🙂

Note that these are just basic tests, in the long run I will try to implement some kind of CI solution that runs complex logical tests of the codebase, because with every new commit there comes dozen of new problems 🙂

Categories: Puppet Tags: , , ,

Array join in Puppet manifest

December 10, 2012 1 comment

Maybe some of you noticed that as of time of writing this blog – there is a split() function available in puppet declarative language, but there is no join()? Well that puzzled me too, because I wanted to join an array and serve it as a content of a file. Well, PuppetLabs made inline_template() available to us lazy users, so we can use all the erb code we want inside our manifests! So, how did I join my array? Here is nice example:

$routes = ['192.168.2.0/24 via 192.168.1.2', '192.168.3.0/24 via 192.168.1.3'],
$fullroute = inline_template("<%= (routes).join('\n') + \"\n\" %>")

Now, I can feed $fullroute to some file as content and that’s it!

file { '/etc/sysconfig/network-scripts/route-eth0':
  ensure  => file,
  content => "$fullroute",
}

Pretty neat, isn’t it? 🙂

Categories: Puppet Tags: , ,

Managing ssh host keys with Puppet

December 4, 2012 Leave a comment

Death of mother Earth
Never a rebirth
Evolution’s end
Never will it mend

(Metallica – Blackened)

Do you get annoyed when after every clean installation of an existing node you have to clean up your known_hosts file, accept new host key? Especially if you manage development machines that are reinstalled often. As we are starting to use Puppet extensively, I’ve thought about solving this problem with it. And I came up with pretty neat solution that I hope will benefit the masses. I’ve created ssh module, which has ssh::server class that looks like this:

class ssh::server {
...
  if generate('/etc/puppet/modules/ssh/scripts/generate_host_keys.sh',
$keys_dir) {
    include ssh::server::keys
  }
...
}

Generate function calls shell script, and if it’s exit status is 0, ssh::server class includes ssh::server::key class. The latter class defines the ssh host keys (as file resources), sets the correct permissions and reloads ssh service if any of the files/keys changes. This is an example of one file resource in that class:

file { '/etc/ssh/ssh_host_rsa_key.pub':
  ensure  => file,
  owner   => root,
  group   => root,
  mode    => '0644',
  source  => [
    'puppet:///private/ssh/ssh_host_rsa_key.pub',
    'puppet:///modules/ssh/ssh_host_rsa_key.pub',
  ],
  require => Package['openssh-server'],
  notify  => Service[$service_name],
}

Private section of puppet server is set up in /etc/puppet/fileserver.conf. Beauty of this approach is that each agent knows only for the existence of directories under his own $fqdn, so one host can’t access other hosts sensitive data (like ssh host keys). Second line in “source” property array value is just as backend if, for some reason, first one is not found. Class ssh:server:keys is not designed to be included directly in node manifest, but to be used as ssh::server class’s helper, so thas should not happen. This is a snippet from fileserver.conf:

[private]
 path /etc/puppet/private/%H
 allow *

and this is the directory structure:

# find /etc/puppet/private/
/etc/puppet/private/
/etc/puppet/private/host1.example.lan
/etc/puppet/private/host2.example.lan
/etc/puppet/private/host2.example.lan/ssh
/etc/puppet/private/host2.example.lan/ssh/ssh_host_key.pub
/etc/puppet/private/host2.example.lan/ssh/ssh_host_dsa_key
/etc/puppet/private/host2.example.lan/ssh/ssh_host_dsa_key.pub
/etc/puppet/private/host2.example.lan/ssh/ssh_host_key
/etc/puppet/private/host2.example.lan/ssh/ssh_host_rsa_key
/etc/puppet/private/host2.example.lan/ssh/ssh_host_rsa_key.pu

As you can see, host2 has keys already generated and host1 doesn’t. And, now the magic glue behind all of this, generate_host_keys.sh script:

#!/bin/bash

# check arg0: dir for keys
[ -z "$1" ] && echo "Please specify directory for key generation" && exit 1
KEYSDIR="$1"

# set umask
umask 0022

# create directory tree if it does not exist
[ ! -d "$KEYSDIR" ] && mkdir -p $KEYSDIR

#
# functions stolen from CentOS 6 sshd init script
#

# Some functions to make the below more readable
KEYGEN=/usr/bin/ssh-keygen
RSA1_KEY=$1/ssh_host_key
RSA_KEY=$1/ssh_host_rsa_key
DSA_KEY=$1/ssh_host_dsa_key

# source function library
. /etc/rc.d/init.d/functions

fips_enabled() {
  if [ -r /proc/sys/crypto/fips_enabled ]; then
    cat /proc/sys/crypto/fips_enabled
  else  
    echo 0
  fi
}

do_rsa1_keygen() {
  if [ ! -s $RSA1_KEY -a `fips_enabled` -eq 0 ]; then
    echo -n $"Generating SSH1 RSA host key: "
    rm -f $RSA1_KEY
    if test ! -f $RSA1_KEY && $KEYGEN -q -t rsa1 -f $RSA1_KEY -C '' -N '' >&/dev/null; then
      chmod 600 $RSA1_KEY
      chmod 644 $RSA1_KEY.pub
      success $"RSA1 key generation"
      echo
    else  
      failure $"RSA1 key generation"
      echo
      exit 1
    fi
  fi
}

do_rsa_keygen() {
  if [ ! -s $RSA_KEY ]; then
    echo -n $"Generating SSH2 RSA host key: "
    rm -f $RSA_KEY
    if test ! -f $RSA_KEY && $KEYGEN -q -t rsa -f $RSA_KEY -C '' -N '' >&/dev/null; then
      chmod 600 $RSA_KEY
      chmod 644 $RSA_KEY.pub
      success $"RSA key generation"
      echo
    else  
      failure $"RSA key generation"
      echo
      exit 1
    fi
  fi
}

do_dsa_keygen() {
  if [ ! -s $DSA_KEY ]; then
    echo -n $"Generating SSH2 DSA host key: "
    rm -f $DSA_KEY
    if test ! -f $DSA_KEY && $KEYGEN -q -t dsa -f $DSA_KEY -C '' -N '' >&/dev/null; then
      chmod 600 $DSA_KEY
      chmod 644 $DSA_KEY.pub
      success $"DSA key generation"
      echo
    else
      failure $"DSA key generation"
      echo
      exit 1
    fi
  fi
}

# main
do_rsa1_keygen
do_rsa_keygen
do_dsa_keygen
chmod -R 644 $KEYSDIR/*
exit 0

Hope you like the solution, and I promise to publish the whole module on the forge when it’s polished enough!

Categories: Puppet, RedHat Tags: , , ,

Vim and Puppet

August 25, 2012 Leave a comment

The dreamer and the wine
Poet without a rhyme
A widow writer torn apart by chains of Hell
(Nightwish – The poet and the pendulum)

One nice thing about vim is ability to extend it with additional plugins. So I’ve stumbled across nice plugin for syntax and code styling checking tools. As I know I’ll be doing lots of my time editing puppet code, I guess I really need this.
For this magic to work, workstation has to have some tools installed:

# yum install puppet rubygem-puppet-lint

Now let’s do some vim magic:

$ mkdir -p ~/.vim/bundle ~/.vim/autoload
$ cd /tmp
$ git clone https://github.com/tpope/vim-pathogen
$ mv vim-pathogen/autoload/pathogen.vim ~/.vim/autoload/
$ cd ~/.vim/bundle
$ git clone https://github.com/scrooloose/syntastic.git

Also you have to have this lines in your .vimrc:

call pathogen#infect()
filetype indent on

And that’s about it!

Now, when you make a code style error for example, vim will detect it and inform you of it!

Vim reporting synthax violations

Categories: Puppet Tags: ,

Hiera + Puppet = winning combination

August 22, 2012 Leave a comment

Lean on me,
when you’re not strong
and I’ll be your friend
I’ll help you carry on
(Bill Withers)

I’m currently in the middle of a process of implementing puppet as configuration management, so every now and then I still get surprised at some nice things it has to offer. When writing puppet manifests for heterogenic environments, one has to assure that manifests can be reusable for different purposes. For example, not every host in our environment has same NTP servers, although configuration is pretty much the same. Writing two classes is out of question. So, one has to either use parametrized classes and inherit parameters or use variables and set them up in code. Both ways assume mixing code with data. As the code grows, so does the reconfiguration become an issue. How to find data, and not disturb the code? Welcome, Hiera!
Hiera is rubygem for implementing hierarchical data stores. What does it mean? It means that if queried value isn’t found in top level datastore, hiera will proceed deeper in the hierarchy. First answer will be accepted. This is ideal for puppet, so they have now been connected. As of 3.0 hiera will be a part of Puppet, but even if you use stable version (as I do) you can install hiera as an addon. I’ve built RPM’s of two rubygems, because I hate deploying software that’s not packaged. So first step to install hiera is (you have to have SRCE yum repos to make this work):

# yum install rubygem-hiera-puppet

which will also pull rubygem-hiera as a dependency. Next step is to link hiera’s directory to puppet master’s modulepath. To check which modulepath you use, run the following command:

# puppet master --configprint modulepath

In my case, it’s /etc/puppet/modules and /usr/share/puppet/modules. So:

# cd /usr/share/puppet/modules
# ln -s /usr/lib/ruby/gems/1.8/gems/hiera-puppet-0.3.0 hiera-puppet

Now we need to tell puppet to create libraries in /var/lib/puppet:

# puppet agent -t --pluginsync

This will produce some errors if you don’t manage your puppet master machine with puppet, but will create files that are needed for Hiera to work.
Now, let’s make a datastore. First, create /etc/puppet/hiera.yaml, that looks something like this:

---
:hierarchy:
    - %{hostname}
    - %{operatingsystem}
    - common
:backends:
    - yaml
:yaml:
    :datadir: '/etc/puppet/hieradata'

Also don’t forget to create /etc/puppet/hieradata directory. This configuration says to hiera that data is stored in yaml, and in three levels of hierarchy. First level is per hostname, second is per operating system and third is common. So you will have the following tree:

# find /etc/puppet/hieradata
/etc/puppet/hieradata
/etc/puppet/hieradata/myhost1.yaml
/etc/puppet/hieradata/myhost2.yaml
/etc/puppet/hieradata/CentOS.yaml
/etc/puppet/hieradata/Debian.yaml
/etc/puppet/hieradata/common.yaml

This is the example of common.yaml:

---
ntpserverip: 10.200.211.5

If I define ntpserverip in some of the top layers (OS or hostname), then that value will be used and not the value from common.yaml. So common is in this case a default which will be used is any other value isn’t defined.
And that’s it, to use it in your manifests, simply assign the value of hiera’s function output to some variable:

$ntpserver = hiera('ntpserverip')

and that’s it! You can lean on Hiera 😉

Categories: Puppet Tags: , ,
%d bloggers like this: