Archive for December, 2012

Puppet module for managing Cobbler

December 29, 2012 3 comments

I was working on a Puppet module for managing Cobbler last few months. It’s my first module with custom types, so I was on a steep learning curve. We’ve been using it in production for few weeks, but I felt it wasn’t as good as it should have been. Puppet agent runs were very slow (longer than a minute) because custom providers did several command line queries to get the current state of the system from Cobbler. These were all ordinary strings and providers had to cut out relevant information and store it into organized data formats like hashes. So, I’ve decided to rewrite the providers to use XMLRPC interface Cobbler offers and to fetch the state of the whole system with one single XMLRPC query. See the results of rewrite yourself:

Site1 (22 x cobblersystem)
- agent run before: 68.1s
- agent run after:   9.7s

Site2 (19 x cobblersystem)
- agent run before: 54.0s
- agent run after:  10.0s

Site3: (11 x cobblersystem)
- agent run before: 40.9s
- agent run after:   9.7s

Site4: (22 x cobblersystem)
- agent run before: 80.1s
- agent run after:  10.6s

As you see, cobbler custom providers did really big impact on agent run times before I rewrote them to use ‘self.instances’ function. So finally I was happy with the module. Now I can proudly publish it 🙂 Puppet module is available at Puppet Forge:

Code is available at BitBucket, as is the issue tracker:

Hope you like it!

Categories: Cobbler, Puppet

Precommit hook for Puppet code

December 13, 2012 1 comment

Please God, give me the strength to lead
It’s my charge to keep
Make the sorrow the fuel I need
It’s my charge to keep

(Iced Earth – A charge to keep)

As my colleagues slowly pick up the pace with Puppet modules development I’ve charged, we stumbled upon a problem experienced developers see so often. Code that was pushed to our repository started to get messy. Syntax errors were rare, but the coding style standard went berzerk with the first devop that joined the puppet “team”. My initial reaction was to re-factor the code every now and then, and by re-factor I mean running checks on my own working copy. And that started to consume too much of my time… rewriting modules, committing, checking if they they work as intended after my changes… and finally I was pissed… Being obsessive compulsive may seem counter-productive at times but some rules just have to be obeyed.
So, instead of continuing to fix other’s mistakes, I’ve decided to put a stop on them. This is what I was doing on our modules repository:

for i in $(find ~/puppet -regex ".*pp$"); do
  echo $i;
  puppet parser validate $i;
  if [[ ! $? -eq 0 ]]; then
    echo"* puppet syntax errors found!"
  puppet-lint --no-80chars-check --fail-on-warnings $i
  if [[ ! $? -eq 0 ]]; then
    echo "* puppet code style broken!"

This would spit out any syntax error or code style divergence. So, my idea now was to offload that to people writing code. So, I decided to make a precommit hook. We use mercurial, so my .hg/hgrc looks like this:

default =

pretxncommit.puppet = .hg/check_puppet.rb

And finally, check_puppet.rb script (which has to be set executable):


def puppet_parser_validate(file)
    if !system('puppet parser validate ' + file + ' > /dev/null 2>&1')
    print('!!! Syntax error in file: ' + file + "\n")
        system('puppet parser validate ' + file)

def puppet_lint(file)
    if !system('puppet-lint --no-80chars-check --fail-on-warnings ' + file + ' > /dev/null 2>&1')
    print('!!! Coding style error in file: ' + file + "\n")
        system('puppet-lint --no-80chars-check --fail-on-warnings ' + file)

def puppet_erb_check(file)
    if !system('erb -x -T \'-\' ' + file + ' | ruby -c > /dev/null 2>&1')
    print('!!! Syntax error in erb template: ' + file + "\n")
        system('erb -x -T \'-\' ' + file + ' | ruby -c')

def puppet_ruby_check(file)
    if !system('ruby -c ' + file + ' > /dev/null 2>&1')
    print('!!! Syntax error in erb template: ' + file + "\n")
        system('ruby -c ' + file)

# go through list of files, and call adequate checks
IO.popen('hg status | grep -v ^R').readlines.each { |file| 
    file.sub!(/^[\w|\?] (.*)\n/,'\1')
    if file.match('.pp$')
        puppet_parser_validate file
        puppet_lint file
    elsif file.match('.erb$')
    puppet_erb_check file
    elsif file.match('.rb$')
    puppet_ruby_check file

Script checks every file that occurs in the output of hg status. Some may not be happy with that solution, so if you want to skip files that are not yet added to your repository, you could change grep -v ^R to egrep -v “^(p|\?)”. If you want to run this script, the machine has to have puppet and rubygem puppet-lint installed. You can fetch the latter from SRCE repositories if you use RHEL/CentOS:

Now, next problem is how to force every member of a team to adhere to precommit and not turn it off? Well, I haven’t come to that part yet 🙂 But social engineering seems to work so far. So, if the people understand the benefit of such approach, they will in most cases respect it and use it. Otherwise I’ll be hanging at their necks 🙂

Note that these are just basic tests, in the long run I will try to implement some kind of CI solution that runs complex logical tests of the codebase, because with every new commit there comes dozen of new problems 🙂

Categories: Puppet Tags: , , ,

Array join in Puppet manifest

December 10, 2012 1 comment

Maybe some of you noticed that as of time of writing this blog – there is a split() function available in puppet declarative language, but there is no join()? Well that puzzled me too, because I wanted to join an array and serve it as a content of a file. Well, PuppetLabs made inline_template() available to us lazy users, so we can use all the erb code we want inside our manifests! So, how did I join my array? Here is nice example:

$routes = [' via', ' via'],
$fullroute = inline_template("<%= (routes).join('\n') + \"\n\" %>")

Now, I can feed $fullroute to some file as content and that’s it!

file { '/etc/sysconfig/network-scripts/route-eth0':
  ensure  => file,
  content => "$fullroute",

Pretty neat, isn’t it? 🙂

Categories: Puppet Tags: , ,

Managing ssh host keys with Puppet

December 4, 2012 Leave a comment

Death of mother Earth
Never a rebirth
Evolution’s end
Never will it mend

(Metallica – Blackened)

Do you get annoyed when after every clean installation of an existing node you have to clean up your known_hosts file, accept new host key? Especially if you manage development machines that are reinstalled often. As we are starting to use Puppet extensively, I’ve thought about solving this problem with it. And I came up with pretty neat solution that I hope will benefit the masses. I’ve created ssh module, which has ssh::server class that looks like this:

class ssh::server {
  if generate('/etc/puppet/modules/ssh/scripts/',
$keys_dir) {
    include ssh::server::keys

Generate function calls shell script, and if it’s exit status is 0, ssh::server class includes ssh::server::key class. The latter class defines the ssh host keys (as file resources), sets the correct permissions and reloads ssh service if any of the files/keys changes. This is an example of one file resource in that class:

file { '/etc/ssh/':
  ensure  => file,
  owner   => root,
  group   => root,
  mode    => '0644',
  source  => [
  require => Package['openssh-server'],
  notify  => Service[$service_name],

Private section of puppet server is set up in /etc/puppet/fileserver.conf. Beauty of this approach is that each agent knows only for the existence of directories under his own $fqdn, so one host can’t access other hosts sensitive data (like ssh host keys). Second line in “source” property array value is just as backend if, for some reason, first one is not found. Class ssh:server:keys is not designed to be included directly in node manifest, but to be used as ssh::server class’s helper, so thas should not happen. This is a snippet from fileserver.conf:

 path /etc/puppet/private/%H
 allow *

and this is the directory structure:

# find /etc/puppet/private/

As you can see, host2 has keys already generated and host1 doesn’t. And, now the magic glue behind all of this, script:


# check arg0: dir for keys
[ -z "$1" ] && echo "Please specify directory for key generation" && exit 1

# set umask
umask 0022

# create directory tree if it does not exist
[ ! -d "$KEYSDIR" ] && mkdir -p $KEYSDIR

# functions stolen from CentOS 6 sshd init script

# Some functions to make the below more readable

# source function library
. /etc/rc.d/init.d/functions

fips_enabled() {
  if [ -r /proc/sys/crypto/fips_enabled ]; then
    cat /proc/sys/crypto/fips_enabled
    echo 0

do_rsa1_keygen() {
  if [ ! -s $RSA1_KEY -a `fips_enabled` -eq 0 ]; then
    echo -n $"Generating SSH1 RSA host key: "
    rm -f $RSA1_KEY
    if test ! -f $RSA1_KEY && $KEYGEN -q -t rsa1 -f $RSA1_KEY -C '' -N '' >&/dev/null; then
      chmod 600 $RSA1_KEY
      chmod 644 $
      success $"RSA1 key generation"
      failure $"RSA1 key generation"
      exit 1

do_rsa_keygen() {
  if [ ! -s $RSA_KEY ]; then
    echo -n $"Generating SSH2 RSA host key: "
    rm -f $RSA_KEY
    if test ! -f $RSA_KEY && $KEYGEN -q -t rsa -f $RSA_KEY -C '' -N '' >&/dev/null; then
      chmod 600 $RSA_KEY
      chmod 644 $
      success $"RSA key generation"
      failure $"RSA key generation"
      exit 1

do_dsa_keygen() {
  if [ ! -s $DSA_KEY ]; then
    echo -n $"Generating SSH2 DSA host key: "
    rm -f $DSA_KEY
    if test ! -f $DSA_KEY && $KEYGEN -q -t dsa -f $DSA_KEY -C '' -N '' >&/dev/null; then
      chmod 600 $DSA_KEY
      chmod 644 $
      success $"DSA key generation"
      failure $"DSA key generation"
      exit 1

# main
chmod -R 644 $KEYSDIR/*
exit 0

Hope you like the solution, and I promise to publish the whole module on the forge when it’s polished enough!

Categories: Puppet, RedHat Tags: , , ,
%d bloggers like this: