Archive
Puppet module for managing Cobbler
I was working on a Puppet module for managing Cobbler last few months. It’s my first module with custom types, so I was on a steep learning curve. We’ve been using it in production for few weeks, but I felt it wasn’t as good as it should have been. Puppet agent runs were very slow (longer than a minute) because custom providers did several command line queries to get the current state of the system from Cobbler. These were all ordinary strings and providers had to cut out relevant information and store it into organized data formats like hashes. So, I’ve decided to rewrite the providers to use XMLRPC interface Cobbler offers and to fetch the state of the whole system with one single XMLRPC query. See the results of rewrite yourself:
Site1 (22 x cobblersystem) - agent run before: 68.1s - agent run after: 9.7s Site2 (19 x cobblersystem) - agent run before: 54.0s - agent run after: 10.0s Site3: (11 x cobblersystem) - agent run before: 40.9s - agent run after: 9.7s Site4: (22 x cobblersystem) - agent run before: 80.1s - agent run after: 10.6s
As you see, cobbler custom providers did really big impact on agent run times before I rewrote them to use ‘self.instances’ function. So finally I was happy with the module. Now I can proudly publish it 🙂 Puppet module is available at Puppet Forge:
http://forge.puppetlabs.com/jsosic/cobbler
Code is available at BitBucket, as is the issue tracker:
https://bitbucket.org/jsosic/puppet-cobbler
Hope you like it!
Precommit hook for Puppet code
Please God, give me the strength to lead
It’s my charge to keep
Make the sorrow the fuel I need
It’s my charge to keep
(Iced Earth – A charge to keep)
As my colleagues slowly pick up the pace with Puppet modules development I’ve charged, we stumbled upon a problem experienced developers see so often. Code that was pushed to our repository started to get messy. Syntax errors were rare, but the coding style standard went berzerk with the first devop that joined the puppet “team”. My initial reaction was to re-factor the code every now and then, and by re-factor I mean running checks on my own working copy. And that started to consume too much of my time… rewriting modules, committing, checking if they they work as intended after my changes… and finally I was pissed… Being obsessive compulsive may seem counter-productive at times but some rules just have to be obeyed.
So, instead of continuing to fix other’s mistakes, I’ve decided to put a stop on them. This is what I was doing on our modules repository:
for i in $(find ~/puppet -regex ".*pp$"); do
echo $i;
puppet parser validate $i;
if [[ ! $? -eq 0 ]]; then
echo"* puppet syntax errors found!"
break
fi
puppet-lint --no-80chars-check --fail-on-warnings $i
if [[ ! $? -eq 0 ]]; then
echo "* puppet code style broken!"
break
fi
done
This would spit out any syntax error or code style divergence. So, my idea now was to offload that to people writing code. So, I decided to make a precommit hook. We use mercurial, so my .hg/hgrc looks like this:
[paths] default = https://code.google.com/p/lutak/ [hooks] pretxncommit.puppet = .hg/check_puppet.rb
And finally, check_puppet.rb script (which has to be set executable):
#!/usr/bin/ruby
def puppet_parser_validate(file)
if !system('puppet parser validate ' + file + ' > /dev/null 2>&1')
print('!!! Syntax error in file: ' + file + "\n")
system('puppet parser validate ' + file)
exit(1)
end
end
def puppet_lint(file)
if !system('puppet-lint --no-80chars-check --fail-on-warnings ' + file + ' > /dev/null 2>&1')
print('!!! Coding style error in file: ' + file + "\n")
system('puppet-lint --no-80chars-check --fail-on-warnings ' + file)
exit(1)
end
end
def puppet_erb_check(file)
if !system('erb -x -T \'-\' ' + file + ' | ruby -c > /dev/null 2>&1')
print('!!! Syntax error in erb template: ' + file + "\n")
system('erb -x -T \'-\' ' + file + ' | ruby -c')
exit(1)
end
end
def puppet_ruby_check(file)
if !system('ruby -c ' + file + ' > /dev/null 2>&1')
print('!!! Syntax error in erb template: ' + file + "\n")
system('ruby -c ' + file)
exit(1)
end
end
# go through list of files, and call adequate checks
IO.popen('hg status | grep -v ^R').readlines.each { |file|
file.sub!(/^[\w|\?] (.*)\n/,'\1')
if file.match('.pp$')
puppet_parser_validate file
puppet_lint file
elsif file.match('.erb$')
puppet_erb_check file
elsif file.match('.rb$')
puppet_ruby_check file
end
}
Script checks every file that occurs in the output of hg status. Some may not be happy with that solution, so if you want to skip files that are not yet added to your repository, you could change grep -v ^R to egrep -v “^(p|\?)”. If you want to run this script, the machine has to have puppet and rubygem puppet-lint installed. You can fetch the latter from SRCE repositories if you use RHEL/CentOS:
ftp://ftp.srce.hr/redhat/base/el6/x86_64/rubygem-puppet-lint-0.3.2-1.el6.srce.noarch.rpm ftp://ftp.srce.hr/redhat/base/el5/x86_64/rubygem-puppet-lint-0.3.2-1.el5.srce.noarch.rpm
Now, next problem is how to force every member of a team to adhere to precommit and not turn it off? Well, I haven’t come to that part yet 🙂 But social engineering seems to work so far. So, if the people understand the benefit of such approach, they will in most cases respect it and use it. Otherwise I’ll be hanging at their necks 🙂
Note that these are just basic tests, in the long run I will try to implement some kind of CI solution that runs complex logical tests of the codebase, because with every new commit there comes dozen of new problems 🙂
Array join in Puppet manifest
Maybe some of you noticed that as of time of writing this blog – there is a split() function available in puppet declarative language, but there is no join()? Well that puzzled me too, because I wanted to join an array and serve it as a content of a file. Well, PuppetLabs made inline_template() available to us lazy users, so we can use all the erb code we want inside our manifests! So, how did I join my array? Here is nice example:
$routes = ['192.168.2.0/24 via 192.168.1.2', '192.168.3.0/24 via 192.168.1.3'],
$fullroute = inline_template("<%= (routes).join('\n') + \"\n\" %>")
Now, I can feed $fullroute to some file as content and that’s it!
file { '/etc/sysconfig/network-scripts/route-eth0':
ensure => file,
content => "$fullroute",
}
Pretty neat, isn’t it? 🙂
Managing ssh host keys with Puppet
Death of mother Earth
Never a rebirth
Evolution’s end
Never will it mend
(Metallica – Blackened)
Do you get annoyed when after every clean installation of an existing node you have to clean up your known_hosts file, accept new host key? Especially if you manage development machines that are reinstalled often. As we are starting to use Puppet extensively, I’ve thought about solving this problem with it. And I came up with pretty neat solution that I hope will benefit the masses. I’ve created ssh module, which has ssh::server class that looks like this:
class ssh::server {
...
if generate('/etc/puppet/modules/ssh/scripts/generate_host_keys.sh',
$keys_dir) {
include ssh::server::keys
}
...
}
Generate function calls shell script, and if it’s exit status is 0, ssh::server class includes ssh::server::key class. The latter class defines the ssh host keys (as file resources), sets the correct permissions and reloads ssh service if any of the files/keys changes. This is an example of one file resource in that class:
file { '/etc/ssh/ssh_host_rsa_key.pub':
ensure => file,
owner => root,
group => root,
mode => '0644',
source => [
'puppet:///private/ssh/ssh_host_rsa_key.pub',
'puppet:///modules/ssh/ssh_host_rsa_key.pub',
],
require => Package['openssh-server'],
notify => Service[$service_name],
}
Private section of puppet server is set up in /etc/puppet/fileserver.conf. Beauty of this approach is that each agent knows only for the existence of directories under his own $fqdn, so one host can’t access other hosts sensitive data (like ssh host keys). Second line in “source” property array value is just as backend if, for some reason, first one is not found. Class ssh:server:keys is not designed to be included directly in node manifest, but to be used as ssh::server class’s helper, so thas should not happen. This is a snippet from fileserver.conf:
[private] path /etc/puppet/private/%H allow *
and this is the directory structure:
# find /etc/puppet/private/ /etc/puppet/private/ /etc/puppet/private/host1.example.lan /etc/puppet/private/host2.example.lan /etc/puppet/private/host2.example.lan/ssh /etc/puppet/private/host2.example.lan/ssh/ssh_host_key.pub /etc/puppet/private/host2.example.lan/ssh/ssh_host_dsa_key /etc/puppet/private/host2.example.lan/ssh/ssh_host_dsa_key.pub /etc/puppet/private/host2.example.lan/ssh/ssh_host_key /etc/puppet/private/host2.example.lan/ssh/ssh_host_rsa_key /etc/puppet/private/host2.example.lan/ssh/ssh_host_rsa_key.pu
As you can see, host2 has keys already generated and host1 doesn’t. And, now the magic glue behind all of this, generate_host_keys.sh script:
#!/bin/bash
# check arg0: dir for keys
[ -z "$1" ] && echo "Please specify directory for key generation" && exit 1
KEYSDIR="$1"
# set umask
umask 0022
# create directory tree if it does not exist
[ ! -d "$KEYSDIR" ] && mkdir -p $KEYSDIR
#
# functions stolen from CentOS 6 sshd init script
#
# Some functions to make the below more readable
KEYGEN=/usr/bin/ssh-keygen
RSA1_KEY=$1/ssh_host_key
RSA_KEY=$1/ssh_host_rsa_key
DSA_KEY=$1/ssh_host_dsa_key
# source function library
. /etc/rc.d/init.d/functions
fips_enabled() {
if [ -r /proc/sys/crypto/fips_enabled ]; then
cat /proc/sys/crypto/fips_enabled
else
echo 0
fi
}
do_rsa1_keygen() {
if [ ! -s $RSA1_KEY -a `fips_enabled` -eq 0 ]; then
echo -n $"Generating SSH1 RSA host key: "
rm -f $RSA1_KEY
if test ! -f $RSA1_KEY && $KEYGEN -q -t rsa1 -f $RSA1_KEY -C '' -N '' >&/dev/null; then
chmod 600 $RSA1_KEY
chmod 644 $RSA1_KEY.pub
success $"RSA1 key generation"
echo
else
failure $"RSA1 key generation"
echo
exit 1
fi
fi
}
do_rsa_keygen() {
if [ ! -s $RSA_KEY ]; then
echo -n $"Generating SSH2 RSA host key: "
rm -f $RSA_KEY
if test ! -f $RSA_KEY && $KEYGEN -q -t rsa -f $RSA_KEY -C '' -N '' >&/dev/null; then
chmod 600 $RSA_KEY
chmod 644 $RSA_KEY.pub
success $"RSA key generation"
echo
else
failure $"RSA key generation"
echo
exit 1
fi
fi
}
do_dsa_keygen() {
if [ ! -s $DSA_KEY ]; then
echo -n $"Generating SSH2 DSA host key: "
rm -f $DSA_KEY
if test ! -f $DSA_KEY && $KEYGEN -q -t dsa -f $DSA_KEY -C '' -N '' >&/dev/null; then
chmod 600 $DSA_KEY
chmod 644 $DSA_KEY.pub
success $"DSA key generation"
echo
else
failure $"DSA key generation"
echo
exit 1
fi
fi
}
# main
do_rsa1_keygen
do_rsa_keygen
do_dsa_keygen
chmod -R 644 $KEYSDIR/*
exit 0
Hope you like the solution, and I promise to publish the whole module on the forge when it’s polished enough!
Vim and Puppet
The dreamer and the wine
Poet without a rhyme
A widow writer torn apart by chains of Hell
(Nightwish – The poet and the pendulum)
One nice thing about vim is ability to extend it with additional plugins. So I’ve stumbled across nice plugin for syntax and code styling checking tools. As I know I’ll be doing lots of my time editing puppet code, I guess I really need this.
For this magic to work, workstation has to have some tools installed:
# yum install puppet rubygem-puppet-lint
Now let’s do some vim magic:
$ mkdir -p ~/.vim/bundle ~/.vim/autoload $ cd /tmp $ git clone https://github.com/tpope/vim-pathogen $ mv vim-pathogen/autoload/pathogen.vim ~/.vim/autoload/ $ cd ~/.vim/bundle $ git clone https://github.com/scrooloose/syntastic.git
Also you have to have this lines in your .vimrc:
call pathogen#infect() filetype indent on
And that’s about it!
Now, when you make a code style error for example, vim will detect it and inform you of it!
Hiera + Puppet = winning combination
Lean on me,
when you’re not strong
and I’ll be your friend
I’ll help you carry on
(Bill Withers)
I’m currently in the middle of a process of implementing puppet as configuration management, so every now and then I still get surprised at some nice things it has to offer. When writing puppet manifests for heterogenic environments, one has to assure that manifests can be reusable for different purposes. For example, not every host in our environment has same NTP servers, although configuration is pretty much the same. Writing two classes is out of question. So, one has to either use parametrized classes and inherit parameters or use variables and set them up in code. Both ways assume mixing code with data. As the code grows, so does the reconfiguration become an issue. How to find data, and not disturb the code? Welcome, Hiera!
Hiera is rubygem for implementing hierarchical data stores. What does it mean? It means that if queried value isn’t found in top level datastore, hiera will proceed deeper in the hierarchy. First answer will be accepted. This is ideal for puppet, so they have now been connected. As of 3.0 hiera will be a part of Puppet, but even if you use stable version (as I do) you can install hiera as an addon. I’ve built RPM’s of two rubygems, because I hate deploying software that’s not packaged. So first step to install hiera is (you have to have SRCE yum repos to make this work):
# yum install rubygem-hiera-puppet
which will also pull rubygem-hiera as a dependency. Next step is to link hiera’s directory to puppet master’s modulepath. To check which modulepath you use, run the following command:
# puppet master --configprint modulepath
In my case, it’s /etc/puppet/modules and /usr/share/puppet/modules. So:
# cd /usr/share/puppet/modules # ln -s /usr/lib/ruby/gems/1.8/gems/hiera-puppet-0.3.0 hiera-puppet
Now we need to tell puppet to create libraries in /var/lib/puppet:
# puppet agent -t --pluginsync
This will produce some errors if you don’t manage your puppet master machine with puppet, but will create files that are needed for Hiera to work.
Now, let’s make a datastore. First, create /etc/puppet/hiera.yaml, that looks something like this:
---
:hierarchy:
- %{hostname}
- %{operatingsystem}
- common
:backends:
- yaml
:yaml:
:datadir: '/etc/puppet/hieradata'
Also don’t forget to create /etc/puppet/hieradata directory. This configuration says to hiera that data is stored in yaml, and in three levels of hierarchy. First level is per hostname, second is per operating system and third is common. So you will have the following tree:
# find /etc/puppet/hieradata /etc/puppet/hieradata /etc/puppet/hieradata/myhost1.yaml /etc/puppet/hieradata/myhost2.yaml /etc/puppet/hieradata/CentOS.yaml /etc/puppet/hieradata/Debian.yaml /etc/puppet/hieradata/common.yaml
This is the example of common.yaml:
--- ntpserverip: 10.200.211.5
If I define ntpserverip in some of the top layers (OS or hostname), then that value will be used and not the value from common.yaml. So common is in this case a default which will be used is any other value isn’t defined.
And that’s it, to use it in your manifests, simply assign the value of hiera’s function output to some variable:
$ntpserver = hiera('ntpserverip')
and that’s it! You can lean on Hiera 😉
It's all bullshit 