Archive

Archive for August, 2012

Vim and Puppet

August 25, 2012 Leave a comment

The dreamer and the wine
Poet without a rhyme
A widow writer torn apart by chains of Hell
(Nightwish – The poet and the pendulum)

One nice thing about vim is ability to extend it with additional plugins. So I’ve stumbled across nice plugin for syntax and code styling checking tools. As I know I’ll be doing lots of my time editing puppet code, I guess I really need this.
For this magic to work, workstation has to have some tools installed:

# yum install puppet rubygem-puppet-lint

Now let’s do some vim magic:

$ mkdir -p ~/.vim/bundle ~/.vim/autoload
$ cd /tmp
$ git clone https://github.com/tpope/vim-pathogen
$ mv vim-pathogen/autoload/pathogen.vim ~/.vim/autoload/
$ cd ~/.vim/bundle
$ git clone https://github.com/scrooloose/syntastic.git

Also you have to have this lines in your .vimrc:

call pathogen#infect()
filetype indent on

And that’s about it!

Now, when you make a code style error for example, vim will detect it and inform you of it!

Vim reporting synthax violations

Categories: Puppet Tags: ,

Hiera + Puppet = winning combination

August 22, 2012 Leave a comment

Lean on me,
when you’re not strong
and I’ll be your friend
I’ll help you carry on
(Bill Withers)

I’m currently in the middle of a process of implementing puppet as configuration management, so every now and then I still get surprised at some nice things it has to offer. When writing puppet manifests for heterogenic environments, one has to assure that manifests can be reusable for different purposes. For example, not every host in our environment has same NTP servers, although configuration is pretty much the same. Writing two classes is out of question. So, one has to either use parametrized classes and inherit parameters or use variables and set them up in code. Both ways assume mixing code with data. As the code grows, so does the reconfiguration become an issue. How to find data, and not disturb the code? Welcome, Hiera!
Hiera is rubygem for implementing hierarchical data stores. What does it mean? It means that if queried value isn’t found in top level datastore, hiera will proceed deeper in the hierarchy. First answer will be accepted. This is ideal for puppet, so they have now been connected. As of 3.0 hiera will be a part of Puppet, but even if you use stable version (as I do) you can install hiera as an addon. I’ve built RPM’s of two rubygems, because I hate deploying software that’s not packaged. So first step to install hiera is (you have to have SRCE yum repos to make this work):

# yum install rubygem-hiera-puppet

which will also pull rubygem-hiera as a dependency. Next step is to link hiera’s directory to puppet master’s modulepath. To check which modulepath you use, run the following command:

# puppet master --configprint modulepath

In my case, it’s /etc/puppet/modules and /usr/share/puppet/modules. So:

# cd /usr/share/puppet/modules
# ln -s /usr/lib/ruby/gems/1.8/gems/hiera-puppet-0.3.0 hiera-puppet

Now we need to tell puppet to create libraries in /var/lib/puppet:

# puppet agent -t --pluginsync

This will produce some errors if you don’t manage your puppet master machine with puppet, but will create files that are needed for Hiera to work.
Now, let’s make a datastore. First, create /etc/puppet/hiera.yaml, that looks something like this:

---
:hierarchy:
    - %{hostname}
    - %{operatingsystem}
    - common
:backends:
    - yaml
:yaml:
    :datadir: '/etc/puppet/hieradata'

Also don’t forget to create /etc/puppet/hieradata directory. This configuration says to hiera that data is stored in yaml, and in three levels of hierarchy. First level is per hostname, second is per operating system and third is common. So you will have the following tree:

# find /etc/puppet/hieradata
/etc/puppet/hieradata
/etc/puppet/hieradata/myhost1.yaml
/etc/puppet/hieradata/myhost2.yaml
/etc/puppet/hieradata/CentOS.yaml
/etc/puppet/hieradata/Debian.yaml
/etc/puppet/hieradata/common.yaml

This is the example of common.yaml:

---
ntpserverip: 10.200.211.5

If I define ntpserverip in some of the top layers (OS or hostname), then that value will be used and not the value from common.yaml. So common is in this case a default which will be used is any other value isn’t defined.
And that’s it, to use it in your manifests, simply assign the value of hiera’s function output to some variable:

$ntpserver = hiera('ntpserverip')

and that’s it! You can lean on Hiera 😉

Categories: Puppet Tags: , ,

Using mercurial as SVN client

August 17, 2012 2 comments

Somewhere over the rainbow
Skies are blue,
And the dreams that you dare to dream
Really do come true.
(Harold Arlen and E.Y. Harburg)

There are two things I can’t stand with SVN. The whole unnecessary forest of .svn sub-directories scattered all around my working copy. Second is the propset way of ignoring files. Can I add third? 🙂 It’s the manual svn add/remove for every file I create/delete…  Ay my current company (srce.hr) we use SVN as central VCS. I use it for RPM packages and puppet modules/manifests.
Now, if you want to distribute same working copy on more than one location, and be able to pull and push changes, all those .svn directories are really clumsy. So I decided to try HGsubversion extension. I already use Mercurial a lot (for example with etckeeper) and love its simplicity. So, on my workstation running CentOS 6, I first fetched hgsubversion extension:

% hg clone http://bitbucket.org/durin42/hgsubversion/ ~/.hgsubversion
 % echo "[extensions]" > ~/.hgrc
 % echo "hgsubversion = ~/.hgsubversion/hgsubversion" >> ~/.hgrc

How many of you actually read README’s? 🙂 I surely don’t until things break out. And so they broke…

% hg clone svn+https://svnserver/svn/repo
 destination directory: repo
 Auth realm:  SVN login
 Password for jsosic:
 [r1] jsosic: import modula
 [r2] jsosic: Redesigned.
 [r3] jsosic: something
 [r4] jsosic: something else
 pulled 4 revisions
 updating to branch default
 151 files updated, 0 files merged, 0 files removed, 0 files unresolved
 ** unknown exception encountered, details follow
 ** report bug details to http://mercurial.selenic.com/bts/
 ** or mercurial@selenic.com
 ** Mercurial Distributed SCM (version 1.4)
 ** Extensions loaded: hgsubversion
 Traceback (most recent call last):
 File "/usr/bin/hg", line 27, in
 mercurial.dispatch.run()
 File "/usr/lib64/python2.6/site-packages/mercurial/dispatch.py", line 16, in run
 sys.exit(dispatch(sys.argv[1:]))
 File "/usr/lib64/python2.6/site-packages/mercurial/dispatch.py", line 30, in dispatch
 return _runcatch(u, args)
 File "/usr/lib64/python2.6/site-packages/mercurial/dispatch.py", line 46, in _runcatch
 return _dispatch(ui, args)
 File "/usr/lib64/python2.6/site-packages/mercurial/dispatch.py", line 449, in _dispatch
 return runcommand(lui, repo, cmd, fullargs, ui, options, d)
 File "/usr/lib64/python2.6/site-packages/mercurial/dispatch.py", line 319, in runcommand
 ret = _runcommand(ui, options, cmd, d)
 File "/usr/lib64/python2.6/site-packages/mercurial/dispatch.py", line 500, in _runcommand
 return checkargs()
 File "/usr/lib64/python2.6/site-packages/mercurial/dispatch.py", line 454, in checkargs
 return cmdfunc()
 File "/usr/lib64/python2.6/site-packages/mercurial/dispatch.py", line 448, in
 d = lambda: util.checksignature(func)(ui, *args, **cmdoptions)
 File "/usr/lib64/python2.6/site-packages/mercurial/util.py", line 386, in check
 return func(*args, **kwargs)
 File "/usr/lib64/python2.6/site-packages/mercurial/extensions.py", line 116, in wrap
 util.checksignature(origfn), *args, **kwargs)
 File "/usr/lib64/python2.6/site-packages/mercurial/util.py", line 386, in check
 return func(*args, **kwargs)
 File "/home/jsosic/.hgsubversion/hgsubversion/wrappers.py", line 540, in clone
 fd = dst.opener("hgrc", "a", text=True)
 AttributeError: 'bool' object has no attribute 'opener'
 [1]    15118 exit 1     hg clone svn+https://svnserver/svn/repo

It seems that I didn’t have subvertpy installed, which is availabe in EPEL for RHEL/CentOS so…

# yum install python-subvertpy

After that – another error…

% hg clone svn+https://svnserver/svn/repo
 destination directory: repo
 ** unknown exception encountered, details follow
 ** report bug details to http://mercurial.selenic.com/bts/
 ** or mercurial@selenic.com
 ** Mercurial Distributed SCM (version 1.4)
 ** Extensions loaded: hgsubversion
 Traceback (most recent call last):
 File "/usr/bin/hg", line 27, in
 mercurial.dispatch.run()
 File "/usr/lib64/python2.6/site-packages/mercurial/dispatch.py", line 16, in run
 sys.exit(dispatch(sys.argv[1:]))
 File "/usr/lib64/python2.6/site-packages/mercurial/dispatch.py", line 30, in dispatch
 return _runcatch(u, args)
 File "/usr/lib64/python2.6/site-packages/mercurial/dispatch.py", line 46, in _runcatch
 return _dispatch(ui, args)
 File "/usr/lib64/python2.6/site-packages/mercurial/dispatch.py", line 449, in _dispatch
 return runcommand(lui, repo, cmd, fullargs, ui, options, d)
 File "/usr/lib64/python2.6/site-packages/mercurial/dispatch.py", line 319, in runcommand
 ret = _runcommand(ui, options, cmd, d)
 File "/usr/lib64/python2.6/site-packages/mercurial/dispatch.py", line 500, in _runcommand
 return checkargs()
 File "/usr/lib64/python2.6/site-packages/mercurial/dispatch.py", line 454, in checkargs
 return cmdfunc()
 File "/usr/lib64/python2.6/site-packages/mercurial/dispatch.py", line 448, in
 d = lambda: util.checksignature(func)(ui, *args, **cmdoptions)
 File "/usr/lib64/python2.6/site-packages/mercurial/util.py", line 386, in check
 return func(*args, **kwargs)
 File "/usr/lib64/python2.6/site-packages/mercurial/extensions.py", line 116, in wrap
 util.checksignature(origfn), *args, **kwargs)
 File "/usr/lib64/python2.6/site-packages/mercurial/util.py", line 386, in check
 return func(*args, **kwargs)
 File "/home/jsosic/.hgsubversion/hgsubversion/wrappers.py", line 528, in clone
 orig(ui, source, dest, **opts)
 File "/usr/lib64/python2.6/site-packages/mercurial/util.py", line 386, in check
 return func(*args, **kwargs)
 File "/usr/lib64/python2.6/site-packages/mercurial/commands.py", line 649, in clone
 update=opts.get('updaterev') or not opts.get('noupdate'))
 File "/usr/lib64/python2.6/site-packages/mercurial/extensions.py", line 128, in wrap
 return wrapper(origfn, *args, **kwargs)
 File "/home/jsosic/.hgsubversion/hgsubversion/wrappers.py", line 517, in hgclonewrapper
 data['srcrepo'], data['dstrepo'] = orig(ui, *args, **opts)
 File "/usr/lib64/python2.6/site-packages/mercurial/hg.py", line 292, in clone
 dest_repo.clone(src_repo, heads=revs, stream=stream)
 File "/usr/lib64/python2.6/site-packages/mercurial/localrepo.py", line 2144, in clone
 return self.pull(remote, heads)
 File "/home/jsosic/.hgsubversion/hgsubversion/svnrepo.py", line 81, in wrapper
 return fn(self, *args, **opts)
 File "/home/jsosic/.hgsubversion/hgsubversion/svnrepo.py", line 104, in pull
 return wrappers.pull(self, remote, heads, force)
 File "/home/jsosic/.hgsubversion/hgsubversion/wrappers.py", line 271, in pull
 svn_url = source.svnurl
 File "/home/jsosic/.hgsubversion/hgsubversion/svnrepo.py", line 154, in svnurl
 return self.svn.svn_url
 File "/usr/lib64/python2.6/site-packages/mercurial/util.py", line 150, in __get__
 result = self.func(obj)
 File "/home/jsosic/.hgsubversion/hgsubversion/svnrepo.py", line 159, in svn
 return svnwrap.SubversionRepo(*self.svnauth, password_stores=self.password_stores)
 File "/home/jsosic/.hgsubversion/hgsubversion/svnwrap/subvertpy_wrapper.py", line 179, in __init__
 self.init_ra_and_client()
 File "/home/jsosic/.hgsubversion/hgsubversion/svnwrap/subvertpy_wrapper.py", line 231, in init_ra_and_client
 auth=auth)
 File "/usr/lib64/python2.6/site-packages/subvertpy/ra.py", line 47, in RemoteAccess
 return url_handlers[type](url, *args, **kwargs)
 subvertpy.SubversionException: ("OPTIONS of 'https://svnserver/svn/repo': authorization failed:
 Could not authenticate to server: rejected Basic challenge (https://svnserver)", 170001)
 [1]    15247 exit 1     hg clone svn+https://svnserver/svn/repo

Now this second error is due to subvertpy not asking for authentication credentials… So what you have to do to fix this one is to store SVN password in either gnome-keyring, kwallet or simply plaintext… After that, HG works finaly 🙂

Edit: I’ve built hgsubversion RPM package and put it into SRCE rpm repositories.

Categories: Development Tags: , , ,

Multiple DHCP interfaces on CentOS

August 14, 2012 1 comment

I can feel you inside me
You’re not stronger than me
I won’t let you
No, you can’t have me!
(Iced Earth – Jekyll & Hyde)

Today I encountered interesting problem. I had one interface (eth0) in VirtualBox testing machine, which was bridged to host interface and being configured by dhclient (DHCP). Default route was being provided by DHCP server.

Now, I wanted to add another interface (eth1), with type NAT. NAT interface in VirtualBox gets it’s IP address by built-in DHCP server.

So I now had two interfaces – eth0 & eth1 both getting addresses from DHCP, and to make the matters worse – from different servers. Because of the linear nature of network init script in CentOS 5, eth0 starts up first, and eth1 second. Problem is that dhclient run for second interface overwrites default route set by the dhclient of the first interface.

One obvious solution was to rename interface eth0 to eth2, which would change the order of startup and in this case eth2 (ex eth0) interface would overwrite default route set by the eth1. But this is a hack, and I love to do things by the book. Advantage of doing things this way is that you can automate and script more predictably than if you hack too much with the system.

Fortunately, there is a way to do this a little smarter than brute-force renaming. Secret is GATEWAYDEV variable that can be set in /etc/sysconfig/network file. That variable says to network init script which interface should set up default gateway. So, all I had to do is:

# echo "GATEWAYDEV=eth0" >> /etc/sysconfig/network
# service network restart

And that’s it!

Accessing host filesystem from VirtualBox guest

August 13, 2012 9 comments

I came into this world
A screaming infant
(Iced Earth – Life and death)

VirtualBox is a greate piece of software for desktop and workstation virtualization. I use it for both testing and production environments – if it is applicable in the latter.

One good idea for VirtualBox is if you are web/app developer for Linux and you want to keep Windows as your primary OS. One can install Linux distribution of choice in VirtualBox VM and reap both benefits of Windows as primary os (gaaameeees) and Linux as a server platform.

So why am I blogging about something that seems to work? Devil is in the details… “it seems to work” is crucial.

There are 3 ways AFAIK of sharing files between Win host and Linux guest:

  • keep files in guest (Linux) and share with host through Samba
  • keep files on host (Win) and use VirtualBox Shared Folders
  • keep files on host (Win) and mount CIFS from within guest

For someone like me, an experienced Linux sysadmin, first step is the easiest. But it has it’s drawbacks too. For example – I don’t wanna keep files in VM, in case of corruption of VM, reinstallation, or whatever. I want them on native Windows disk. So first option is clearly out.

Shared Folders are really great idea. You set folder in settings of a VM, start it up, install VirtualBox Guest Additions (which bring mount.vboxsf binary and vboxsf kernel module), mount it in guest and voila! But there’s a catch – performance. Shared Folders are really really slow… which in turn kills the idea of programming and testing code without loosing hair because of the slowness.

Analyzing the performance, it’s obvious that kernel spends really large amount of time in IO mode – so vboxsf emulation is behaving something like ZFSonLinux or FUSE… And we don’t like that, do we 😦

It seems that only viable solution is sharing folder though Windows, and mounting it via CIFS….

I did some tests, to show the performance difference. There are three directories I do my tests on:

  • /var/www/repo            ( vboxsf )
  • /var/www/repo2          ( local disk )
  • /var/www/repo3          ( windows export )

I’m doing 2 tests – find and du -sh on a 500MB svn working copy. Also I drop all caches before every test. Here are the results of vboxsf mounted test:

# time find /var/www/repo
real 2m8.864s
user 0m0.200s
sys 0m14.638s

# time du -sh /var/www/repo
592M    /var/www/repo
real 0m24.951s
user 0m0.340s
sys 0m4.859s

Next, the folder mounted via CIFS (windows share/export):

# time find /var/www/repo3
real    0m23.844s
user    0m0.280s
sys     0m1.590s

# time du -sh /var/www/repo3
653M    /var/www/repo3
real    0m3.903s
user    0m0.060s
sys     0m0.330s

Notice the difference in size, not to mention the 10 fold increase in speed… And finaly as a reference, tests of files stored localy on ext3 partition in guest:

# time find /var/www/repo3
real	0m29.775s
user	0m0.180s
sys	0m2.500s

# time du -sh /var/www/repo2
719M	/var/www/repo2
real	0m15.032s
user	0m0.530s
sys	0m0.640s

Notice once more increase in size. Clearly, CIFS mounted Windows exports win… Only drawback – you cannot do symlinks on them.

%d bloggers like this: