Archive for the ‘Cobbler’ Category

XenServer PV guests and Cobbler

May 23, 2013 Leave a comment

Far, far beyond the island
We dwelt in shades of twilight
Through dread and weary days
Through grief and endless pain

(Blind Guardian – Mirror, mirror)

After automatizing deployment of machines, one starts to really hate manual installs ūüôā They are long, repetitive task that bores one to hell… One thing we didn’t automatize yet were installations of Citrix XenServer paravirtualized guests. After searching through Citrix forums, we found a neat solution.

First step is to configure new system in Cobbler. I’m using my own cobbler-puppet module ( for managing cobbler, so I’ll post the puppet code instead of classic Cobbler CLI for adding new system:

   cobblersystem { '':
    ensure     => present,
    profile    => 'CentOS-6.4-x86_64-xen',
    interfaces => {
      'eth0' => {
        ip_address  => '',
        netmask     => '',
        mac_address => 'EE:B8:80:CB:B3:04',
        static      => true,
        management  => true,
    kernel_options => {
      clocksource  => 'acpi_pm',
      text         => '~',
      kssendmac    => '~',
    gateway    => '',
    hostname   => '',

As you can notice, we added “clocksource=acpi_pm” kernel parameter, so that kernel at installation time uses acpi_pm as source for it’s clock. This parameter is needed for the VM to boot properly in Rescue mode. So, how does a kickstart file look like?¬† Important parts are about partitioning:

# System bootloader configuration
bootloader --location=mbr --driveorder=xvda --append="console=hvc0"
# Clear the Master Boot Record
# Partition clearing information
clearpart --drives=xvda --all
# Disk partitioning information
part /boot --ondisk=xvda --asprimary --fstype="ext2" --size=100
part swap  --ondisk=xvda --asprimary --fstype="swap" --size=4096
part /     --ondisk=xvda --asprimary --fstype="ext3" --size=20480
part /data --ondisk=xvda --asprimary --fstype="ext4" --size=4096 --grow

Notice that we changed classic ‘sda’ with ‘xvda’, because it’s the block device name in XenServer guests, and added “console=hvc0” parameter to kernel cmdline in “bootloader” section.

After we set up Cobbler, we can advance to creating new virtual machine in XenServer. In this example I choose CentOS 6.0 template. The most important thing is not to boot the VM after creation:

XenServer PV VM creation wizzard

I will emphasize it once more: it’s important to turn off the “Start the new VM automatically” option. Also, to ensure the boot from PXE, you must leave the virtual DVD drive empty. Now, there is one more issue I’ve stumbled upon. For some reason, if the first boot of the VM is done in Rescue Mode, vbd-param “bootable” is false. So, neat trick I use to overcome this issue is to boot the VM in normal mode, and power it off afterwards, and then boot it with empty DVD drive into Rescue Mode, which will use PXE…. And that’s it! Cobbler will install the VM, and after installation is complete, VM will reboot into PV mode.

If the VM refuses to boot after PXE/kickstart installation is done, you have to use little XenServer magic to set the bootable flag to true:

# xe vm-disk-list uuid=<our_new_shiny_vm_uuid>
# xe vbd-param-set uuid=<vbd_uuid_from_previous_command> bootable=true

Although this is not fully automatized solution which can spawn VM’s on it’s own, it surely helps to advance to fully managed environment.


Puppet module for managing Cobbler

December 29, 2012 3 comments

I was working on a Puppet module for managing Cobbler last few months. It’s my first module with custom types, so I was on a steep learning curve. We’ve been using it in production for few weeks, but I felt it wasn’t as good as it should have been. Puppet agent runs were very slow (longer than a minute) because custom providers did several command line queries to get the current state of the system from Cobbler. These were all ordinary strings and providers had to cut out relevant information and store it into organized data formats like hashes. So, I’ve decided to rewrite the providers to use XMLRPC interface Cobbler offers and to fetch the state of the whole system with one single XMLRPC query. See the results of rewrite yourself:

Site1 (22 x cobblersystem)
- agent run before: 68.1s
- agent run after:   9.7s

Site2 (19 x cobblersystem)
- agent run before: 54.0s
- agent run after:  10.0s

Site3: (11 x cobblersystem)
- agent run before: 40.9s
- agent run after:   9.7s

Site4: (22 x cobblersystem)
- agent run before: 80.1s
- agent run after:  10.6s

As you see, cobbler custom providers did really big impact on agent run times before I rewrote them to use ‘self.instances’ function. So finally I was happy with the module. Now I can proudly publish it ūüôā Puppet module is available at Puppet Forge:

Code is available at BitBucket, as is the issue tracker:

Hope you like it!

Categories: Cobbler, Puppet

Problems with grub and (U)EFI

October 16, 2012 5 comments

You keep this love, thing, child, toy
You keep this love, fist, scar, break
You keep this love
(Pantera – This love)

I really hate those unproductive hours (hopefully not days) when one needs to debug some strange problems whose solution won’t be reusable. All of us have some of those. Although IBM delivers some fine hardware, sometimes strange problems arise. So this was one of those days…

We use Cobbler + PXE for unattended CentOS installations, and didn’t have problems on many different machines. But the day arrived ūüôā There were two kind of servers at our disposal: x3550 M4 1U and x3630M4 2U with attached DAS. First equipped with 2x900GB SAS drives and second with bunch of disks ūüôā x3550M4 were set up in RAID0 (striping) configuration, and with stock BIOS settings PXE installation went like a charm. On the other hand, two of the system drives (another pair of 900GB SAS) on x3630M4 were set up as mirror, and installation went smoothly but grub refused to start. Only output we had was a blinking cursor on the top left corner of empty black screen. And it’s kinda hard to google for solution of that error ūüėČ

CentOS 6 uses old Grub 0.97, but that was not the issue. Booting livecd, manually forcing grub installs, installing on MBR of other block devices present on the system – nothing worked…. After two days of pain, solution finally came around the block… Two simple steps we’re needed to fix it forever:

  • enter bios
  • Load Default Settings
  • System Settings => Legacy support
  • disable BBS Boot
  • back to main menu (ESC)
  • Save Settings
  • Exit Setup

We’ve tried another dozens of combinations but final solution was to 1) load defaults and 2) disable BBS Boot. We haven’t tried booting from GPT partition table, so maybe that would work too. But why bother if virtual block device is 900GB – msdos partition table would sufice. So if anyone comes across of this kind of problem: grub not loading from msdos partition table on a system that also has GPT patition tables on another set of block devices, with or without EFI/UEFI setting in BIOS => hope this blog entry saves you some time by disabling Bunch of Bull Shit BIOS option.

As someone over at Pantera’s video wrote:

You know the world sucks when you search “This Love” and the results are maroon 5

Categories: Cobbler, Linux Tags:
%d bloggers like this: