Archive for July, 2013

DRAC6 fencing through IPMI

July 17, 2013 Leave a comment

So understand
Don’t waste your time always searching for those wasted years,
Face up… make your stand,
And realise you’re living in the golden years.
(Iron Maiden – Wasted Years)

RedHat Cluster Suite uses fencing as a safety measure for split brain scenario. Split brain is a situation when a cluster creates two or more partitions. Each partition thinks it is the only active and tries to start services. Now, the most benign situation that can occur in this scenario is IP conflict. Another, much worse situation is if two nodes try to write to same (shared) filesystem. Data corruption is obviously much worse then IP conflict 😉

To circumvent these situation, some kind of eviction of nodes has to occur. RedHat Cluster Suite leans on fencing mechanism. Fencing usually does some of the following:

  • node reboot (via mgmt console)
  • node power off (via PDU)
  • SCSI reservation

The most often used method is reboot via IPMI or APC PDU’s. In my case, I often use IPMI because APC PDU’s are quite expensive and are rarely available. RPM package ‘fence-agents’ offers many fencing mechanisms. Old Dell DRAC5 has it’s own fencing agent, but new, DRAC6 which is available on machines like R610/R620 or R710/R720 isn’t covered by fence agents. But there is a way to fence through DRAC 6 – via IPMI.

To enable IPMI fencing, few things have to be set in DRAC web GUI.

First, IPMI has to be enabled. To enable IPMI, you choose “iDRAC settings” from the menu on the left side of the screen, and then choose “Network/Security => Network” in top menu. At the bottom of that page, you can find settings section for IPMI. Set it up like it’s shown in the image and apply the settings.

DRAC6 IPMI settings

DRAC6 IPMI settings

Now all you have to do is to create a user with IPMI privileges. You can use ‘root’, but I strongly advise against. This user/password combo is blanktext in cluster.conf, so if one of your cluster nodes is compromised, attacker will find DRAC passwords for all the cluster members.

So, to create user, choose the “Users” subtab under “Network/Security” tab. Choose one of the free numbers, and after the wizzard stars choose “Configure User”. In the user configuration window, I recommend following settings:

  • Enable User: ON
  • User name: fencer
  • Maximum LAN User Privilege Granted: Administrator
  • Maximum Serial Port User Privilege Granted: Non
  • Leave all the iDRAC user privileges turned off

After you apply the settings, it’s time to test if they work. Login to some RHEL/CentOS machine and install fence agents:

# yum -y install fence-agents

Now, try running fence_ipmi agent:

# fence_ipmilan -P -a <drac_IP> -l fencer -p <password> -o status -v
Getting status of IPMI:<drac_IP_address>...Spawning:
  '/usr/bin/ipmitool -I lanplus -H '<drac_IP_address>'
   -U 'fencer' -P '[set]' -v chassis power status'...
Chassis power = On

If you get the OK output, and not the “Unknown”, you’re all set! You can also test hard reboot, if you wish:

# fence_ipmilan -P -a <drac_IP> -l fencer -p <password> -o reboot -v
Rebooting machine @ IPMI:<drac_IP>...
Spawning: 'ipmitool .... -v chassis power status'...
Spawning: 'ipmitool .... -v chassis power off'...
Spawning: 'ipmitool .... -v chassis power status'...
Spawning: 'ipmitool .... -v chassis power off'...
Spawning: 'ipmitool .... -v chassis power status'...
Spawning: 'ipmitool .... -v chassis power off'...
Spawning: 'ipmitool .... -v chassis power status'...
Spawning: 'ipmitool .... -v chassis power off'...
Spawning: 'ipmitool .... -v chassis power status'...
Spawning: 'ipmitool .... -v chassis power status'...
Spawning: 'ipmitool .... -v chassis power on'...
Spawning: 'ipmitool .... -v chassis power status'...

Congratulations, your server has just been rebooted! 🙂
Now, to use fence_ipmi with lanplus in your cluster.conf, you set up your fence device along these lines:

<fencedevice agent="fence_ipmilan" name="drac_fqdn" ipaddr="drac_IP" 
login="fencer" passwd="pass" lanplus="1"/>

And that’s it.

Categories: Linux, RedHat
%d bloggers like this: