Home > Linux, Storage > SRP tools problems

SRP tools problems

It’s like rain on your wedding day
It’s a free ride when you’ve already paid
It’s the good advice that you just didn’t take
Who would’ve thought… it figures

(Alanis Morissette – Ironic)

SRP stands for SCSI RDMA Protocol. If you still don’t know what it is – a protocol that allows you to connect to SCSI devices attached to other computer, over Remote Direct Memory Access. Remote-Direct? Yeah, oxymoron. So if you wish to use RDMA, underlying network has to support it. So far, the most common usage is on InfiniBand networks.
I had a chance to play with SRP target / SRP client connection, and from my impressions – whole SRP field is still – let’s put it this way – not tested enough. I wasn’t impressed with what I saw. Software isn’t mature yet… But, since I’ve already started, lets dive in.

My distro of choice for production environment is CentOS, so I’ll talk about implementations available on CentOS. So, if you have Infiniband adapter, how should you start?

# yum install rdma infiniband-diags mstflint qperf
# yum install librdmacm libmlx4 libmthca srptools opensm
# /etc/init.d/rdma start
# /etc/init.d/opensm start

As you can notice, I’m using mlx4 adapter. And now the fun starts.
I have the targets already set up on port 1 of two-port Mellanox adapter. Port 2 is not connected (even IB cables cost as hell), so I won’t be using multipathing. I just want to simply connect to the target and use the disk. There is ibsrpdm command that allows you to scan the network for available targets, so we can use it:

# ibsrpdm -c
id_ext=0002c9030051060c,ioc_guid=0002c9030051060c,\
dgid=fe800000000000000002c9030051060d,pkey=ffff,service_id=0002c9030051060c

The output of the command can be fed to your ‘/sys/class/infiniband_srp/srp-_0-/add_target‘, and client will connect to target and see all the disks available to him on that target.
Now, to automate this task for you, and to do periodic re-scans, you can use srp daemon. At least that documentation says.

First, let me describe the way daemon starts on CentOS.
After you start the init script, it calls another bash script as daemon, ‘/usr/sbin/srp_daemon.sh‘. This script tries to trap handles, manage log, and after that, goes through all available adapters and ports, and runs new bash script, named ‘run_srp_daemon‘, for each of them. And finally, ‘run_srp_daemon‘ actually runs ‘srp_daemon‘ binary.

First problem I’ve noticed is that ‘srp_daemon‘ is called for available interfaces, no matter if they are up or down. This results in these errors:

25/10/12 17:34:58 : No response to inform info registration
25/10/12 17:34:58 : Fail to register to traps, maybe there is no opensm running on fabric
25/10/12 17:34:58 : SM LID is 0, maybe no opensm is running

Next thing I decided was to grab and unpack Debian Sid package, to see if how they did it. I’ve found a piece of code that checks if the interfaces is actually up and running before starting the daemon. So I ported that code to RHEL script. So that fixed that problem, but now I was onto new issue. ‘srp_daemon‘ just didn’t add targets at all. If I run the following command:

for i in `/usr/sbin/ibsrpdm -c`; do \
 echo $i > /sys/class/infiniband_srp/srp-mlx4_0-1/add_target;\
done

all the block devices are added and registered correctly, and can be seen in ‘fdisk -l‘ output. So, for a first couple of days I was running this command in ‘/etc/rc.local‘ script, but eventually get pissed off and decided to debug this thing further. So, running ‘srp_daemon‘ in verbose mode (-V) shows following information that can be seen as useful:

enter do_port
Found an SRP target with id_ext 0002c90300510648 - check if it allowed by rules file
Found an SRP target with id_ext 0002c90300510648 - check if it is already connected
id_ext=0002c90300510648,ioc_guid=0002c90300510648,dgid=fe800000000000000002c90300510649,pkey=ffff,service_id=0002c90300510648
Adding target returned 156

Now, this returned 156 sounds like something didn’t go quite right. So, I’ve fetched source code of srptools, and applied this patch:

--- srptools-0.0.4/srp_daemon/srp_daemon.c	2009-08-30 15:56:11.000000000 +0200
+++ srp_daemon.c	2013-01-12 03:11:09.000000000 +0100
@@ -183,6 +183,7 @@
 		}
 		ret = write(fd, target_str, strlen(target_str));
 		pr_debug("Adding target returned %d\n", ret);
+		pr_debug("Target string: %s\n", target_str);
 		close(fd);
 	}
 }

This actually lead me to see that srp_daemon is trying to add target with initiator_ext=… So, my solution was to erase ‘-n’ option. Finally, the patch looks like this:

--- srp_daemon.sh	2013-01-12 02:17:37.000000000 +0100
+++ srp_daemon.sh_patched	2013-01-12 02:17:52.000000000 +0100
@@ -108,8 +108,14 @@
 do
     for port in `/bin/ls -1 ${ibdir}/${hca_id}/ports/`
     do
-        ${prog} -e -c -n -i ${hca_id} -p ${port} -R ${retries} ${params}&
-        pids="$pids $!"
+        STATUS=`/usr/sbin/ibstat $hca_id $port | grep "State:"`
+        if [ "$STATUS" = "State: Active" ] ; then
+            ${prog} -e -c -i ${hca_id} -p ${port} -R ${retries} ${params}&
+            pids="$pids $!"
+        fi 
     done
 done

Now, the script was starting ‘srp_daemon‘ only if interface was up, and targets were added correctly. Logs were finally quiet.

But, next problem came after the reboot. Daemon didn’t start at all, although it was in the correct runlevel. After some more reboots of machine and few dozen restarts of services I’ve noticed that the issue was with ‘opensm‘ slow warmup… Now, init script of Subnet Manager has a ‘sleep‘ command after the start procedure, but it sleeps only for a second. Increasing that to 18 seconds did the trick. Yeah, I know, pretty much! But servers these days tend to do a lot of self-checks at each startup, so some additional seconds to allow ‘srp_daemon‘ to start sanely isn’t all that much. I’ve opened a bug report: https://bugzilla.redhat.com/show_bug.cgi?id=894546 , and here’s my patch:

--- opensm	2013-01-12 02:40:35.000000000 +0100
+++ opensm_patched	2013-01-12 03:04:52.000000000 +0100
@@ -61,7 +61,7 @@
     else
         $prog -B $prio >/dev/null 2>&1
     fi
-    sleep 1
+    sleep 20
     OSM_PID=`pidof $prog`
     checkpid $OSM_PID
     RC=$?

I will write about SCST target daemon in another post so stay tuned 😉

Advertisements
  1. No comments yet.
  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: