Archive for September, 2013

Is Not A Child Of A Service

September 8, 2013 1 comment

The mirror in your eyes (the mirror in your eyes)
Is telling me what I’ve become alive and that is why
There’s pain inside your eyes (the mirror never lies)
The mirror in your eyes
(Rage – The Mirror in Your Eyes)

Developing a resource agents for RedHat Cluster Suite is always a fun ride. Resource agents are pretty simple and I’m really amused with the fact that RedHat distributes only a handful of finished resource agents… So, I’m thinking about developing a whole set of them and reporting it back to RedHat (or Fedora) so that they can get included in official distro.

Today, I decided to make an agent for rsyslog. We are making highly available rsyslog instance, so why duplicate init script when I can spare some time and develop something that will be useful to general public. (Well, not so general after all – but RHCS/Pacemaker involved public at least).

So, after I polish the shell script, I encountered a first obstacle:

Sep  8 03:45:54 testhost rgmanager[22364]: [rsyslog]
   Service rsyslog:testresource Is Not A Child Of A Service
Sep  8 03:46:11 testhost rgmanager[23126]: [rsyslog]
   Service rsyslog:testresource Is Not A Child Of A Service

So, let the fun begin. After almost two hours of diff-ing the script versus other resource agents like apache, named, samba, running all the possible RHCS debug/verbose commands that I know, and meddling with cluster.conf options and positioning of the <rsyslog> resource, and offcourse googling for a solution – eureka!!!! I had the similar issue with pgsql-9.1 resource agent few months ago… A-HAAA, so I take a peak at cluster.rng file, and immediately I can see where’s the problem… I did not have a service_name parameter in my rsyslog.metadata! I added the following block:

<parameter name="service_name" inherit="service%name">
    <longdesc lang="en">
        Inherit the service name.  We need to know
        the service name in order to determine file
        systems and IPs for this smb service.
    <shortdesc lang="en">
        Inherit the service name.
    <content type="string"/>

and problem solved! So for the benefit of (probably non-existing) resource agent developers, hope you find this solution googling around 🙂 If you do, please post a comment so that I know this particular post really helped someone 🙂 I would be glad….

Load balancing app that’s opening random local ports

September 6, 2013 2 comments

She started crying like a baby
Screamed for her mother, cursed the day of birth
But the reaper took her hand
(Rage – Innocent)

Moodle VPL, or Virtual Prorgramming Lab is a moodle addon created for managing programming assigments inside moodle. Students can edit and execute programs, while professors can enable the automatic and continuous assessment.

So, what is so special about this piece of (crap) software? The way it works.

Client runs Java applet in his browser. Java applet talks to a web server via standard http protocol (tcp/80). Applet is typical text editor with functions like save, run, etc. Purpose is to write code and execute it in a sandbox, so that students can concentrate on task itself and not on bringing up development environment for the task. Very noble idea indeed. For very lazy students too. I’m not even going to try to go down the path why is that idea broken in so many fundamental ways….

So, what happens when student runs code from that applet? Applet sends instruction to a web server via http, web server (php) opens listening socket, and returns the port number back to Java applet. Java applet then uses getDocumentBase().getHost() function to get the $SERVER_NAME, and then finally connects to <SERVER_NAME>:<PORT>.

Now, everything works just fine if you have only one web server with public IP address. All you have to do is open the predefined port range in firewall of that single machine.


Things get little messy when there’s a load balancing going on between client and application server. First problem that occurs is that client cannot connect to the port that PHP opened, because port is located on the backend web server while client tries to connect to that port on the load balancer. Simple solution port forwarding:

balancer # iptables -t nat -A PREROUTING \
           -p tcp -m multiport --dports 62000:62299 \
           -j DNAT --to-destination <web_node01>

Problem solved. TCP/SYN packets from client pass through balancer, and are forwarded to application server (web_node01), but new problem occurs. Web node responds to the client, but since client didn’t initiate connection to the IP of the web node but to the IP of the load balancer, client doesn’t recognize the answer. Packet has SRC header set to web node’s IP and not blancer’s IP. So packet ends dropped on the client, even if it reaches it.

This problems is little more difficult to solve, but since we already know port range that application uses, we can manage it. Welcome to port based routing

First we need to create new routing table:

web_node # cat '200   moodle_vpl' >> /etc/iproute2/rt_table

After creating table, lets add load balancer as a default gateway in that table:

web_node # ip route add default via <balancer_ip> table moodle_vpl

Now, let’s mark packets we want to route

web_node # iptables -t mangle -A OUTPUT \
           -p tcp -m multiport --sports 62000:62199 \
           -j MARK --set-mark 0x99

This iptables rule basicly sets mark 0x99 on every package that has source port in the range between 62000 to 62199. Finally tell kernel to route marked packages via table moodle_vpl:

web_node # ip rule add fwmark 0x99 lookup moodle_vpl

Now, if ip_forward is enabled on balancer, everything should work. This is how the network traffic scheme looks like now:


Please don’t forget to use different range of ports for every other server in the load balancing farm.

Now, isn’t this setup fun? Sigh…

%d bloggers like this: