Home > Linux > Load balancing app that’s opening random local ports

Load balancing app that’s opening random local ports

She started crying like a baby
Screamed for her mother, cursed the day of birth
But the reaper took her hand
(Rage – Innocent)

Moodle VPL, or Virtual Prorgramming Lab is a moodle addon created for managing programming assigments inside moodle. Students can edit and execute programs, while professors can enable the automatic and continuous assessment.

So, what is so special about this piece of (crap) software? The way it works.

Client runs Java applet in his browser. Java applet talks to a web server via standard http protocol (tcp/80). Applet is typical text editor with functions like save, run, etc. Purpose is to write code and execute it in a sandbox, so that students can concentrate on task itself and not on bringing up development environment for the task. Very noble idea indeed. For very lazy students too. I’m not even going to try to go down the path why is that idea broken in so many fundamental ways….

So, what happens when student runs code from that applet? Applet sends instruction to a web server via http, web server (php) opens listening socket, and returns the port number back to Java applet. Java applet then uses getDocumentBase().getHost() function to get the $SERVER_NAME, and then finally connects to <SERVER_NAME>:<PORT>.

Now, everything works just fine if you have only one web server with public IP address. All you have to do is open the predefined port range in firewall of that single machine.

vpl_standard

Things get little messy when there’s a load balancing going on between client and application server. First problem that occurs is that client cannot connect to the port that PHP opened, because port is located on the backend web server while client tries to connect to that port on the load balancer. Simple solution port forwarding:

balancer # iptables -t nat -A PREROUTING \
           -p tcp -m multiport --dports 62000:62299 \
           -j DNAT --to-destination <web_node01>

Problem solved. TCP/SYN packets from client pass through balancer, and are forwarded to application server (web_node01), but new problem occurs. Web node responds to the client, but since client didn’t initiate connection to the IP of the web node but to the IP of the load balancer, client doesn’t recognize the answer. Packet has SRC header set to web node’s IP and not blancer’s IP. So packet ends dropped on the client, even if it reaches it.

This problems is little more difficult to solve, but since we already know port range that application uses, we can manage it. Welcome to port based routing

First we need to create new routing table:

web_node # cat '200   moodle_vpl' >> /etc/iproute2/rt_table

After creating table, lets add load balancer as a default gateway in that table:

web_node # ip route add default via <balancer_ip> table moodle_vpl

Now, let’s mark packets we want to route

web_node # iptables -t mangle -A OUTPUT \
           -p tcp -m multiport --sports 62000:62199 \
           -j MARK --set-mark 0x99

This iptables rule basicly sets mark 0x99 on every package that has source port in the range between 62000 to 62199. Finally tell kernel to route marked packages via table moodle_vpl:

web_node # ip rule add fwmark 0x99 lookup moodle_vpl

Now, if ip_forward is enabled on balancer, everything should work. This is how the network traffic scheme looks like now:

vpl_balanced

Please don’t forget to use different range of ports for every other server in the load balancing farm.

Now, isn’t this setup fun? Sigh…

Advertisements
  1. nobody
    September 19, 2014 at 7:00 pm

    These problems are solved in VPL 3.0 and upper. The Java applet has been removed

    • October 17, 2014 at 5:06 pm

      Thank you, since the blog post I moved to other company so I’m not working on that any more. Sorry for harshe comments 😉

  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: