System Administration: Load Balancing with Apache

System Administration: Load Balancing with Apache

This year starts with a topic that doesn't show up that often on the blog, mostly because is not one of my main area of interest, system administration.

Since last month, this blog is running on a new application server.
With the addition of a new server some configuration had to be made on the network, giving me the chance to do some of the things that were on my to-do list.

One of those things was to create a decent fail system that is automatically activated when the application server is being deployed or, is down due to some sort of maintenance that requires application server downtime.

The current system configuration is a single application server running Jetty, being front-ended by the Apache httpd server using mod_proxy. Although this is the configuration being used, the only requirement for the solution that will be presented is the Apache httpd server the application server can be any other than Jetty such as Tomcat, JBoss, etc.

A simple but not effective solution

A very simple solution to take care of the application server errors is to specify a ErrorDocument for the 503 - the service unavailable code - in the apache configuration file.

The problem with such solution is that when the error, or errors, specified in the configuration are received from the application server apache sends a redirect to the client with the url provided in the configuration.
This means that the client is sent to the error page and by refreshing it won't request the page that was being initially requested from the application server.

In my opinion this is a problem. Specially because I already knew that it is possible to serve an unavailable page at any url without redirecting, it was Jorge Matias - one of the sysadmins at work - that pointed me out the solution. Thanks Jorge!

A good solution (in my opinion)

Since Apache Http v2.2 a Load Balancing module is available. The idea is that the Apache server behaves as a balancer that distributes the traffic among the application servers. Since, 2.2.4 there's an option which allows a server to be marked as hot standby (pay special attention: the documentation talks about this feature since 2.2.0 but it is only available since 2.2.4). An application server marked as an hot standby server will only be reached if and only if any other application server is unavailable.

As soon as the application servers are once again available the traffic will start being redirected to them.

This is a typical scenario for the application deployment. The application servers are unavailable, although there is other server that replies to any request with the same page saying that the service is unavailable.

This way if a client requests /something/something2/abc and the application server isn't available it won't be redirected but instead will get a virtual page served by the hot standby server.

Below is the explanation on how to create such environment.

Configuration

For this configuration example there will be two different instances of Apache running - the balancer and the hot standby - and one application server instance.

Let's first examine the balancer configuration.

The Proxy balancer://cluster defines a new balancer cluster called cluster and it contains two members, defined between the Proxy tags: http://theOnlyServer:8668/ and http://someOtherServer/. The last one is marked with a status=+H which defines it's a hot standby server.

Another common pitfall is defining a single ProxyPassReverse for a cluster's ProxyPass. If a ProxyPass is defined for the cluster then, a ProxyPassReverse has to be defined for each BalancerMember.

Although there are various options that allow to define load factors, maximum of clients per Balancer Member, among other things the configuration can be as simple as presented above. To read more about the various options in the balancer module, please refer to the Apache Documentation.

On the hot standby server the configuration is also very simple. The apache's rewrite engine is used to rewrite any requested url into the not available page url. Let's then examine the hot standby apache's configuration (the one who's running in someOtherServer host):

The previous configuration enables the rewrite engine on apache and rewrites any url to /unavailableServer.html. This is, once again, the simplest scenario. More complex configuration can be created, such as, creating specific rules in order to use css files and images.

Conclusions

With minimum effort a hot standby configuration can be created, improving the user experience when the application server is being deployed. At least, using this configuration the user gets a personalised screen rather than a huge sloppy "503 Service Unavailable".