need load balancing advice

[Ray Zimmerman <rz...@cornell.edu>]

We have a web app which consists of some very expensive dynamic 
pages. Network bandwidth isn't an issue, but web-server CPU load is. 
We used to run a front-end webserver, back-end webserver and database 
server on the same machine. Recently we tried moving the back-end 
webserver to two separate machines which the front-end server proxies 
to in round-robin style. With our typical load this has solved the 
problem quite nicely.

However, we find that under extremely high load the system is 
unstable and "falls over" onto one or the other of the two backend 
servers. In other words, the load on one of them gets really high, 
while the other looks basically idle (while serving the same number 
of requests, mind you) and it tends to stay that way. I think it's 
due to the way our app works, where the slow running requests end up 
with more cleanup work to do, and the fast requests typically end up 
doing less work.

Any suggestions for a simple method of allocating a higher number of 
requests to the machine with the lower load?  I've looked a little 
bit at mod_backhand, which looks like it might do what we want ... 
are there any other simple tools available to handle this?

Right now we're simply using a PerlTransHandler on the front end 
server. I wouldn't mind a customized approach like this, but I don't 
know of a good, efficient way to get the info on the system load of 
the two backend servers.

-- 
  Ray Zimmerman  / e-mail: rz10@cornell.edu / 428-B Phillips Hall
   Sr Research  /   phone: (607) 255-9645  /  Cornell University
    Associate  /      FAX: (815) 377-3932 /   Ithaca, NY  14853

Re: need load balancing advice

[Stas Bekman <st...@stason.org>]

Ray Zimmerman wrote:
> At 5:10 PM +0100 10/16/03, Ged Haywood wrote:
> 
>> Have you searched the archives?  There was a thread talking about load
>> balancing earlier in the year, see
>>
>> http://groups.yahoo.com/group/modperl/message/49334
>>
>> for example which might help.
> 
> 
> Thanks, I hadn't seen this thread. The pointer to the URL Rewriting 
> Guide were helpful, but I'm still not sure what's a good strategy for 
> querying the backend servers about their current load?

Apache::Scoreboard has an API for retrieval of the server scores, which 
includes a lot of useful information about the Apache processes, but not the 
system information. But that could be useful enough as you will know how many 
processes are busy churning requests, and can probably make some good 
judgements based on that information.

In case you want to know the real load (CPU/mem usage), you could write a 
simple handler which will return this data to the front end server on request. 
You could use GTop if libgtop is available on your OS. See Apache::VMonitor 
for an example of its usage.

Of course you may want to try using mosix/openmosix which come with tools to 
do that internally. But clusters is not a topic to discuss on this list, you 
are better off to do that on the scalable list.
http://perl.apache.org/docs/offsite/other.html#Performance_and_Scalability

__________________________________________________________________
Stas Bekman            JAm_pH ------> Just Another mod_perl Hacker
http://stason.org/     mod_perl Guide ---> http://perl.apache.org
mailto:stas@stason.org http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org   http://ticketmaster.com

Re: need load balancing advice

[Ray Zimmerman <rz...@cornell.edu>]

At 5:10 PM +0100 10/16/03, Ged Haywood wrote:
>Have you searched the archives?  There was a thread talking about load
>balancing earlier in the year, see
>
>http://groups.yahoo.com/group/modperl/message/49334
>
>for example which might help.

Thanks, I hadn't seen this thread. The pointer to the URL Rewriting 
Guide were helpful, but I'm still not sure what's a good strategy for 
querying the backend servers about their current load?

-- 
  Ray Zimmerman  / e-mail: rz10@cornell.edu / 428-B Phillips Hall
   Sr Research  /   phone: (607) 255-9645  /  Cornell University
    Associate  /      FAX: (815) 377-3932 /   Ithaca, NY  14853

Re: need load balancing advice

[Ged Haywood <ge...@www2.jubileegroup.co.uk>]

Hi there,

On Thu, 16 Oct 2003, Ray Zimmerman wrote:

> We have a web app which consists of some very expensive dynamic 
> pages. Network bandwidth isn't an issue, but web-server CPU load is. 
> [snip]
> the front-end server proxies to in round-robin style. With our
> typical load this has solved the problem quite nicely.
> [snip]
> under extremely high load the system is unstable and "falls over"
> onto one or the other of the two backend servers
> Any suggestions for a simple method of allocating a higher number of 
> requests to the machine with the lower load?

Have you searched the archives?  There was a thread talking about load
balancing earlier in the year, see

http://groups.yahoo.com/group/modperl/message/49334

for example which might help.

73,
Ged.