Prevent Apache Backup when a single tomcat stops responding

[Akash Jauhar <aj...@sapient.com>]

All

Here is an interesting problem that we are facing.

Our Production architecture is as follows:

There are 4 Machines each hosting 1 Apache and 3 Tomcat 3.3.1a servlet engines. Hence there are a total of 4 Apache and 12 tomcat workers. Each apache can talk to all the 12 tomcat workers. 

Sometimes (during peak load) when tomcat worker (lets name it tc1) is doing garbage collection for an extended period of time, all requests for this tomcat worker tc1 get queued up. We are noticing some weird GC behaviour when the tomcat does 
Minor GC. I am pasting some garbage collection logs below
233191.652: [GC 233191.652: [DefNew: 127232K->3840K(127232K),204.1755072 secs] 293286K->176929K(385272K), 204.1757747 secs] 233396.272: [GC 233396.272: [DefNew: 127232K->3840K(127232K),101.8621946 secs] 300321K->183426K(385272K), 101.8624498 secs]

A number of such simultaneous Garbage collections and the requests for this  tomcat gets queued up.  

We are using AJP 12 threads and the maxThreads attribute is set to 100. Hence 100 requests for tc1 take those 100 threads. Any more requests start getting queued up in the apache servers causing it to backup and ultimately reach the value set in maxClients attribute in httpd.conf. thus bringing down the site.

Is there a way by which once the 100 threads of the AJP12 connector are taken up, the tomcat sends a server busy signal to the end user and not queue up the apache and bring down the whole site (even though there are 11 other tomcat workers available). I read something about the backlog parameter in server.xml at http://jakarta.apache.org/tomcat/tomcat-3.3-doc/serverxml.html. can someone explain what this parameter does.

Any recommendations on how to prevent this one tomcat taking down the whole site would really help

Thanks
Akash