Load balance problem

[dantam sreedhar <sr...@yahoo.com>]

Hi,
   
  We have one apache and two tomcats running on separate machines in production. There is a firewall between apache and tomcat servers. Sticky session is enabled for load balancing. Load balancing between these tomcat is happening properly, however fail-over has problem. 
   
  When we shutdown one tomcat, the other tomcat was getting hung with in two minutes. The number of users on tomcat1 were around 400 and almost equal number of users on tomcat2 (load was around 700 requests per min). When we took thread dump of second tomcat, all AJP connectors threads (around 650) were stuck with the following stack trace. Could you help me to solve this problem? Do you think "connectionTimeout=-1" in server.xml causing this problem?
   
  at java.net.SocketInputStream.socketRead0(Native Method)
              at java.net.SocketInputStream.read(SocketInputStream.java:129)
              at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
              at java.io.BufferedInputStream.read1(BufferedInputStream.java:222)
              at java.io.BufferedInputStream.read(BufferedInputStream.java:277)
              - locked <0x7e19ebe8> (a java.io.BufferedInputStream)
              at org.apache.jk.common.ChannelSocket.read(ChannelSocket.java:601)
              at org.apache.jk.common.ChannelSocket.receive(ChannelSocket.java:538)
              at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:666)
              at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
              at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
              at java.lang.Thread.run(Thread.java:534)
   
  Setup:
   
  OS - SolarisOS,
  Apache/1.3.33 (Unix) mod_ssl/2.8.22 OpenSSL/0.9.7e
  mod_jk - 1.2.14
  jdk - 1.4.2_03 
  Tomcat - 5.5.8
   
  workers.properties:
  ---------------------------
  workers.tomcat_home=/opt/tomcat
ps=/
worker.list=tomcat1, tomcat2, loadbalancer, jkstatus
  worker.tomcat1.port=8099
worker.tomcat1.host=xxxx
worker.tomcat1.type=ajp13
worker.tomcat1.lbfactor=1
worker.tomcat1.socket_keepalive=1
worker.tomcat1.socket_timeout=15
worker.tomcat1.recycle_timeout=60
worker.tomcat1.retries=3
  worker.tomcat2.port=9009
worker.tomcat2.host=yyyy
worker.tomcat2.type=ajp13
worker.tomcat2.lbfactor=1
worker.tomcat2.socket_keepalive=1
worker.tomcat2.socket_timeout=15
worker.tomcat2.recycle_timeout=60
worker.tomcat2.retries=3
  worker.loadbalancer.type=lb
worker.loadbalancer.balanced_workers=tomcat1, tomcat2
worker.loadbalancer.sticky_session=true
worker.jkstatus.type=status

  server.xml
  --------------
  <Connector port="8099" enableLookups="false"  protocol="AJP/1.3" 
     minSpareThreads="100" maxThreads="700" maxSpareThreads="80" acceptCount="60" debug="0" 
     connectionTimeout="-1" request.registerRequests="false" />

  Thanks,
  -Sreedhar

 		
---------------------------------
Do you Yahoo!?
 Get on board. You're invited to try the new Yahoo! Mail Beta.