[users@httpd] BALANCER: (balancer://cluster). All workers are in error state for route in apache error logs and service temporarily unavailanble for tomcat

[Amol Puglia <am...@yahoo.com>]

Hello Team,

I have configured apache to load balanced six tomcat instances in the backend using mod_proxy_balance and mod_proxy_ajp modules.

I am frequently getting service temporary unavailable message and following error in the apache error logs.

[Mon Feb 06 09:07:28 2012] [error] 
proxy: BALANCER: (balancer://cluster). All workers are in error state 
for route (marsprod_rmiserver_3)
[Mon Feb 06 09:07:32 2012] [error] proxy: BALANCER: 
(balancer://cluster). All workers are in error state for route 
(marsprod_rmiserver_3)
[Mon Feb 06 09:07:36 2012] [error] proxy: BALANCER: 
(balancer://cluster). All workers are in error state for route 
(marsprod_rmiserver_3)
[Mon Feb 06 09:07:37 2012] [error] proxy: BALANCER: 
(balancer://cluster). All workers are in error state for route 
(marsprod_rmiserver_3)
[Mon Feb 06 09:07:42 2012] [error] proxy: BALANCER: 
(balancer://cluster). All workers are in error state for route 
(marsprod_rmiserver_3)
[Mon Feb 06 09:07:45 2012] [error] proxy: BALANCER: 
(balancer://cluster). All workers are in error state for route 
(marsprod_rmiserver_3)
[Mon Feb 06 09:08:02 2012] [error] proxy: BALANCER: 
(balancer://cluster). All workers are in error state for route 
(marsprod_rmiserver_3)


I am using following version of apache and tomcat.

Apache version :- Apache/2.2.16

Tomcat :- 6.0.26

Jdk :-1.6.0_24

Below is the configuration of my apache and tomcat.

# Port 80
   Listen server_name:80
   
   <VirtualHost _default_:80>
      ServerName server_name
      ServerAlias server_name
        ServerAlias server_name
       

<Directory /eMatrix>

        Order deny,allow

        Deny from all

        Allow from 153.88.251.174 153.88.251.160 153.88.251.165 153.88.251.170 153.88.251.212 153.88.251.199

      </Directory>

      Timeout 1800
      ProxyTimeout 1800
      ProxyRequests Off


       ProxyPass /eMatrix balancer://cluster stickysession=JSESSIONID|jsessionid nofailover=On
       #ProxyPass /eMatrix balancer://cluster stickysession=JSESSIONID nofailover=On

       ProxyPreserveHost On


       ProxyPass /eMatrix ajp://server_name:8009/eMatrix

       ProxyPass /eMatrix ajp://server_name:8010/eMatrix

      ProxyPass /eMatrix ajp://server_name:8011/eMatrix

      ProxyPass /eMatrix ajp://server_name:9009/eMatrix

      ProxyPass /eMatrix ajp://server_name:9010/eMatrix

      ProxyPass /eMatrix ajp://server_name:9011/eMatrix

       ProxyPassReverse /eMatrix ajp://server_name:8009/eMatrix

       ProxyPassReverse /eMatrix ajp://server_name:8010/eMatrix

       ProxyPassReverse /eMatrix ajp://server_name:8011/eMatrix

       ProxyPassReverse /eMatrix ajp://server_name:9009/eMatrix

       ProxyPassReverse /eMatrix ajp://server_name:9010/eMatrix

       ProxyPassReverse /eMatrix ajp://server_name:9011/eMatrix

       <Proxy balancer://cluster>
         BalancerMember ajp://server_name:8009/eMatrix route=marsprod_rmiserver_1 loadfactor=33 retry=60
         BalancerMember ajp://server_name:8010/eMatrix route=marsprod_rmiserver_2 loadfactor=33 retry=60
         BalancerMember ajp://server_name:8011/eMatrix route=marsprod_rmiserver_3 loadfactor=33 retry=60
         BalancerMember ajp://server_name:9009/eMatrix route=marsprod_rmiserver_4 loadfactor=33 retry=60
         BalancerMember ajp://server_name:9010/eMatrix route=marsprod_rmiserver_5 loadfactor=33 retry=60
         #load balancing performed based on number of user requests
         #ProxySet lbmethod=byrequests
         # Report server is for Report purpose only so balancing is required now.
         BalancerMember ajp://server_name:9011/eMatrix route=marsprod_rmiserver_6 loadfactor=33 retry=60
         ProxySet lbmethod=byrequests
       </Proxy>

       #Status page for balancer

       <Location /balancer>
       SetHandler balancer-manager
       Order Deny,Allow
       Deny from all
       Allow from all
      </Location>

      ProxyStatus On

      <Location /status>
      SetHandler server-status
      Order Deny,Allow
      Deny from all
      Allow from all
      </Location>

      ErrorLog /opt/web/apache/app/mxora/logs/error.log

 </VirtualHost>


Tomcat Configuration Server.xml

<!-- for Native tomcat setup-->
    <Connector port="8082" protocol="org.apache.coyote.http11.Http11AprProtocol"
        maxHttpHeaderSize="8192"
        maxThreads="60"
        maxPostSize="0"
        enableLookups="false"
        redirectPort="8443"
        acceptCount="150"
        compression="on"
        connectionTimeout="60000"
        disableUploadTimeout="true"
        URIEncoding="UTF-8"/>


    <!-- Define an AJP 1.3 Connector on port 9084 -->
    <Connector port="8010"
        protocol="AJP/1.3"
        redirectPort="8443"
        maxThreads="60"
        enableLookups="false"
        maxPostSize="0"
        connectionTimeout="60000"
        URIEncoding="UTF-8"/>



Kindly help me to trobleshoot the issue.