[users@httpd] TimeOut expired | Multiple Balancers using multiple workers | Apache 2.2.17

["King Holger (CI/AFP2)" <Ho...@de.bosch.com>]

Hi group,

we have the following topology up and running:

         Apache2 (works as load-balancer)
         |           |
Tomcat1        Tomcat2
(rb-wcmstc1)  (rb-wcmstc2)

using the following TWO balancers named "fs4_javaclient_over_http" and "fs4_webclient_over_http"
specifying DIFFERENT TIMEOUT VALUES with "90" and "660" seconds:

<Proxy balancer://fs4_javaclient_over_http>
    BalancerMember ajp://rb-wcmstc1.xx.xxxxx.xxx:8009 loadfactor=100 retry=10 route=rb-wcmstc1 connectiontimeout=5 timeout=90
    BalancerMember ajp://rb-wcmstc2.xx.xxxxx.xxx:8009 loadfactor=100 retry=10 route=rb-wcmstc2 connectiontimeout=5 timeout=90

    ProxySet stickysession=JSESSIONID|jsessionid
    ProxySet lbmethod=byrequests
    ProxySet scolonpathdelim=On
    ProxySet nofailover=Off
</Proxy>

<Proxy balancer://fs4_webclient_over_http>
    BalancerMember ajp://rb-wcmstc1.xx.xxxxx.xxx:8009 loadfactor=100 retry=10 route=rb-wcmstc1 connectiontimeout=5 timeout=660
    BalancerMember ajp://rb-wcmstc2.xx.xxxxx.xxx:8009 loadfactor=100 retry=10 route=rb-wcmstc2 connectiontimeout=5 timeout=660

    ProxySet stickysession=JSESSIONID|jsessionid
    ProxySet lbmethod=byrequests
    ProxySet scolonpathdelim=On
    ProxySet nofailover=Off
</Proxy>

Incoming requests for ONE TYPE of HTTP-Client are using the "fs4_javaclient_over_http" balancer via:
RewriteCond %{REQUEST_URI} ^/servlet/ClientIO/* [NC]
RewriteRule ^/(.*) balancer://fs4_javaclient_over_http/$1 [proxy,last]

the other HTTP-CLIENT is using "fs4_webclient_over_http":
RewriteCond %{REQUEST_URI} !^/servlet/ClientIO/* [NC]
RewriteCond %{REQUEST_URI} !^/index.html
RewriteRule ^/(.*) balancer://fs4_webclient_over_http/$1 [proxy,last]

This is done to separate connection losts for long running requests. Formerly, when one Client provoked
a request that took longer than the configured "timeout" value, both clients lost their connection due to a Tomcat switch.

THE PROBLEM
- in Apache2 "error_log" we see that a TIMEOUT HAS EXPIRED:
[Wed Feb 16 19:43:08 2011] [error] (70007)The timeout specified has expired: ajp_ilink_receive() can't receive header

This leads to a switching of the Tomcat from "rb-wcmstc1" to "rb-wcmstc2"
or vice versa only for requests that run over the "fs4_webclient_over_http"
balancer.

But It's really strange, that we can't find any request that takes longer
than the configured TIMEOUT parameter of "660" seconds (see config
above) within the Apache2 ACCESS-Log. What we can see is: there
are long running requests - but they do not reach the value of 660 seconds.

When enabling "debug" LogLevel we see more details (see excerpt enclosed).


Our questions:
- is the configuration okay?
- which timeout is expired? (connectiontimeout?, replytimeout?, etc.)
- why does it happen?
- is it a Tomcat connection-problem?