[users@httpd] Apache2 and Tomcat load balancing | Timeouts: request, connection, cping/cpong

["King Holger (CI/AFP2)" <Ho...@de.bosch.com>]

Dear Apache2 community,

having installed Apache2 2.4.3 in combination with:
-       mod_proxy_balancer (not "mod_jk"!)
-       mod_proxy_ajp
-       mod_proxy

and the following virtual host configuration:

BalancerMember ajp://rb-wcmstc1.de.bosch.com:8010 loadfactor=50 ping=5 retry=10 route=rb-wcmstc1 connectiontimeout=5 timeout=300
BalancerMember ajp://rb-wcmstc2.de.bosch.com:8010 loadfactor=50 ping=5 retry=10 route=rb-wcmstc2 connectiontimeout=5 timeout=300

to allow and setup a Tomcat 6 load balancing infrastructure, we regularly see the above configured timeouts in seconds (see http://httpd.apache.org/docs/2.2/mod/mod_proxy.html#proxypass):
- request-timeout (300 seconds)
- connection-timeout (5 seconds)
- cping/cpong-timeout (5 seconds)

being exceeded. The Apache2 error logs prove that:

(1)     [Mon Apr 08 18:23:16.954749 2013] [proxy_ajp:error] [pid 5030:tid 48] (70007)The timeout specified has expired: AH01030: ajp_ilink_receive() can't receive header
(2)     [Mon Apr 08 18:25:28.422599 2013] [proxy_ajp:error] [pid 6623:tid 52] (70007)The timeout specified has expired: [client 10.25.131.11:28706] AH00897: cping/cpong failed to xx.xx.xx.xxx:8010 (rb-wcmstc1.xx.xxxxx.xxx), referer: https://rb-xxxxx.xx.xxxxx.xxx/fs4webedit/message.jsp

(1)     might be logged when a request took longer >300 seconds
(2)     might be logged when ping request took longer >5 seconds

Taking a look at the HTTPS Access logs, we see for:
(1)     a request really taking 300 sec. It leads to the above error log entry at 18:23:16 - five minutes later:
xx.xx.xx.xxx - - [08/Apr/2013:18:18:16 +0200] "POST /fs4webedit/Dispatcher HTTP/1.1" 302 295 "..." "F0FDAFA023B323A62813598C985C3AB1.rb-wcmstc1" "-" TLSv1 RC4-MD5 300004793


With an important BUG fix provided since Apache2 version 2.2.22 we have a solution that prevents marking a complete AJP-worker (used for communication between Apache2 and Tomcat) in ERROR state when a single long running request occurred. This bugfix only works when setting the "ping" attribute for the BalancerMember. Details see ChangeLog enclosed: http://www.apache.org/dist/httpd/CHANGES_2.2.22 respectively http://mail-archives.apache.org/mod_mbox/httpd-cvs/201201.mbox/%3C20120104201529.54EA823888D2@eris.apache.org%3E

Now, our questions are:
-       what provokes the 2nd error message: "AH01012: ajp_handle_cping_cpong: ajp_ilink_receive failed" (a stop-the-world garbage collection on Tomcat side?)
-       what are the consequences if that happens?
-       how to prevent it (simply increasing the ping timeout?)

Mit freundlichen Grüßen / Best regards,
Holger King
CI/AFP2

Tel. 0711-811-59112

BeQIK