You are viewing a plain text version of this content. The canonical link for it is here.
Posted to bugs@httpd.apache.org by bu...@apache.org on 2009/04/09 22:32:34 UTC
DO NOT REPLY [Bug 47011] New: mod_proxy/mod_proxy_balancer
hot-standby BalancerMembers not taking over immediately
https://issues.apache.org/bugzilla/show_bug.cgi?id=47011
Summary: mod_proxy/mod_proxy_balancer hot-standby
BalancerMembers not taking over immediately
Product: Apache httpd-2
Version: 2.2.8
Platform: PC
OS/Version: Linux
Status: NEW
Severity: regression
Priority: P2
Component: mod_proxy_balancer
AssignedTo: bugs@httpd.apache.org
ReportedBy: mwhiteley@materialogic.com
After upgrading from Apache 2.2.6 to Apache 2.2.8 or greater, when gracefully
shutting down one of our Embedded Tomcats on our application server, we notice
a temporary outage (503) from our proxy balancer before the hot-standby
(status=+H) takes over.
Layout:
- Application proxy server (Apache 2.2.8)
- Proxies requests via mod_proxy/mod_proxy_balancer/mod_proxy_ajp to
application server
- Application server (Java)
- Runs master Java application server with Embedded Tomcat (Tomcat/5.5.17)
on port 8009
- Runs slave Java application server with Embedded Tomcat (Tomcat/5.5.17) on
port 8008
When the application proxy server was Apache 2.2.6, we were able to gracefully
shutdown the master Tomcat server (calling embedded.stop()), and the Hot
Standby (status=+H) BalancerMember would immediately start serving requests.
After the upgrade to 2.2.8, we see "HTTP/1.1 503 This application is not
currently available" while the active BalancerMember is shutting down before
the hot-standby takes over (~1 second). These errors appear in both the Tomcat
access log and the Apache access log.
I have recompiled and tested Apache versions 2.2.6, 2.2.8, 2.2.9 and 2.2.11 to
verify that this problem exists after version 2.2.6. I still had the problem
when switching the BalancerMember protocol from ajp:// to http://, so I think
this rules out AJP-specific issues. I was unable to reproduce the problem
using Apache servers as the BalancerMembers, so I'm speculating this has
something to do with the interaction with Tomcat.
Proxy conf:
=================================
ProxyPassMatch /(.+/)?application.server$ balancer://production_server
<Proxy balancer://production_server/>
BalancerMember ajp://server.domain.tld:8009/ lbset=1 retry=10
loadfactor=100
BalancerMember ajp://server.domain.tld:8008/ lbset=2 retry=10
status=+H
ProxySet lbmethod=bytraffic
</Proxy>
=================================
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 47011] mod_proxy/mod_proxy_balancer hot-standby
BalancerMembers not taking over immediately
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=47011
thosor <th...@btc-ag.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Priority|P2 |P1
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 47011] mod_proxy/mod_proxy_balancer hot-standby
BalancerMembers not taking over immediately
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=47011
thosor <th...@btc-ag.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |thomas.soring@btc-ag.com
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 47011] mod_proxy/mod_proxy_balancer hot-standby
BalancerMembers not taking over immediately
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=47011
--- Comment #4 from Ruediger Pluem <rp...@apache.org> 2009-04-22 13:35:51 PST ---
Which connector are you using in Tomcat? The APR connector or the classic
blocking connector?
Does it help when you add
ping=1
to the other BalancerMember parameters?
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 47011] mod_proxy/mod_proxy_balancer hot-standby
BalancerMembers not taking over immediately
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=47011
thosor <th...@btc-ag.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC|thomas.soring@btc-ag.com |
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 47011] mod_proxy/mod_proxy_balancer hot-standby
BalancerMembers not taking over immediately
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=47011
--- Comment #1 from Ruediger Pluem <rp...@apache.org> 2009-04-09 13:40:10 PST ---
Can you please check if the following patch fixes your issue
(:http://svn.apache.org/viewvc/httpd/httpd/branches/2.2.x/modules/proxy/mod_proxy.c?r1=713145&r2=739610&view=patch):
--- httpd/httpd/branches/2.2.x/modules/proxy/mod_proxy.c 2008/11/11 20:01:59
713145
+++ httpd/httpd/branches/2.2.x/modules/proxy/mod_proxy.c 2009/01/31 20:58:07
739610
@@ -1002,8 +1002,10 @@
* We can not failover to another worker.
* Mark the worker as unusable if member of load balancer
*/
- if (balancer)
+ if (balancer) {
worker->s->status |= PROXY_WORKER_IN_ERROR;
+ worker->s->error_time = apr_time_now();
+ }
break;
}
else if (access_status == HTTP_SERVICE_UNAVAILABLE) {
@@ -1013,6 +1015,7 @@
*/
if (balancer) {
worker->s->status |= PROXY_WORKER_IN_ERROR;
+ worker->s->error_time = apr_time_now();
}
}
else {
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 47011] mod_proxy/mod_proxy_balancer hot-standby
BalancerMembers not taking over immediately
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=47011
--- Comment #6 from Ruediger Pluem <rp...@apache.org> 2009-04-22 23:16:11 PST ---
Ah, my bad. To get this working with 2.2.11 you should apply
http://svn.apache.org/viewvc/apr/apr/branches/1.3.x/strings/apr_strings.c?r1=727605&r2=727604&pathrev=727605&view=patch
first (its from http://svn.apache.org/viewvc?view=rev&revision=727605).
Please try again afterwards with 2.2.11.
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 47011] mod_proxy/mod_proxy_balancer hot-standby
BalancerMembers not taking over immediately
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=47011
mwhiteley@materialogic.com changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |mwhiteley@materialogic.com
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 47011] mod_proxy/mod_proxy_balancer hot-standby
BalancerMembers not taking over immediately
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=47011
--- Comment #3 from mwhiteley@materialogic.com 2009-04-22 12:30:27 PST ---
After further research, i've traced this to proxy_util.c r582620 ( https://svn.apache.org/viewcvs.cgi?view=rev&rev=582620 )(which
references PR 43472). Reverting just this change in 2.2.11 resolves the issue
described in this report.
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 47011] mod_proxy/mod_proxy_balancer hot-standby
BalancerMembers not taking over immediately
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=47011
--- Comment #2 from mwhiteley@materialogic.com 2009-04-09 14:25:17 PST ---
i have tried the patch you sent in a clean httpd-2.2.8 and also tried patching
mod_proxy, mod_proxy_http, and mod_proxy_balancer in httpd-2.2.11 up to
revision 763402 ( https://svn.apache.org/viewcvs.cgi?view=rev&rev=763402 ) with the same results.
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 47011] mod_proxy/mod_proxy_balancer hot-standby
BalancerMembers not taking over immediately
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=47011
--- Comment #7 from mwhiteley@materialogic.com 2009-04-23 10:30:19 PST ---
After applying the apr_strings.c patch on 2.2.11, when gracefully shutting down
the primary tomcat, Apache logs the following once:
[Thu Apr 23 12:07:59 2009] [error] (111)Connection refused: proxy: AJP: attempt
to connect to xxx.xxx.xxx.xxx:8009 (xxxxxxx.xxxxxxx.com) failed
[Thu Apr 23 12:07:59 2009] [error] ap_proxy_connect_backend disabling worker
for (xxxxxxx.xxxxxxx.com)
[Thu Apr 23 12:07:59 2009] [error] proxy: AJP: failed to make connection to
backend: xxxxxxx.xxxxxxx.com
And the hot-standby takes the request as expected without an outage on the
front-end.
The above worked with and without "ping=1".
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 47011] mod_proxy/mod_proxy_balancer hot-standby
BalancerMembers not taking over immediately
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=47011
--- Comment #5 from mwhiteley@materialogic.com 2009-04-22 14:22:14 PST ---
i'm not exactly sure if this is the answer you need but, we're using an
Embedded tomcat, and setting up the AJP connector as follows:
Connector ajpConnector =
embedded.createConnector((java.net.InetAddress) null, this.ajpPort, "ajp");
embedded.addConnector(ajpConnector);
Results of adding "ping=1" to the BalancerMembers:
Apache 2.2.11:
- Wouldn't start, output the following:
BalancerMember Ping/Pong timeout has wrong format
Apache 2.2.9:
- During graceful shutdown outlined previously, balancer returned:
HTTP/1.1 500 Internal Server Error
- Logged:
[Wed Apr 22 16:05:53 2009] [error] (70014)End of file found:
ajp_ilink_receive() can't receive header
Apache 2.2.8:
- Balancer always returned:
HTTP/1.1 503 Service Temporarily Unavailable
- Logged:
[Wed Apr 22 16:00:50 2009] [error] ajp_msg_append_uint8():
BufferOverflowException 4 4
[Wed Apr 22 16:00:50 2009] [error] ajp_handle_cping_cpong:
ajp_marshal_into_msgb failed
[Wed Apr 22 16:00:50 2009] [error] (120001)APR does not understand this error
code: proxy: AJP: cping/cpong failed to (null) (xxx.xxxxxxxxxx.com)
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org