You are viewing a plain text version of this content. The canonical link for it is here.
Posted to bugs@httpd.apache.org by bu...@apache.org on 2013/03/09 09:07:04 UTC
[Bug 54657] New: DNS lookup failure for load balancer member does
not stop affected member from receiving and failing requests
https://issues.apache.org/bugzilla/show_bug.cgi?id=54657
Bug ID: 54657
Summary: DNS lookup failure for load balancer member does not
stop affected member from receiving and failing
requests
Product: Apache httpd-2
Version: 2.2.21
Hardware: PC
OS: Linux
Status: NEW
Severity: minor
Priority: P2
Component: mod_proxy_balancer
Assignee: bugs@httpd.apache.org
Reporter: domainspam@bellsouth.net
Classification: Unclassified
the problem that i am reporting here is that when name resolution of a load
balancer member fails, the affected member is not marked as disabled (or in
error state), and taken out of the loop of actively load balanced members. the
bad member continues to get requests and fail them.
i believe what should occur is that the bad member should be marked as disabled
(or in error state), a log entry made to the error log (this already occurs),
and the request sent to a good member, if one is available. also, subsequent
requests should not be sent to the bad member until/unless it becomes
available. if a good member is not available to handle a request, then an error
response is appropriate.
this is a minor issue because the member can be disabled at runtime through the
balancer manager, or the name can be mapped to an address (even if it points to
an address where nothing is listening on the right port -- because as long as
the name resolves and the connection fails, then the member gets marked as
being in error state and removed from the load balancing loop), etc...
this problem was noticed in an environment where the same httpd config is used
in multiple testing environments. the testing environments have varying numbers
of cluster members actually provisioned and available. the idea was that the
loadbalancer would be able to determine when members were not available and
skip over them.
given this simplified config:
<Proxy balancer://api-cluster>
BalancerMember http://box01:8182/api
BalancerMember http://box02:8182/api
</Proxy>
ProxyPass /api/ balancer://api-cluster/
when the box02 name is not resolvable, then every other client request coming
into the load balancer generates a response similar to:
HTTP/1.1 502 Proxy Error
Date: Fri, 08 Mar 2013 15:12:03 GMT
Content-Length: 400
Content-Type: text/html; charset=iso-8859-1
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>502 Proxy Error</title>
</head><body>
<h1>Proxy Error</h1>
<p>The proxy server received an invalid
response from an upstream server.<br />
The proxy server could not handle the request <em><a
href="/api/whatever">GET /api/whatever</a></em>.<p>
Reason: <strong>DNS lookup failure for: box02</strong></p></p>
</body></html>
also, this goes to the error_log at error level:
[Fri Mar 08 10:12:03 2013] [error] [client 127.0.0.1] proxy: DNS lookup failure
for: box02 returned by /api/whatever
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 54657] DNS lookup failure for load balancer member does not
stop affected member from receiving and failing requests
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=54657
--- Comment #2 from abiacco@formatdynamics.com ---
I can confirm this happens on my 2.2.24 also.
The app-03.local member listed below has no entry in my hosts file.
My member numbers also vary based on environment.
centos 6.4
kernel 2.6.32-358.14.1.el6.x86_64
httpd 2.2.24
apr 1.4.6
apr-util 1.5.2
Sch Host Stat Route Redir F Set Acc Wr Rd
ajp app-03.local Ok app-03 1 0 2 0 0
[08/Nov/2013:08:59:14 -0700] GET /XXX 502 Sz 17 BR 246 BS 166 TMSec 12412 TSec
0 Bal balancer://loadbalancer SessWrk - RealWrk app-03 WrkName
ajp://app-03.local:8009 PID 31861 TID 140004374312704 UID
Un0KUgoKCtMAAHx1OF8AAAAM VHost XXX
<IfModule mod_proxy_balancer.c>
ProxyPass /balancer-manager !
ProxyPass /XXX balancer://loadbalancer/XXX
ProxyPassReverse /XXX balancer://loadbalancer/XXX
</IfModule>
<IfModule mod_proxy.c>
ProxyRequests off
ProxyStatus On
# Enable/disable the handling of HTTP/1.1 "Via:" headers.
# ("Full" adds the server version; "Block" removes all outgoing Via:
headers)
# Set to one of: Off | On | Full | Block
ProxyVia Off
ProxyPreserveHost On
<IfModule mod_proxy_balancer.c>
<Proxy balancer://loadbalancer>
# Max is equal to the max threads a single tomcat can handle, devided by the
number of tomcats being balanced
# So if a single tomcat is configured for 300 max threads and there are 3
tomcats, you would set Max to 100 for each balancer member
BalancerMember ajp://app-01.local:8009 route=app-01 loadfactor=1 max=200
acquire=2000 connectiontimeout=2 disablereuse=off keepalive=off ping=2
timeout=60 retry=60 ttl=120 flushpackets=on
BalancerMember ajp://app-02.local:8009 route=app-02 loadfactor=1 max=200
acquire=2000 connectiontimeout=2 disablereuse=off keepalive=off ping=2
timeout=60 retry=60 ttl=120 flushpackets=on
BalancerMember ajp://app-03.local:8009 route=app-03 loadfactor=1 max=200
acquire=2000 connectiontimeout=2 disablereuse=off keepalive=off ping=2
timeout=60 retry=60 ttl=120 flushpackets=on
ProxySet stickysession=JSESSIONID|jsessionid
ProxySet lbmethod=bybusyness
ProxySet scolonpathdelim=On
# Balancer timeout in seconds. If set this will be the maximum time to wait for
a free worker
# Default is not to wait. (Acquire time * number of workers) / 1000?
ProxySet timeout=6
# A single or comma-separated list of HTTP status codes. If set this will force
the worker
# into error state when the backend returns any status code in the list.
ProxySet failonstatus=500,503,502
</Proxy>
</IfModule>
# end mod_proxy_balancer
</IfModule>
# end mod_proxy
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 54657] DNS lookup failure for load balancer member does not
stop affected member from receiving and failing requests
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=54657
mike <do...@bellsouth.net> changed:
What |Removed |Added
----------------------------------------------------------------------------
Component|mod_proxy_balancer |mod_proxy
Version|2.2.21 |2.4.4
Severity|minor |normal
--- Comment #1 from mike <do...@bellsouth.net> ---
just adding notes on how to reproduce this problem with the current latest
version, 2.4.4:
build process:
- mkdir /tmp/apache/
- cd /tmp/apache/
- wget http://www.gtlib.gatech.edu/pub/apache/httpd/httpd-2.4.4.tar.gz
- wget http://www.gtlib.gatech.edu/pub/apache/apr/apr-1.4.6.tar.gz
- wget http://www.gtlib.gatech.edu/pub/apache/apr/apr-util-1.5.1.tar.gz
- wget ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre/pcre-8.32.tar.gz
- tar xzf httpd-2.4.4.tar.gz
- tar xzf apr-1.4.6.tar.gz
- tar xzf apr-util-1.5.1.tar.gz
- tar xzf pcre-8.32.tar.gz
- mv apr-1.4.6 httpd-2.4.4/srclib/apr/
- mv apr-util-1.5.1 httpd-2.4.4/srclib/apr-util/
- cd /tmp/apache/pcre-8.32/
- ./configure --prefix=/tmp/apache/mypcre
- make && make install
- cd /tmp/apache/httpd-2.4.4/
- ./configure --prefix=/tmp/apache/myapache/ --enable-mods-shared=all
--enable-mpms-shared=all --with-mpm=worker --disable-cgid --enable-proxy=shared
--with-included-apr --with-pcre=/tmp/apache/mypcre/
- make && make install
test config:
/tmp/apache/myapache/conf> cat httpd.conf
ServerRoot /tmp/apache/myapache/
Listen 8080
LoadModule authz_core_module modules/mod_authz_core.so
LoadModule mpm_worker_module modules/mod_mpm_worker.so
LoadModule unixd_module modules/mod_unixd.so
LoadModule slotmem_shm_module modules/mod_slotmem_shm.so
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_http_module modules/mod_proxy_http.so
LoadModule proxy_balancer_module modules/mod_proxy_balancer.so
LoadModule lbmethod_byrequests_module modules/mod_lbmethod_byrequests.so
<Directory />
AllowOverride None
Require all denied
</Directory>
DocumentRoot /tmp/apache/myapache/htdocs
<Directory /tmp/apache/myapache/htdocs>
AllowOverride None
Require all granted
</Directory>
ErrorLog logs/error_log
LogLevel warn
<Proxy balancer://api-cluster>
BalancerMember http://box01:8182/api
BalancerMember http://box02:8182/api
</Proxy>
ProxyPass /api/ balancer://api-cluster/
- start the server
- reproduce problem by repeatedly doing: curl
http://localhost:8080/api/whatever
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org