You are viewing a plain text version of this content. The canonical link for it is here.
Posted to bugs@httpd.apache.org by bu...@apache.org on 2011/02/18 20:11:06 UTC
DO NOT REPLY [Bug 50807] New: mod_proxy fails to send FIN response
when a FIN is received from a backend
https://issues.apache.org/bugzilla/show_bug.cgi?id=50807
Summary: mod_proxy fails to send FIN response when a FIN is
received from a backend
Product: Apache httpd-2
Version: 2.2.16
Platform: PC
OS/Version: Linux
Status: NEW
Severity: normal
Priority: P2
Component: mod_proxy
AssignedTo: bugs@httpd.apache.org
ReportedBy: gregory.boyce@gmail.com
I'm currently responsible for managing some reverse proxy servers using apache
and mod_proxy, and I stumbled across an apparent bug that I'm hoping someone
might have some insights into.
Mod_proxy is configured to frontend various webservers, typically with
persistent connections enabled on both the frontend and on the backend. What
I've discovered is that when there is an established connection to the backend
and the remote side of the connection sends a FIN, mod_proxy gets into a bad
state. For some reason we never send the corresponding FIN to finish closing
the connection.
Connections where backend persistent connections are disabled work fine.
I replicated this on my Ubuntu desktop running Apache 2.2.16 with a very simple
proxy configuration:
ProxyPass /apache/ http://72.52.221.58/
ProxyPassReverse /apache/ http://apache.com/
I issue a curl against http://localhost/apache/ and I see the following in a
packet capture:
13:56:52.377520 IP 192.168.1.57.58103 > 72.52.221.58.80: Flags [S], seq
3154899214, win 5840, options [mss 1460,sackOK,TS val 465793064 ecr
0,nop,wscale 7], length 0
13:56:52.428168 IP 72.52.221.58.80 > 192.168.1.57.58103: Flags [S.], seq
1968453344, ack 3154899215, win 5840, options [mss 1460,nop,nop,sackOK], length
0
13:56:52.428210 IP 192.168.1.57.58103 > 72.52.221.58.80: Flags [.], ack 1, win
5840, length 0
13:56:52.428453 IP 192.168.1.57.58103 > 72.52.221.58.80: Flags [P.], seq 1:266,
ack 1, win 5840, length 265
13:56:52.480281 IP 72.52.221.58.80 > 192.168.1.57.58103: Flags [.], ack 266,
win 6432, length 0
13:56:52.484575 IP 72.52.221.58.80 > 192.168.1.57.58103: Flags [P.], seq 1:648,
ack 266, win 6432, length 647
13:56:52.484593 IP 192.168.1.57.58103 > 72.52.221.58.80: Flags [.], ack 648,
win 7117, length 0
Interesting part comes after a few seconds:
13:56:57.484840 IP 72.52.221.58.80 > 192.168.1.57.58103: Flags [F.], seq 648,
ack 266, win 6432, length 0
13:56:57.519404 IP 192.168.1.57.58103 > 72.52.221.58.80: Flags [.], ack 649,
win 7117, length 0
In netstat I see an apache thread in a CLOSE_WAIT state:
tcp 1 0 192.168.1.57:58103 72.52.221.58:80 CLOSE_WAIT
apachectl-status shows one of the threads in a "W" Sending Reply status.
I haven't found any existing bugs on this, but I do see an old e-mail with a
similar behavior for mod_proxy's ftp data connection back in 2000:
http://mail-archives.apache.org/mod_mbox/www-apache-bugdb/200001.mbox/%3C20000110120507.66088.qmail@locus.apache.org%3E
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 50807] mod_proxy fails to send FIN response when a FIN is
received from a backend
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=50807
--- Comment #11 from Andy Wang <do...@moonteeth.com> ---
Looking at the mod_proxy documentation there's this little tidbit:
ttl can be used to avoid using a connection which is subject to closing because
of the backend server's keep-alive timeout.
So wouldn't that be the proper configurable solution?
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 50807] mod_proxy fails to send FIN response when a
FIN is received from a backend
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=50807
--- Comment #2 from Gregory Boyce <gr...@gmail.com> 2011-02-18 14:39:36 EST ---
(In reply to comment #1)
> Isn't this just the state the connection sits in until a subsequent request to
> the proxy finds the connection in the pool and tries to use it? The request
> that established the connection has moved on by the time the origin server is
> performing a keepalive close.
I'm not quite sure what you mean.
The client request that triggered the backend connection has definitely been
responded to and closed.
The backend connection itself is was kept alive until the backend webserver's
timeout was reached, and it attempted to close the connection by sending a FIN.
I would expect that Apache would then send its own FIN and open a new
connection for subsequent requests.
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 50807] mod_proxy fails to send FIN response when a
FIN is received from a backend
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=50807
Adam Woodworth <mi...@gmail.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Priority|P2 |P1
Severity|normal |major
--- Comment #4 from Adam Woodworth <mi...@gmail.com> 2011-06-30 15:23:10 UTC ---
Has anyone looked into this problem? Is a fix available? This is a critical
bug with mod_proxy. We've now seen this problem 3 times at different sites.
When using backend keepalives mod_proxy needs to be fixed to send a FIN to the
backend server (i.e. close the connection) when it gets a FIN from the backend
server. Otherwise the connection is half opened and in many cases a firewall
is tracking that and will not allow that same source port to be reused until a
FIN is received or until the firewall's timeout expires for that type of
tracking.
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 50807] mod_proxy fails to send FIN response when a
FIN is received from a backend
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=50807
--- Comment #5 from Adam Woodworth <mi...@gmail.com> 2011-07-07 16:06:09 UTC ---
This bug is made worse by the fact that
net.netfilter.nf_conntrack_tcp_timeout_close_wait in the Linux kernel (at least
on some versions) has a default value of 60 seconds. This value is used when
iptables is tracking connections, and it is how long conntrack will wait for a
responding FIN after seeing a FIN from one side. In this case, with the
default value, when Apache finally tries to close a half-closed connection, if
it has been longer than 60 seconds then Apache will not be able to send its
FIN, making this problem worse. Increasing the value, of course, is an option,
but probably not often done or even known about.
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 50807] mod_proxy fails to send FIN response when a FIN is
received from a backend
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=50807
--- Comment #9 from William A. Rowe Jr. <wr...@apache.org> ---
As a short-term band-aid, should all disconnected proxy connections which
hadn't reached their timeout emit a warning like
'Proxy connection to %s unexpectedly disconnected; the keepalive timeout of
that backend server or an intermediate proxy, router, load balancer or firewall
appliance is too short, all must be greater than %d seconds as configured in
httpd'
?
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 50807] mod_proxy fails to send FIN response when a
FIN is received from a backend
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=50807
--- Comment #1 from Eric Covener <co...@gmail.com> 2011-02-18 14:30:01 EST ---
Isn't this just the state the connection sits in until a subsequent request to
the proxy finds the connection in the pool and tries to use it? The request
that established the connection has moved on by the time the origin server is
performing a keepalive close.
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 50807] mod_proxy fails to send FIN response when a FIN is
received from a backend
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=50807
Jim Jagielski <ji...@apache.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |NEEDINFO
--- Comment #6 from Jim Jagielski <ji...@apache.org> ---
Can you confirm that the issue still exists in 2.4.x?
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 50807] mod_proxy fails to send FIN response when a
FIN is received from a backend
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=50807
taffy-tyler6464@hotmail.co.uk changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |taffy-tyler6464@hotmail.co.
| |uk
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 50807] mod_proxy fails to send FIN response when a FIN is
received from a backend
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=50807
--- Comment #7 from sebmoule <se...@orange.com> ---
Is there any correction for this problem ? does apache 2.4 correct it ?
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 50807] mod_proxy fails to send FIN response when a FIN is
received from a backend
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=50807
Orion Poplawski <or...@cora.nwra.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |orion@cora.nwra.com
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 50807] mod_proxy fails to send FIN response when a
FIN is received from a backend
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=50807
drop@tchorbadjiev.com changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |drop@tchorbadjiev.com
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 50807] mod_proxy fails to send FIN response when a FIN is
received from a backend
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=50807
William A. Rowe Jr. <wr...@apache.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEEDINFO |NEW
--- Comment #8 from William A. Rowe Jr. <wr...@apache.org> ---
This is not needinfo, as Eric properly identified the condition and there is
no change I'm aware of to the 2.4 pools which would close them early.
That said, there is no clear fix because those connections are already gone.
The workaround, AIUI (as noted in comment 3) is to ensure the backend keepalive
timeouts (and intervening firewalls, routers and load balancers) are always
longer than httpd's proxy keepalive timeouts. Remaining _WAIT connections
should be small enough in number to handle gracefully. Better yet; since proxy
workers should never need recycling, use an unlimited keepalive window where
sensible.
Perhaps a proxy pool management thread could park unused workers and respond to
the signalled poll condition.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 50807] mod_proxy fails to send FIN response when a FIN is
received from a backend
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=50807
Matt Lockwood <md...@gmail.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |mdlockwood@gmail.com
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 50807] mod_proxy fails to send FIN response when a
FIN is received from a backend
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=50807
--- Comment #3 from Gregory Boyce <gr...@gmail.com> 2011-02-18 14:46:32 EST ---
Ah, I see. It appears that Apache does eventually send a FIN packet the next
time that the specific process attempts to establish a new connection. Until
that point, both ends of the connection are left in a half closed state until
either the next request triggers the FIN and a subsequent SYN, or the servers
timeout the connection.
This ended up biting us when a load balancer had a very long timeout value (30
minutes) and we were apparently not sending enough requests through in order to
flush out the FINs. Eventually a request would attempt to reuse the same
source port and the load balancer wouldn't respond to the SYN requests since it
was still waiting on a FIN from us on that source port.
We've worked around it for now by dropping the timeout on the load balancer.
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 50807] mod_proxy fails to send FIN response when a FIN is
received from a backend
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=50807
--- Comment #10 from Orion Poplawski <or...@cora.nwra.com> ---
Adam's not in comment #5 applies to just standard systems as well where the
net.ipv4.tcp_fin_timeout defaults to 60 seconds. So when apache tries to send
the final FIN, we get a flood of FIN and RST packets due to tcp retransmission:
29 5.076937000 BACKEND -> PROXY TCP 66 8141 > 59404 [FIN, ACK] Seq=7266
Ack=2506 Win=35968 Len=0 TSval=372987155 TSecr=373232386
30 5.116993000 PROXY -> BACKEND TCP 66 59404 > 8141 [ACK] Seq=2506 Ack=7267
Win=32000 Len=0 TSval=373237391 TSecr=372987155
616 168.070814000 PROXY -> BACKEND TCP 66 59404 > 8141 [FIN, ACK] Seq=2506
Ack=7267 Win=32000 Len=0 TSval=373400344 TSecr=372987155
617 168.070868000 BACKEND -> PROXY TCP 54 8141 > 59404 [RST] Seq=7267 Win=0
Len=0
632 168.274981000 PROXY -> BACKEND TCP 66 [TCP Retransmission] 59404 > 8141
[FIN, ACK] Seq=2506 Ack=7267 Win=32000 Len=0 TSval=373400549 TSecr=372987155
633 168.275009000 BACKEND -> PROXY TCP 54 8141 > 59404 [RST] Seq=7267 Win=0
Len=0
634 168.685140000 PROXY -> BACKEND TCP 66 [TCP Retransmission] 59404 > 8141
[FIN, ACK] Seq=2506 Ack=7267 Win=32000 Len=0 TSval=373400959 TSecr=372987155
635 168.685196000 BACKEND -> PROXY TCP 54 8141 > 59404 [RST] Seq=7267 Win=0
Len=0
636 169.504941000 PROXY -> BACKEND TCP 66 [TCP Retransmission] 59404 > 8141
[FIN, ACK] Seq=2506 Ack=7267 Win=32000 Len=0 TSval=373401779 TSecr=372987155
637 169.504968000 BACKEND -> PROXY TCP 54 8141 > 59404 [RST] Seq=7267 Win=0
Len=0
640 171.144951000 PROXY -> BACKEND TCP 66 [TCP Retransmission] 59404 > 8141
[FIN, ACK] Seq=2506 Ack=7267 Win=32000 Len=0 TSval=373403419 TSecr=372987155
641 171.144982000 BACKEND -> PROXY TCP 54 8141 > 59404 [RST] Seq=7267 Win=0
Len=0
644 174.454315000 PROXY -> BACKEND TCP 66 [TCP Retransmission] 59404 > 8141
[FIN, ACK] Seq=2506 Ack=7267 Win=32000 Len=0 TSval=373406728 TSecr=372987155
645 174.454348000 BACKEND -> PROXY TCP 54 8141 > 59404 [RST] Seq=7267 Win=0
Len=0
673 181.013929000 PROXY -> BACKEND TCP 66 [TCP Retransmission] 59404 > 8141
[FIN, ACK] Seq=2506 Ack=7267 Win=32000 Len=0 TSval=373413288 TSecr=372987155
674 181.013955000 BACKEND -> PROXY TCP 54 8141 > 59404 [RST] Seq=7267 Win=0
Len=0
732 194.133990000 PROXY -> BACKEND TCP 66 [TCP Retransmission] 59404 > 8141
[FIN, ACK] Seq=2506 Ack=7267 Win=32000 Len=0 TSval=373426408 TSecr=372987155
733 194.134013000 BACKEND -> PROXY TCP 54 8141 > 59404 [RST] Seq=7267 Win=0
Len=0
877 220.374086000 PROXY -> BACKEND TCP 66 [TCP Retransmission] 59404 > 8141
[FIN, ACK] Seq=2506 Ack=7267 Win=32000 Len=0 TSval=373452648 TSecr=372987155
878 220.374149000 BACKEND -> PROXY TCP 54 8141 > 59404 [RST] Seq=7267 Win=0
Len=0
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org