You are viewing a plain text version of this content. The canonical link for it is here.
Posted to bugs@httpd.apache.org by bu...@apache.org on 2006/06/21 11:01:41 UTC
DO NOT REPLY [Bug 39849] New: - Issues with Timeout in httpd.conf and large downloads
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG�
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
<http://issues.apache.org/bugzilla/show_bug.cgi?id=39849>.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND�
INSERTED IN THE BUG DATABASE.
http://issues.apache.org/bugzilla/show_bug.cgi?id=39849
Summary: Issues with Timeout in httpd.conf and large downloads
Product: Apache httpd-2
Version: 2.2.2
Platform: Other
OS/Version: Linux
Status: NEW
Severity: normal
Priority: P3
Component: Core
AssignedTo: bugs@httpd.apache.org
ReportedBy: maarten@daemon.be
Hi,
It could be that I've identified a bug in Apache that prevents large
files from being downloaded successfully when a 'lower than default'
Timeout setting is used in httpd.conf. I sent this to the mailing list first
and waited for 6 days, but as there was no reply I'd thought I'd open a bug
report.
My apologies for not yet investigating this issue in more depth and providing a
full patch, but I just don't have the time and thought it would be better to
report it, to help creating a better Apache, than just to ignore the issue.
A roundup of the issue:
- We publish the following file:
115120550 (size) May 22 17:21 Syllable-0.6.1-dev4.zip
When this file is downloaded, the download halts intermediately, at
around 24 MB. From the point of view of the server, it looks like the
download completed succesfully. This is the same on the client side.
This was tested and identified as an issue with Apache 1.3.24, 2.0.58
(both with added modules) and Apache 2.2.2 (clean install per bug
reporting requirements, no additional modules). This was on the same
server, running RedHat 7.1 with Apache installed from source. A test
with another Apache 2.0.58 on a separate server showed the same issue,
but at a different point in the download.
This interruption occurred on one server after approximately 7 minutes,
when the transfer had completed 24 MB. On the other server I tested,
this happened after 9 minutes, with a 20 MB completion rate. Downloads
from faster connections (less than 6 minutes of transfer time) went
smoothly and without interruption.
The issue only occurs when the Timeout parameter in httpd.conf is
lowered from its default of 300 seconds to 60 seconds. Note that this is
a common change, made to either improve performance or mitigate the
impact of certain types of denial of service attacks against the server.
When changing the value back to its default of 300 seconds the download
does complete correctly.
No entry appears in the error log, but a regular 200 request appears in
the access_log:
203.59.90.155 - - [16/Jun/2006:08:19:14 +0000] "GET
/Syllable-0.6.1-dev4.zip HTTP/1.1" 200 115120550
This was tested with Firefox, Internet Explorer and Wget, which all gave
identical results (with wget automatically retrying using a specified
byte-range so it did in fact complete the download).
Apache documentation mentions that Timeout defines the amount of time
Apache will wait for any of the following three things:
- The total amount of time it takes to receive a GET request.
- The amount of time between receipt of TCP packets on a POST or PUT
request.
- The amount of time between ACKs on transmissions of TCP packets in
responses.
I've made a PCAP file of an interrupted transfer:
http://www.daemon.be/apache-failed-down.cap
If you look at this file you can see that everything seems to go fine,
and then suddenly at frame 25947, 7.35 minutes after the connection was
established, the Apache server suddenly sends a packet with the fin,
push and ack flags set. This disrupts the connection.
As such, none of the above conditions apply: (1) a GET request was
received and is being served, (2) I don't really see any of both parties
in the transaction being unresponsive. This shouldn't match the Timeout
value. It appears somewhere in the download code the counter used to
measure this timeout isn't being increased.
If you have issues replicating it with a different file, the original
which caused us issues for quite a while prior us discovering the link
with the Timeout option can be found here:
http://www.cilinder.be/syllable/Syllable-0.6.1-dev4.zip
Cheers,
Maarten
--
Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 39849] - Issues with Timeout in httpd.conf and large downloads
Posted by bu...@apache.org.
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG�
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
<http://issues.apache.org/bugzilla/show_bug.cgi?id=39849>.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND�
INSERTED IN THE BUG DATABASE.
http://issues.apache.org/bugzilla/show_bug.cgi?id=39849
jorton@redhat.com changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |NEEDINFO
------- Additional Comments From jorton@redhat.com 2006-07-18 12:17 -------
Any update?
--
Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 39849] - Issues with Timeout in httpd.conf and large downloads
Posted by bu...@apache.org.
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG�
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
<http://issues.apache.org/bugzilla/show_bug.cgi?id=39849>.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND�
INSERTED IN THE BUG DATABASE.
http://issues.apache.org/bugzilla/show_bug.cgi?id=39849
------- Additional Comments From jorton@redhat.com 2006-06-21 14:09 -------
The poll() call is a 60 second write timeout, which implicates packet
loss/network congestion/network issues of some kind, as would a high send queue
value from netstat. There's nothing here really to indicate anything other than
that.
--
Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 39849] - Issues with Timeout in httpd.conf and large downloads
Posted by bu...@apache.org.
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG�
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
<http://issues.apache.org/bugzilla/show_bug.cgi?id=39849>.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND�
INSERTED IN THE BUG DATABASE.
http://issues.apache.org/bugzilla/show_bug.cgi?id=39849
------- Additional Comments From maarten@daemon.be 2006-06-21 13:21 -------
Hi Joe,
Thanks. Below is the output:
poll([{fd=8, events=POLLOUT}], 1, 60000) = 0
read(8, 0x8167960, 8000) = -1 EAGAIN (Resource temporarily
unavailable)
write(7, "203.59.90.155 - - [21/Jun/2006:1"..., 101) = 101
close(8) = 0
read(4, 0xbffff8e3, 1) = -1 EAGAIN (Resource temporarily
unavailable)
close(9) = 0
accept(3, <unfinished ...>
Despite the <unfinished ...> the process no longer showed up with netstat at the
time that I ended the strace.
You may also be interested in the fact that immediately after the download
starts (of a file which is 109.8 MB in size), the Send-Q value is immediately
set to the size of the part that effectively downloads (so 24 MB).
While doing the "slow" download as I described (60kb/sec) it goes into FIN_WAIT
state shortly after the download starts:
tcp 0 4021417 64.246.44.158:80 203.59.90.155:60185 FIN_WAIT1 -
When doing a "fast" download (900kb/sec), the Send-Q value increases again
automatically and the connection remains alive.
--
Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 39849] - Issues with Timeout in httpd.conf and large downloads
Posted by bu...@apache.org.
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG�
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
<http://issues.apache.org/bugzilla/show_bug.cgi?id=39849>.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND�
INSERTED IN THE BUG DATABASE.
http://issues.apache.org/bugzilla/show_bug.cgi?id=39849
maarten@daemon.be changed:
What |Removed |Added
----------------------------------------------------------------------------
AssignedTo|bugs@httpd.apache.org |maarten@daemon.be
Status|ASSIGNED |NEW
--
Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 39849] - Issues with Timeout in httpd.conf and large downloads
Posted by bu...@apache.org.
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG�
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
<http://issues.apache.org/bugzilla/show_bug.cgi?id=39849>.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND�
INSERTED IN THE BUG DATABASE.
http://issues.apache.org/bugzilla/show_bug.cgi?id=39849
------- Additional Comments From maarten@daemon.be 2006-07-25 17:23 -------
Dear Joe,
My apologies for my late response. In the meanwhile we've tested this again on a
physically different network and indeed, the issue is no longer occuring. As
such, this was likely indeed related to a timeout issue, and Apache was acting
correctly. We're currently reviewing the network setup to pinpoint the exact
issue, but are confident it is not related to the Apache software.
Thank you for your assistance in troubleshooting this issue.
Best regards,
Maarten
--
Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 39849] - Issues with Timeout in httpd.conf and large downloads
Posted by bu...@apache.org.
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG�
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
<http://issues.apache.org/bugzilla/show_bug.cgi?id=39849>.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND�
INSERTED IN THE BUG DATABASE.
http://issues.apache.org/bugzilla/show_bug.cgi?id=39849
chip@force-elite.com changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |NEEDINFO
------- Additional Comments From chip@force-elite.com 2006-06-21 09:25 -------
Have you tried adding the following to your configuration file:
EnableSendfile off
EnableMMAP off
I would guess on such an old version of redhat, it might be a broken sendfile.
--
Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 39849] - Issues with Timeout in httpd.conf and large downloads
Posted by bu...@apache.org.
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG�
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
<http://issues.apache.org/bugzilla/show_bug.cgi?id=39849>.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND�
INSERTED IN THE BUG DATABASE.
http://issues.apache.org/bugzilla/show_bug.cgi?id=39849
jorton@redhat.com changed:
What |Removed |Added
----------------------------------------------------------------------------
AssignedTo|maarten@daemon.be |bugs@httpd.apache.org
------- Additional Comments From jorton@redhat.com 2006-06-21 15:43 -------
[please don't change bug owners, we rely on the default assignment to see updates]
--
Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 39849] - Issues with Timeout in httpd.conf and large downloads
Posted by bu...@apache.org.
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG�
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
<http://issues.apache.org/bugzilla/show_bug.cgi?id=39849>.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND�
INSERTED IN THE BUG DATABASE.
http://issues.apache.org/bugzilla/show_bug.cgi?id=39849
maarten@daemon.be changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEEDINFO |ASSIGNED
------- Additional Comments From maarten@daemon.be 2006-06-21 14:44 -------
Hi Joe,
That makes sense, however it's still strange that it's happening on two servers
simultaneously. To make sure no network issues are involved I'll build a new box
with a default install OS on a clean (local) network to try and replicate this
issue. Will get back to you should it continue to appear.
Thanks again for your assistance.
Best regards,
Maarten
--
Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 39849] - Issues with Timeout in httpd.conf and large downloads
Posted by bu...@apache.org.
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG�
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
<http://issues.apache.org/bugzilla/show_bug.cgi?id=39849>.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND�
INSERTED IN THE BUG DATABASE.
http://issues.apache.org/bugzilla/show_bug.cgi?id=39849
------- Additional Comments From jorton@redhat.com 2006-06-21 12:32 -------
Can you run strace against the httpd child when it is serving this file? The
technique is:
1. start the download
2. run mod_status or "netstat -pant" or similar to find out the pid of the httpd
process handling the download (compare against the client address)
3. run "strace -o /tmp/httpd.trace -p <pid>" using the pid found in step 2
then attach the /tmp/httpd.trace here.
--
Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 39849] - Issues with Timeout in httpd.conf and large downloads
Posted by bu...@apache.org.
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG�
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
<http://issues.apache.org/bugzilla/show_bug.cgi?id=39849>.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND�
INSERTED IN THE BUG DATABASE.
http://issues.apache.org/bugzilla/show_bug.cgi?id=39849
jorton@redhat.com changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEEDINFO |RESOLVED
Resolution| |INVALID
------- Additional Comments From jorton@redhat.com 2006-07-26 11:07 -------
No problem, thanks for reporting back.
--
Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 39849] - Issues with Timeout in httpd.conf and large downloads
Posted by bu...@apache.org.
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG�
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
<http://issues.apache.org/bugzilla/show_bug.cgi?id=39849>.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND�
INSERTED IN THE BUG DATABASE.
http://issues.apache.org/bugzilla/show_bug.cgi?id=39849
------- Additional Comments From maarten@daemon.be 2006-06-21 10:08 -------
Hi Paul,
Thanks for your reply. On the RedHat box, I stopped the daemon, added these into
httpd.conf and restart it. Afterwards tested the download again, but it still
fails. Tested this twice to make sure. Also note that the second machine on
which I tested this was a Debian 3.1.
Cheers,
Maarten
--
Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org