You are viewing a plain text version of this content. The canonical link for it is here.
Posted to bugs@httpd.apache.org by bu...@apache.org on 2021/06/02 16:26:16 UTC
[Bug 65348] New: mod_ratelimit of proxied content aborts with long
fetches (over 20 seconds)
https://bz.apache.org/bugzilla/show_bug.cgi?id=65348
Bug ID: 65348
Summary: mod_ratelimit of proxied content aborts with long
fetches (over 20 seconds)
Product: Apache httpd-2
Version: 2.4.38
Hardware: PC
OS: Linux
Status: NEW
Severity: normal
Priority: P2
Component: mod_ratelimit
Assignee: bugs@httpd.apache.org
Reporter: bugs@m8y.org
Target Milestone: ---
Debian (well, Devuan, but same package) stable Apache httpd 2.4.38
Using settings below to simulate a fetch that is about 50 seconds with this
data which is ~2,700,000 bytes long
SetOutputFilter RATE_LIMIT
SetEnv rate-limit 50
SetEnv rate-initial-burst 50
ProxyPass "/apitest" "http://localhost:8085/api"
If I run in wget I get:
2021-06-02 12:16:11 (50.0 KB/s) - Connection closed at byte 2605056. Retrying.
(it then tries infinitely since the remote doesn't support partial)
If I try in Firefox it just aborts with a partial read.
I get the following error in the apache httpd error log.
[Tue Jun 01 17:02:44.531073 2021] [proxy_http:error] [pid 58736:tid
140291765540608] (70008)Partial results are valid but processing is incomplete:
[client 12.34.56.78:60626] AH01110: error reading response
I tried trace6 but didn't see anything else that was particularly interesting.
I've tried the following settings:
Timeout 300
ProxyTimeout 300
SetEnv proxy-sendchunked 1
SetEnv proxy-sendchunks 1 (how are these not default O_o)
SetEnv proxy-interim-request RFC
On the ProxyPass
connectiontimeout=300 timeout=300 disablereuse=On enablereuse=Off
I've confimed that *ALL* of the following work with no issues:
wget --limit-rate=15k http://localhost:8085/api/json (direct connection to
local API)
wget --limit-rate=15k https://server/apitest/json (rate limit on client
side, apache proxy pass, NO rate limit set in apache)
wget https://server/static.json (Rate limit of 50 + static /var/www/html
file)
Only the combination of a rate_limit plus mod_proxy seem to break anything.
If I increase the rate-limit to 200 so that the content fetches in under 15
seconds I no longer get errors.
If I leave the rate-limit at 50 but I reduce the content fetched so it fetches
more quickly (under 15 seconds in this case) I no longer get errors.
It only seems to abort if the content fetches in ~20s or more.
I hunted for related bugs and did find bug #62362 which involved mod proxy and
ratelimit and mentioned a significant recent refactor. I don't know if it is
related in any way though.
--------------------------------1494138182410193280
Content-Disposition: form-data; name="data"; filename=""
Content-Type: application/octet-stream
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 65348] mod_ratelimit of proxied content aborts with long
fetches (over 20 seconds)
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=65348
--- Comment #2 from nemo <bu...@m8y.org> ---
So, I just tried using netcat:
{ echo -ne "HTTP/1.0 200 OK\r\nContent-Type:
application/json;utf-8\r\nContent-Length: $(wc -c <test.json)\r\n\r\n"; cat
test.json; } | nc -l -s 127.0.0.1 -p 8080
And this *ALSO* worked fine...
So, that eliminates mod_proxy I think.
This *only* seems to impact the combination of the actual endpoint (rocket 0.4)
and rate_limit.
wget --limit-rate does not seem to cause issues.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 65348] mod_ratelimit of proxied content aborts with long
fetches (over 20 seconds)
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=65348
--- Comment #10 from nemo <bu...@m8y.org> ---
Final update. I've been moved off onto other emergencies so won't be able to
test my "Apache RATE_LIMIT is suddenly halting all traffic towards the end of a
large transfer" theory.
I also noticed one additional bug with Apache RATE_LIMIT that I may have time
to write up properly someday.
If the proxied server responds with 304 plus an empty body (as it does with
If-None-Match of an existing ETag, I get an invalid request response with
RATE_LIMIT enabled).
========
Read error (The request is invalid.) in headers.
Retrying.
========
This happens in all browsers/clients - above is wget. No errors with
rate_limit disabled.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 65348] mod_ratelimit of proxied content aborts with long
fetches (over 20 seconds)
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=65348
--- Comment #4 from nemo <bu...@m8y.org> ---
Hm. write blocks? Can you elaborate? This is a GET, BTW, not a POST. So I'm
just fetching data from the backend, not writing to it.
Haven't tried a full dump yet, apart from trace8 which didn't seem unusual.
Actually I was pretty hopeful this was just some odd interaction of mod_proxy
and rate_limit right up until the point where my netcat test worked.
I guess that's the only thing left to try though ☹ (I do so hate poking through
packet traces)
Apart from hunting the Rocket documentation or testing the 0.5 beta..
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 65348] mod_ratelimit of proxied content aborts with long
fetches (over 20 seconds)
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=65348
--- Comment #1 from nemo <bu...@m8y.org> ---
Found one more interesting variant that works fine.
ProxyPass "/apitest" "http://localhost"
That is. Proxy apache to itself. Once again. No issues!
I'm now going to try a second webserver.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 65348] mod_ratelimit of proxied content aborts with long
fetches (over 20 seconds)
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=65348
--- Comment #5 from nemo <bu...@m8y.org> ---
BTW, another variant that failed. I tried pointing ProxyPass at the production
apache.
So. Rate limited test apache → https://prodserver/api →
http://localhost:8085/api
This one failed in the same fashion (any rate limit resulting in a fetch longer
than 20 seconds causing partial fetch).
Once again, curl and wget rate limits against the production server worked
fine. Even with fetch times of several minutes.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 65348] mod_ratelimit of proxied content aborts with long
fetches (over 20 seconds)
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=65348
--- Comment #6 from nemo <bu...@m8y.org> ---
Bit of good news for me - I reproduced it with latest release Rocket v0.4.10
and their reference examples/static_files.
STR (sorry for mercurialisms, I don't care for git's user interface or
featureset at all). Also this assumes rust stable installed. Rustup takes care
of that.
① hg clone git+https://github.com/SergioBenitez/Rocket.git;hg up -r v0.4.10
② cd Rocket/examples/static_files
③ cargo build
④ scp ../../target/debug/static_files testserver:
⑤ ssh testserver
⑥ mkdir static
⑦ cp /var/www/html/test.json static # this is the large 2.7MB json file.
I'm pretty sure this will be a problem with any random large file though. I
will verify with something large and public like a PG text next.
⑧ ./static_files
And, in Apache 2.4.38 I was using:
SetOutputFilter RATE_LIMIT
SetEnv rate-limit 50
SetEnv rate-initial-burst 50
Timeout 120
ProxyTimeout 300
ProxyPass "/apitest" "http://localhost:8000/"
Then just hitting http://testserver/apitest/test.json
Once again, the RATE_LIMITed Apache 2.4.38 caused truncation but a wget or curl
--limit-rate directly to port 8000 was not a problem.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 65348] mod_ratelimit of proxied content aborts with long
fetches (over 20 seconds)
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=65348
--- Comment #3 from Eric Covener <co...@gmail.com> ---
maybe the backend fails if a write blocks? Have you looked at a packet capture
between Apache and the backend?
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 65348] mod_ratelimit of proxied content aborts with long
fetches (over 20 seconds)
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=65348
--- Comment #7 from nemo <bu...@m8y.org> ---
Rocket master (which is probably close to 0.5 release any-day-now) did *NOT*
trigger this issue.
So... this appears to be specific to the combination of Apache's RATE_LIMIT and
Rocket 0.4.x
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 65348] mod_ratelimit of proxied content aborts with long
fetches (over 20 seconds)
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=65348
--- Comment #8 from nemo <bu...@m8y.org> ---
18:16 < SergioB> There are read/write timeouts in 0.4 that aren't present in
0.5.
Disabling those prevented the issue. (read_timeout = 0 write_timeout = 0)
The timeout values by default are 5 seconds. There was no point during
trace6/trace8 where packets took longer than 5 seconds it seems to me... But
perhaps there was some bug in the implementation.
Still odd it only triggered with Apache's RATE_LIMIT and not wget/curl - I
guess that might be the only thing the Apache people might be interested in, in
case the module has some odd behaviour. I know httpd has a long history of
working around issues in other clients/servers ☺
But under circumstances going to guess this is a low priority bug. ☺
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 65348] mod_ratelimit of proxied content aborts with long
fetches (over 20 seconds)
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=65348
--- Comment #9 from nemo <bu...@m8y.org> ---
One more thought. In all the cases where the transaction aborted, it tended to
happen very close to the end of the large file.
Like - 90%.
What if the read_timeout and write_timeout were *not* failing?
What if the Apache RATE_LIMIT has a bug, and overcorrects an accumulated
RATE_LIMIT average rate error towards the end?
Perhaps that's why curl and wget rate limits do not fail.
If so, there may have actually been a 5s stall.
I will try testing this in a bit.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org