You are viewing a plain text version of this content. The canonical link for it is here.
Posted to bugs@httpd.apache.org by bu...@apache.org on 2018/05/08 20:41:26 UTC
[Bug 62362] New: Proxied content not properly rate-limited
https://bz.apache.org/bugzilla/show_bug.cgi?id=62362
Bug ID: 62362
Summary: Proxied content not properly rate-limited
Product: Apache httpd-2
Version: 2.4.33
Hardware: PC
OS: Mac OS X 10.1
Status: NEW
Severity: normal
Priority: P2
Component: mod_ratelimit
Assignee: bugs@httpd.apache.org
Reporter: toscano.luca@gmail.com
Target Milestone: ---
On users@ it was reported the following scenario:
"""
I'm using Apache 2.4.24 on Debian 9 Stable, behind a DSL connection, with an
estimated upload capacity of ~130kB/s.
I'm trying to limit the bandwidth available to my users (per-connection limit
is fine).
However, it seems to me that the rate-limit parameter is coarsely grained :
- if I set it to 8, users are limited to 8 kB/s
- if I set it to 20, or 30, users are limited to 40 kB/s
- if I set it to 50, 60 or 80, users are limited to my BW, so ~120 kB/s
"""
After following up with the user it seems that the issue happens with proxied
content. So I've set up the following experiment:
- Directory with a 4MB file inside
- Simple Location that proxies content via mod_proxy_http to a Python process
running a webserver, capable of returning the same 4MB file outlined above.
I tested the rate limit using curl's summary (average Dload speed for example).
This is what I gathered:
- when httpd serves the file directly, mod_ratelimit's output filter is called
once and the bucket brigade contains all the data contained in the file. This
is probably due to how bucket brigates work when morphing a file content?
- when httpd serves the file via mod_proxy, the output filter is called
multiple times, and each time the buckets are maximum the size of
ProxyIOBufferSize (8192 by default). Still not completely sure about this one,
so please let me know if I am totally wrong :)
The main problem is, IIUC, in the output's filter logic that does this: it
calculates the size of a chunk, based on the rate-limit set in the httpd's
conf, and then it splits the bucket brigade, if necessary, in buckets of that
chunk size, interleaving them with FLUSH buckets (and sleeping 200ms).
So a trace of execution with say a chunk size of 8192 would be something like:
First call of the filter: 8192 --> FLUSH --> sleep(200ms) --> 8192 --> ... ->
last chunk (either 8192 or something less).
This happens correctly when httpd serves directly the content, but not when
proxied:
First call of the filter: 8192 -> FLUSH (no sleep, since do_sleep turns to 1
only after the first flush)
Second call of the filter: 8192 -> FLUSH (no sleep)
...
So one way to alleviate this issue is to move do_sleep to the ctx data
structure, so if the filter gets called multiple times it will "remember" to
sleep between flushes (with the assumption that it is allocated for each
request). It remains the problem that when the rate-limit speed sets a chunk
size different than the ProxyIOBufferSize (8192 by default) then the client
will be rate limited to the speed dictated by the buffer size (for example,
8192 should correspond to ~40KB/s).
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 62362] Proxied content not properly rate-limited
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=62362
--- Comment #4 from Luca Toscano <to...@gmail.com> ---
Committed r1833875 in httpd trunk, feedback/testing is welcome!
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 62362] Proxied content not properly rate-limited
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=62362
Luca Toscano <to...@gmail.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Resolution|--- |FIXED
Status|NEW |RESOLVED
--- Comment #5 from Luca Toscano <to...@gmail.com> ---
Backported to 2.4.x in http://svn.apache.org/r1835168, the fix will be part of
2.4.34.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 62362] Proxied content not properly rate-limited
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=62362
Luca Toscano <to...@gmail.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Attachment #35950|0 |1
is obsolete| |
--- Comment #3 from Luca Toscano <to...@gmail.com> ---
Created attachment 35957
--> https://bz.apache.org/bugzilla/attachment.cgi?id=35957&action=edit
httpd-trunk-mod_ratelimit-rate_limit_proxied_content.patch
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 62362] Proxied content not properly rate-limited
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=62362
--- Comment #6 from Luca Toscano <to...@gmail.com> ---
For posterity: this change lead to
https://bz.apache.org/bugzilla/show_bug.cgi?id=62568
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 62362] Proxied content not properly rate-limited
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=62362
--- Comment #2 from Luca Toscano <to...@gmail.com> ---
The attached patch seems to work on my test case scenarios. What I did was:
* created a ~4M text file containing lines like TEST 0, TEST 1, TEST 2, etc..
* served the file directly from httpd (via Directory config) and also via
Python 3's http server and proxied via mod_proxy_http.
* calculated the sha1sum of the files generated redirecting curl's output to a
file.
* noted time spent in both cases to rate limit the content with various rate
limit speeds (via curl's average dload speed).
Everything seems consistent and fine as far as I can see, the patch seems to
work as expected. This is a rather invasive refactoring of the rate-limit
filter so I might have missed something, testing/feedback/suggestions/reviews
are really welcome :)
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 62362] Proxied content not properly rate-limited
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=62362
Luca Toscano <to...@gmail.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Keywords| |FixedInTrunk
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 62362] Proxied content not properly rate-limited
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=62362
--- Comment #1 from Luca Toscano <to...@gmail.com> ---
Created attachment 35950
--> https://bz.apache.org/bugzilla/attachment.cgi?id=35950&action=edit
httpd-trunk-mod_ratelimit-rate_limit_proxied_content.patch
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org