You are viewing a plain text version of this content. The canonical link for it is here.
Posted to cvs@httpd.apache.org by tr...@apache.org on 2004/12/29 21:39:47 UTC
svn commit: r123674 - /httpd/httpd/branches/2.0.x/STATUS
Author: trawick
Date: Wed Dec 29 12:39:46 2004
New Revision: 123674
URL: http://svn.apache.org/viewcvs?view=rev&rev=123674
Log:
does anyone remember this proxy issue?
Modified:
httpd/httpd/branches/2.0.x/STATUS
Modified: httpd/httpd/branches/2.0.x/STATUS
Url: http://svn.apache.org/viewcvs/httpd/httpd/branches/2.0.x/STATUS?view=diff&rev=123674&p1=httpd/httpd/branches/2.0.x/STATUS&r1=123673&p2=httpd/httpd/branches/2.0.x/STATUS&r2=123674
==============================================================================
--- httpd/httpd/branches/2.0.x/STATUS (original)
+++ httpd/httpd/branches/2.0.x/STATUS Wed Dec 29 12:39:46 2004
@@ -260,6 +260,14 @@
+1: stoddard, striker, jim
-1: brianp (we need a more robust solution than what's in 2.1 right now),
jerenkrantz (should be fixed, but I don't have time to do this)
+ trawick: Reviewing the thread in Dec '02, I don't see any
+ proposal about more robust solutions once it was made clear
+ that Connection: Close can't be used. If Justin is correct in
+ his assertion that many servers don't handle TE: chunked, then the
+ current solution is reasonable. If Justin is incorrect in that
+ assertion, then something more robust is possible, such as
+ buffering up to N bytes in hopes of reading the entire body,
+ before deciding to send C-L or T-E: chunked. Alternate recollections?
* Allow mod_dav to do weak entity comparison functions.
modules/dav/main/util.c: r1.45
Re: C-L or T-E: chunked for proxied request bodies
Posted by Jeff Trawick <tr...@gmail.com>.
On Thu, 30 Dec 2004 13:54:05 -0500, Jeff Trawick <tr...@gmail.com> wrote:
> An interesting issue is that what is in 2.0 now is optimal when it
> works, and it works in a common real-world scenario (client sends C-L
> and no filters modify the request body size). At present, we don't
> have any proposed solution which would preserve this common, optimal
> situation. That will be a painful regression for some, since the
> request body will be buffered in memory in order to re-calculate the
> C-L.
Here is a patch which preserves the C-L setting and streams the
request body when the client specified C-L and there are no input
filters which could possibly muck with the request body. (Some
filters might be in place which don't modify the request body length,
but we can't check that. An envvar could be used to indicate that it
is safe to preserve the C-L even though there is an input filter.)
Re: C-L or T-E: chunked for proxied request bodies
Posted by Nick Kew <ni...@webthing.com>.
On Thu, 30 Dec 2004, Jeff Trawick wrote:
> 1. special case:
> if client specified C-L and either C-L is zero or there are no
> input filters or some magic envvar is set to indicate that filters
> don't change the size:
> pass through the C-L header and stream the request body, if any
>
> 2. normal:
> read enough request body to see if there is more than 32K of
> request body...
> if <= 32K:
> send the C-L and the small request body;
> else if some magic variable is set to force calculation of C-L:
> get temporary file;
> read entire request body into file;
> send calculated C-L;
> send request body from the file;
> else:
> use chunked encoding and stream the request body
Hey, I was just drafting a reply in this thread, but you beat me to it:-)
+1 to the above.
> > Will there really remain no active filters in the common case?
>
> strictly speaking, there are always filters; but we're just worried
> about filters which can modify the request body; it is common to have
> a configuration in which there are no filters which modify the request
> body...
This is essentially what the protocol handling in mod_filter offers
for output filters. I wonder if you've just made the case (always
"potentially useful/interesting") to extend mod_filter to input filters?
--
Nick Kew
Re: C-L or T-E: chunked for proxied request bodies
Posted by Jeff Trawick <tr...@gmail.com>.
On Thu, 30 Dec 2004 21:12:16 +0100, Jan Kratochvil
<rc...@jankratochvil.net> wrote:
> Hi,
>
> On Thu, 30 Dec 2004 19:54:05 +0100, Jeff Trawick wrote:
> > On Thu, 30 Dec 2004 18:47:48 +0100, Jan Kratochvil
> > <rc...@jankratochvil.net> wrote:
> ...
> > An interesting issue is that what is in 2.0 now is optimal when it
> > works, and it works in a common real-world scenario (client sends C-L
> > and no filters modify the request body size).
>
> Sorry for chatting without code sample etc.
>
> * If no active filters are detected after headers parsing pass the
> single request unchanged through this connection by its C-L despite
> its possible "keepalive".
that's the goal of the patch I posted earlier this afternoon; I didn't
consider keepalive setting and hope it is irrelevant ;)
> * Use currently 2.1-implemented C-E "chunked" as fallback.
for the very short term, I agree; if my prior patch to try to preserve
some of the optimal handling in 2.0 is acceptable, then we could fall
back to chunked
for the long term, here is my current understanding of a better solution.
1. special case:
if client specified C-L and either C-L is zero or there are no
input filters or some magic envvar is set to indicate that filters
don't change the size:
pass through the C-L header and stream the request body, if any
2. normal:
read enough request body to see if there is more than 32K of
request body...
if <= 32K:
send the C-L and the small request body;
else if some magic variable is set to force calculation of C-L:
get temporary file;
read entire request body into file;
send calculated C-L;
send request body from the file;
else:
use chunked encoding and stream the request body
> Will there really remain no active filters in the common case?
strictly speaking, there are always filters; but we're just worried
about filters which can modify the request body; it is common to have
a configuration in which there are no filters which modify the request
body...
> At least ap_byterange_filter() gives up but chunk_filter() never does.
> Is it possible to patch such filters? It would be performance wise. :-)
those two filters aren't used on input, and they aren't used when
forwarding the request to the origin server
Re: C-L or T-E: chunked for proxied request bodies
Posted by Jan Kratochvil <rc...@jankratochvil.net>.
Hi,
On Thu, 30 Dec 2004 19:54:05 +0100, Jeff Trawick wrote:
> On Thu, 30 Dec 2004 18:47:48 +0100, Jan Kratochvil
> <rc...@jankratochvil.net> wrote:
...
> An interesting issue is that what is in 2.0 now is optimal when it
> works, and it works in a common real-world scenario (client sends C-L
> and no filters modify the request body size).
Sorry for chatting without code sample etc.
* If no active filters are detected after headers parsing pass the
single request unchanged through this connection by its C-L despite
its possible "keepalive".
* Use currently 2.1-implemented C-E "chunked" as fallback.
Will there really remain no active filters in the common case?
At least ap_byterange_filter() gives up but chunk_filter() never does.
Is it possible to patch such filters? It would be performance wise. :-)
Regards,
Lace
Re: C-L or T-E: chunked for proxied request bodies
Posted by Jeff Trawick <tr...@gmail.com>.
On Thu, 30 Dec 2004 18:47:48 +0100, Jan Kratochvil
<rc...@jankratochvil.net> wrote:
> On Thu, 30 Dec 2004 14:05:30 +0100, Jeff Trawick wrote:
> > On 29 Dec 2004 20:39:47 -0000, trawick@apache.org <tr...@apache.org> wrote:
> ...
> > Any agreement to use the non-DOS-able and protocol-compliant
> > implementation by default, and allow the C-L method to be enabled by a
> > special setting?
>
> Is it going to be backported to 2.0 or it waits for 2.2?
It is very likely that whatever ends up in 2.2, possibly different
than in current snapshots, will be backported. The only reason it
hasn't been backported yet is that there are some concerns with what
is in the current snapshots. And such concerns surely apply to 2.2 as
well, so something has to be done ;)
An interesting issue is that what is in 2.0 now is optimal when it
works, and it works in a common real-world scenario (client sends C-L
and no filters modify the request body size). At present, we don't
have any proposed solution which would preserve this common, optimal
situation. That will be a painful regression for some, since the
request body will be buffered in memory in order to re-calculate the
C-L.
--/--
Oddity in the current 2.1 snapshot solution: If you set the
proxy-sendchunked variable, then the origin server will always get a
request body, even if the client didn't send one. A simple GET from
the client turns into this:
G E T / s i l l y / / s e n d c h u n k s / ? r e a d _ b o d
y H T T P / 1 . 1\r\n H o s t : 1 2 7 . 0 . 0 . 1 : 8 0 8 1
\r\n M a x - F o r w a r d s : 1 0\r\n C o n n e c t i o n :
c l o s e\r\n X - F o r w a r d e d - F o r : 1 2 7 . 0 . 0 .
1\r\n X - F o r w a r d e d - H o s t : 1 2 7 . 0 . 0 . 1\r\n
X - F o r w a r d e d - S e r v e r : 1 9 2 . 1 6 8 . 1 . 1 1
\r\n T r a n s f e r - E n c o d i n g : c h u n k e d\r\n\r\n
0\r\n\r\n
Re: C-L or T-E: chunked for proxied request bodies
Posted by Jan Kratochvil <rc...@jankratochvil.net>.
Hi Jeff,
On Thu, 30 Dec 2004 14:05:30 +0100, Jeff Trawick wrote:
> On 29 Dec 2004 20:39:47 -0000, trawick@apache.org <tr...@apache.org> wrote:
...
> Any agreement to use the non-DOS-able and protocol-compliant
> implementation by default, and allow the C-L method to be enabled by a
> special setting?
Is it going to be backported to 2.0 or it waits for 2.2?
Thanks,
Lace
Re: C-L or T-E: chunked for proxied request bodies
Posted by Dirk-Willem van Gulik <di...@webweaving.org>.
On Sat, 1 Jan 2005, Justin Erenkrantz wrote:
> Apache 1.3 does not support chunked request bodies.
Eh - I am fairly sure it does (and has for quite a while) -except- for
things whcih do not know their length - such as life output from CGI.
> (All 2.x GA releases do though.) So, that factors very much in the
> reason that I don't think we should send chunks by default: all requests
Dw.
Re: C-L or T-E: chunked for proxied request bodies
Posted by Justin Erenkrantz <ju...@erenkrantz.com>.
On Tue, Jan 04, 2005 at 02:35:25PM -0500, Jeff Trawick wrote:
> does this look correct?
>
> svn copy https://svn.apache.org/repos/asf/httpd/httpd/trunk \
> https://svn.apache.org/repos/asf/httpd/httpd/branches/proxy-reqbody
Yup. -- justin
Re: C-L or T-E: chunked for proxied request bodies
Posted by Jeff Trawick <tr...@gmail.com>.
On Tue, 04 Jan 2005 09:45:13 -0800, Justin Erenkrantz
<ju...@erenkrantz.com> wrote:
> --On Tuesday, January 4, 2005 8:48 AM -0500 Jeff Trawick
> <tr...@gmail.com> wrote:
>
> > Here is current patch-in-progress. I'd guess there many bugs. I
> > don't mean to waste anyone's time with an in-progress mess, but any
> > comments on the current state would be very valuable.
>
> How about creating a branch in Subversion? You can check in the patch
> there so that we can review it and make improvements to it there. I think
> we're mostly in agreement that we'll change our current behavior.
does this look correct?
svn copy https://svn.apache.org/repos/asf/httpd/httpd/trunk \
https://svn.apache.org/repos/asf/httpd/httpd/branches/proxy-reqbody
Re: C-L or T-E: chunked for proxied request bodies
Posted by Justin Erenkrantz <ju...@erenkrantz.com>.
--On Tuesday, January 4, 2005 8:48 AM -0500 Jeff Trawick
<tr...@gmail.com> wrote:
> Here is current patch-in-progress. I'd guess there many bugs. I
> don't mean to waste anyone's time with an in-progress mess, but any
> comments on the current state would be very valuable.
How about creating a branch in Subversion? You can check in the patch
there so that we can review it and make improvements to it there. I think
we're mostly in agreement that we'll change our current behavior.
I've always disliked patch iterations on a mailing list: we have a cool
version control system with cheap branches. Let's use it! =) -- justin
Re: C-L or T-E: chunked for proxied request bodies
Posted by Jeff Trawick <tr...@gmail.com>.
On Sun, 2 Jan 2005 15:28:54 -0500, Jeff Trawick <tr...@gmail.com> wrote:
> On Sun, 2 Jan 2005 11:31:36 -0800 (PST), Justin Erenkrantz
> <ju...@erenkrantz.com> wrote:
>
> > Turning on sendchunks by default is going to result in broken behavior on
> > all sorts of popular sites:
>
> I believe you, though there are plenty of chunks-are-beautiful
> scenarios as well. Also, there is a limit to regression since new
> logic can preserve the existing C-L and stream the request body when
> there is no input filter.
>
> I don't mind greatly changing the pseudo-code I mentioned earlier to
> default to spooling the request body in order to compute the C-L. The
> bulk of the logic should be able to handle either preference. But I
> will add some logic to check if the client sent us chunks, in which
> case we'll send the origin server chunks as well. (I really want to
> avoid the spooling if at all possible.)
>
> FWIW, it seems likely that I'll have a barely tested patch in the next 24 hours.
Here is current patch-in-progress. I'd guess there many bugs. I
don't mean to waste anyone's time with an in-progress mess, but any
comments on the current state would be very valuable.
Thanks!
Re: C-L or T-E: chunked for proxied request bodies
Posted by Jeff Trawick <tr...@gmail.com>.
On Sun, 2 Jan 2005 11:31:36 -0800 (PST), Justin Erenkrantz
<ju...@erenkrantz.com> wrote:
> Turning on sendchunks by default is going to result in broken behavior on
> all sorts of popular sites:
I believe you, though there are plenty of chunks-are-beautiful
scenarios as well. Also, there is a limit to regression since new
logic can preserve the existing C-L and stream the request body when
there is no input filter.
I don't mind greatly changing the pseudo-code I mentioned earlier to
default to spooling the request body in order to compute the C-L. The
bulk of the logic should be able to handle either preference. But I
will add some logic to check if the client sent us chunks, in which
case we'll send the origin server chunks as well. (I really want to
avoid the spooling if at all possible.)
FWIW, it seems likely that I'll have a barely tested patch in the next 24 hours.
Re: C-L or T-E: chunked for proxied request bodies
Posted by Justin Erenkrantz <ju...@erenkrantz.com>.
> On Sat, 1 Jan 2005, Roy T.Fielding wrote:
>
>> On Jan 1, 2005, at 1:38 PM, Justin Erenkrantz wrote:
>> > Apache 1.3 does not support chunked request bodies. (All 2.x GA
>> > releases
>> > do though.) So, that factors very much in the reason that I don't
>> > think
>> > we should send chunks by default: all requests to 1.3.x would receive
>> > some 4xx (413??) error if we tried to pass chunks. -- justin
>>
>> Umm, yes it does support chunked request bodies -- I wrote that code
>> long before 2.0 even started. Only the CGI module doesn't support it
>> because CGI depends on knowing the length in advance.
>
> Exactly... it definitely does support chunked request bodies. Remember
> CVE-2002-0392?
Well, it depends upon which httpd modules are installed. For example, a
server with PHP installed will break because PHP forbids chunking:
The offending code is at line 595 of mod_php5.c:
http://cvs.php.net/co.php/php-src/sapi/apache/mod_php5.c?r=1.12:
...
if ((retval = setup_client_block(r, REQUEST_CHUNKED_ERROR))) {
...
Here's a POST request against php.net:
% telnet www.php.net 80
Trying 64.246.30.37...
Connected to rs1.php.net.
Escape character is '^]'.
POST /submit-event.php HTTP/1.1
Host: www.php.net
Transfer-Encoding: chunked
HTTP/1.1 411 Length Required
Date: Sun, 02 Jan 2005 19:19:35 GMT
Server: Apache/1.3.26 (Unix) mod_gzip/1.3.26.1a PHP/4.3.3-dev
Connection: close
Transfer-Encoding: chunked
Content-Type: text/html; charset=iso-8859-1
11e
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<HTML><HEAD>
<TITLE>411 Length Required</TITLE>
</HEAD><BODY>
<H1>Length Required</H1>
A request of the requested method POST requires a valid Content-length.<P>
chunked Transfer-Encoding forbidden: /submit-event.php<P>
</BODY></HTML>
0
Connection closed by foreign host.
And, here's a beauty:
% telnet www.microsoft.com 80
Trying 207.46.250.222...
Connected to origin2.microsoft.com.
Escape character is '^]'.
GET / HTTP/1.1
Host: www.microsoft.com
Transfer-Encoding: chunked
HTTP/1.1 404 Not Found
Content-Length: 103
Content-Type: text/html
Server: Microsoft-IIS/6.0
Date: Sun, 02 Jan 2005 19:13:28 GMT
Connection: close
<html><head><title>Error</title></head><body>The system cannot find the
file specified.
</body></html>
---
Turning on sendchunks by default is going to result in broken behavior on
all sorts of popular sites: anything with PHP or IIS installed. Perhaps
it might be worth it to do a 'retry' mechanism: try with the T-E: chunked
and if we get a 4xx response, then try with C-L. I don't know how that'd
work though.
But, chunked request bodies don't work in practice. -- justin
Re: C-L or T-E: chunked for proxied request bodies
Posted by Cliff Woolley <jw...@virginia.edu>.
On Sat, 1 Jan 2005, Roy T.Fielding wrote:
> On Jan 1, 2005, at 1:38 PM, Justin Erenkrantz wrote:
> > Apache 1.3 does not support chunked request bodies. (All 2.x GA
> > releases
> > do though.) So, that factors very much in the reason that I don't
> > think
> > we should send chunks by default: all requests to 1.3.x would receive
> > some 4xx (413??) error if we tried to pass chunks. -- justin
>
> Umm, yes it does support chunked request bodies -- I wrote that code
> long before 2.0 even started. Only the CGI module doesn't support it
> because CGI depends on knowing the length in advance.
Exactly... it definitely does support chunked request bodies. Remember
CVE-2002-0392?
--Cliff
Re: C-L or T-E: chunked for proxied request bodies
Posted by "Roy T. Fielding" <fi...@gbiv.com>.
On Jan 1, 2005, at 1:38 PM, Justin Erenkrantz wrote:
> Apache 1.3 does not support chunked request bodies. (All 2.x GA
> releases
> do though.) So, that factors very much in the reason that I don't
> think
> we should send chunks by default: all requests to 1.3.x would receive
> some 4xx (413??) error if we tried to pass chunks. -- justin
Umm, yes it does support chunked request bodies -- I wrote that code
long before 2.0 even started. Only the CGI module doesn't support it
because CGI depends on knowing the length in advance.
....Roy
Re: C-L or T-E: chunked for proxied request bodies
Posted by Justin Erenkrantz <ju...@erenkrantz.com>.
Jeff wrote:
> Any agreement to use the non-DOS-able and protocol-compliant
> implementation by default, and allow the C-L method to be enabled by a
> special setting? Even if somebody reworks the C-L method to be able
> to buffer on disk, that is wasted effort (and still a DOS concern) in
> many cases.
Apache 1.3 does not support chunked request bodies. (All 2.x GA releases
do though.) So, that factors very much in the reason that I don't think
we should send chunks by default: all requests to 1.3.x would receive
some 4xx (413??) error if we tried to pass chunks. -- justin
C-L or T-E: chunked for proxied request bodies
Posted by Jeff Trawick <tr...@gmail.com>.
On 29 Dec 2004 20:39:47 -0000, trawick@apache.org <tr...@apache.org> wrote:
> Author: trawick
> Date: Wed Dec 29 12:39:46 2004
> New Revision: 123674
>
> URL: http://svn.apache.org/viewcvs?view=rev&rev=123674
> Log:
> does anyone remember this proxy issue?
>
> Modified:
> httpd/httpd/branches/2.0.x/STATUS
>
> Modified: httpd/httpd/branches/2.0.x/STATUS
> Url: http://svn.apache.org/viewcvs/httpd/httpd/branches/2.0.x/STATUS?view=diff&rev=123674&p1=httpd/httpd/branches/2.0.x/STATUS&r1=123673&p2=httpd/httpd/branches/2.0.x/STATUS&r2=123674
> ==============================================================================
> --- httpd/httpd/branches/2.0.x/STATUS (original)
> +++ httpd/httpd/branches/2.0.x/STATUS Wed Dec 29 12:39:46 2004
> @@ -260,6 +260,14 @@
> +1: stoddard, striker, jim
> -1: brianp (we need a more robust solution than what's in 2.1 right now),
> jerenkrantz (should be fixed, but I don't have time to do this)
> + trawick: Reviewing the thread in Dec '02, I don't see any
> + proposal about more robust solutions once it was made clear
> + that Connection: Close can't be used. If Justin is correct in
> + his assertion that many servers don't handle TE: chunked, then the
> + current solution is reasonable. If Justin is incorrect in that
> + assertion, then something more robust is possible, such as
> + buffering up to N bytes in hopes of reading the entire body,
> + before deciding to send C-L or T-E: chunked. Alternate recollections?
Sleep helps. Consuming arbitrary amounts of memory to gather the
entire request body in order to generate C-L could certainly be called
non-robust ;)
Any agreement to use the non-DOS-able and protocol-compliant
implementation by default, and allow the C-L method to be enabled by a
special setting? Even if somebody reworks the C-L method to be able
to buffer on disk, that is wasted effort (and still a DOS concern) in
many cases.