You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Ewald Dieterich <ew...@mailbox.org> on 2015/01/08 10:38:56 UTC

Reverse proxy: invalid Content-Length leads to 413 + 400 errors mixed up

I set up a simple reverse proxy with Apache 2.4.10 on Debian unstable:

ProxyPass / http://backend/
ProxyPassreverse / http://backend/

When I send a request to the reverse proxy with an invalid 
Content-Length header, I get two 413 and 400 response bodies concatenated:

$ curl -i -H "Content-Length: a" http://frontend/
HTTP/1.1 413 Request Entity Too Large
Date: Thu, 08 Jan 2015 09:00:33 GMT
Server: Apache/2.4.10 (Debian)
Connection: close
Content-Type: text/html; charset=iso-8859-1

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>413 Request Entity Too Large</title>
</head><body>
<h1>Request Entity Too Large</h1>
The requested resource<br />/<br />
does not allow request data with GET requests, or the amount of data 
provided in
the request exceeds the capacity limit.
<hr>
<address>Apache/2.4.10 (Debian) Server at frontend Port 80</address>
</body></html>
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>400 Bad Request</title>
</head><body>
<h1>Bad Request</h1>
<p>Your browser sent a request that this server could not understand.<br />
</p>
<hr>
<address>Apache/2.4.10 (Debian) Server at frontend Port 80</address>
</body></html>

Even though the HTTP status code in the response is 413, the access log 
shows a 400 error:

[...] "GET / HTTP/1.1" 400 904 "-" "curl/7.26.0"

I think that first a 413 error is created that later gets replaced 
partly by a 400 error.

Here are some log entries:

[...] AH01587: Invalid Content-Length

=> That's where a 413 (HTTP_REQUEST_ENTITY_TOO_LARGE) is created.

[...] (-102)Unknown error -102: [client 10.128.128.95:46000] AH01095: 
prefetch request body failed to 10.8.19.114:80 (frontend) from 
10.128.128.95 ()

=> That's were where a 400 (HTTP_BAD_REQUEST) is returned.

Any ideas how to fix this so that this situation is handled as a single 
error and not as two errors mixed up?

Re: Reverse proxy: invalid Content-Length leads to 413 + 400 errors mixed up

Posted by Micha Lenk <mi...@lenk.info>.
Sorry, this shouldn't have gone to dev@httpd.apache.org.

Re: Reverse proxy: invalid Content-Length leads to 413 + 400 errors mixed up

Posted by Micha Lenk <mi...@lenk.info>.
Hi Ewald,

zunächst mal ein gutes neues Jahr!

Am 08.01.2015 um 17:29 schrieb Ewald Dieterich:
> On 01/08/2015 04:15 PM, Yann Ylavic wrote:
>>> Can you test this (attached) patch please (without yours applied)?
>>
>> Or with yours and just changing "return
>> ap_map_http_request_error(status, HTTP_BAD_REQUEST);" by "return
>> (status == AP_FILTER_ERROR) ? DONE : ap_map_http_request_error(status,
>> HTTP_BAD_REQUEST);"
> 
> Looks good. I tested both your patch and my modified one and now error
> response and access log entry are both OK.

Was meinst Du? Wären diese Änderungen auch für's mod_proxy_msrpc hilfreich?

Viele Grüße aus Zürich,
Micha

Re: Reverse proxy: invalid Content-Length leads to 413 + 400 errors mixed up

Posted by Eric Covener <co...@gmail.com>.
On Thu, Jan 8, 2015 at 1:02 PM, Eric Covener <co...@gmail.com> wrote:
> I am not following (the existing) AH01579. Why is it related to a
> custom error page at that point?

http://svn.apache.org/viewvc/httpd/httpd/trunk/modules/http/http_request.c?r1=722213&r2=722212&pathrev=722213

There is a short thread responding to the commit email.

I think it's a good check minus the "custom errordocument" slant, and
I think I agree with Yann's remix of it where we just do this check
any time we ap_die with a non-HTTP status before headers have been
written.

I think currently this branch should be pretty rare, as
ap_invoke_handler is filtering this status code for most places we'd
expect it to show up. A non-handler hook or filter returning it
directly are the other paths. Those seem unlikely to pass it to
ap_die().

-- 
Eric Covener
covener@gmail.com

AW: Reverse proxy: invalid Content-Length leads to 413 + 400 errors mixed up

Posted by Plüm, Rüdiger, Vodafone Group <ru...@vodafone.com>.

> -----Ursprüngliche Nachricht-----
> Von: Eric Covener [mailto:covener@gmail.com]
> Gesendet: Donnerstag, 8. Januar 2015 19:03
> An: Apache HTTP Server Development List
> Betreff: Re: Reverse proxy: invalid Content-Length leads to 413 + 400
> errors mixed up
> 
> On Thu, Jan 8, 2015 at 12:29 PM, Yann Ylavic <yl...@gmail.com>
> wrote:
> > On Thu, Jan 8, 2015 at 6:17 PM, Eric Covener <co...@gmail.com>
> wrote:
> >> On Thu, Jan 8, 2015 at 12:12 PM, Yann Ylavic <yl...@gmail.com>
> wrote:
> >>> Also, it seems that ap_map_http_request_error() would better map
> >>> AP_FILTER_ERROR to DONE, any issue with this?
> >>
> >> Does that affect ap_die adversely? It can no longer tell the
> difference.
> >
> > As we already discussed in [1] and [2], we'd probably handle AP_FILTER
> > better in ap_die() and apply patch [3], so that we don't have to
> > bother whether a handler returns DONE or AP_FILTER_ERROR, ap_die()
> > should do the right thing (ie. nothing if AP_FILTER_ERROR already
> > generated a response).
> >
> > [1] http://mail-archives.apache.org/mod_mbox/httpd-
> dev/201401.mbox/%3CCALK=YjMb2DHo415PuAt4_DsBjXFsvB-
> ag=QjixX6UcLhN62Ysw@mail.gmail.com%3E
> > [2] http://issues.apache.org/bugzilla/show_bug.cgi?id=56035
> > [3] http://issues.apache.org/bugzilla/attachment.cgi?id=31245
> 
> I am not following (the existing) AH01579. Why is it related to a
> custom error page at that point?

Me neither.

Regards

Rüdiger


Re: Reverse proxy: invalid Content-Length leads to 413 + 400 errors mixed up

Posted by Eric Covener <co...@gmail.com>.
On Thu, Jan 8, 2015 at 12:29 PM, Yann Ylavic <yl...@gmail.com> wrote:
> On Thu, Jan 8, 2015 at 6:17 PM, Eric Covener <co...@gmail.com> wrote:
>> On Thu, Jan 8, 2015 at 12:12 PM, Yann Ylavic <yl...@gmail.com> wrote:
>>> Also, it seems that ap_map_http_request_error() would better map
>>> AP_FILTER_ERROR to DONE, any issue with this?
>>
>> Does that affect ap_die adversely? It can no longer tell the difference.
>
> As we already discussed in [1] and [2], we'd probably handle AP_FILTER
> better in ap_die() and apply patch [3], so that we don't have to
> bother whether a handler returns DONE or AP_FILTER_ERROR, ap_die()
> should do the right thing (ie. nothing if AP_FILTER_ERROR already
> generated a response).
>
> [1] http://mail-archives.apache.org/mod_mbox/httpd-dev/201401.mbox/%3CCALK=YjMb2DHo415PuAt4_DsBjXFsvB-ag=QjixX6UcLhN62Ysw@mail.gmail.com%3E
> [2] http://issues.apache.org/bugzilla/show_bug.cgi?id=56035
> [3] http://issues.apache.org/bugzilla/attachment.cgi?id=31245

I am not following (the existing) AH01579. Why is it related to a
custom error page at that point?

Re: Reverse proxy: invalid Content-Length leads to 413 + 400 errors mixed up

Posted by Yann Ylavic <yl...@gmail.com>.
On Thu, Jan 8, 2015 at 6:17 PM, Eric Covener <co...@gmail.com> wrote:
> On Thu, Jan 8, 2015 at 12:12 PM, Yann Ylavic <yl...@gmail.com> wrote:
>> Also, it seems that ap_map_http_request_error() would better map
>> AP_FILTER_ERROR to DONE, any issue with this?
>
> Does that affect ap_die adversely? It can no longer tell the difference.

As we already discussed in [1] and [2], we'd probably handle AP_FILTER
better in ap_die() and apply patch [3], so that we don't have to
bother whether a handler returns DONE or AP_FILTER_ERROR, ap_die()
should do the right thing (ie. nothing if AP_FILTER_ERROR already
generated a response).

[1] http://mail-archives.apache.org/mod_mbox/httpd-dev/201401.mbox/%3CCALK=YjMb2DHo415PuAt4_DsBjXFsvB-ag=QjixX6UcLhN62Ysw@mail.gmail.com%3E
[2] http://issues.apache.org/bugzilla/show_bug.cgi?id=56035
[3] http://issues.apache.org/bugzilla/attachment.cgi?id=31245

Re: Reverse proxy: invalid Content-Length leads to 413 + 400 errors mixed up

Posted by Eric Covener <co...@gmail.com>.
On Thu, Jan 8, 2015 at 12:12 PM, Yann Ylavic <yl...@gmail.com> wrote:
> Also, it seems that ap_map_http_request_error() would better map
> AP_FILTER_ERROR to DONE, any issue with this?

Does that affect ap_die adversely? It can no longer tell the difference.

Re: Reverse proxy: invalid Content-Length leads to 413 + 400 errors mixed up

Posted by Yann Ylavic <yl...@gmail.com>.
On Thu, Jan 8, 2015 at 6:08 PM, Yann Ylavic <yl...@gmail.com> wrote:
> On Thu, Jan 8, 2015 at 5:29 PM, Ewald Dieterich <ew...@mailbox.org> wrote:
>>
>> Looks good. I tested both your patch and my modified one and now error
>> response and access log entry are both OK.
>
> Thanks Ewald for testing.
>
> Eric, all proxy modules concerned in trunk could be fixed with
> "(status == AP_FILTER_ERROR) ? DONE :
> ap_map_http_request_error(status, HTTP_BAD_REQUEST)", but not 2.4.x
> where "(status == AP_FILTER_ERROR) ? DONE : HTTP_BAD_REQUEST)" is a
> simple fix.
> Should we also (propose to )backport to 2.4.x the whole
> ap_map_http_request_error() stuff (r1482522 + r1529991 + r1482918,
> quite huge...) so that we don't lose specific errors returned by
> ap_http_filter (ie. turned to 400), or is the simple fix OK?

Also, it seems that ap_map_http_request_error() would better map
AP_FILTER_ERROR to DONE, any issue with this?

Re: Reverse proxy: invalid Content-Length leads to 413 + 400 errors mixed up

Posted by Yann Ylavic <yl...@gmail.com>.
Backport proposed in r1665737 (+ r1665743).

On Sun, Jan 11, 2015 at 10:37 PM, Jim Jagielski <ji...@jagunet.com> wrote:
> +1
>> On Jan 8, 2015, at 12:12 PM, Eric Covener <co...@gmail.com> wrote:
>>
>> On Thu, Jan 8, 2015 at 12:08 PM, Yann Ylavic <yl...@gmail.com> wrote:
>>> Eric, all proxy modules concerned in trunk could be fixed with
>>> "(status == AP_FILTER_ERROR) ? DONE :
>>> ap_map_http_request_error(status, HTTP_BAD_REQUEST)", but not 2.4.x
>>> where "(status == AP_FILTER_ERROR) ? DONE : HTTP_BAD_REQUEST)" is a
>>> simple fix.
>>> Should we also (propose to )backport to 2.4.x the whole
>>> ap_map_http_request_error() stuff (r1482522 + r1529991 + r1482918,
>>> quite huge...) so that we don't lose specific errors returned by
>>> ap_http_filter (ie. turned to 400), or is the simple fix OK?
>>
>> my own 2 cents -- would prefer to maintain the commonality.
>

Re: Reverse proxy: invalid Content-Length leads to 413 + 400 errors mixed up

Posted by Jim Jagielski <ji...@jaguNET.com>.
+1
> On Jan 8, 2015, at 12:12 PM, Eric Covener <co...@gmail.com> wrote:
> 
> On Thu, Jan 8, 2015 at 12:08 PM, Yann Ylavic <yl...@gmail.com> wrote:
>> Eric, all proxy modules concerned in trunk could be fixed with
>> "(status == AP_FILTER_ERROR) ? DONE :
>> ap_map_http_request_error(status, HTTP_BAD_REQUEST)", but not 2.4.x
>> where "(status == AP_FILTER_ERROR) ? DONE : HTTP_BAD_REQUEST)" is a
>> simple fix.
>> Should we also (propose to )backport to 2.4.x the whole
>> ap_map_http_request_error() stuff (r1482522 + r1529991 + r1482918,
>> quite huge...) so that we don't lose specific errors returned by
>> ap_http_filter (ie. turned to 400), or is the simple fix OK?
> 
> my own 2 cents -- would prefer to maintain the commonality.


Re: Reverse proxy: invalid Content-Length leads to 413 + 400 errors mixed up

Posted by Eric Covener <co...@gmail.com>.
On Thu, Jan 8, 2015 at 12:08 PM, Yann Ylavic <yl...@gmail.com> wrote:
> Eric, all proxy modules concerned in trunk could be fixed with
> "(status == AP_FILTER_ERROR) ? DONE :
> ap_map_http_request_error(status, HTTP_BAD_REQUEST)", but not 2.4.x
> where "(status == AP_FILTER_ERROR) ? DONE : HTTP_BAD_REQUEST)" is a
> simple fix.
> Should we also (propose to )backport to 2.4.x the whole
> ap_map_http_request_error() stuff (r1482522 + r1529991 + r1482918,
> quite huge...) so that we don't lose specific errors returned by
> ap_http_filter (ie. turned to 400), or is the simple fix OK?

my own 2 cents -- would prefer to maintain the commonality.

Re: Reverse proxy: invalid Content-Length leads to 413 + 400 errors mixed up

Posted by Yann Ylavic <yl...@gmail.com>.
On Thu, Jan 8, 2015 at 5:29 PM, Ewald Dieterich <ew...@mailbox.org> wrote:
>
> Looks good. I tested both your patch and my modified one and now error
> response and access log entry are both OK.

Thanks Ewald for testing.

Eric, all proxy modules concerned in trunk could be fixed with
"(status == AP_FILTER_ERROR) ? DONE :
ap_map_http_request_error(status, HTTP_BAD_REQUEST)", but not 2.4.x
where "(status == AP_FILTER_ERROR) ? DONE : HTTP_BAD_REQUEST)" is a
simple fix.
Should we also (propose to )backport to 2.4.x the whole
ap_map_http_request_error() stuff (r1482522 + r1529991 + r1482918,
quite huge...) so that we don't lose specific errors returned by
ap_http_filter (ie. turned to 400), or is the simple fix OK?

Re: Reverse proxy: invalid Content-Length leads to 413 + 400 errors mixed up

Posted by Ewald Dieterich <ew...@mailbox.org>.
On 01/08/2015 04:15 PM, Yann Ylavic wrote:
>> Can you test this (attached) patch please (without yours applied)?
>
> Or with yours and just changing "return
> ap_map_http_request_error(status, HTTP_BAD_REQUEST);" by "return
> (status == AP_FILTER_ERROR) ? DONE : ap_map_http_request_error(status,
> HTTP_BAD_REQUEST);"

Looks good. I tested both your patch and my modified one and now error 
response and access log entry are both OK.

Re: Reverse proxy: invalid Content-Length leads to 413 + 400 errors mixed up

Posted by Yann Ylavic <yl...@gmail.com>.
On Thu, Jan 8, 2015 at 4:11 PM, Yann Ylavic <yl...@gmail.com> wrote:
> On Thu, Jan 8, 2015 at 3:47 PM, Ewald Dieterich <ew...@mailbox.org> wrote:
>>
>> But the access log entry is still wrong. Now a 200 is logged:
>>
>> [...] "GET / HTTP/1.1" 200 590 "-" "curl/7.26.0"
>>
>> I still see the -102 error:
>>
>> [...] (-102)Unknown error -102: [client 10.128.128.95:46766] AH01095:
>> prefetch request body failed to 10.8.19.114:80 (backend) from 10.128.128.95
>> ()
>>
>> I guess there are more changes in trunk that I would need to add?
>
> Can you test this (attached) patch please (without yours applied)?

Or with yours and just changing "return
ap_map_http_request_error(status, HTTP_BAD_REQUEST);" by "return
(status == AP_FILTER_ERROR) ? DONE : ap_map_http_request_error(status,
HTTP_BAD_REQUEST);"

>
> Regards,
> Yann.

Re: Reverse proxy: invalid Content-Length leads to 413 + 400 errors mixed up

Posted by Yann Ylavic <yl...@gmail.com>.
On Thu, Jan 8, 2015 at 3:47 PM, Ewald Dieterich <ew...@mailbox.org> wrote:
>
> But the access log entry is still wrong. Now a 200 is logged:
>
> [...] "GET / HTTP/1.1" 200 590 "-" "curl/7.26.0"
>
> I still see the -102 error:
>
> [...] (-102)Unknown error -102: [client 10.128.128.95:46766] AH01095:
> prefetch request body failed to 10.8.19.114:80 (backend) from 10.128.128.95
> ()
>
> I guess there are more changes in trunk that I would need to add?

Can you test this (attached) patch please (without yours applied)?

Regards,
Yann.

Re: Reverse proxy: invalid Content-Length leads to 413 + 400 errors mixed up

Posted by Eric Covener <co...@gmail.com>.
On Thu, Jan 8, 2015 at 9:47 AM, Ewald Dieterich <ew...@mailbox.org> wrote:
> Hope I tested the right thing. ap_map_http_request_error() is not available
> in 2.4.x, so I added it from trunk and replaced the return statements in the
> stanzas above as suggested. I attached a patch with my changes to 2.4.10.

Sorry, I did not realize it was trunk-only.  I see Yann replied while
I had this in draft -- sorry for the misfire.

Re: Reverse proxy: invalid Content-Length leads to 413 + 400 errors mixed up

Posted by Ewald Dieterich <ew...@mailbox.org>.
On 01/08/2015 01:39 PM, Eric Covener wrote:
> On Thu, Jan 8, 2015 at 4:38 AM, Ewald Dieterich <ew...@mailbox.org> wrote:
>> Any ideas how to fix this so that this situation is handled as a single
>> error and not as two errors mixed up?
>
> in mod_proxy.c you will see at least 1 stanza like this:
>
>          status = ap_get_brigade(r->input_filters, temp_brigade,
>                                  AP_MODE_READBYTES, APR_BLOCK_READ,
>                                  MAX_MEM_SPOOL - bytes_read);
>          if (status != APR_SUCCESS) {
>              ap_log_rerror(APLOG_MARK, APLOG_ERR, status, r, APLOGNO(01095)
>                            "prefetch request body failed to %pI (%s)"
>                            " from %s (%s)",
>                            p_conn->addr, p_conn->hostname ? p_conn->hostname: "",
>                            c->client_ip, c->remote_host ? c->remote_host: "");
>              return HTTP_BAD_REQUEST;
>          }
>
> The proper pattern in 2.4.x and later is to not return an error like that:
>
>              return ap_map_http_request_error(status, HTTP_BAD_REQUEST);
>
> In the case of that -102 error, the -102 will be returned verbatim
> instead (AP_FILTER_ERROR). Are you able to test and verify?

Hope I tested the right thing. ap_map_http_request_error() is not 
available in 2.4.x, so I added it from trunk and replaced the return 
statements in the stanzas above as suggested. I attached a patch with my 
changes to 2.4.10.

The response looks good now:

$ curl -i -H "Content-Length: a" http://frontend/
HTTP/1.1 413 Request Entity Too Large
Date: Thu, 08 Jan 2015 14:22:09 GMT
Server: Apache/2.4.10 (Debian)
Connection: close
Content-Type: text/html; charset=iso-8859-1

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>413 Request Entity Too Large</title>
</head><body>
<h1>Request Entity Too Large</h1>
The requested resource<br />/<br />
does not allow request data with GET requests, or the amount of data 
provided in
the request exceeds the capacity limit.
<hr>
<address>Apache/2.4.10 (Debian) Server at frontend Port 80</address>
</body></html>

But the access log entry is still wrong. Now a 200 is logged:

[...] "GET / HTTP/1.1" 200 590 "-" "curl/7.26.0"

I still see the -102 error:

[...] (-102)Unknown error -102: [client 10.128.128.95:46766] AH01095: 
prefetch request body failed to 10.8.19.114:80 (backend) from 
10.128.128.95 ()

I guess there are more changes in trunk that I would need to add?

Re: Reverse proxy: invalid Content-Length leads to 413 + 400 errors mixed up

Posted by Eric Covener <co...@gmail.com>.
On Thu, Jan 8, 2015 at 4:38 AM, Ewald Dieterich <ew...@mailbox.org> wrote:
> Any ideas how to fix this so that this situation is handled as a single
> error and not as two errors mixed up?
>

in mod_proxy.c you will see at least 1 stanza like this:

        status = ap_get_brigade(r->input_filters, temp_brigade,
                                AP_MODE_READBYTES, APR_BLOCK_READ,
                                MAX_MEM_SPOOL - bytes_read);
        if (status != APR_SUCCESS) {
            ap_log_rerror(APLOG_MARK, APLOG_ERR, status, r, APLOGNO(01095)
                          "prefetch request body failed to %pI (%s)"
                          " from %s (%s)",
                          p_conn->addr, p_conn->hostname ? p_conn->hostname: "",
                          c->client_ip, c->remote_host ? c->remote_host: "");
            return HTTP_BAD_REQUEST;
        }


The proper pattern in 2.4.x and later is to not return an error like that:

            return ap_map_http_request_error(status, HTTP_BAD_REQUEST);

In the case of that -102 error, the -102 will be returned verbatim
instead (AP_FILTER_ERROR). Are you able to test and verify?

-- 
Eric Covener
covener@gmail.com