Problem with mod_fcgid handling ErrorDocuments

[Edgar Frank <ef...@email.de>]

Hi mod_fcgid developers,

I'm currently exploring a potential problem with mod_fcgid.
Let's assume a setup with mod_security and mod_fcgid
(has nothing to do with mod_security itself - it just helps to
trigger the problem).

Now we have a large POST request which mod_security blocks
(by SecRequestBodyLimit) with "413 Request Entity Too Large".

The ErrorDocument for 413 is configured to a Location which
mod_fcgid serves. (Please don't argue that it's this way - I know
the problems and I'm not happy with it, but it's not my decision
to do it that way.)

HTTPD issues a GET subrequest for the ErrorDocument and
mod_fcgid kicks in. But now it starts consuming the request body
we just blocked - or if the request body size is larger than
FcgidMaxRequestLen, ErrorDocument generation fails.

I wonder how to circumvent this. In fcgid_bridge.c:bridge_request
I found:

if (role == FCGI_RESPONDER) {
 rc = add_request_body( [...] );
}

Could one change this to something like the following without
causing trouble?

if (role == FCGI_RESPONDER && !ap_is_HTTP_ERROR(r->status)) {
 rc = add_request_body( [...] );
}

Or maybe something like a HTTP method check? (Is there a
reliable way to detect if we're in ErrorDocument generation
anyway?) But at this point we have put the Content-Length header
already into the stream to the FCGI backend, so one would also
have to take action earlier.

What do you think in general of handling this? I'd really
appreciate an elaborate answer - if you find it fix-worthy,
first ideas how to fix it - and if not, why not. 

Regards,
Edgar

FYI:
mod_fcgid 2.3.5
with httpd 2.2.15
on CentOS 5.4 x64
built from source with gcc 4.2.4

Re: Problem with mod_fcgid handling ErrorDocuments

[Edgar Frank <ef...@email.de>]

2010/07/07 Graham Dumpleton:
>On 7 July 2010 11:43, Graham Dumpleton <gr...@gmail.com>
>wrote:
>> On 6 July 2010 22:56, Edgar Frank <ef...@email.de> wrote:
>>> Hi mod_fcgid developers,
>>>
>>> I'm currently exploring a potential problem with mod_fcgid.
>>> Let's assume a setup with mod_security and mod_fcgid
>>> (has nothing to do with mod_security itself - it just helps to
>>> trigger the problem).
>>>
>>> Now we have a large POST request which mod_security blocks
>>> (by SecRequestBodyLimit) with "413 Request Entity Too Large".
>>
>> Presumably it might also occur with LimitRequestBody and not even
>> need mod_security. Unless that is that this is a problem with
>> mod_security itself and how it handles a 413.
>>
>> So, is the issue with mod_fcgid. If there is an issue here with
>> ErrorDocument for a 413 where the handler is a proxy of some form,
>> then likely could affect other modules besides mod_fcgid.
>>
>> I would be investigating where ErrorDocument for 413 is handed off
>> to URL implemented by CGI or even mod_proxy to see what happens.

Sorry, this isn't clear to me. Could you please clarify?

Debugging showed, that when mod_security rejects the request because
it's oversized, ap_run_post_read_request in protocol.c:1011 returns 413,
in which case regular ap_die() handling kicks in in protocol.c:1012.
(line numbers refer to the 2.2.15 tag).
Then, in http_request.c:425 (internal_internal_redirect calls
ap_run_post_read_request) mod_security returns DECLINED, allowing an
errorpage to be served instead of recursively returning 413.

So I'd assume that it's not a mod_security issue?

>>> The ErrorDocument for 413 is configured to a Location which
>>> mod_fcgid serves. (Please don't argue that it's this way - I know
>>> the problems and I'm not happy with it, but it's not my decision
>>> to do it that way.)
>>>
>>> HTTPD issues a GET subrequest for the ErrorDocument and
>>> mod_fcgid kicks in. But now it starts consuming the request body
>>> we just blocked - or if the request body size is larger than
>>> FcgidMaxRequestLen, ErrorDocument generation fails.
>>>
>>> I wonder how to circumvent this. In fcgid_bridge.c:bridge_request
>>> I found:
>>>
>>> if (role == FCGI_RESPONDER) {
>>>  rc = add_request_body( [...] );
>>> }
>>>
>>> Could one change this to something like the following without
>>> causing trouble?
>>>
>>> if (role == FCGI_RESPONDER && !ap_is_HTTP_ERROR(r->status)) {
>>>  rc = add_request_body( [...] );
>>> }
>>>
>>> Or maybe something like a HTTP method check? (Is there a
>>> reliable way to detect if we're in ErrorDocument generation
>>> anyway?)
>
> Addressing this question, r->prev attribute should refer to original
> request and so can possibly check r->prev->status.
>
> Also, the following gets set:
>
> apr_table_setn(new->subprocess_env, "REDIRECT_STATUS",
> apr_itoa(r->pool, r->status));
>
> before internal redirect. This is more of interest where CGI, or
> FASTCGI script actually gets executed as that wouldn't have access to
> the original request object.

Oh, my fault - it's an internal redirect, not a subrequest. Thanks for
pointing this out.

At this point in mod_fcgid, why don't access r->prev->status? Or have I
mistaken you? This would save a table lookup and a string-to-int. Or did
you mean, in the CGI-script (after a request-body might have been
dropped)?

I though about it a while and came to the conclusion that one would have
to check ap_status_drops_connection( r->prev->status) (assuming, that
r->prev != NULL). If this is false, we'd have to consume the request
body from the input filter chain to allow future (keep-alive) requests
to be parsed correctly. If it's true, the input filter chain shouldn't
be touched (connection will be dropped anyway) and Content-Length,
Transfer-Encoding and Expect headers should be removed to not confuse
the (Fast)CGI script.

But this are just my thoughts and I'm interested, if I'm totally off or
you maybe agree. Looking forward to hear your comments.

Regards,
Edgar

Re: Problem with mod_fcgid handling ErrorDocuments

[Graham Dumpleton <gr...@gmail.com>]

On 7 July 2010 11:43, Graham Dumpleton <gr...@gmail.com> wrote:
> On 6 July 2010 22:56, Edgar Frank <ef...@email.de> wrote:
>> Hi mod_fcgid developers,
>>
>> I'm currently exploring a potential problem with mod_fcgid.
>> Let's assume a setup with mod_security and mod_fcgid
>> (has nothing to do with mod_security itself - it just helps to
>> trigger the problem).
>>
>> Now we have a large POST request which mod_security blocks
>> (by SecRequestBodyLimit) with "413 Request Entity Too Large".
>
> Presumably it might also occur with LimitRequestBody and not even need
> mod_security. Unless that is that this is a problem with mod_security
> itself and how it handles a 413.
>
> So, is the issue with mod_fcgid. If there is an issue here with
> ErrorDocument for a 413 where the handler is a proxy of some form,
> then likely could affect other modules besides mod_fcgid.
>
> I would be investigating where ErrorDocument for 413 is handed off to
> URL implemented by CGI or even mod_proxy to see what happens.
>
> Graham
>
>> The ErrorDocument for 413 is configured to a Location which
>> mod_fcgid serves. (Please don't argue that it's this way - I know
>> the problems and I'm not happy with it, but it's not my decision
>> to do it that way.)
>>
>> HTTPD issues a GET subrequest for the ErrorDocument and
>> mod_fcgid kicks in. But now it starts consuming the request body
>> we just blocked - or if the request body size is larger than
>> FcgidMaxRequestLen, ErrorDocument generation fails.
>>
>> I wonder how to circumvent this. In fcgid_bridge.c:bridge_request
>> I found:
>>
>> if (role == FCGI_RESPONDER) {
>>  rc = add_request_body( [...] );
>> }
>>
>> Could one change this to something like the following without
>> causing trouble?
>>
>> if (role == FCGI_RESPONDER && !ap_is_HTTP_ERROR(r->status)) {
>>  rc = add_request_body( [...] );
>> }
>>
>> Or maybe something like a HTTP method check? (Is there a
>> reliable way to detect if we're in ErrorDocument generation
>> anyway?)

Addressing this question, r->prev attribute should refer to original
request and so can possibly check r->prev->status.

Also, the following gets set:

    apr_table_setn(new->subprocess_env, "REDIRECT_STATUS",
                   apr_itoa(r->pool, r->status));

before internal redirect. This is more of interest where CGI, or
FASTCGI script actually gets executed as that wouldn't have access to
the original request object.

Graham

> But at this point we have put the Content-Length header
>> already into the stream to the FCGI backend, so one would also
>> have to take action earlier.
>>
>> What do you think in general of handling this? I'd really
>> appreciate an elaborate answer - if you find it fix-worthy,
>> first ideas how to fix it - and if not, why not.
>>
>> Regards,
>> Edgar
>>
>> FYI:
>> mod_fcgid 2.3.5
>> with httpd 2.2.15
>> on CentOS 5.4 x64
>> built from source with gcc 4.2.4
>>
>

Re: Problem with mod_fcgid handling ErrorDocuments

[Graham Dumpleton <gr...@gmail.com>]

On 6 July 2010 22:56, Edgar Frank <ef...@email.de> wrote:
> Hi mod_fcgid developers,
>
> I'm currently exploring a potential problem with mod_fcgid.
> Let's assume a setup with mod_security and mod_fcgid
> (has nothing to do with mod_security itself - it just helps to
> trigger the problem).
>
> Now we have a large POST request which mod_security blocks
> (by SecRequestBodyLimit) with "413 Request Entity Too Large".

Presumably it might also occur with LimitRequestBody and not even need
mod_security. Unless that is that this is a problem with mod_security
itself and how it handles a 413.

So, is the issue with mod_fcgid. If there is an issue here with
ErrorDocument for a 413 where the handler is a proxy of some form,
then likely could affect other modules besides mod_fcgid.

I would be investigating where ErrorDocument for 413 is handed off to
URL implemented by CGI or even mod_proxy to see what happens.

Graham

> The ErrorDocument for 413 is configured to a Location which
> mod_fcgid serves. (Please don't argue that it's this way - I know
> the problems and I'm not happy with it, but it's not my decision
> to do it that way.)
>
> HTTPD issues a GET subrequest for the ErrorDocument and
> mod_fcgid kicks in. But now it starts consuming the request body
> we just blocked - or if the request body size is larger than
> FcgidMaxRequestLen, ErrorDocument generation fails.
>
> I wonder how to circumvent this. In fcgid_bridge.c:bridge_request
> I found:
>
> if (role == FCGI_RESPONDER) {
>  rc = add_request_body( [...] );
> }
>
> Could one change this to something like the following without
> causing trouble?
>
> if (role == FCGI_RESPONDER && !ap_is_HTTP_ERROR(r->status)) {
>  rc = add_request_body( [...] );
> }
>
> Or maybe something like a HTTP method check? (Is there a
> reliable way to detect if we're in ErrorDocument generation
> anyway?) But at this point we have put the Content-Length header
> already into the stream to the FCGI backend, so one would also
> have to take action earlier.
>
> What do you think in general of handling this? I'd really
> appreciate an elaborate answer - if you find it fix-worthy,
> first ideas how to fix it - and if not, why not.
>
> Regards,
> Edgar
>
> FYI:
> mod_fcgid 2.3.5
> with httpd 2.2.15
> on CentOS 5.4 x64
> built from source with gcc 4.2.4
>