You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by "Paul J. Reder" <re...@remulak.net> on 2002/06/10 23:50:29 UTC
Recursive error processing.
While Allan Edwards and I were doing some testing of SSL we ran into a case
where we were able to send Apache into an infinite loop which eventually
consumed the machine's resources.
The problem occurs if you send a request to "http://some.where.com:443" (instead
of "https://some.where.com:443".
Apache tries to return a custom error but gets confused while trying to
send the error response to a non-secure request over an apparently secure
connection.
A small snip of the back trace follows:
#590 0x080a1549 in ap_die (type=400, r=0x81e0e58) at http_request.c:198
#591 0x080a1b51 in internal_internal_redirect (new_uri=0x8188770 "/error/HTTP_BAD_REQUEST.html.var", r=0x81e0650) at http_request.c:408
#592 0x080a1e10 in ap_internal_redirect (new_uri=0x8188770 "/error/HTTP_BAD_REQUEST.html.var", r=0x81e0650) at http_request.c:483
#593 0x080a1549 in ap_die (type=400, r=0x81e0650) at http_request.c:198
#594 0x080a1b51 in internal_internal_redirect (new_uri=0x8188770 "/error/HTTP_BAD_REQUEST.html.var", r=0x81df1c0) at http_request.c:408
#595 0x080a1e10 in ap_internal_redirect (new_uri=0x8188770 "/error/HTTP_BAD_REQUEST.html.var", r=0x81df1c0) at http_request.c:483
#596 0x080a1549 in ap_die (type=400, r=0x81df1c0) at http_request.c:198
#597 0x080e4c24 in ap_read_request (conn=0x81cce58) at protocol.c:982
#598 0x0809bc63 in ap_process_http_connection (c=0x81cce58) at http_core.c:284
#599 0x080dfc17 in ap_run_process_connection (c=0x81cce58) at connection.c:85
#600 0x080dffbe in ap_process_connection (c=0x81cce58, csd=0x81ccd88) at connection.c:207
#601 0x080d2123 in child_main (child_num_arg=0) at prefork.c:671
#602 0x080d21f4 in make_child (s=0x8132908, slot=0) at prefork.c:711
---Type <return> to continue, or q <return> to quit---
#603 0x080d2321 in startup_children (number_to_start=5) at prefork.c:783
#604 0x080d274b in ap_mpm_run (_pconf=0x812e9f0, plog=0x8172b00, s=0x8132908) at prefork.c:999
#605 0x080d8e98 in main (argc=5, argv=0xbffff884) at main.c:646
#606 0x402be627 in __libc_start_main (main=0x80d8600 <main>, argc=5, ubp_av=0xbffff884, init=0x80658f0 <_init>, fini=0x80f81e0 <_fini>,
rtld_fini=0x4000dcc4 <_dl_fini>,
stack_end=0xbffff87c) at ../sysdeps/generic/libc-start.c:129
The loop (ap_die, ap_internal_redirect, internal_internal_redirect, ap_die...) happens
until the system dies (due to newly allocated request_recs).
The problem seems to be related to the fact that ap_die should be killing
the custom_response and just dropping the connection (which is what 1.3 does)
rather than falling through and trying to send a custom response via internal_redirect.
Is this an artifact of the recent changes for 401/413 processing? Is this symptomatic
of a bigger problem of infinite loops during error redirects?
This all starts because the SSL post_read_request hook function (ssl_hook_ReadReq)
returns HTTP_BAD_REQUEST after finding sslconn->non_ssl_request set to 1 (by
ssl_io_filter_input after it notices ssl_connect fails in ssl_hook_process_connection).
Thanks for any pointers here.
--
Paul J. Reder
-----------------------------------------------------------
"The strength of the Constitution lies entirely in the determination of each
citizen to defend it. Only if every single citizen feels duty bound to do
his share in this defense are the constitutional rights secure."
-- Albert Einstein
Re: Recursive error processing.
Posted by Justin Erenkrantz <je...@apache.org>.
On Mon, Jun 10, 2002 at 06:52:52PM -0400, Paul J. Reder wrote:
> I'm running with CVS head as of Friday morning with
> OpenSSL 0.9.6b [engine] 9 Jul 2001 on Linux (RedHat 7.2).
rbb's changes went in on 2002/06/07 22:31:34 GMT. =)
You should update. -- justin
RE: Recursive error processing.
Posted by Ryan Bloom <rb...@covalent.net>.
> From: Cliff Woolley [mailto:jwoolley@virginia.edu]
>
> On Mon, 10 Jun 2002, Ryan Bloom wrote:
>
> > I don't have any ideas. I can't reproduce this problem though.
I'll
> > keep debugging on my end. Cliff, this may take some time.
>
> Any progress? I *can* reproduce this and am looking at it. I
basically
> just took Paul's httpd.conf and ssl.conf (and tweaked the paths)...
though
> they're very default-looking config files for the most part. I used
my
> config.nice, though it's largely similar to Paul's in that it enables
all
> of the modules.
Still can't dup. :-( I'll keep trying.
> Why is ap_run_post_read_request() returning 400?
Because that is when mod_ssl notices that the request was HTTP on the
HTTPS port.
Ryan
RE: Recursive error processing.
Posted by Cliff Woolley <jw...@virginia.edu>.
On Mon, 10 Jun 2002, Ryan Bloom wrote:
> I don't have any ideas. I can't reproduce this problem though. I'll
> keep debugging on my end. Cliff, this may take some time.
Any progress? I *can* reproduce this and am looking at it. I basically
just took Paul's httpd.conf and ssl.conf (and tweaked the paths)... though
they're very default-looking config files for the most part. I used my
config.nice, though it's largely similar to Paul's in that it enables all
of the modules.
Why is ap_run_post_read_request() returning 400?
--Cliff
RE: Recursive error processing.
Posted by Ryan Bloom <rb...@covalent.net>.
I don't have any ideas. I can't reproduce this problem though. I'll
keep debugging on my end. Cliff, this may take some time.
Ryan
----------------------------------------------
Ryan Bloom rbb@covalent.net
645 Howard St. rbb@apache.org
San Francisco, CA
> -----Original Message-----
> From: Paul J. Reder [mailto:rederpj@remulak.net]
> Sent: Monday, June 10, 2002 4:51 PM
> To: dev@httpd.apache.org
> Subject: Re: Recursive error processing.
>
> Bad news. I just finished running
>
> cvs update -dP httpd-2.0;cd httpd-2.0;make
> distclean;buildconf;config.nice;make;make install
>
> and tested it. The same thing still happens with the config I
referenced
> earlier.
>
> Any other ideas?
>
> Paul J. Reder wrote:
>
> > Hmmm, I missed them. I'm updating and building now, I'll have an
answer
> > shortly
> > after dinner.
> >
> > Ryan Bloom wrote:
> >
> >>> I'm running with CVS head as of Friday morning with
> >>> OpenSSL 0.9.6b [engine] 9 Jul 2001 on Linux (RedHat 7.2). I've
> >>> attached my httpd.conf, ssl.conf, and config.nice files.
> >>> I have been able to reproduce it on worker and prefork on two
> >>> different Linux boxes (both redhat 7.2).
> >>>
> >>> All I do is bring the box up and use Mozilla to send the request
> >>> "http://sol.Reders:443" and watch the cpu start spinning.
> >>>
> >>
> >> Please update your tree. There were changes to how Apache handles
> >> calling ap_die and ap_discard_request_body() on Friday evening.
> >>
> >> Ryan
> >>
> >>
> >>
> >>
> >
> >
>
>
> --
> Paul J. Reder
> -----------------------------------------------------------
> "The strength of the Constitution lies entirely in the determination
of
> each
> citizen to defend it. Only if every single citizen feels duty bound
to do
> his share in this defense are the constitutional rights secure."
> -- Albert Einstein
>
Re: Recursive error processing.
Posted by "Paul J. Reder" <re...@remulak.net>.
Bad news. I just finished running
cvs update -dP httpd-2.0;cd httpd-2.0;make distclean;buildconf;config.nice;make;make install
and tested it. The same thing still happens with the config I referenced earlier.
Any other ideas?
Paul J. Reder wrote:
> Hmmm, I missed them. I'm updating and building now, I'll have an answer
> shortly
> after dinner.
>
> Ryan Bloom wrote:
>
>>> I'm running with CVS head as of Friday morning with
>>> OpenSSL 0.9.6b [engine] 9 Jul 2001 on Linux (RedHat 7.2). I've
>>> attached my httpd.conf, ssl.conf, and config.nice files.
>>> I have been able to reproduce it on worker and prefork on two
>>> different Linux boxes (both redhat 7.2).
>>>
>>> All I do is bring the box up and use Mozilla to send the request
>>> "http://sol.Reders:443" and watch the cpu start spinning.
>>>
>>
>> Please update your tree. There were changes to how Apache handles
>> calling ap_die and ap_discard_request_body() on Friday evening.
>>
>> Ryan
>>
>>
>>
>>
>
>
--
Paul J. Reder
-----------------------------------------------------------
"The strength of the Constitution lies entirely in the determination of each
citizen to defend it. Only if every single citizen feels duty bound to do
his share in this defense are the constitutional rights secure."
-- Albert Einstein
Re: Recursive error processing.
Posted by "Paul J. Reder" <re...@remulak.net>.
Hmmm, I missed them. I'm updating and building now, I'll have an answer shortly
after dinner.
Ryan Bloom wrote:
>>I'm running with CVS head as of Friday morning with
>>OpenSSL 0.9.6b [engine] 9 Jul 2001 on Linux (RedHat 7.2). I've
>>attached my httpd.conf, ssl.conf, and config.nice files.
>>I have been able to reproduce it on worker and prefork on two
>>different Linux boxes (both redhat 7.2).
>>
>>All I do is bring the box up and use Mozilla to send the request
>>"http://sol.Reders:443" and watch the cpu start spinning.
>>
>
> Please update your tree. There were changes to how Apache handles
> calling ap_die and ap_discard_request_body() on Friday evening.
>
> Ryan
>
>
>
>
--
Paul J. Reder
-----------------------------------------------------------
"The strength of the Constitution lies entirely in the determination of each
citizen to defend it. Only if every single citizen feels duty bound to do
his share in this defense are the constitutional rights secure."
-- Albert Einstein
RE: Recursive error processing.
Posted by Ryan Bloom <rb...@covalent.net>.
> I'm running with CVS head as of Friday morning with
> OpenSSL 0.9.6b [engine] 9 Jul 2001 on Linux (RedHat 7.2). I've
> attached my httpd.conf, ssl.conf, and config.nice files.
> I have been able to reproduce it on worker and prefork on two
> different Linux boxes (both redhat 7.2).
>
> All I do is bring the box up and use Mozilla to send the request
> "http://sol.Reders:443" and watch the cpu start spinning.
Please update your tree. There were changes to how Apache handles
calling ap_die and ap_discard_request_body() on Friday evening.
Ryan
Re: Recursive error processing.
Posted by "Paul J. Reder" <re...@remulak.net>.
I'm running with CVS head as of Friday morning with
OpenSSL 0.9.6b [engine] 9 Jul 2001 on Linux (RedHat 7.2). I've
attached my httpd.conf, ssl.conf, and config.nice files.
I have been able to reproduce it on worker and prefork on two
different Linux boxes (both redhat 7.2).
All I do is bring the box up and use Mozilla to send the request
"http://sol.Reders:443" and watch the cpu start spinning.
Ryan Bloom wrote:
>>From: Cliff Woolley [mailto:jwoolley@virginia.edu]
>>
>>On Mon, 10 Jun 2002, Ryan Bloom wrote:
>>
>>
>>>Please make sure that your code is up to date, because the server is
>>>supposed to have logic that protects us from getting into an
>>>
> infinite
>
>>>loop.
>>>
>>Paul, I notice the line numbers in your back trace don't quite match
>>
> up
>
>>with mine... is this HEAD? Or are there local mods?
>>
>>
>>>Wait a sec, the problem could be the ErrorDocument path. The test
>>>
> suite
>
>>>doesn't exercise that path......... Will report back soon.
>>>
>>Ah. Well I'll wait for Ryan to check that then.
>>
>
> I've tried everything I can think of to make this fail. It refuses to
> fail for me. Please make sure that your code is up to date, and let me
> know what version of the SSL libraries you are using. For completeness,
> here are my test cases:
>
> 1) Run the test suite (this tests http://localhost:8350 where 8350 is
> the SSL port). Also requested a page through telnet and Konqueror.
>
> 2) Add a plain text ErrorDocument for 400 requests. Request a page
>
> 3) Copy the HTTP_BAD_REQUEST.html.var files and the config to my test
> server, request a page.
>
> All three scenarios work for me on Linux. There is a problem in the 3rd
> case, which looks to be from a non-terminated string (bad, but not a
> buffer overflow, we just forgot to add a \0). I'll fix that quickly.
> Paul or Allen, can either of you provide more details? There really is
> logic in the server to stop the ap_die calls from being recursive, so
> this bug really surprises me.
>
> Ryan
>
>
>
>
--
Paul J. Reder
-----------------------------------------------------------
"The strength of the Constitution lies entirely in the determination of each
citizen to defend it. Only if every single citizen feels duty bound to do
his share in this defense are the constitutional rights secure."
-- Albert Einstein
RE: Recursive error processing.
Posted by Ryan Bloom <rb...@covalent.net>.
> From: Cliff Woolley [mailto:jwoolley@virginia.edu]
>
> On Mon, 10 Jun 2002, Ryan Bloom wrote:
>
> > Please make sure that your code is up to date, because the server is
> > supposed to have logic that protects us from getting into an
infinite
> > loop.
>
> Paul, I notice the line numbers in your back trace don't quite match
up
> with mine... is this HEAD? Or are there local mods?
>
> > Wait a sec, the problem could be the ErrorDocument path. The test
suite
> > doesn't exercise that path......... Will report back soon.
>
> Ah. Well I'll wait for Ryan to check that then.
I've tried everything I can think of to make this fail. It refuses to
fail for me. Please make sure that your code is up to date, and let me
know what version of the SSL libraries you are using. For completeness,
here are my test cases:
1) Run the test suite (this tests http://localhost:8350 where 8350 is
the SSL port). Also requested a page through telnet and Konqueror.
2) Add a plain text ErrorDocument for 400 requests. Request a page
3) Copy the HTTP_BAD_REQUEST.html.var files and the config to my test
server, request a page.
All three scenarios work for me on Linux. There is a problem in the 3rd
case, which looks to be from a non-terminated string (bad, but not a
buffer overflow, we just forgot to add a \0). I'll fix that quickly.
Paul or Allen, can either of you provide more details? There really is
logic in the server to stop the ap_die calls from being recursive, so
this bug really surprises me.
Ryan
RE: Recursive error processing.
Posted by Cliff Woolley <jw...@virginia.edu>.
On Mon, 10 Jun 2002, Ryan Bloom wrote:
> Please make sure that your code is up to date, because the server is
> supposed to have logic that protects us from getting into an infinite
> loop.
Paul, I notice the line numbers in your back trace don't quite match up
with mine... is this HEAD? Or are there local mods?
> Wait a sec, the problem could be the ErrorDocument path. The test suite
> doesn't exercise that path......... Will report back soon.
Ah. Well I'll wait for Ryan to check that then.
--Cliff
RE: Recursive error processing.
Posted by Ryan Bloom <rb...@covalent.net>.
I can't reproduce this. This test case is actually tested for in the
test suite. Which SSL library are you using? I was going off of the
assumption that the ap_discard_request_body() changes had broken this,
but since I have the most up-to-date code, I don't believe that the two
are related.
Please make sure that your code is up to date, because the server is
supposed to have logic that protects us from getting into an infinite
loop.
Wait a sec, the problem could be the ErrorDocument path. The test suite
doesn't exercise that path......... Will report back soon.
Ryan
----------------------------------------------
Ryan Bloom rbb@covalent.net
645 Howard St. rbb@apache.org
San Francisco, CA
> -----Original Message-----
> From: Ryan Bloom [mailto:rbb@covalent.net]
> Sent: Monday, June 10, 2002 2:55 PM
> To: dev@httpd.apache.org
> Subject: RE: Recursive error processing.
>
> > From: Paul J. Reder [mailto:rederpj@remulak.net]
> >
> > While Allan Edwards and I were doing some testing of SSL we ran into
a
> > case
> > where we were able to send Apache into an infinite loop which
> eventually
> > consumed the machine's resources.
> >
> > The problem occurs if you send a request to
> "http://some.where.com:443"
> > (instead
> > of "https://some.where.com:443".
>
> This was working a few days ago, and is in fact a part of the test
> suite.
>
> > The problem seems to be related to the fact that ap_die should be
> killing
> > the custom_response and just dropping the connection (which is what
> 1.3
> > does)
> > rather than falling through and trying to send a custom response via
> > internal_redirect.
>
> 1.3 doesn't drop the connection, it sends a custom response.
>
> > Is this an artifact of the recent changes for 401/413 processing? Is
> this
> > symptomatic
> > of a bigger problem of infinite loops during error redirects?
> >
> > This all starts because the SSL post_read_request hook function
> > (ssl_hook_ReadReq)
> > returns HTTP_BAD_REQUEST after finding sslconn->non_ssl_request set
to
> 1
> > (by
> > ssl_io_filter_input after it notices ssl_connect fails in
> > ssl_hook_process_connection).
>
> Hold on, I think I know what the problem is, I'll try to commit a fix
in
> a few minutes.
>
> Ryan
>
RE: Recursive error processing.
Posted by Ryan Bloom <rb...@covalent.net>.
> From: Paul J. Reder [mailto:rederpj@remulak.net]
>
> While Allan Edwards and I were doing some testing of SSL we ran into a
> case
> where we were able to send Apache into an infinite loop which
eventually
> consumed the machine's resources.
>
> The problem occurs if you send a request to
"http://some.where.com:443"
> (instead
> of "https://some.where.com:443".
This was working a few days ago, and is in fact a part of the test
suite.
> The problem seems to be related to the fact that ap_die should be
killing
> the custom_response and just dropping the connection (which is what
1.3
> does)
> rather than falling through and trying to send a custom response via
> internal_redirect.
1.3 doesn't drop the connection, it sends a custom response.
> Is this an artifact of the recent changes for 401/413 processing? Is
this
> symptomatic
> of a bigger problem of infinite loops during error redirects?
>
> This all starts because the SSL post_read_request hook function
> (ssl_hook_ReadReq)
> returns HTTP_BAD_REQUEST after finding sslconn->non_ssl_request set to
1
> (by
> ssl_io_filter_input after it notices ssl_connect fails in
> ssl_hook_process_connection).
Hold on, I think I know what the problem is, I'll try to commit a fix in
a few minutes.
Ryan