You are viewing a plain text version of this content. The canonical link for it is here.
Posted to modperl@perl.apache.org by Jeff Ambrosino <jb...@gmail.com> on 2005/05/01 22:29:56 UTC
how to trap connection reset/abort in Apache::Filter?
I've got an HTTP output filter running on mp2-RC4 + httpd2.0.54 +
libapreq2-2.04_03-dev, and occasionally I see these messages in my
error_log:
[Sun May 01 05:12:21 2005] [error] [client X.X.X.X]
Apache::Filter::print: (104)Connection reset by peer at
/var/httpd/lib/perl/MyOutputFilter.pm line
453n\tMyOutputFilter::handler('Apache::Filter=SCALAR(0x87106a8)',
'APR::Brigade=SCALAR(0x86eac6c)') called at -e line 0\n\teval {...}
called at -e line 0\nApache::Filter: (103) Software caused connection
abort at -e line 0
Line 453, as referenced in this error, is:
$f->print ($f->ctx) if ( defined($f->ctx) );
I assume the errors are due to the end user clicking "stop" or
navigating away from the page before it has finished being sent...
but is there a [more elegant] way of trapping these events from within
my mod_perl filter?
thanks
Jeff
Re: how to trap connection reset/abort in Apache::Filter?
Posted by Stas Bekman <st...@stason.org>.
Jeff Ambrosino wrote:
> Stas, just a quick "closure"... the patch has survived several ScanAlert
> scans without any aborted httpd children. This was the original way I
> discovered the problem (www.scanalert.com - they periodically throw
> all sorts of weird hack/exploit traffic at our server as an audit.)
> So from my end, the problem is officially fixed.
Excellent. Now committed.
Thanks Jeff.
--
__________________________________________________________________
Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide ---> http://perl.apache.org
mailto:stas@stason.org http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org http://ticketmaster.com
Re: how to trap connection reset/abort in Apache::Filter?
Posted by Stas Bekman <st...@stason.org>.
Jeff Ambrosino wrote:
> Stas, just a quick "closure"... the patch has survived several ScanAlert
> scans without any aborted httpd children. This was the original way I
> discovered the problem (www.scanalert.com - they periodically throw
> all sorts of weird hack/exploit traffic at our server as an audit.)
> So from my end, the problem is officially fixed.
Excellent. Now committed.
Thanks Jeff.
--
__________________________________________________________________
Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide ---> http://perl.apache.org
mailto:stas@stason.org http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org http://ticketmaster.com
Re: how to trap connection reset/abort in Apache::Filter?
Posted by Stas Bekman <st...@stason.org>.
Jeff? Could you please try that patch that I've sent earlier? I think it's
right, but I was still not very successful at reproducing your problem on
the HTTP level. Thanks.
Stas Bekman wrote:
> Stas Bekman wrote:
>
>> Jeff Ambrosino wrote:
>>
>>> Ok, this is going to be tough for me to test because I'm still on RC4,
>>> so it's not yet converted to the new Apache2 naming. I'll still try
>>> to make/test the svn snapshot, though (which also gives me a good
>>> reason to get more familiar with subversion.) I'll email what I find
>>> out.
>>>
>>> thanks Stas!
>>
>>
>>
>> OK, may be just try comparing to:
>>
>> $@ == APR::Const::ECONNABORTED or $@ == APR::Const::ECONNRESET
>>
>> The new subs just supposed to work on more platforms, where other
>> similar codes may be sent by the system.
>>
>> You can still play with the test
>> t/protocol/TestProtocol/echo_nonblock.pm from RC4.
>>
>> but looking again at your report, I can see that the issue is elsewhere.
>>
>> I believe it's coming from src/modules/perl/modperl_filter.c:
>>
>> if (filter->mode == MP_INPUT_FILTER_MODE) {
>> ...
>> }
>> else {
>> MP_RUN_CROAK(modperl_output_filter_flush(filter),
>> "Apache2::Filter");
>> }
>>
>> is that it? if you change the "Apache2::Filter" string (drop the 2 in
>> your case) to something else?
>
>
> If this is the case, please try this patch, I hope it applies to RC4
> (I've dropped '2' for you)
>
> Index: src/modules/perl/modperl_filter.c
> ===================================================================
> --- src/modules/perl/modperl_filter.c (revision 169708)
> +++ src/modules/perl/modperl_filter.c (working copy)
> @@ -535,11 +535,15 @@
> filter->bb_in = NULL;
> }
> MP_RUN_CROAK(modperl_input_filter_flush(filter),
> - "Apache::Filter");
> + "Apache::Filter internal flush");
> }
> else {
> - MP_RUN_CROAK(modperl_output_filter_flush(filter),
> - "Apache::Filter");
> + apr_status_t rc = modperl_output_filter_flush(filter);
> + if (!(rc == APR_SUCCESS ||
> + APR_STATUS_IS_ECONNRESET(rc) ||
> + APR_STATUS_IS_ECONNABORTED(rc))) {
> + modperl_croak(aTHX_ rc, "Apache::Filter internal flush");
> + }
> }
>
> MP_FILTER_RESTORE_ERRSV(errsv);
>
> I need to figure out how to write a test for this, so I can decide
> what's the right way to handle this.
>
--
__________________________________________________________________
Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide ---> http://perl.apache.org
mailto:stas@stason.org http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org http://ticketmaster.com
Re: how to trap connection reset/abort in Apache::Filter?
Posted by Stas Bekman <st...@stason.org>.
Stas Bekman wrote:
> Jeff Ambrosino wrote:
>
>> Ok, this is going to be tough for me to test because I'm still on RC4,
>> so it's not yet converted to the new Apache2 naming. I'll still try
>> to make/test the svn snapshot, though (which also gives me a good
>> reason to get more familiar with subversion.) I'll email what I find
>> out.
>>
>> thanks Stas!
>
>
> OK, may be just try comparing to:
>
> $@ == APR::Const::ECONNABORTED or $@ == APR::Const::ECONNRESET
>
> The new subs just supposed to work on more platforms, where other
> similar codes may be sent by the system.
>
> You can still play with the test
> t/protocol/TestProtocol/echo_nonblock.pm from RC4.
>
> but looking again at your report, I can see that the issue is elsewhere.
>
> I believe it's coming from src/modules/perl/modperl_filter.c:
>
> if (filter->mode == MP_INPUT_FILTER_MODE) {
> ...
> }
> else {
> MP_RUN_CROAK(modperl_output_filter_flush(filter),
> "Apache2::Filter");
> }
>
> is that it? if you change the "Apache2::Filter" string (drop the 2 in
> your case) to something else?
If this is the case, please try this patch, I hope it applies to RC4 (I've
dropped '2' for you)
Index: src/modules/perl/modperl_filter.c
===================================================================
--- src/modules/perl/modperl_filter.c (revision 169708)
+++ src/modules/perl/modperl_filter.c (working copy)
@@ -535,11 +535,15 @@
filter->bb_in = NULL;
}
MP_RUN_CROAK(modperl_input_filter_flush(filter),
- "Apache::Filter");
+ "Apache::Filter internal flush");
}
else {
- MP_RUN_CROAK(modperl_output_filter_flush(filter),
- "Apache::Filter");
+ apr_status_t rc = modperl_output_filter_flush(filter);
+ if (!(rc == APR_SUCCESS ||
+ APR_STATUS_IS_ECONNRESET(rc) ||
+ APR_STATUS_IS_ECONNABORTED(rc))) {
+ modperl_croak(aTHX_ rc, "Apache::Filter internal flush");
+ }
}
MP_FILTER_RESTORE_ERRSV(errsv);
I need to figure out how to write a test for this, so I can decide what's
the right way to handle this.
--
__________________________________________________________________
Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide ---> http://perl.apache.org
mailto:stas@stason.org http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org http://ticketmaster.com
Re: how to trap connection reset/abort in Apache::Filter?
Posted by Stas Bekman <st...@stason.org>.
Jeff Ambrosino wrote:
> Ok, this is going to be tough for me to test because I'm still on RC4,
> so it's not yet converted to the new Apache2 naming. I'll still try
> to make/test the svn snapshot, though (which also gives me a good
> reason to get more familiar with subversion.) I'll email what I find
> out.
>
> thanks Stas!
OK, may be just try comparing to:
$@ == APR::Const::ECONNABORTED or $@ == APR::Const::ECONNRESET
The new subs just supposed to work on more platforms, where other similar
codes may be sent by the system.
You can still play with the test t/protocol/TestProtocol/echo_nonblock.pm
from RC4.
but looking again at your report, I can see that the issue is elsewhere.
I believe it's coming from src/modules/perl/modperl_filter.c:
if (filter->mode == MP_INPUT_FILTER_MODE) {
...
}
else {
MP_RUN_CROAK(modperl_output_filter_flush(filter),
"Apache2::Filter");
}
is that it? if you change the "Apache2::Filter" string (drop the 2 in your
case) to something else?
--
__________________________________________________________________
Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide ---> http://perl.apache.org
mailto:stas@stason.org http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org http://ticketmaster.com
Re: how to trap connection reset/abort in Apache::Filter?
Posted by Jeff Ambrosino <jb...@gmail.com>.
Ok, this is going to be tough for me to test because I'm still on RC4,
so it's not yet converted to the new Apache2 naming. I'll still try
to make/test the svn snapshot, though (which also gives me a good
reason to get more familiar with subversion.) I'll email what I find
out.
thanks Stas!
Jeff
On 5/11/05, Stas Bekman <st...@stason.org> wrote:
> I'm playing with this issue. I've notice that we already have a reliably
> reproducable issue with t/protocol/echo_nonblock.t which calls
> t/protocol/TestProtocol/echo_nonblock.pm. Could you please try the current
> svn (if you go for the snapshot please wait for 6 hours, since I've just
> committed things) [1]
>
> There I now use the code:
[.......]
Re: how to trap connection reset/abort in Apache::Filter?
Posted by Stas Bekman <st...@stason.org>.
Jeff Ambrosino wrote:
> Hi Stas & MP list,
>
> I tried causing this error by opening sockets to the web server, then
> killing (-9) the client, but that didn't work. However, every morning
> like clockwork the scanning service (ScanAlert) hits our site and
> causes this problem... so, while I cannot intentionally cause the
> error by myself, I know when our scanning service will do so, and I
> could put some additional code in place to watch for it. If there are
> additional debug/traps that you could suggest, I'd like to try. Any
> ideas?
I'm playing with this issue. I've notice that we already have a reliably
reproducable issue with t/protocol/echo_nonblock.t which calls
t/protocol/TestProtocol/echo_nonblock.pm. Could you please try the current
svn (if you go for the snapshot please wait for 6 hours, since I've just
committed things) [1]
There I now use the code:
die $@ unless ref $@ eq 'APR::Error'
&& (APR::Status::is_ECONNABORTED($@) ||
APR::Status::is_ECONNRESET($@));
can you also see that die doesn't appear in the error_log of the test
suite? and if you just do
die $@;
it logs the error?
t/TEST -v t/protocol/echo_nonblock.t
and check t/logs/error_log
[1]
http://perl.apache.org/download/source.html#Development_mod_perl_2_0_Source_Distribution
--
__________________________________________________________________
Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide ---> http://perl.apache.org
mailto:stas@stason.org http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org http://ticketmaster.com
Re: how to trap connection reset/abort in Apache::Filter?
Posted by Jeff Ambrosino <jb...@gmail.com>.
Hi Stas & MP list,
I tried causing this error by opening sockets to the web server, then
killing (-9) the client, but that didn't work. However, every morning
like clockwork the scanning service (ScanAlert) hits our site and
causes this problem... so, while I cannot intentionally cause the
error by myself, I know when our scanning service will do so, and I
could put some additional code in place to watch for it. If there are
additional debug/traps that you could suggest, I'd like to try. Any
ideas?
thanks
Jeff
On 5/2/05, Stas Bekman <st...@stason.org> wrote:
> Jeff Ambrosino wrote:
> > Ok, I tried this approach, and it only partially worked; unfortunately
> > my httpd procs are still dying, and APR::Error doesn't seem to pass
> > anything. Here's the error I got _before_ wrapping $f->print in the
> > eval:
> >
> > [Sun May 01 05:12:21 2005] [error] [client X.X.X.X]
> > Apache::Filter::print: (104)Connection reset by peer at
> > /var/httpd/lib/perl/MyFilter.pm line
> > 453n\tMyFilter::handler('Apache::Filter=SCALAR0x87106a8)',
> > 'APR::Brigade=SCALAR(0x86eac6c)') called at -e line 0\n\teval {...}
> > called at -e line 0\n
> > Apache::Filter: (103) Software caused connection abort at -e line 0
> >
> > And here's how it looks in the error+log _after_ putting in the eval wrapper:
> >
> > Apache::Filter: (103) Software caused connection abort at -e line 0
> >
> > Notice there's no timestamp next to the Apache::Filter error, so it
> > appears that something is writing that message directly to STDERR.
> >
> > The main problem is that with or without the eval, the httpd process
> > is getting killed, and it seems to be taking a while for the httpd
> > parent to realize this and start new procs (I'm using MPM prefork.)
> >
> > Here's the code I'm using to trap the $f->print failure:
> >
> > if ( defined($f->ctx) ) {
> > eval { $f->print ($f->ctx) };
> >
> > if ($@ && ref $@) {
> > $r->log_error("WARNING " . "\$f->print
> > exception: $@");
> > }
> > }
> >
> > Any suggestions?
> >
> > And FWIW, I haven't been able to reproduce these conditions
> > artificially, I just have to wait until it occurs in the wild. A
> > "wild" event that occurs regularly, however, is a daily vulnerability
> > scan by ScanAlert (a service that we pay for.) They throw all sorts
> > of potential http exploits at our server and notify us of any problems
> > - I'm guessing some of their tests do some funky socket operations
> > that cause Apache::FIlter to choke.
>
> I think the first step is be able to reliably reproduce that. Let me see
> if I can figure out how to do that.
>
> --
> __________________________________________________________________
> Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
> http://stason.org/ mod_perl Guide ---> http://perl.apache.org
> mailto:stas@stason.org http://use.perl.org http://apacheweek.com
> http://modperlbook.org http://apache.org http://ticketmaster.com
>
Re: how to trap connection reset/abort in Apache::Filter?
Posted by Stas Bekman <st...@stason.org>.
Jeff Ambrosino wrote:
> Ok, I tried this approach, and it only partially worked; unfortunately
> my httpd procs are still dying, and APR::Error doesn't seem to pass
> anything. Here's the error I got _before_ wrapping $f->print in the
> eval:
>
> [Sun May 01 05:12:21 2005] [error] [client X.X.X.X]
> Apache::Filter::print: (104)Connection reset by peer at
> /var/httpd/lib/perl/MyFilter.pm line
> 453n\tMyFilter::handler('Apache::Filter=SCALAR0x87106a8)',
> 'APR::Brigade=SCALAR(0x86eac6c)') called at -e line 0\n\teval {...}
> called at -e line 0\n
> Apache::Filter: (103) Software caused connection abort at -e line 0
>
> And here's how it looks in the error+log _after_ putting in the eval wrapper:
>
> Apache::Filter: (103) Software caused connection abort at -e line 0
>
> Notice there's no timestamp next to the Apache::Filter error, so it
> appears that something is writing that message directly to STDERR.
>
> The main problem is that with or without the eval, the httpd process
> is getting killed, and it seems to be taking a while for the httpd
> parent to realize this and start new procs (I'm using MPM prefork.)
>
> Here's the code I'm using to trap the $f->print failure:
>
> if ( defined($f->ctx) ) {
> eval { $f->print ($f->ctx) };
>
> if ($@ && ref $@) {
> $r->log_error("WARNING " . "\$f->print
> exception: $@");
> }
> }
>
> Any suggestions?
>
> And FWIW, I haven't been able to reproduce these conditions
> artificially, I just have to wait until it occurs in the wild. A
> "wild" event that occurs regularly, however, is a daily vulnerability
> scan by ScanAlert (a service that we pay for.) They throw all sorts
> of potential http exploits at our server and notify us of any problems
> - I'm guessing some of their tests do some funky socket operations
> that cause Apache::FIlter to choke.
I think the first step is be able to reliably reproduce that. Let me see
if I can figure out how to do that.
--
__________________________________________________________________
Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide ---> http://perl.apache.org
mailto:stas@stason.org http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org http://ticketmaster.com
Re: how to trap connection reset/abort in Apache::Filter?
Posted by Jeff Ambrosino <jb...@gmail.com>.
Ok, I tried this approach, and it only partially worked; unfortunately
my httpd procs are still dying, and APR::Error doesn't seem to pass
anything. Here's the error I got _before_ wrapping $f->print in the
eval:
[Sun May 01 05:12:21 2005] [error] [client X.X.X.X]
Apache::Filter::print: (104)Connection reset by peer at
/var/httpd/lib/perl/MyFilter.pm line
453n\tMyFilter::handler('Apache::Filter=SCALAR0x87106a8)',
'APR::Brigade=SCALAR(0x86eac6c)') called at -e line 0\n\teval {...}
called at -e line 0\n
Apache::Filter: (103) Software caused connection abort at -e line 0
And here's how it looks in the error+log _after_ putting in the eval wrapper:
Apache::Filter: (103) Software caused connection abort at -e line 0
Notice there's no timestamp next to the Apache::Filter error, so it
appears that something is writing that message directly to STDERR.
The main problem is that with or without the eval, the httpd process
is getting killed, and it seems to be taking a while for the httpd
parent to realize this and start new procs (I'm using MPM prefork.)
Here's the code I'm using to trap the $f->print failure:
if ( defined($f->ctx) ) {
eval { $f->print ($f->ctx) };
if ($@ && ref $@) {
$r->log_error("WARNING " . "\$f->print
exception: $@");
}
}
Any suggestions?
And FWIW, I haven't been able to reproduce these conditions
artificially, I just have to wait until it occurs in the wild. A
"wild" event that occurs regularly, however, is a daily vulnerability
scan by ScanAlert (a service that we pay for.) They throw all sorts
of potential http exploits at our server and notify us of any problems
- I'm guessing some of their tests do some funky socket operations
that cause Apache::FIlter to choke.
thanks
Jeff
On 5/2/05, Stas Bekman <st...@stason.org> wrote:
> > Actually, it looks like using APR::Error [1] and wrapping $f->print in
> > an eval would do the trick...?
>
> That's correct, Jeff. eval and check for APR::Const::ECONNRESET as
> explained in [1]
>
> > [1] http://perl.apache.org/docs/2.0/api/APR/Error.html
>
> Please report whether this did the trick. We ought to document this one,
> since many will encounter this issue.
Re: how to trap connection reset/abort in Apache::Filter?
Posted by Stas Bekman <st...@stason.org>.
Jeff Ambrosino wrote:
> Actually, it looks like using APR::Error [1] and wrapping $f->print in
> an eval would do the trick...?
That's correct, Jeff. eval and check for APR::Const::ECONNRESET as
explained in [1]
> [1] http://perl.apache.org/docs/2.0/api/APR/Error.html
Please report whether this did the trick. We ought to document this one,
since many will encounter this issue.
> On 5/1/05, Jeff Ambrosino <jb...@gmail.com> wrote:
>
>>[Sun May 01 05:12:21 2005] [error] [client X.X.X.X]
>>Apache::Filter::print: (104)Connection reset by peer at
>>/var/httpd/lib/perl/MyOutputFilter.pm line
>>453n\tMyOutputFilter::handler('Apache::Filter=SCALAR(0x87106a8)',
>>'APR::Brigade=SCALAR(0x86eac6c)') called at -e line 0\n\teval {...}
>>called at -e line 0\nApache::Filter: (103) Software caused connection
>>abort at -e line 0
>
> ....
>
>>I assume the errors are due to the end user clicking "stop" or
>>navigating away from the page before it has finished being sent...
>>but is there a [more elegant] way of trapping these events from within
>>my mod_perl filter
>
> ?
--
__________________________________________________________________
Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide ---> http://perl.apache.org
mailto:stas@stason.org http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org http://ticketmaster.com
Re: how to trap connection reset/abort in Apache::Filter?
Posted by Jeff Ambrosino <jb...@gmail.com>.
Actually, it looks like using APR::Error [1] and wrapping $f->print in
an eval would do the trick...?
Jeff
[1] http://perl.apache.org/docs/2.0/api/APR/Error.html
On 5/1/05, Jeff Ambrosino <jb...@gmail.com> wrote:
> [Sun May 01 05:12:21 2005] [error] [client X.X.X.X]
> Apache::Filter::print: (104)Connection reset by peer at
> /var/httpd/lib/perl/MyOutputFilter.pm line
> 453n\tMyOutputFilter::handler('Apache::Filter=SCALAR(0x87106a8)',
> 'APR::Brigade=SCALAR(0x86eac6c)') called at -e line 0\n\teval {...}
> called at -e line 0\nApache::Filter: (103) Software caused connection
> abort at -e line 0
....
> I assume the errors are due to the end user clicking "stop" or
> navigating away from the page before it has finished being sent...
> but is there a [more elegant] way of trapping these events from within
> my mod_perl filter?