You are viewing a plain text version of this content. The canonical link for it is here.
Posted to modperl@perl.apache.org by Heiko Weber <he...@wecos.de> on 2010/02/05 11:25:27 UTC
Re: [mp2] mod_perl closes apache's stdin and/or stdout
Dear List-Members,
with interest I found the below thread. Starting in Oct. or Nov. last year I am getting a lot of messages in apaches error_log like:
[Fri Feb 5 11:07:09 2010] -e: Software caused connection abort at ...
And it always happen in a print to STDOUT. I notice that it also happen with smaller scripts (running under mod_perl) with no database connection, i.e. scripts which do the following:
use CGI;
use IO::File;
my $q = CGI->new();
print $q->header() . $q->start_html();
my $fp = IO::File->new('< /somewhere/');
if ($fp) {
binmode STDOUT;
while (read($fp, $buffer, 1024)) {
print $buffer; # << abort points to here !!!
}
$fh->close();
}
print $q->end_html();
So I am really wondering whats going on here. The above file works for years now, has not been touched and the content of the opened files isn't empty. The server is a FreeBSD 7.0, apache apache-2.2.14, prefork MPM, mod_perl2-2.0.4 everything from a current freebsd ports.
I use a perl-startup script for apache:
---snip--snip---
#/usr/bin/perl
use CGI();
CGI->compile(':all');
use DBI;
DBI->install_driver("mysql");
use Carp;
---snap---snap---
which is loaded within httpd.conf with:
<IfModule mod_perl.c>
PerlWarn On
PerlTaintCheck On
PerlModule Apache::DBI
PerlRequire /usr/local/etc/apache22/perl-startup.pl
<Perl>
use CGI::Carp;
$SIG{__DIE__} = sub { confess shift };
$SIG{__WARN__} = \&Carp::cluck;
</Perl>
<Files *.pl>
SetHandler perl-script
PerlHandler ModPerl::Registry
Options ExecCGI
PerlSendHeader On
</Files>
</IfModule>
I would appreciate any help or ideas to get rid of the aborts.
Heiko
> On Tue, Jan 26, 2010 at 5:20 PM, Jonathan Swartz <sw...@pobox.com> wrote:
>> This never got a response. Which surprises me, since I think it is a
>> legitimate and nasty bug.
>>
>> So is the silence because
>> 1) people don't think it's really a bug
>> 2) people glazed over while reading the description
>> 3) ??
>
> 4) Don't understand how widespread or common this issue is, or if you
> are the only one seeing it.
>
>>
>> Thanks :)
>> Jon
>>
>> On Jan 8, 2010, at 6:15 AM, Jonathan Swartz wrote:
>>
>>> (A continuation of:
>>> http://marc.info/?l=apache-modperl&m=117507879929572&w=2
>>> http://marc.info/?l=apache-modperl&m=119072925228529&w=2
>>> )
>>>
>>> I've just spent many frustrating days debugging a situation that turned
>>> out to be caused by mod_perl's closing of file descriptor 1 (STDOUT).
>>>
>>> Here's the reproducible case I ultimately got it down to. Using mod_perl
>>> 2, with a dead-simple configuration and this handler:
>>>
>>> use DBI;
>>> sub handler {
>>> my $dbh = DBI->connect( "DBI:mysql:$database", $user, $pass, {
>>> RaiseError => 1 } );
>>> system('echo "hello"');
>>> eval { $dbh->do("select 1") };
>>> print "dbh - " . ( $@ ? "error: $@\n" : "ok" ) . "\n";
>>> return 0;
>>> }
>>>
>>> This outputs:
>>>
>>> dbh - error: DBD::mysql::db do failed: Lost connection to MySQL server
>>> during query at...
>>>
>>> The DBI connection dies because mod_perl closes fd 1 (STDOUT). So the next
>>> open - in this case the remote mysql connection created by DBI - gets fd 1.
>>> The child process created by system() writes innocently to STDOUT, which
>>> goes to our mysql socket, causing havoc.
>>>
>>> We can confirm this by inserting this at the beginning of the handler:
>>>
>>> sub handler {
>>> open(my $fh, ">/dev/null");
>>> print "fh - " . fileno($fh) . "\n";
>>> ...
>>>
>>> Now this outputs:
>>>
>>> fh - 1
>>> dbh - ok
>>>
>>> The initial open grabs fd 1, which means that DBI gets a different fd, and
>>> the connection doesn't die.
>>>
>>> Now this example is contrived, but replace 'echo "hello"' with any
>>> innocuous system() or backtick call that accidentally sends a bit of output
>>> to STDOUT, and you can see how this would cause all kinds of baffling bugs.
>>> In my case, it was originally a call to "sendmail" that intermittently sent
>>> a warning to STDOUT, and thus destroyed our first database connection. It
>>> worked fine in mod_perl 1, but started breaking when we upgraded to mod_perl
>>> 2.
>>>
>>> Is there really no way to fix this in mod_perl, perhaps by automatically
>>> opening a /dev/null handle as I did above? Other future developers will
>>> surely endure the same hours of frustration I did if it is left alone.
>>>
>>> Thanks
>>> Jon
>>>
>>
>>
Fwd: [mp2] mod_perl closes apache's stdin and/or stdout
Posted by Fred Moyer <fr...@redhotpenguin.com>.
Does httpd provide an api through which file descriptors can be
reallocated (probably with XS I'm guessing) into Apache::Fd or
something near?
---------- Forwarded message ----------
From: Heiko Weber <he...@wecos.de>
Date: Thu, Feb 18, 2010 at 12:21 AM
Subject: Re: [mp2] mod_perl closes apache's stdin and/or stdout
To: Salvador Ortiz Garcia <so...@msg.com.mx>
Cc: mod_perl list <mo...@perl.apache.org>
Salvador,
to avoid such issues my "external" tasks don't use STDOUT, STDIN or
STDERR. They take their parameters from control files and write their
results back to a status file. This tasks don't send any output back
to the browsers. As I said, usually some "sudo's to change some system
settings.
Well, I could replace all system() calls and just store the task jobs
into a database table, to schedule a background job with cron to check
and complete this tasks, but then I lost the immediately feedback to
the user/browser ... AND this is a lot of work for me - unless I can
exactly repeat the issue I am not sure if it is worth to try it.
Currently it feels to me like a "leakage", sometimes a
httpd/mod_perl/process do something, and later (maybe when working on
the next client request) STDOUT is closed. This makes it hard to
create a sample program to repeat it. Within a single script I can do
almost everything: call system(), open DBI connections, write to
STDOUT, ... everything seems to be fine.
Heiko
Am 16.02.2010 um 23:26 schrieb Salvador Ortiz Garcia:
On 02/05/2010 12:16 PM, Heiko Weber wrote:
Hi André,
I know what you mean, and I can't agree with you - the server response
time is really low - most pages are finished loading in less 1-2
seconds, and the overall load of the server is at a low level. I
believe there is an issue, maybe something what Jon is talking about,
I also using some "system()" call's to execute sendmail or sudo tasks,
so maybe STDOUT really gets closed - I have no idea. I only see the
abort messages in errorlog very frequent, maybe 3-4 per minute.
Heiko
Of course STDOUT, STDIN, and STDERR get closed, not by mod_perl but by
apache. Every UNIX process when demonized must close them to detach
from its controling terminal.
Apache reopens STDERR to its log file, but STDOUT and STDIN remain closed.
Maybe de confusion arises from the fact that when executing a CGI the
server connects them to the client socket, but in mod_perl you are in
the deamon process space.
Please check "Advanced Programing in the UNIX environment", chapter
13: "Daemon processes" by W. Richard Stevens.
Regards.
Salvador.
--
Wecos <> Heiko Weber Computer Systeme
D-21644 Sauensiek <> Immenweg 5
heiko@wecos.de <> http://www.wecos.de
Tel. +49 (4169) 91000 <> Fax +49 (4169) 919033
_______________________________________________________________
This email may contain confidential and privileged material for
the sole use of the intended recipient. Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient please contact the sender and delete all copies.
_______________________________________________________________
Diese E-Mail kann digital signiert sein. Falls Ihr E-Mail-Programm
nicht ueber die notwendigen Prueffunktionen verfuegt, ignorieren
Sie bitte die angehaengte Signatur-Datei.
_______________________________________________________________
This email may be digitally signed. If your email software does
not support the necessary validation feature, please disregard the
attached signature file.
_______________________________________________________________
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@perl.apache.org
For additional commands, e-mail: dev-help@perl.apache.org
Re: [mp2] mod_perl closes apache's stdin and/or stdout
Posted by Heiko Weber <he...@wecos.de>.
Wow, thanks Eric ! That seems to be very nice, I'll give it a try and will report later here if avoiding system() calls reduce the number of aborted connections.
Heiko
Am 18.02.2010 um 17:24 schrieb eric.berg@barclayscapital.com:
> I'm starting to use Gearman to get around this whole problem. We use a lot of external processes for many things, so this issue wtih Apache2 really bit me hard in the bee-hind. I've gone to great lengths to work around it, but so far the job queue approach seems to be the most elegant and least problematic approach. Of course, the recommendation came from the gentle folks on this list originally.
>
> Eric
>
> From: Heiko Weber [mailto:heiko@wecos.de]
> Sent: Thursday, February 18, 2010 3:22 AM
> To: Salvador Ortiz Garcia
> Cc: mod_perl list
> Subject: Re: [mp2] mod_perl closes apache's stdin and/or stdout
>
> Salvador,
>
> to avoid such issues my "external" tasks don't use STDOUT, STDIN or STDERR. They take their parameters from control files and write their results back to a status file. This tasks don't send any output back to the browsers. As I said, usually some "sudo's to change some system settings.
>
> Well, I could replace all system() calls and just store the task jobs into a database table, to schedule a background job with cron to check and complete this tasks, but then I lost the immediately feedback to the user/browser ... AND this is a lot of work for me - unless I can exactly repeat the issue I am not sure if it is worth to try it.
>
> Currently it feels to me like a "leakage", sometimes a httpd/mod_perl/process do something, and later (maybe when working on the next client request) STDOUT is closed. This makes it hard to create a sample program to repeat it. Within a single script I can do almost everything: call system(), open DBI connections, write to STDOUT, ... everything seems to be fine.
>
> Heiko
>
> Am 16.02.2010 um 23:26 schrieb Salvador Ortiz Garcia:
>
>> On 02/05/2010 12:16 PM, Heiko Weber wrote:
>>> Hi André,
>>>
>>> I know what you mean, and I can't agree with you - the server response time is really low - most pages are finished loading in less 1-2 seconds, and the overall load of the server is at a low level. I believe there is an issue, maybe something what Jon is talking about, I also using some "system()" call's to execute sendmail or sudo tasks, so maybe STDOUT really gets closed - I have no idea. I only see the abort messages in errorlog very frequent, maybe 3-4 per minute.
>>>
>>> Heiko
>>>
>> Of course STDOUT, STDIN, and STDERR get closed, not by mod_perl but by apache. Every UNIX process when demonized must close them to detach from its controling terminal.
>>
>> Apache reopens STDERR to its log file, but STDOUT and STDIN remain closed.
>>
>> Maybe de confusion arises from the fact that when executing a CGI the server connects them to the client socket, but in mod_perl you are in the deamon process space.
>>
>> Please check "Advanced Programing in the UNIX environment", chapter 13: "Daemon processes" by W. Richard Stevens.
>>
>> Regards.
>>
>> Salvador.
>
> --
> Wecos <> Heiko Weber Computer Systeme
> D-21644 Sauensiek <> Immenweg 5
> heiko@wecos.de <> http://www.wecos.de
> _______________________________________________
>
> This e-mail may contain information that is confidential, privileged or otherwise protected from disclosure. If you are not an intended recipient of this e-mail, do not duplicate or redistribute it by any means. Please delete it and any attachments and notify the sender that you have received it in error. Unless specifically indicated, this e-mail is not an offer to buy or sell or a solicitation to buy or sell any securities, investment products or other financial product or service, an official confirmation of any transaction, or an official statement of Barclays. Any views or opinions presented are solely those of the author and do not necessarily represent those of Barclays. This e-mail is subject to terms available at the following link: www.barcap.com/emaildisclaimer. By messaging with Barclays you consent to the foregoing. Barclays Capital is the investment banking division of Barclays Bank PLC, a company registered in England (number 1026167) with its registered office at 1 Churchill Place, London, E14 5HP. This email may relate to or be sent from other members of the Barclays Group.
> _______________________________________________
--
Wecos <> Heiko Weber Computer Systeme
D-21644 Sauensiek <> Immenweg 5
heiko@wecos.de <> http://www.wecos.de
Tel. +49 (4169) 91000 <> Fax +49 (4169) 919033
_______________________________________________________________
This email may contain confidential and privileged material for
the sole use of the intended recipient. Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient please contact the sender and delete all copies.
_______________________________________________________________
Diese E-Mail kann digital signiert sein. Falls Ihr E-Mail-Programm
nicht ueber die notwendigen Prueffunktionen verfuegt, ignorieren
Sie bitte die angehaengte Signatur-Datei.
_______________________________________________________________
This email may be digitally signed. If your email software does
not support the necessary validation feature, please disregard the
attached signature file.
_______________________________________________________________
RE: [mp2] mod_perl closes apache's stdin and/or stdout
Posted by er...@barclayscapital.com.
I'm starting to use Gearman to get around this whole problem. We use a lot of external processes for many things, so this issue wtih Apache2 really bit me hard in the bee-hind. I've gone to great lengths to work around it, but so far the job queue approach seems to be the most elegant and least problematic approach. Of course, the recommendation came from the gentle folks on this list originally.
Eric
________________________________
From: Heiko Weber [mailto:heiko@wecos.de]
Sent: Thursday, February 18, 2010 3:22 AM
To: Salvador Ortiz Garcia
Cc: mod_perl list
Subject: Re: [mp2] mod_perl closes apache's stdin and/or stdout
Salvador,
to avoid such issues my "external" tasks don't use STDOUT, STDIN or STDERR. They take their parameters from control files and write their results back to a status file. This tasks don't send any output back to the browsers. As I said, usually some "sudo's to change some system settings.
Well, I could replace all system() calls and just store the task jobs into a database table, to schedule a background job with cron to check and complete this tasks, but then I lost the immediately feedback to the user/browser ... AND this is a lot of work for me - unless I can exactly repeat the issue I am not sure if it is worth to try it.
Currently it feels to me like a "leakage", sometimes a httpd/mod_perl/process do something, and later (maybe when working on the next client request) STDOUT is closed. This makes it hard to create a sample program to repeat it. Within a single script I can do almost everything: call system(), open DBI connections, write to STDOUT, ... everything seems to be fine.
Heiko
Am 16.02.2010 um 23:26 schrieb Salvador Ortiz Garcia:
On 02/05/2010 12:16 PM, Heiko Weber wrote:
Hi André,
I know what you mean, and I can't agree with you - the server response time is really low - most pages are finished loading in less 1-2 seconds, and the overall load of the server is at a low level. I believe there is an issue, maybe something what Jon is talking about, I also using some "system()" call's to execute sendmail or sudo tasks, so maybe STDOUT really gets closed - I have no idea. I only see the abort messages in errorlog very frequent, maybe 3-4 per minute.
Heiko
Of course STDOUT, STDIN, and STDERR get closed, not by mod_perl but by apache. Every UNIX process when demonized must close them to detach from its controling terminal.
Apache reopens STDERR to its log file, but STDOUT and STDIN remain closed.
Maybe de confusion arises from the fact that when executing a CGI the server connects them to the client socket, but in mod_perl you are in the deamon process space.
Please check "Advanced Programing in the UNIX environment", chapter 13: "Daemon processes" by W. Richard Stevens.
Regards.
Salvador.
--
Wecos <> Heiko Weber Computer Systeme
D-21644 Sauensiek <> Immenweg 5
heiko@wecos.de<ma...@wecos.de> <> http://www.wecos.de<http://www.wecos.de/>
_______________________________________________
This e-mail may contain information that is confidential, privileged or otherwise protected from disclosure. If you are not an intended recipient of this e-mail, do not duplicate or redistribute it by any means. Please delete it and any attachments and notify the sender that you have received it in error. Unless specifically indicated, this e-mail is not an offer to buy or sell or a solicitation to buy or sell any securities, investment products or other financial product or service, an official confirmation of any transaction, or an official statement of Barclays. Any views or opinions presented are solely those of the author and do not necessarily represent those of Barclays. This e-mail is subject to terms available at the following link: www.barcap.com/emaildisclaimer. By messaging with Barclays you consent to the foregoing. Barclays Capital is the investment banking division of Barclays Bank PLC, a company registered in England (number 1026167) with its registered office at 1 Churchill Place, London, E14 5HP. This email may relate to or be sent from other members of the Barclays Group.
_______________________________________________
Re: [mp2] mod_perl closes apache's stdin and/or stdout
Posted by Heiko Weber <he...@wecos.de>.
Salvador,
to avoid such issues my "external" tasks don't use STDOUT, STDIN or STDERR. They take their parameters from control files and write their results back to a status file. This tasks don't send any output back to the browsers. As I said, usually some "sudo's to change some system settings.
Well, I could replace all system() calls and just store the task jobs into a database table, to schedule a background job with cron to check and complete this tasks, but then I lost the immediately feedback to the user/browser ... AND this is a lot of work for me - unless I can exactly repeat the issue I am not sure if it is worth to try it.
Currently it feels to me like a "leakage", sometimes a httpd/mod_perl/process do something, and later (maybe when working on the next client request) STDOUT is closed. This makes it hard to create a sample program to repeat it. Within a single script I can do almost everything: call system(), open DBI connections, write to STDOUT, ... everything seems to be fine.
Heiko
Am 16.02.2010 um 23:26 schrieb Salvador Ortiz Garcia:
> On 02/05/2010 12:16 PM, Heiko Weber wrote:
>> Hi André,
>>
>> I know what you mean, and I can't agree with you - the server response time is really low - most pages are finished loading in less 1-2 seconds, and the overall load of the server is at a low level. I believe there is an issue, maybe something what Jon is talking about, I also using some "system()" call's to execute sendmail or sudo tasks, so maybe STDOUT really gets closed - I have no idea. I only see the abort messages in errorlog very frequent, maybe 3-4 per minute.
>>
>> Heiko
>>
> Of course STDOUT, STDIN, and STDERR get closed, not by mod_perl but by apache. Every UNIX process when demonized must close them to detach from its controling terminal.
>
> Apache reopens STDERR to its log file, but STDOUT and STDIN remain closed.
>
> Maybe de confusion arises from the fact that when executing a CGI the server connects them to the client socket, but in mod_perl you are in the deamon process space.
>
> Please check "Advanced Programing in the UNIX environment", chapter 13: "Daemon processes" by W. Richard Stevens.
>
> Regards.
>
> Salvador.
--
Wecos <> Heiko Weber Computer Systeme
D-21644 Sauensiek <> Immenweg 5
heiko@wecos.de <> http://www.wecos.de
Tel. +49 (4169) 91000 <> Fax +49 (4169) 919033
_______________________________________________________________
This email may contain confidential and privileged material for
the sole use of the intended recipient. Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient please contact the sender and delete all copies.
_______________________________________________________________
Diese E-Mail kann digital signiert sein. Falls Ihr E-Mail-Programm
nicht ueber die notwendigen Prueffunktionen verfuegt, ignorieren
Sie bitte die angehaengte Signatur-Datei.
_______________________________________________________________
This email may be digitally signed. If your email software does
not support the necessary validation feature, please disregard the
attached signature file.
_______________________________________________________________
Re: [mp2] mod_perl closes apache's stdin and/or stdout
Posted by Salvador Ortiz Garcia <so...@msg.com.mx>.
On 02/05/2010 12:16 PM, Heiko Weber wrote:
> Hi André,
>
> I know what you mean, and I can't agree with you - the server response time is really low - most pages are finished loading in less 1-2 seconds, and the overall load of the server is at a low level. I believe there is an issue, maybe something what Jon is talking about, I also using some "system()" call's to execute sendmail or sudo tasks, so maybe STDOUT really gets closed - I have no idea. I only see the abort messages in errorlog very frequent, maybe 3-4 per minute.
>
> Heiko
>
Of course STDOUT, STDIN, and STDERR get closed, not by mod_perl but by
apache. Every UNIX process when demonized must close them to detach
from its controling terminal.
Apache reopens STDERR to its log file, but STDOUT and STDIN remain closed.
Maybe de confusion arises from the fact that when executing a CGI the
server connects them to the client socket, but in mod_perl you are in
the deamon process space.
Please check "Advanced Programing in the UNIX environment", chapter 13:
"Daemon processes" by W. Richard Stevens.
Regards.
Salvador.
Re: [mp2] mod_perl closes apache's stdin and/or stdout
Posted by Heiko Weber <he...@wecos.de>.
Hi André,
I know what you mean, and I can't agree with you - the server response time is really low - most pages are finished loading in less 1-2 seconds, and the overall load of the server is at a low level. I believe there is an issue, maybe something what Jon is talking about, I also using some "system()" call's to execute sendmail or sudo tasks, so maybe STDOUT really gets closed - I have no idea. I only see the abort messages in errorlog very frequent, maybe 3-4 per minute.
Heiko
Am 05.02.2010 um 15:49 schrieb André Warnier:
>
> Heiko Weber wrote:
>> Dear List-Members,
>> with interest I found the below thread. Starting in Oct. or Nov. last year I am getting a lot of messages in apaches error_log like:
>> [Fri Feb 5 11:07:09 2010] -e: Software caused connection abort at ...
>> And it always happen in a print to STDOUT. I notice that it also happen with smaller scripts (running under mod_perl) with no database connection, i.e. scripts which do the following:
> ...
>
>> So I am really wondering whats going on here. The above file works for years now, has not been touched and the content of the opened files isn't empty. The server is a FreeBSD 7.0, apache apache-2.2.14, prefork MPM, mod_perl2-2.0.4 everything from a current freebsd ports.
>
> This seems to be happening when your server-side module is trying to send data back to the browser who requested it.
> The most common explanation is that, by the time your module tries to send the answer, the browser connection does not exist anymore, because the browser (or something in-between the server and browser) closed it.
> This happens for example when the user clicks on a link which triggers your module, then changes his mind and clicks somewhere else (or closes the window or the browser) before your module has finished sending the response.
> In other words, he hung up on you.
>
> It is quite frequent and nothing to worry about in principle.
>
> Where it would get more worrying, is that it could indicate that your application is taking too long to send back a result to the user, and that he's losing patience.
> If a user clicks on a link and expects an answer, he will usually wait only 10-20 seconds maximum before starting to worry. He may then just click again on the same link, which would have the same effect.
Re: [mp2] mod_perl closes apache's stdin and/or stdout
Posted by André Warnier <aw...@ice-sa.com>.
Heiko Weber wrote:
> Dear List-Members,
>
> with interest I found the below thread. Starting in Oct. or Nov. last year I am getting a lot of messages in apaches error_log like:
>
> [Fri Feb 5 11:07:09 2010] -e: Software caused connection abort at ...
>
> And it always happen in a print to STDOUT. I notice that it also happen with smaller scripts (running under mod_perl) with no database connection, i.e. scripts which do the following:
>
...
>
> So I am really wondering whats going on here. The above file works for years now, has not been touched and the content of the opened files isn't empty. The server is a FreeBSD 7.0, apache apache-2.2.14, prefork MPM, mod_perl2-2.0.4 everything from a current freebsd ports.
>
This seems to be happening when your server-side module is trying to
send data back to the browser who requested it.
The most common explanation is that, by the time your module tries to
send the answer, the browser connection does not exist anymore, because
the browser (or something in-between the server and browser) closed it.
This happens for example when the user clicks on a link which triggers
your module, then changes his mind and clicks somewhere else (or closes
the window or the browser) before your module has finished sending the
response.
In other words, he hung up on you.
It is quite frequent and nothing to worry about in principle.
Where it would get more worrying, is that it could indicate that your
application is taking too long to send back a result to the user, and
that he's losing patience.
If a user clicks on a link and expects an answer, he will usually wait
only 10-20 seconds maximum before starting to worry. He may then just
click again on the same link, which would have the same effect.