You are viewing a plain text version of this content. The canonical link for it is here.
Posted to modperl@perl.apache.org by Alex Greg <ag...@digitallook.com> on 2004/09/30 14:50:21 UTC
mod_perl crashing / hanging
Hi,
We've been having some issues with our mod_perl web servers recently.
Basically what happens is the server becomes very very slow, to the
point of being almost totally unresponsive. At the time this happens, we
see this in the error log:
[Thu Sep 30 09:44:43 2004] [error] (12)Cannot allocate memory: fork:
Unable to fork new process
This leaves us with one Apache child running, which doesn't respond to
requests. The only way to restore service to the machine is to kill
Apache manually then start it up again.
This may be related, but at the time this occurs mod_perl seems to have
difficulty getting the date from the server, i.e. we get this in our
error logs as well:
[Thu Sep 30 00:20:20 2004] [error] Hour '2004-- 00' out of range 0..23
at /www/cgi-bin/Shared/DateStuff.pm line 1111\n
This is because a shell call to date ($date = `date`;) is failing - this
is most likely what's causing the failure to fork we see above.
We're running Fedora Core 1 on IBM eServers with a custom-built mod_perl
httpd. Apache is built with the following command:
perl Makefile.PL EVERYTHING=1 USE_APACI=1 DYNAMIC=1 APACHE_PREFIX=/www/
APACHE_SRC=/usr/local/src/apache_1.3.31/src DO_HTTPD=1
APACI_ARGS="--enable-shared=max"
make && make test && make install
This is what we saw last time the problem happened (this morning):
[root@webapps3 root]# ps -auxww | grep ht
root 7619 0.0 4.5 677332 46944 ? S Sep24 2:06 /www/bin/httpd
nobody 25661 5.0 12.4 749172 128132 ? S 09:54 0:43 /www/bin/httpd
nobody 26374 3.9 8.2 705296 85016 ? S 09:57 0:28 /www/bin/httpd
nobody 28181 4.2 8.2 707132 85600 ? S 10:03 0:13 /www/bin/httpd
nobody 28480 4.7 8.2 706288 84652 ? S 10:04 0:12 /www/bin/httpd
nobody 28713 3.8 7.0 694656 72312 ? S 10:05 0:08 /www/bin/httpd
nobody 28920 3.8 7.6 700608 78600 ? S 10:05 0:07 /www/bin/httpd
nobody 29013 3.8 7.0 694076 72320 ? S 10:06 0:06 /www/bin/httpd
nobody 29109 3.8 7.1 696480 73940 ? S 10:06 0:05 /www/bin/httpd
nobody 29158 4.1 7.1 695772 73780 ? S 10:06 0:05 /www/bin/httpd
nobody 29557 4.1 6.4 689356 66168 ? S 10:07 0:02 /www/bin/httpd
nobody 29596 5.0 6.2 687308 64420 ? S 10:08 0:02 /www/bin/httpd
root 29880 0.0 0.0 5580 572 pts/0 R 10:08 0:00 grep ht
[root@webapps3 root]# /www/bin/apachectl stop
/www/bin/apachectl stop: httpd stopped
[root@webapps3 root]# ps -auxww | grep ht
root 7619 0.0 4.5 677332 46944 ? S Sep24 2:06 /www/bin/httpd
nobody 25661 5.0 0.0 0 0 ? Z 09:54 0:43 [httpd <defunct>]
nobody 26374 4.0 0.0 0 0 ? Z 09:57 0:28 [httpd <defunct>]
nobody 28181 4.3 0.0 0 0 ? Z 10:03 0:13 [httpd <defunct>]
nobody 28480 4.7 8.8 706288 91780 ? R 10:04 0:12 /www/bin/httpd
nobody 28713 3.9 7.0 695188 72812 ? R 10:05 0:08 /www/bin/httpd
nobody 29013 3.9 0.0 0 0 ? Z 10:06 0:06 [httpd <defunct>]
nobody 29109 3.9 0.0 0 0 ? Z 10:06 0:05 [httpd <defunct>]
nobody 29158 4.4 0.0 0 0 ? RW 10:06 0:06 [httpd]
nobody 29557 4.2 0.0 0 0 ? Z 10:07 0:02 [httpd <defunct>]
nobody 29596 5.2 0.0 0 0 ? Z 10:08 0:03 [httpd <defunct>]
nobody 29909 0.0 7.0 695188 72812 ? R 10:09 0:00 /www/bin/httpd
root 29911 0.0 0.0 4852 576 pts/0 R 10:09 0:00 grep ht
[root@webapps3 root]# ps -auxww | grep ht
root 7619 0.0 4.5 677332 46944 ? S Sep24 2:06 /www/bin/httpd
nobody 28713 3.9 7.4 695188 77176 ? S 10:05 0:08 /www/bin/httpd
nobody 29909 1.0 7.4 695188 77176 ? S 10:09 0:00 /www/bin/httpd
root 29913 0.0 0.0 4516 576 pts/0 R 10:09 0:00 grep ht
[root@webapps3 root]# ps -auxww | grep ht
root 7619 0.0 4.5 677332 46944 ? S Sep24 2:06 /www/bin/httpd
nobody 28713 3.8 7.4 695188 77176 ? S 10:05 0:08 /www/bin/httpd
nobody 29909 0.6 7.4 695188 77176 ? S 10:09 0:00 /www/bin/httpd
root 29915 0.0 0.0 4720 576 pts/0 R 10:09 0:00 grep ht
[root@webapps3 root]# ps -auxww | grep ht
root 7619 0.0 4.5 677332 46944 ? S Sep24 2:06 /www/bin/httpd
nobody 28713 3.8 7.4 695188 77176 ? S 10:05 0:08 /www/bin/httpd
nobody 29909 0.5 7.4 695188 77176 ? S 10:09 0:00 /www/bin/httpd
root 29917 0.0 0.0 4856 560 pts/0 R 10:09 0:00 grep ht
[root@webapps3 root]# killall httpd
[root@webapps3 root]# ps -auxww | grep ht
nobody 29909 0.2 7.4 695188 77176 ? S 10:09 0:00 /www/bin/httpd
root 29923 0.0 0.0 4528 576 pts/0 R 10:09 0:00 grep ht
[root@webapps3 root]# ps -auxww | grep ht
nobody 29909 0.2 7.4 695188 77176 ? S 10:09 0:00 /www/bin/httpd
root 29925 0.0 0.0 5384 572 pts/0 R 10:09 0:00 grep ht
[root@webapps3 root]# kill -9 29909
[root@webapps3 root]# ps -auxww | grep ht
root 29930 0.0 0.0 4776 576 pts/0 R 10:09 0:00 grep ht
[root@webapps3 root]# /www/bin/apachectl start
/www/bin/apachectl start: httpd started
[root@webapps3 root]# ps -auxww | grep ht
root 29946 92.0 6.0 68676 62348 ? R 10:09 0:00 /www/bin/httpd
root 29968 0.0 0.0 4964 576 pts/0 R 10:09 0:00 grep ht
[root@webapps3 root]# ps -auxww | grep ht
root 29946 68.0 7.0 78476 72312 ? S 10:09 0:01 /www/bin/httpd
nobody 29971 0.0 7.0 78476 72324 ? S 10:09 0:00 /www/bin/httpd
nobody 29972 0.0 7.0 78476 72324 ? S 10:09 0:00 /www/bin/httpd
nobody 29973 0.0 7.0 78476 72324 ? S 10:09 0:00 /www/bin/httpd
nobody 29974 0.0 7.0 78476 72324 ? S 10:09 0:00 /www/bin/httpd
nobody 29975 0.0 7.0 78476 72324 ? S 10:09 0:00 /www/bin/httpd
nobody 29976 0.0 7.0 78476 72324 ? S 10:09 0:00 /www/bin/httpd
nobody 29977 0.0 7.0 78476 72324 ? S 10:09 0:00 /www/bin/httpd
nobody 29978 0.0 7.0 78476 72324 ? S 10:09 0:00 /www/bin/httpd
nobody 29979 0.0 7.0 78476 72324 ? S 10:09 0:00 /www/bin/httpd
nobody 29980 0.0 7.0 78476 72324 ? S 10:09 0:00 /www/bin/httpd
root 29982 0.0 0.0 5052 572 pts/0 R 10:09 0:00 grep ht
Any ideas as to what might be causing this problem would be greatly
appreciated.
Regards,
-- Alex
--
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html
Re: mod_perl crashing / hanging
Posted by Alex Greg <ag...@digitallook.com>.
Tom Schindl wrote:
> Alex Greg wrote:
>
>> Geoffrey Young wrote:
>>
>>>
> [...]
>
>>
>> Thanks for your response. Sorry I wasn't clear; the httpd processes
>> don't actually kill the server, they just slow Apache right down to
>> the point at which it doesn't respond to requests any more. I am still
>> able to get onto the box and kill the processes etc. without having to
>> do a hard reboot or anything. My question is what's causing Apache to
>> get into the state where it can't fork any more shells from Perl.
>>
>>>> This is because a shell call to date ($date = `date`;) is failing -
>>>> this
>>>> is most likely what's causing the failure to fork we see above.
>>>
>>>
>>> if you want to avoid that shell call try POSIX::strftime() instead.
>>
>>
>> We have quite a lot of calls via backticks throughout our code - and
>> not just for things like date that are trivially replaceable. Would
>> you recommend replacing these as far as possible?
>
>
> Yes because calling the shell is fairly slow if there are any
> perl-modules you could use these instead and it will certainly give you
> a performance boost.
>
> What calls do you make, are these custom programs? If not I'd take a
> look at CPAN maybe someone has already written a Perl-Module which does
> exactly what you need else you could maybe write it your own. I have
> done this for example librsvg because I wanted to avoid calls to the
> shell inside my httpd-processes.
Hi,
We have some calls to ps that parse the output - I suppose we could
replace these with Proc::ProcessTable. The rest of it is just date, and
stuff that we can replace with either system() or Perl built-ins.
My concern though, is that this is a symptom of another problem, as has
been suggested earlier. I have activated mod_status, but since this
problem doesn't occur very often (and when it does, renders the server
unable to process requests, hence I won't be able to view the status),
I'm not sure how it will help. Looking at it now, I can see 7 servers in
state "Waiting for connection", and 4 in state "Sending reply", which I
assume is fairly normal.
I will keep watching this and start work on replacing the backticks. Any
further suggestions would be very much appreciated.
Regards,
-- Alex
--
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html
Re: mod_perl crashing / hanging
Posted by Tom Schindl <to...@gmx.at>.
Alex Greg wrote:
> Geoffrey Young wrote:
>
>>
[...]
>
> Thanks for your response. Sorry I wasn't clear; the httpd processes
> don't actually kill the server, they just slow Apache right down to the
> point at which it doesn't respond to requests any more. I am still able
> to get onto the box and kill the processes etc. without having to do a
> hard reboot or anything. My question is what's causing Apache to get
> into the state where it can't fork any more shells from Perl.
>
>>> This is because a shell call to date ($date = `date`;) is failing - this
>>> is most likely what's causing the failure to fork we see above.
>>
>>
>>
>> if you want to avoid that shell call try POSIX::strftime() instead.
>
>
>
> We have quite a lot of calls via backticks throughout our code - and not
> just for things like date that are trivially replaceable. Would you
> recommend replacing these as far as possible?
Yes because calling the shell is fairly slow if there are any
perl-modules you could use these instead and it will certainly give you
a performance boost.
What calls do you make, are these custom programs? If not I'd take a
look at CPAN maybe someone has already written a Perl-Module which does
exactly what you need else you could maybe write it your own. I have
done this for example librsvg because I wanted to avoid calls to the
shell inside my httpd-processes.
[...]
Tom
Reclaim Your Inbox!
http://www.mozilla.org/products/thunderbird
--
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html
Re: mod_perl crashing / hanging
Posted by Stas Bekman <st...@stason.org>.
Torsten Förtsch wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On Thursday 30 September 2004 19:48, Alex Greg wrote:
>
>>Thanks for your response. Sorry I wasn't clear; the httpd processes
>>don't actually kill the server, they just slow Apache right down to the
>>point at which it doesn't respond to requests any more. I am still able
>>to get onto the box and kill the processes etc. without having to do a
>>hard reboot or anything. My question is what's causing Apache to get
>>into the state where it can't fork any more shells from Perl.
>
>
> Activate mod_status and look at it's output. I suspect you'll see a lot of
> W's, L's or C's. This can give you a hint what your server processes are
> doing.
Or even better -- use Apache::VMonitor :)
--
__________________________________________________________________
Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide ---> http://perl.apache.org
mailto:stas@stason.org http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org http://ticketmaster.com
--
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html
Re: mod_perl crashing / hanging
Posted by Torsten Förtsch <to...@gmx.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On Thursday 30 September 2004 19:48, Alex Greg wrote:
> Thanks for your response. Sorry I wasn't clear; the httpd processes
> don't actually kill the server, they just slow Apache right down to the
> point at which it doesn't respond to requests any more. I am still able
> to get onto the box and kill the processes etc. without having to do a
> hard reboot or anything. My question is what's causing Apache to get
> into the state where it can't fork any more shells from Perl.
Activate mod_status and look at it's output. I suspect you'll see a lot of
W's, L's or C's. This can give you a hint what your server processes are
doing.
Torsten
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
iD8DBQFBXHvvwicyCTir8T4RApx3AJ4rG8aCbUPbI9rcotPK8E4ErrHH4QCcC/5j
DnyF7Iq/iXtbWzIT8n7YITk=
=NLCV
-----END PGP SIGNATURE-----
--
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html
Re: mod_perl crashing / hanging
Posted by Alex Greg <ag...@digitallook.com>.
Geoffrey Young wrote:
>
> Alex Greg wrote:
>
>>Hi,
>>
>>
>> [...]
>>
>>[Thu Sep 30 09:44:43 2004] [error] (12)Cannot allocate memory: fork:
>>Unable to fork new process
>
>
> there is lots of documentation around about how to tune your httpd processes
> so that they don't kill your box. see, for example
>
> http://perl.apache.org/docs/1.0/guide/performance.html
>
> as well as
>
> chapters 11 and 12 in practical mod_perl
> chapter 9 in the mod_perl developer's cookbook
Hi Geoff,
Thanks for your response. Sorry I wasn't clear; the httpd processes
don't actually kill the server, they just slow Apache right down to the
point at which it doesn't respond to requests any more. I am still able
to get onto the box and kill the processes etc. without having to do a
hard reboot or anything. My question is what's causing Apache to get
into the state where it can't fork any more shells from Perl.
>>This is because a shell call to date ($date = `date`;) is failing - this
>>is most likely what's causing the failure to fork we see above.
>
>
> if you want to avoid that shell call try POSIX::strftime() instead.
We have quite a lot of calls via backticks throughout our code - and not
just for things like date that are trivially replaceable. Would you
recommend replacing these as far as possible?
> but I
> suspect that the httpd failure isn't caused by calling out to shell, but
> rather that the shell call failure is a symptom of the real problem.
>
Agreed. I will have a thorough read of the performance guide. Can you
suggest what the real problem might be, or what might be causing it?
Thanks very much,
-- Alex
--
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html
Re: mod_perl crashing / hanging
Posted by Geoffrey Young <ge...@modperlcookbook.org>.
Alex Greg wrote:
> Hi,
>
>
> We've been having some issues with our mod_perl web servers recently.
> Basically what happens is the server becomes very very slow, to the
> point of being almost totally unresponsive. At the time this happens, we
> see this in the error log:
>
>
> [Thu Sep 30 09:44:43 2004] [error] (12)Cannot allocate memory: fork:
> Unable to fork new process
there is lots of documentation around about how to tune your httpd processes
so that they don't kill your box. see, for example
http://perl.apache.org/docs/1.0/guide/performance.html
as well as
chapters 11 and 12 in practical mod_perl
chapter 9 in the mod_perl developer's cookbook
>
> This is because a shell call to date ($date = `date`;) is failing - this
> is most likely what's causing the failure to fork we see above.
if you want to avoid that shell call try POSIX::strftime() instead. but I
suspect that the httpd failure isn't caused by calling out to shell, but
rather that the shell call failure is a symptom of the real problem.
HTH
--Geoff
--
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html