You are viewing a plain text version of this content. The canonical link for it is here.
Posted to modperl@perl.apache.org by Henrik Steffen <st...@topconcepts.de> on 2006/10/17 08:54:10 UTC
Cannot restart or stop Apache after several days of running
Hello all,
I am using Apache 2.0.59 on Linux with mod_perl 2.0.2
in a productive enviornment, with intense usage of perl-scripts.
If changes to the perl-scripts are necessary
I'll sometimes have to restart the apache webserver
to make use of the changed modules etc.
This works fine in the first couple of days (sometimes
weeks) after a reboot but then suddenly I am no longer able to
perform a normal "apachectl restart". After entering this command
actually nothing happens. The command execution just
hangs with no visible result. The number of apache
processes does not decrease at all. After minutes of waiting
I can just hit CTRL-C.
Even "apachectl -k stop" leads to the same result.
I could of course "killall" the apache processes brutally, but
even then I can't do "apachectl start". The server won't
start up again!
The only thing I can do to activate my module-changes
at the moment, is a complete reboot of the machine,
which is not such a Good Thing (TM).
If I didn't need to restart the apache-server for the
modules sake, I wouldn't bother at all, because the server
is running completely fine all the time. The only
thing not working is the apache restarts (but this only happens
as I said before, after a few days or sometimes weeks)
I used to think that this had something to do with mod_perl 1.99
which I have been using until recently. But now - after the upgrade
to 2.0.2 - I am still encountering the same problem.
Anyone ever experienced this?
Any ideas what to do, or how to debug any further, or how to solve this?
Help appreciated!
Thank you,
--
Kind regards
Henrik Steffen
Re: Cannot restart or stop Apache after several days of running
Posted by Malcolm J Harwood <mj...@liminalflux.net>.
On Tue, 2006-10-17 at 08:54 +0200, Henrik Steffen wrote:
> This works fine in the first couple of days (sometimes
> weeks) after a reboot but then suddenly I am no longer able to
> perform a normal "apachectl restart". After entering this command
> actually nothing happens. The command execution just
> hangs with no visible result. The number of apache
> processes does not decrease at all. After minutes of waiting
> I can just hit CTRL-C.
> Even "apachectl -k stop" leads to the same result.
Are you running with ssl enabled?
One thing I've noticed (at least on linux) is that if your system is low on
entropy random number generator is low on randomness
('cat /proc/sys/kernel/random/entropy_avail' to get the number on linux) then
apache wont shut down until it gets enough to make ssl happy.
If your webserver is running on a box will little disk access and no one on
the console, it can easily get too low for apache to stop. (I get this
problem on my dev boxes if I'm restarting apache a lot but not giving it much
traffic - as happens when I'm doing some handler development work).
AW: Cannot restart or stop Apache after several days of running
Posted by Henrik Steffen <st...@city-map.de>.
Dear Torsten,
> This can happen if apache is compiled with SysV shared memory
> support. If a segment is used for example as scoreboard and apache is
> killed with SIGKILL the segment remains and prevents further
> starting. After a reboot the segment disappears and the server will
> start again.
this sounds exactly like the symptoms I can see!
> You can check this with ipcs. A named scoreboard is not necessary.
> The way shared memory is allocated is defined when APR is configured.
> Linux supports all 3 choices. Unfortunately SysV is default. But for
> me mmap() based shared memory works best. It avoids this problem.
uff, never used ipcs before and I have no idea how to. This is what it shows
me right now (sorry for the German os-output):
----- Gemeinsamer Speicher: Segmente -----
Schlüssel shmid Besitzer Rechte Bytes nattch Status
----- Semaphorenfelder -----
Schlüssel SemID Besitzer Rechte nsems
0x00000000 1998848 wwwrun 600 1
0x00000000 2031617 wwwrun 600 1
0x00000000 688130 modperl 600 1
0x00000000 720899 modperl 600 1
0x00000000 1179652 modperl 600 1
0x00000000 1212421 modperl 600 1
0x00000000 1933318 modperl 600 1
0x00000000 1966087 modperl 600 1
0x00000000 2097160 modperl 600 1
0x00000000 2129929 modperl 600 1
----- Nachrichtenwarteschlangen -----
Schlüssel msqid Besitzer Rechte used-bytes messages
modperl ist the owner of my mod_perl webserver. wwwrun is just another
light httpd not affected by this topic at all.
> But maybe I am all wrong and your problem is somewhere else.
no, I think this could be it! But I don't know how to debug it any further.
Could you give me any additional hint?
Thank you,
--
Mit freundlichem Gruß
Henrik Steffen
Geschäftsführer
top concepts Internetmarketing GmbH
Am Steinkamp 7 - D-21684 Stade - Germany
------------------------------------------------------------
Damit wir Ihnen bestmöglichen Service bieten können, senden
Sie Ihre E-Mail-Anfragen bitte an support@topconcepts.de
------------------------------------------------------------
http://www.topconcepts.de Tel. +49 1805 9977 501*
mail: steffen@topconcepts.de Fax. +49 1805 9977 502*
------------------------------------------------------------
SMS Versand ab 9.9 Cent: http://sms-gw.topconcepts.de
------------------------------------------------------------
Handelsregister: AG Tostedt HRB 100687 - UstId: DE 213645563
------------------------------------------------------------
*) EUR 0,12/Min. (CNS24)
openBC - We're Networking People
www.openbc.com/hp/Henrik_Steffen/
Re: Cannot restart or stop Apache after several days of running
Posted by Torsten Foertsch <to...@gmx.net>.
On Tuesday 17 October 2006 19:15, Perrin Harkins wrote:
> > I could of course "killall" the apache processes brutally, but
> > even then I can't do "apachectl start". The server won't
> > start up again!
>
> So if you send the kill yourself, it does stop? And what do you mean by
> "won't start up again"? What happens when you try?
This can happen if apache is compiled with SysV shared memory support. If a
segment is used for example as scoreboard and apache is killed with SIGKILL
the segment remains and prevents further starting. After a reboot the segment
disappears and the server will start again.
You can check this with ipcs. A named scoreboard is not necessary. The way
shared memory is allocated is defined when APR is configured. Linux supports
all 3 choices. Unfortunately SysV is default. But for me mmap() based shared
memory works best. It avoids this problem.
But maybe I am all wrong and your problem is somewhere else.
Torsten
AW: Cannot restart or stop Apache after several days of running
Posted by Henrik Steffen <st...@city-map.de>.
> I could understand the ignoring the SIGHUP that "restart" sends, if
> something in your code was ignoring signals. The safe
> signals stuff in
> perl 5.8+ can be an issue there. I can't understand ignoring kill
> signals though.
well, meanwhile I found out:
kill -HUP
works fine. However, "apachectl restart" which is the same
as "httpd -k restart" does not.
> Have you tried running it through strace, to
> make sure
> it is sending the signals?
no , not yet.
> So if you send the kill yourself, it does stop? And what do
> you mean by
> "won't start up again"? What happens when you try?
yes. It does stop and vanishes completely from "ps ax".
If I then try to fire it up again with either "httpd -k start"
or "apachectl start" NOTHING happens. It just hangs.
--
Mit freundlichem Gruß
Henrik Steffen
Geschäftsführer
top concepts Internetmarketing GmbH
Am Steinkamp 7 - D-21684 Stade - Germany
------------------------------------------------------------
Damit wir Ihnen bestmöglichen Service bieten können, senden
Sie Ihre E-Mail-Anfragen bitte an support@topconcepts.de
------------------------------------------------------------
http://www.topconcepts.de Tel. +49 1805 9977 501*
mail: steffen@topconcepts.de Fax. +49 1805 9977 502*
------------------------------------------------------------
SMS Versand ab 9.9 Cent: http://sms-gw.topconcepts.de
------------------------------------------------------------
Handelsregister: AG Tostedt HRB 100687 - UstId: DE 213645563
------------------------------------------------------------
*) EUR 0,12/Min. (CNS24)
openBC - We're Networking People
www.openbc.com/hp/Henrik_Steffen/
Re: Cannot restart or stop Apache after several days of running
Posted by Perrin Harkins <pe...@elem.com>.
On Tue, 2006-10-17 at 08:54 +0200, Henrik Steffen wrote:
> This works fine in the first couple of days (sometimes
> weeks) after a reboot but then suddenly I am no longer able to
> perform a normal "apachectl restart". After entering this command
> actually nothing happens. The command execution just
> hangs with no visible result. The number of apache
> processes does not decrease at all. After minutes of waiting
> I can just hit CTRL-C.
>
> Even "apachectl -k stop" leads to the same result.
I could understand the ignoring the SIGHUP that "restart" sends, if
something in your code was ignoring signals. The safe signals stuff in
perl 5.8+ can be an issue there. I can't understand ignoring kill
signals though. Have you tried running it through strace, to make sure
it is sending the signals?
> I could of course "killall" the apache processes brutally, but
> even then I can't do "apachectl start". The server won't
> start up again!
So if you send the kill yourself, it does stop? And what do you mean by
"won't start up again"? What happens when you try?
- Perrin
Re: AW: Cannot restart or stop Apache after several days of running
Posted by Jonathan <mo...@2xlp.com>.
On Oct 17, 2006, at 2:55 PM, Henrik Steffen wrote:
>
> oops, I didn't know that. I have always done it with
> "apachectl restart" until now, and never had any trouble with it.
> Note though: apachectl is just a bash script which does a simple
> "/usr/sbin/httpd -k restart". Is this still a problem?
no one does! its very well documented, but in the worst documented
place ( not in 1.0/2.0 docs, but elsewhere ). everyone who has used
mod_perl finds out about it some time or another.
the problem is in the the restart signal to apache. it doesn't
recycle the memory. you need to do an explicit httpd stop , followed
by a start
> On a different machine e.g. it works fine, without even the
> problems restarting after a time.
that 'bug'/behavior seems to affect different os's and mp versions
differently. i think someone was trying to fix it and almost did
once. (maybe it is done in the newer ones )
> Really?? I have never seen something like that. My server has been
> running for 42 days now, with maybe 10 "apachectl restarts"
> since the last reboot, and the biggest httpd is still only 44 MB.
> At the moment only 3.8 GB of the available 4 GB are occupied.
> That's pretty normal for this server.
which os? i might have to switch.
> I read this article. And I got some help from it, because the
> kill -HUP works fine for me. The module changes are loaded and
> I don't need to perform a reboot anymore. That's very good!!
great!
> Still, I find it strange, that a usual "httpd -k restart" does not
> work.
the only way to get past it, is to forget thinking about it as
apache, and think about it as modperl.
you can still restart apache
you just can't restart modperl
> When typing "httpd -k start" or "apachectl start": nothing
> happening, no logs, no info, nothing.
try tailing /var/log/messages , /var/log/syslog & /var/log/security
when you do that next
it might be writing an error to something you don't expect.
AW: Cannot restart or stop Apache after several days of running
Posted by Henrik Steffen <st...@city-map.de>.
Hi Jonathan,
>
> You should never "apachectl restart" under mod perl
>
> You must always "apachectl stop" , wait , "apachectl start"
oops, I didn't know that. I have always done it with
"apachectl restart" until now, and never had any trouble with it.
Note though: apachectl is just a bash script which does a simple
"/usr/sbin/httpd -k restart". Is this still a problem?
On a different machine e.g. it works fine, without even the
problems restarting after a time.
> when you restart, you end up cloning a lot of memory-- MP2
> isn't very
> good at tearing out the old stuff. On my machines, I'll see an mp2
> process go from 90-180-270-360 mb if i restart 3 times.
Really?? I have never seen something like that. My server has been
running for 42 days now, with maybe 10 "apachectl restarts"
since the last reboot, and the biggest httpd is still only 44 MB.
At the moment only 3.8 GB of the available 4 GB are occupied.
That's pretty normal for this server.
> That *might* have someting to do with your issue. I don't know if
> you restarted before it would freeze or not.
yes I did.
> read up on
>
> http://perl.apache.org/docs/general/control/control.html
>
> There are ~5 sections on restarting in there
I read this article. And I got some help from it, because the
kill -HUP works fine for me. The module changes are loaded and
I don't need to perform a reboot anymore. That's very good!!
Still, I find it strange, that a usual "httpd -k restart" does not work.
I just had to reboot my machine. But before this I tried
"kill -TERM " on the apache. The httpd was completely shut down,
but unfortunately, then I couldn't start it up again.
When typing "httpd -k start" or "apachectl start": nothing
happening, no logs, no info, nothing.
So I just could reboot... and now everything works fine again.
If I remove the "LoadModule mod_perl" from the httpd.conf however,
the apache can start. That's why , I believe it got something todo
with mod_perl.
--
Mit freundlichem Gruß
Henrik Steffen
Geschäftsführer
top concepts Internetmarketing GmbH
Am Steinkamp 7 - D-21684 Stade - Germany
------------------------------------------------------------
Damit wir Ihnen bestmöglichen Service bieten können, senden
Sie Ihre E-Mail-Anfragen bitte an support@topconcepts.de
------------------------------------------------------------
http://www.topconcepts.de Tel. +49 1805 9977 501*
mail: steffen@topconcepts.de Fax. +49 1805 9977 502*
------------------------------------------------------------
SMS Versand ab 9.9 Cent: http://sms-gw.topconcepts.de
------------------------------------------------------------
Handelsregister: AG Tostedt HRB 100687 - UstId: DE 213645563
------------------------------------------------------------
*) EUR 0,12/Min. (CNS24)
openBC - We're Networking People
www.openbc.com/hp/Henrik_Steffen/
Re: Cannot restart or stop Apache after several days of running
Posted by Jonathan Vanasco <mo...@2xlp.com>.
On Oct 17, 2006, at 2:54 AM, Henrik Steffen wrote:
> This works fine in the first couple of days (sometimes
> weeks) after a reboot but then suddenly I am no longer able to
> perform a normal "apachectl restart". After entering this command
> actually nothing happens. The command execution just
You should never "apachectl restart" under mod perl
You must always "apachectl stop" , wait , "apachectl start"
when you restart, you end up cloning a lot of memory-- MP2 isn't very
good at tearing out the old stuff. On my machines, I'll see an mp2
process go from 90-180-270-360 mb if i restart 3 times.
That *might* have someting to do with your issue. I don't know if
you restarted before it would freeze or not.
read up on
http://perl.apache.org/docs/general/control/control.html
There are ~5 sections on restarting in there
AW: Cannot restart or stop Apache after several days of running
Posted by Henrik Steffen <st...@city-map.de>.
Hi Helmut,
Helmut Zeilinger wrote:
>
> is this a remote machine administrated via ssh/telnet/etc.
> and with nobody typing on the keyboard?
yes, it's a remote machine. No keyboard attached. Normally
administrated via ssh, for emergencies or kernel-updates
there is a console-switch, too.
--
Mit freundlichem Gruß
Henrik Steffen
Geschäftsführer
top concepts Internetmarketing GmbH
Am Steinkamp 7 - D-21684 Stade - Germany
------------------------------------------------------------
Damit wir Ihnen bestmöglichen Service bieten können, senden
Sie Ihre E-Mail-Anfragen bitte an support@topconcepts.de
------------------------------------------------------------
http://www.topconcepts.de Tel. +49 1805 9977 501*
mail: steffen@topconcepts.de Fax. +49 1805 9977 502*
------------------------------------------------------------
SMS Versand ab 9.9 Cent: http://sms-gw.topconcepts.de
------------------------------------------------------------
Handelsregister: AG Tostedt HRB 100687 - UstId: DE 213645563
------------------------------------------------------------
*) EUR 0,12/Min. (CNS24)
openBC - We're Networking People
www.openbc.com/hp/Henrik_Steffen/
Re: Cannot restart or stop Apache after several days of running
Posted by Helmut Zeilinger <hz...@hzlabs.de>.
Hi Henrik,
is this a remote machine administrated via ssh/telnet/etc.
and with nobody typing on the keyboard?
Helmut
Henrik Steffen schrieb:
> Hello all,
>
> I am using Apache 2.0.59 on Linux with mod_perl 2.0.2
> in a productive enviornment, with intense usage of perl-scripts.
>
> If changes to the perl-scripts are necessary
> I'll sometimes have to restart the apache webserver
> to make use of the changed modules etc.
>
> This works fine in the first couple of days (sometimes
> weeks) after a reboot but then suddenly I am no longer able to
> perform a normal "apachectl restart". After entering this command
> actually nothing happens. The command execution just
> hangs with no visible result. The number of apache
> processes does not decrease at all. After minutes of waiting
> I can just hit CTRL-C.
>
> Even "apachectl -k stop" leads to the same result.
>
> I could of course "killall" the apache processes brutally, but
> even then I can't do "apachectl start". The server won't
> start up again!
>
> The only thing I can do to activate my module-changes
> at the moment, is a complete reboot of the machine,
> which is not such a Good Thing (TM).
>
>
> If I didn't need to restart the apache-server for the
> modules sake, I wouldn't bother at all, because the server
> is running completely fine all the time. The only
> thing not working is the apache restarts (but this only happens
> as I said before, after a few days or sometimes weeks)
>
> I used to think that this had something to do with mod_perl 1.99
> which I have been using until recently. But now - after the upgrade
> to 2.0.2 - I am still encountering the same problem.
>
> Anyone ever experienced this?
> Any ideas what to do, or how to debug any further, or how to solve this?
>
> Help appreciated!
>
> Thank you,
>
> --
>
> Kind regards
>
> Henrik Steffen
>
--
HZ.labs
Dr. Helmut Zeilinger
Wiesengrund 2
D-86 684 Holzheim
Tel. 08276 58767
Fax. 08276 58787
Mobil 0160 91 55 61 68
Internet http://www.hzlabs.de
Mail hz@hzlabs.de