You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Marc Slemko <ma...@znep.com> on 1999/11/18 21:49:48 UTC
general/5336: Apache process hangs, and parent will not die. (fwd)
And yet another linux 2.2.13 hangs bug...
I would have hoped that if there really was such a problem, in a
"stable" series, it would be addressed fairly quickly but... who knows.
It is, of course, possible that it is Apache's fault but that seems
unlikely to me. I guess it is more possible it could be some random
module or library's fault, but that doesn't seem as likely either.
---------- Forwarded message ----------
Date: 18 Nov 1999 19:31:13 -0000
From: Eric Fisher <fi...@osiriscomm.com>
To: apbugs@hyperreal.org
Subject: general/5336: Apache process hangs, and parent will not die.
>Number: 5336
>Category: general
>Synopsis: Apache process hangs, and parent will not die.
>Confidential: no
>Severity: critical
>Priority: medium
>Responsible: apache
>State: open
>Class: sw-bug
>Submitter-Id: apache
>Arrival-Date: Thu Nov 18 11:40:01 PST 1999
>Last-Modified:
>Originator: fisher@osiriscomm.com
>Organization:
apache
>Release: 1.3.9
>Environment:
Linux sobek.osiriscomm.com 2.2.13 #3 SMP Sat Nov 6 01:04:50 PST 1999 i686 unknown
This is a RedHat 6.1 box with a custom build of apache.
512 MB RAM on a dual Pentium III
Apache is built as: Server Version: Apache/1.3.9 (Unix) mod_perl/1.21 PHP/3.0.12
and Server Version: Apache/1.3.9 (Unix) PHP/3.0.12 mod_ssl/2.4.6 OpenSSL/0.9.4
>Description:
We have a failry large web server with 9 separate instances and versions of apache
running. We have named them the following: httpd-d1, httpd-d2, httpd-d3, httpd-d4,
httpd-d5, httpd-h1, httpd-ssl, httpd-fp, and httpd-sava.
About once a day, now, we notice that the machine's load rises to about 1 and stays
there. At this time, we try to restart the the web servers (using kill -HUP `cat /home/logs/<servername>.pid`)
and all but (usually) one server restarts. A typical amount of traffic for one of
these servers is about 125000 connections and 1GB per day. There are rarely any
messages in the error log for that server. Sometimes we will see something like this, but
not always:
[Thu Nov 18 08:48:40 1999] [warn] child process 24339 still did not exit, sending a SIGTERM
[Thu Nov 18 08:48:44 1999] [error] child process 24339 still did not exit, sending a SIGKILL
[Thu Nov 18 08:49:00 1999] [notice] caught SIGTERM, shutting down
strace -p <pid> on the parent process returns nothing for the dead server. The web
server still serves up pages through the children, but as the children die, or are killed,
they are not respawning. So, I guess it looks like the parent is dead. The only way
to get that instance of apache running again is to restart the machine.
If we neglect to watch the load, then eventually the whole linux box locks up. The
load gradually gets higher and higher after several hours of the parent being
unresponsive.
The following is a typical httpd.conf file with some cuts made in the VirtualHost section:
ServerType standalone
ServerRoot "/var/lib/httpd"
PidFile /var/lib/httpd/logs/d1.pid
ScoreBoardFile /home/logs/d1.scoreboard
ResourceConfig /dev/null
AccessConfig /dev/null
Timeout 300
KeepAlive On
MaxKeepAliveRequests 256
KeepAliveTimeout 15
MinSpareServers 5
MaxSpareServers 15
StartServers 10
MaxClients 256
MaxRequestsPerChild 100000
ExtendedStatus On
#Port 80
User nobody
Group www
ServerAdmin support@osiriscomm.com
ServerName domain1.osiriscomm.com
DocumentRoot "/var/lib/httpd/htdocs"
<Directory />
Options All
# AllowOverride None
</Directory>
<Directory "/var/lib/httpd/conf/htdocs">
Options All
AllowOverride None
Order allow,deny
Allow from all
</Directory>
<Location /server-status>
SetHandler server-status
Order deny,allow
Deny from all
Allow from 216.200.46.18
Allow from 209.249.213.154
Allow from 216.102.105.20
</Location>
<Location /server-info>
SetHandler server-info
Order deny,allow
Deny from all
Allow from 216.200.46.18
Allow from 209.249.213.154
Allow from 216.102.105.20
</Location>
#UserDir public_html
DirectoryIndex index.html index.htm index.shtml index.php3 index.cgi
AccessFileName .htaccess
<Files ~ "^\.ht">
Order allow,deny
Deny from all
</Files>
UseCanonicalName On
TypesConfig /etc/httpd/conf/mime.types
DefaultType text/plain
HostnameLookups On
ErrorLog /home/logs/server/d1_error_log
LogLevel warn
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %b" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent
CustomLog /home/logs/server/d1_access_log common
ServerSignature On
Alias /icons/ "/var/lib/httpd/icons/"
Alias /img-osiris/ "/var/lib/httpd/htdocs/images/"
<Directory "/var/lib/httpd/icons">
Options Indexes MultiViews
AllowOverride None
Order allow,deny
Allow from all
</Directory>
ScriptAlias /cgi-bin/ /var/lib/httpd/cgi-bin/
ScriptAlias /cgi-osiris/ /var/lib/httpd/cgi-bin/
#
# "/var/lib/httpd/conf/cgi-bin" should be changed to whatever your ScriptAliased
# CGI directory exists, if you have that configured.
#
<Directory "/var/lib/httpd/cgi-bin">
AllowOverride None
Options None
Order allow,deny
Allow from all
</Directory>
IndexOptions FancyIndexing
AddIconByEncoding (CMP,/icons/compressed.gif) x-compress x-gzip
AddIconByType (TXT,/icons/text.gif) text/*
AddIconByType (IMG,/icons/image2.gif) image/*
AddIconByType (SND,/icons/sound2.gif) audio/*
AddIconByType (VID,/icons/movie.gif) video/*
AddIcon /icons/binary.gif .bin .exe
AddIcon /icons/binhex.gif .hqx
AddIcon /icons/tar.gif .tar
AddIcon /icons/world2.gif .wrl .wrl.gz .vrml .vrm .iv
AddIcon /icons/compressed.gif .Z .z .tgz .gz .zip
AddIcon /icons/a.gif .ps .ai .eps
AddIcon /icons/layout.gif .html .shtml .htm .pdf
AddIcon /icons/text.gif .txt
AddIcon /icons/c.gif .c
AddIcon /icons/p.gif .pl .py
AddIcon /icons/f.gif .for
AddIcon /icons/dvi.gif .dvi
AddIcon /icons/uuencoded.gif .uu
AddIcon /icons/script.gif .conf .sh .shar .csh .ksh .tcl
AddIcon /icons/tex.gif .tex
AddIcon /icons/bomb.gif core
AddIcon /icons/back.gif ..
AddIcon /icons/hand.right.gif README
AddIcon /icons/folder.gif ^^DIRECTORY^^
AddIcon /icons/blank.gif ^^BLANKICON^^
DefaultIcon /icons/unknown.gif
ReadmeName README
HeaderName HEADER
#
# IndexIgnore is a set of filenames which directory indexing should ignore
# and not include in the listing. Shell-style wildcarding is permitted.
#
IndexIgnore .??* *~ *# HEADER* README* RCS CVS *,v *,t
#
# AddEncoding allows you to have certain browsers (Mosaic/X 2.1+) uncompress
# information on the fly. Note: Not all browsers support this.
# Despite the name similarity, the following Add* directives have nothing
# to do with the FancyIndexing customization directives above.
#
AddEncoding x-compress Z
AddEncoding x-gzip gz tgz
#
# AddLanguage allows you to specify the language of a document. You can
# then use content negotiation to give a browser a file in a language
# it can understand. Note that the suffix does not have to be the same
# as the language keyword --- those with documents in Polish (whose
# net-standard language code is pl) may wish to use "AddLanguage pl .po"
# to avoid the ambiguity with the common suffix for perl scripts.
#
AddLanguage en .en
AddLanguage fr .fr
AddLanguage de .de
AddLanguage da .da
AddLanguage el .el
AddLanguage it .it
#
# LanguagePriority allows you to give precedence to some languages
# in case of a tie during content negotiation.
# Just list the languages in decreasing order of preference.
#
LanguagePriority en fr de
#
# AddType allows you to tweak mime.types without actually editing it, or to
# make certain files to be certain types.
#
# For example, the PHP3 module (not part of the Apache distribution - see
# http://www.php.net) will typically use:
#
AddType application/x-httpd-php3 .php3
AddType application/x-httpd-php3-source .phps
AddType application/x-tar .tgz
#
# AddHandler allows you to map certain file extensions to "handlers",
# actions unrelated to filetype. These can be either built into the server
# or added with the Action command (see below)
#
# If you want to use server side includes, or CGI outside
# ScriptAliased directories, uncomment the following lines.
#
# To use CGI scripts:
#
AddHandler cgi-script .cgi
AddHandler cgi-script .pl
#
# To use server-parsed HTML files
#
AddType text/html .shtml
AddHandler server-parsed .shtml
#
# Uncomment the following line to enable Apache's send-asis HTTP file
# feature
#
#AddHandler send-as-is asis
#
# If you wish to use server-parsed imagemap files, use
#
AddHandler imap-file map
#
# To enable type maps, you might want to use
#
#AddHandler type-map var
BrowserMatch "Mozilla/2" nokeepalive
BrowserMatch "MSIE 4\.0b2;" nokeepalive downgrade-1.0 force-response-1.0
BrowserMatch "RealPlayer 4\.0" force-response-1.0
BrowserMatch "Java/1\.0" force-response-1.0
BrowserMatch "JDK/1\.0" force-response-1.0
#
# Enable/disable the handling of HTTP/1.1 "Via:" headers.
# ("Full" adds the server version; "Block" removes all outgoing Via: headers)
# Set to one of: Off | On | Full | Block
#
#ProxyVia On
### Section 3: Virtual Hosts
##==cut==##
>How-To-Repeat:
The problem seems to be the cause. It typically happens every 24 hours or so.
If we increase the MaxRequestsPerChild then it will last longer, if we decrease
it then the server dies sooner. Would happen once a week when it was set to 0...
once every 10 hours when it was 10000
>Fix:
For now we restart the servers regularly and monitor the load. We really need a
fix, though.
>Audit-Trail:
>Unformatted:
[In order for any reply to be added to the PR database, you need]
[to include <ap...@Apache.Org> in the Cc line and make sure the]
[subject line starts with the report component and number, with ]
[or without any 'Re:' prefixes (such as "general/1098:" or ]
["Re: general/1098:"). If the subject doesn't match this ]
[pattern, your message will be misfiled and ignored. The ]
["apbugs" address is not added to the Cc line of messages from ]
[the database automatically because of the potential for mail ]
[loops. If you do not include this Cc, your reply may be ig- ]
[nored unless you are responding to an explicit request from a ]
[developer. Reply only with text; DO NOT SEND ATTACHMENTS! ]
Re: general/5336: Apache process hangs, and parent will not die. (fwd)
Posted by Manoj Kasichainula <ma...@io.com>.
On Thu, Nov 18, 1999 at 03:52:33PM -0500, Rasmus Lerdorf wrote:
> Is this the trampling herd problem that is solved in the 2.3 kernels with
> wake-once support?
I don't think so. Apache already deals with that by using accept
serialization. the purpose of the 2.3 changes, AFAIK, is to allow
turning off accept serialization.
--
Manoj Kasichainula - manojk at io dot com - http://www.io.com/~manojk/
Re: general/5336: Apache process hangs, and parent will not die.
(fwd)
Posted by Rasmus Lerdorf <ra...@apache.org>.
Is this the trampling herd problem that is solved in the 2.3 kernels with
wake-once support?
On Thu, 18 Nov 1999, Marc Slemko wrote:
> And yet another linux 2.2.13 hangs bug...
>
> I would have hoped that if there really was such a problem, in a
> "stable" series, it would be addressed fairly quickly but... who knows.
>
> It is, of course, possible that it is Apache's fault but that seems
> unlikely to me. I guess it is more possible it could be some random
> module or library's fault, but that doesn't seem as likely either.
Re: general/5336: Apache process hangs, and parent will not die.
(fwd)
Posted by Marc Slemko <ma...@znep.com>.
On Thu, 18 Nov 1999, Dean Gaudet wrote:
> i wonder if it's any different than the bugs we've had reported against
> linux since the 2.0.x days.
I think so. There have been a half dozen in the past two months, all
against recent 2.2.x linux kernels.
I mean, it may not be different in nature, but it is different in that
I'm pretty it has only started recently since otherwise this surge in
complaints would be quite odd.
>
> Dean
>
> On Thu, 18 Nov 1999, Marc Slemko wrote:
>
> > And yet another linux 2.2.13 hangs bug...
> >
> > I would have hoped that if there really was such a problem, in a
> > "stable" series, it would be addressed fairly quickly but... who knows.
> >
> > It is, of course, possible that it is Apache's fault but that seems
> > unlikely to me. I guess it is more possible it could be some random
> > module or library's fault, but that doesn't seem as likely either.
>
Re: general/5336: Apache process hangs, and parent will not die.
(fwd)
Posted by Dean Gaudet <dg...@arctic.org>.
i wonder if it's any different than the bugs we've had reported against
linux since the 2.0.x days.
Dean
On Thu, 18 Nov 1999, Marc Slemko wrote:
> And yet another linux 2.2.13 hangs bug...
>
> I would have hoped that if there really was such a problem, in a
> "stable" series, it would be addressed fairly quickly but... who knows.
>
> It is, of course, possible that it is Apache's fault but that seems
> unlikely to me. I guess it is more possible it could be some random
> module or library's fault, but that doesn't seem as likely either.