os-linux/5525: Many sockets left in CLOSE_WAIT, many apache children hang

[Andrew J Barros <an...@nethernet.com>]

>Number:         5525
>Category:       os-linux
>Synopsis:       Many sockets left in CLOSE_WAIT, many apache children hang
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    apache
>State:          open
>Class:          sw-bug
>Submitter-Id:   apache
>Arrival-Date:   Fri Dec 31 01:10:00 PST 1999
>Last-Modified:
>Originator:     andy@nethernet.com
>Organization:
apache
>Release:        1.3.9
>Environment:
Linux 2.2.7-ac2 
gcc:
gcc version egcs-2.91.66 19990314/Linux (egcs-1.1.2 release)
uname -a:
Linux srv1 2.2.7-ac2 #2 Sun May 9 23:53:58 EDT 1999 i586 unknown
>Description:
Apache has the following modules staticly linked:
Compiled-in modules:
  http_core.c
  mod_mmap_static.c
  mod_env.c
  mod_log_config.c
  mod_log_agent.c
  mod_log_referer.c
  mod_mime_magic.c
  mod_mime.c
  mod_negotiation.c
  mod_status.c
  mod_info.c
  mod_include.c
  mod_autoindex.c
  mod_dir.c
  mod_cgi.c
  mod_asis.c
  mod_imap.c
  mod_actions.c
  mod_userdir.c
  mod_alias.c
  mod_rewrite.c
  mod_access.c
  mod_auth.c
  mod_auth_dbm.c
  mod_auth_db.c
  mod_digest.c
  mod_proxy.c
  mod_expires.c
  mod_headers.c
  mod_usertrack.c
  mod_unique_id.c
  mod_so.c
  mod_setenvif.c
It Dynamicly loads:

LoadModule pam_auth_module    libexec/mod_auth_pam.so
LoadModule perl_module        libexec/libperl.so
LoadModule php3_module        libexec/libphp3.so
LoadModule roaming_module     libexec/mod_roaming.so
LoadModule mysql_auth_module  libexec/mod_auth_mysql.so
LoadModule throttle_module    libexec/mod_throttle.so

Apache starts up and begins to use more and more memory. 
It leaves many sockets in CLOSE_WAIT and the children hand and idle. CPU usage 
hits 100% and the load average has gotten as high as 62.00. A strace of a hung 
child reveals many calls to munmap(). Partial output is include as it spans many 
many similar lines
munmap(0x403be000, 4096)                = 0
munmap(0x403bd000, 4096)                = 0
munmap(0x403bc000, 4096)                = 0
munmap(0x403bb000, 4096)                = 0
munmap(0x403ba000, 4096)                = 0
munmap(0x403b9000, 4096)                = 0
munmap(0x403b8000, 4096)                = 0
munmap(0x403b7000, 4096)                = 0
munmap(0x403b6000, 4096)                = 0
munmap(0x40392000, 4096)                = 0
munmap(0x40391000, 4096)                = 0
munmap(0x40390000, 4096)                = 0
munmap(0x4038f000, 4096)                = 0
munmap(0x4038e000, 4096)                = 0
munmap(0x4038d000, 4096)                = 0
munmap(0x4018c000, 4096)                = 0
munmap(0x40016000, 4096)                = 0
munmap(0x40015000, 4096)                = 0
munmap(0x40014000, 4096)                = 0
This problem seems similar to one solved on solaris.
If i add the following lines to my httpd.conf:
ClearModuleList

AddModule http_core.c
AddModule mod_env.c
AddModule mod_log_config.c
AddModule mod_log_agent.c
AddModule mod_log_referer.c
AddModule mod_mime_magic.c
AddModule mod_mime.c
AddModule mod_negotiation.c
AddModule mod_status.c
AddModule mod_info.c
AddModule mod_include.c
AddModule mod_autoindex.c
AddModule mod_dir.c
AddModule mod_cgi.c
AddModule mod_asis.c
AddModule mod_imap.c
AddModule mod_actions.c
AddModule mod_userdir.c
AddModule mod_alias.c
AddModule mod_rewrite.c
AddModule mod_access.c
AddModule mod_auth.c
AddModule mod_auth_dbm.c
AddModule mod_auth_db.c
AddModule mod_digest.c
AddModule mod_proxy.c
AddModule mod_expires.c
AddModule mod_headers.c
AddModule mod_usertrack.c
AddModule mod_unique_id.c
AddModule mod_so.c
AddModule mod_setenvif.c

Apache will take a very long time to start and use lots of memory producing this 
error message: Ouch!  malloc failed in malloc_block()
 
Contact me at andy@nethernet.com for more info
>How-To-Repeat:
Problem happens regularly. No changes were made to teh configuration.
>Fix:

>Audit-Trail:
>Unformatted:
[In order for any reply to be added to the PR database, you need]
[to include <ap...@Apache.Org> in the Cc line and make sure the]
[subject line starts with the report component and number, with ]
[or without any 'Re:' prefixes (such as "general/1098:" or      ]
["Re: general/1098:").  If the subject doesn't match this       ]
[pattern, your message will be misfiled and ignored.  The       ]
["apbugs" address is not added to the Cc line of messages from  ]
[the database automatically because of the potential for mail   ]
[loops.  If you do not include this Cc, your reply may be ig-   ]
[nored unless you are responding to an explicit request from a  ]
[developer.  Reply only with text; DO NOT SEND ATTACHMENTS!     ]