[users@httpd] apache2 processes hanging over a few hours until MaxClients is reached

[Steve Cox <St...@mergermarket.com>]

Hi,

I have an issue that overnight, my server's apache processes seem to get locked and stop responding to users. They have to close browsers sessions and start again. Meanwhile, the server is creating more apache processes - which in turn are eventually locked out too. In the end, MaxClients is reached.

The server used is public but is used by company staff for a CRM database - connections at the time of failure are from company office IPs so it doesn't look like intentional DoS.

The server is predominantly running PHP but I can't see why the situation occurs at roughly the same time each night so I am looking at apache first.

To see an example of how the process usage escalates, see:

http://i823.photobucket.com/albums/zz153/g00seUK/trowselocaldomain-apache_processes-.png

When it occurred yesterday, MaxClients had been reached. Today, apache was reset before that number had been reached. But you can still see the same path.


Apache 2.0.55
Linux kernel 2.6.15
PHP  5.1.2

When maxed out, all apache processes are in CLOSE_WAIT state. Prior to this, the spare servers are in LISTEN but a number of the apache processes are CLOSE_WAIT.

eg: netstat:

tcp6       0  0 ::ffff:192.168.25.7:443 ::ffff:83.244.142.226:16795 TIME_WAIT  -
tcp6       0   0 ::ffff:192.168.25.7:443 ::ffff:83.244.142.226:8346 TIME_WAIT  -
tcp6       0  0 ::ffff:192.168.25.7:443 ::ffff:83.244.142.226:27024 TIME_WAIT  -
tcp6       0  0 ::ffff:192.168.25.7:443 ::ffff:83.244.142.226:31721 TIME_WAIT  -
tcp6       0  0 ::ffff:192.168.25.7:443 ::ffff:83.244.142.226:31476 TIME_WAIT  -
tcp6       0  0 ::ffff:192.168.25.7:443 ::ffff:83.244.142.226:22722 TIME_WAIT  -
tcp6       0  0 ::ffff:192.168.25.7:443 ::ffff:83.244.142.226:12247 TIME_WAIT  -
tcp6      38   0 ::ffff:192.168.25.7:443 ::ffff:202.155.218.27:2791 CLOSE_WAIT 13243/apache2
tcp6      38   0 ::ffff:192.168.25.7:443 ::ffff:202.155.218.27:2050 CLOSE_WAIT 4788/apache2
tcp6      38   0 ::ffff:192.168.25.7:443 ::ffff:202.155.218.27:1282 CLOSE_WAIT 4789/apache2
tcp6      38   0 ::ffff:192.168.25.7:443 ::ffff:202.155.218.27:2064 CLOSE_WAIT 4791/apache2

At the time this is occurring, I used gdb against some of the CLOSE_WAIT processes and got:


(gdb) thread apply all bt full

Thread 1 (Thread 46912528496000 (LWP 12316)):
#0  0x00002aaaac1009a9 in flock () from /lib/libc.so.6
No symbol table info available.
#1  0x00002aaaacf248e0 in zif_session_is_registered () from /usr/lib/apache2/modules/libphp5.so
No symbol table info available.
#2  0x00002aaaacf24a57 in ps_read_files () from /usr/lib/apache2/modules/libphp5.so
No symbol table info available.
#3  0x00002aaaacf2220f in php_session_start () from /usr/lib/apache2/modules/libphp5.so
No symbol table info available.
#4  0x00002aaaacf229c9 in zif_session_start () from /usr/lib/apache2/modules/libphp5.so
No symbol table info available.
#5  0x00002aaaad04e125 in execute () from /usr/lib/apache2/modules/libphp5.so
No symbol table info available.
#6  0x00002aaaad04d813 in execute () from /usr/lib/apache2/modules/libphp5.so
No symbol table info available.
#7  0x00002aaaad032629 in zend_execute_scripts () from /usr/lib/apache2/modules/libphp5.so
No symbol table info available.
#8  0x00002aaaacff4dda in php_execute_script () from /usr/lib/apache2/modules/libphp5.so
No symbol table info available.
#9  0x00002aaaad0c188f in zend_get_zval_ptr () from /usr/lib/apache2/modules/libphp5.so
No symbol table info available.
#10 0x00000000004364f5 in ap_run_handler ()
No symbol table info available.
#11 0x0000000000436952 in ap_invoke_handler ()
No symbol table info available.
#12 0x00000000004288a0 in ap_process_request ()
No symbol table info available.
#13 0x00000000004238a8 in _start ()
No symbol table info available.
#0  0x00002aaaac1009a9 in flock () from /lib/libc.so.6

I did not run it against all the processes - but the ones I did were identical.

More info:

Compiled in modules:
  core.c
  mod_access.c
  mod_auth.c
  mod_log_config.c
  mod_logio.c
  mod_env.c
  mod_setenvif.c
  prefork.c
  http_core.c
  mod_mime.c
  mod_status.c
  mod_autoindex.c
  mod_negotiation.c
  mod_dir.c
  mod_alias.c
  mod_so.c

By config:

cgi, deflate, expires, mod-security, php5, rewrite, ssl, userdir.

Many thanks,
Steve

This email was sent by a company owned by Pearson plc, registered office at 80 Strand, London WC2R 0RL.
Registered in England and Wales with company number 53723


---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org