[users@httpd] Apache abort and semaphore leak

[Manas Garg <ma...@gmail.com>]

Hello,

We had a problem in one of our server where Apache was not coming up
properly. The problem got resolved by reboot.

We are running the following version of apache:

sh-3.00# ../bin/httpd -v
Server version: Apache/2.2.2
Server built:   Aug 18 2006 11:42:41

It is running on the following system:

sh-3.00# uname -a
Linux fltwd-intdep01.nevisnetworks.com 2.6.17-1.2142_FC4smp #1 SMP Sat Aug
12 08:16:08 EDT 2006 i686 i686 i386 GNU/Linux

This system has 8 GB of RAM and 2 dual core CPUs.

The load on the system is moderate. When the problem occurred, the services
were being started on that system. So, load was practically zero.

We run a watchdog to start Apache if it is not running. Yesterday, we saw
that Apache was not coming up and the error log was full of the following
messages:

[Tue Mar 18 18:41:50 2008] [warn] RSA server certificate CommonName (CN)
`fltwd-intdep03.nevisnetworks.com' does NOT match server name!?
[Tue Mar 18 18:41:50 2008] [warn] RSA server certificate CommonName (CN)
`fltwd-intdep03.nevisnetworks.com' does NOT match server name!?
[Tue Mar 18 18:41:50 2008] [notice] Digest: generating secret for digest
authentication ...
[Tue Mar 18 18:41:50 2008] [notice] Digest: done
[Tue Mar 18 18:41:51 2008] [warn] RSA server certificate CommonName (CN)
`fltwd-intdep03.nevisnetworks.com' does NOT match server name!?
[Tue Mar 18 18:41:51 2008] [warn] RSA server certificate CommonName (CN)
`fltwd-intdep03.nevisnetworks.com' does NOT match server name!?
[Tue Mar 18 18:41:51 2008] [warn] pid file
/usr/local/software/platform/apache/logs/httpd.pid overwritten -- Unclean
shutdown of previous Apa
che run?
[Tue Mar 18 18:41:51 2008] [error] (13)Permission denied: could not create
/usr/local/software/platform/apache/logs/httpd.pid
[Tue Mar 18 18:41:51 2008] [error] httpd: could not log pid to file
/usr/local/software/platform/apache/logs/httpd.pid
[Tue Mar 18 18:42:51 2008] [warn] RSA server certificate CommonName (CN)
`fltwd-intdep03.nevisnetworks.com' does NOT match server name!?
[Tue Mar 18 18:42:51 2008] [warn] RSA server certificate CommonName (CN)
`fltwd-intdep03.nevisnetworks.com' does NOT match server name!?
[Tue Mar 18 18:42:51 2008] [notice] Digest: generating secret for digest
authentication ...
[Tue Mar 18 18:42:51 2008] [notice] Digest: done
[Tue Mar 18 18:42:52 2008] [warn] RSA server certificate CommonName (CN)
`fltwd-intdep03.nevisnetworks.com' does NOT match server name!?
[Tue Mar 18 18:42:52 2008] [warn] RSA server certificate CommonName (CN)
`fltwd-intdep03.nevisnetworks.com' does NOT match server name!?
[Tue Mar 18 18:42:52 2008] [warn] pid file
/usr/local/software/platform/apache/logs/httpd.pid overwritten -- Unclean
shutdown of previous Apa
che run?
[Tue Mar 18 18:42:52 2008] [error] (13)Permission denied: could not create
/usr/local/software/platform/apache/logs/httpd.pid
[Tue Mar 18 18:42:52 2008] [error] httpd: could not log pid to file
/usr/local/software/platform/apache/logs/httpd.pid


We checked the permissions on httpd.pid file and it was writable by Apache.
After some time, the error logs become:

[Wed Mar 19 18:48:06 2008] [warn] RSA server certificate CommonName (CN)
`fltwd-intdep03.nevisnetworks.com' does NOT match server name!?
[Wed Mar 19 18:48:06 2008] [warn] RSA server certificate CommonName (CN)
`fltwd-intdep03.nevisnetworks.com' does NOT match server name!?
[Wed Mar 19 18:48:06 2008] [crit] (28)No space left on device: mod_jk: could
not create jk_log_lock
Configuration Failed
[Wed Mar 19 18:49:03 2008] [warn] RSA server certificate CommonName (CN)
`fltwd-intdep03.nevisnetworks.com' does NOT match server name!?
[Wed Mar 19 18:49:03 2008] [warn] RSA server certificate CommonName (CN)
`fltwd-intdep03.nevisnetworks.com' does NOT match server name!?
[Wed Mar 19 18:49:03 2008] [crit] (28)No space left on device: mod_jk: could
not create jk_log_lock
Configuration Failed

We found that the system had run out of the semaphore. Previously, whenever
apache was exiting, it was leaving some semaphores in the system.
Eventually, the system ran out of semaphores and Apache started giving these
errors.

We decided to reboot the machine and after reboot, everything worked fine.
So, before reboot, Apache complained about permissions on httpd.pid but
after reboot, it didn't. And we didn't change the permissions meanwhile. And
anyways, the permissions were correct to begin with.

I tried reproducing this error by stopping Apache, creating httpd.pid as
root without any rights to Apache and then starting apache. In this case,
Apache didn't print anything in the logs and it didn't leave any semaphores.
It just exited silently. So, I feel that the issue that occurred yesterday
was not really about file permissions. But it was something else.

I have attached the conf file with this mail. Please let me know if I need
to provide more information on this issue.

Thanks & Regards,
Manas Garg