[users@httpd] Re: Assistance using server-status to determine if configuration could be causing issues or if it may be system compromise

[Stevie <ig...@yahoo.com>]

Is no one able to provide any feedback or help with this? Have I worded my question in a bad way? I'll word it differently if it helps. Can someone speak to the longevity of the child process states I mentioned below? 


----- Original Message -----
From: Stevie <ig...@yahoo.com>
To: "users@httpd.apache.org" <us...@httpd.apache.org>
Cc: 
Sent: Thursday, April 26, 2012 1:07 PM
Subject: Assistance using server-status to determine if configuration could be causing issues or if it may be system compromise

I have Apache 2.2.14 running on Ubuntu 10.04. The server has been running for almost a year without incident. The server is a front-end web server which passes requests for dynamic content (Java) to Tomcat on other servers using mod_jk. April 24th the web server started responding slugglishly to client requests (as reported by clients) before ultimately becoming unresponsive. It had to be hard rebooted to bring it back up. I found no evidence of what caused the crash after the server came back up, but I had noticed that there were a lot of Apache processes running (~146) right before everything crapped the bed. The MaxClients directive is set to the default of 150 and this has been sufficient all this time. I changed it to 200 and immediately saw that there was a jump in child process numbers almost immediately, like it couldn't wait to launch more child processes. I didn't see anything in the error log about going over the MaxClient limit so I'm not
sure if I'm chasing squirrels here. In a nutshell, what I'm looking for is help is figuring out if MaxClients/KeepAlives are potential performance problems in my config, or whether I might be looking at some outside influence (compromised system). 

The server is running mod_php so it's using the prefork MPM. Here are the relevant lines from apache2.conf:

<IfModule mpm_prefork_module>
    StartServers          5
    MinSpareServers       5
    MaxSpareServers      10
    MaxClients          200
    MaxRequestsPerChild   0
</IfModule>


KeepAlive On
MaxKeepAliveRequests 100

KeepAliveTimeout 15

I enabled the server-status page and briefly turned ExtendedStatus On (didn't want to leave it on if it could decrease performance). Here's what I saw:

KWKKKKKKKKKKKKKKKKKKKKKKKKKCKKKKKKWKCKKKKKKKKCKKKWSSSSS.........
.....K...................K.....W.......K........................
..K.....K....K..........W...G...KK...................W.......... 
.C.....K........................................................

I had a lot of KeepAlives that I recognized, like this:
3-3 886 6/6/6 K 0.01 3 1 7.1 0.01 0.01 24.227.88.162 clientname.com POST /RequestHandler HTTP/1.1 
This makes sense to me, and the question I have regarding this is whether I should decrease the KeepAlive Timeout number since it seems like that is part of what is causing Apache to spawn so many child processes. 

The other bit is that along with the above lines in server-status I am seeing things like:

136-2 747 0/0/0 K 0.00 1335452011 0 0.0 0.00 0.00    
141-2 752 0/0/0 K 0.00 1335452011 0 0.0 0.00 0.00    
152-2 764 0/0/0 W 0.00 1335452011 0 0.0 0.00 0.00    
156-2 768 0/0/0 G 0.00 1335452011 0 0.0 0.00 0.00    
160-2 772 0/0/0 K 0.00 1335452011 0 0.0 0.00 0.00    
161-2 773 0/0/0 K 0.00 1335452011 0 0.0 0.00 0.00    
181-2 793 0/0/0 W 0.00 1335452011 0 0.0 0.00 0.00    
193-2 805 0/0/0 C 0.00 1335452011 0 0.0 0.00 0.00    
199-2 811 0/0/0 K 0.00 1335452011 0 0.0 0.00 0.00 
If I understand the scoreboard key correctly, this means that there are child processes that have been in either a KeepAlive, closing, or replying state that have no connection on the other end. It's essentially like they're stuck or hanging. Is this an accurate understanding? I also converted the seconds in the SS column to get an idea of how long they've been that way, and it comes out to something like 15,000 days! That's not right obviously. Am I doing something wrong in that calculation, or misunderstanding what it really means? 

Thanks in advance,
SC


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org