[users@httpd] Desperate measures...

[Carlos Cajina <ce...@hotmail.com>]

Hi. I'm new to the list and I'm here because a have a huge problem... Straight to the point: I have one Apache like this (apachectl -V):

Server version: Apache/2.0.52
Server built:   Oct  7 2004 23:35:30
Server's Module Magic Number: 20020903:9
Architecture:   32-bit
Server compiled with....
 -D APACHE_MPM_DIR="server/mpm/prefork"
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
 -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
 -D APR_USE_FCNTL_SERIALIZE
 -D APR_USE_PTHREAD_SERIALIZE
 -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
 -D APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D HTTPD_ROOT="/usr/local/etc/httpd"
 -D SUEXEC_BIN="/usr/local/etc/httpd/bin/suexec"
 -D DEFAULT_PIDLOG="logs/httpd.pid"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_LOCKFILE="logs/accept.lock"
 -D DEFAULT_ERRORLOG="logs/error_log"
 -D AP_TYPES_CONFIG_FILE="conf/mime.types"
 -D SERVER_CONFIG_FILE="conf/httpd.conf"

running on a Sun-Fire-480R server. This server has a JBoss/Tomcat4.1.29 instance and acts as a (mod_jk) load balancer to another JBoss/Tomcat4.1.29 instance. Running netstat -a show lots of  SYN_SENT connections as well as ESTABLISHED ones, but response times end up in time outs or pages not shown at all. I guess that the user load is creating a bottleneck, because -and please correct me if I'm wrong- I'm seeing too many lines like this <PROCESS_NUMBER> ?        0:00 httpd when running a ps -ae

Top shows mutiple-sleepy httpd processes:

last pid: 13549;  load averages:  3.66,  3.65,  3.71                                                                                                                     16:44:35
203 processes: 199 sleeping, 2 running, 2 on cpu
CPU states:  0.3% idle, 59.6% user, 40.1% kernel,  0.0% iowait,  0.0% swap
Memory: 4096M real, 2939M free, 1192M swap in use, 6085M swap free

   PID USERNAME LWP PRI NICE  SIZE   RES STATE    TIME    CPU COMMAND
  9918 root       1  10    0   43M 7336K run    148:40 33.60% Xsun
  5586 root       1  10    0 1392K 1160K cpu/0   21.1H 32.42% nano
 12171 root     124  21    0  707M  474M run     95:58 31.61% java
 13549 root       1  59    0 2928K 1808K cpu/2    0:00  0.36% top
 13547 root       1  59    0 4464K 2112K sleep    0:00  0.09% sshd
  9425 root       1  59    0 2760K 2080K sleep    0:00  0.03% bash
  7865 ccajina    1  59    0 5776K 1728K sleep    0:09  0.02% sshd
 13397 nobody     1  59    0 7256K 2608K sleep    0:00  0.02% httpd
 13475 nobody     1  59    0 7256K 2624K sleep    0:00  0.02% httpd
 13411 nobody     1  59    0 7280K 2656K sleep    0:00  0.01% httpd
   346 mysql      9  59    0   43M   28M sleep   13:07  0.00% mysqld
    64 root      10  59    0 4824K 4008K sleep    3:31  0.00% picld
   409 root       1  59    0 3056K 1664K sleep    0:27  0.00% sshd
   244 root      23  59    0 7032K 3768K sleep    0:03  0.00% nscd
 12492 root       1  59    0 7168K 4280K sleep    0:01  0.00% httpd

I can't figure out why suddenly after a while (say 1/2 an hour) Apache unbearable sits accepting connections but apparntly releasing very few.

Can anybody point out tips, tutorials, how-tos, personal experiences, or any other resourse to help me understand what's going on?

Any help... GREATLY appreciated...

Regards,

Carlos