You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by Martijn Moret <ma...@mmoret.com> on 2009/08/11 10:22:05 UTC
[users@httpd] Strange problem - Apache httpd 2.2.11 on Solaris 10 sparc
Hi,
We use Apache httpd 2.2.11 on Solaris 10 (Sparc).
I compiled httpd myself with the following options:
# ./httpd -V
Server version: Apache/2.2.11 (Unix)
Server built: Feb 5 2009 09:00:47
Server's Module Magic Number: 20051115:21
Server loaded: APR 1.3.3, APR-Util 1.3.4
Compiled using: APR 1.3.3, APR-Util 1.3.4
Architecture: 32-bit
Server MPM: Worker
threaded: yes (fixed thread count)
forked: yes (variable process count)
Server compiled with....
-D APACHE_MPM_DIR="server/mpm/worker"
-D APR_HAS_SENDFILE
-D APR_HAS_MMAP
-D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
-D APR_USE_FCNTL_SERIALIZE
-D APR_USE_PTHREAD_SERIALIZE
-D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
-D APR_HAS_OTHER_CHILD
-D AP_HAVE_RELIABLE_PIPED_LOGS
-D DYNAMIC_MODULE_LIMIT=128
-D HTTPD_ROOT="/usr/local/apache2/"
-D SUEXEC_BIN="/usr/local/apache2/bin/suexec"
-D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
-D DEFAULT_ERRORLOG="logs/error_log"
-D AP_TYPES_CONFIG_FILE="conf/mime.types"
-D SERVER_CONFIG_FILE="conf/httpd.conf"
Worker MPM settings:
<IfModule mpm_worker_module>
ServerLimit 64
StartServers 4
MaxClients 1024
MinSpareThreads 32
MaxSpareThreads 128
ThreadsPerChild 16
MaxRequestsPerChild 1024
</IfModule>
Sometimes the server load increases heavily, when looking at the processes, a few httpd are using quitte a lot of CPU.
Output top:
last pid: 19041; load avg: 8.09, 8.08, 8.07; up 65+18:17:24 10:12:08
65 processes: 59 sleeping, 6 on cpu
CPU states: 66.5% idle, 33.4% user, 0.1% kernel, 0.0% iowait, 0.0% swap
Memory: 3968M phys mem, 1402M free mem, 4097M total swap, 4097M free swap
PID USERNAME LWP PRI NICE SIZE RES STATE TIME CPU COMMAND
28877 daemon 4 0 0 12M 9640K cpu/4 62.3H 12.24% httpd
28625 daemon 4 0 0 12M 9448K cpu/11 62.3H 12.24% httpd
28551 daemon 2 0 0 12M 9336K cpu/17 20.8H 4.08% httpd
29187 daemon 2 0 0 12M 9400K cpu/21 20.8H 4.08% httpd
19041 root 1 50 0 3456K 2408K cpu/8 0:00 0.06% top
There are no errors in the error log.
When looking at the server-status, these pids are active.
28625 and 28877 have three threads with Mode of operation "W"
28551 and 29187 have one threads with Mode of operation "W"
All pids have no other threads, so it looks like a problem sending the reply to the client....
Any ideas on how to solve this?
Regards and TIA
Martijn
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Strange problem - Apache httpd 2.2.11 on Solaris 10
sparc
Posted by Eric Covener <co...@gmail.com>.
On Tue, Aug 11, 2009 at 8:00 AM, Martijn Moret<ma...@mmoret.com> wrote:
> fee5c4fc _ZN27WpiWebServerApacheConnector4ReadEPcPm (253038, 26f898, fe5fa92c, fee45c90, ff03e3c0, ff0485b8) + 8
> fee5c2fc _ZN27WpiWebServerApacheConnector7ReadAllEPPcPm (253038, 26f898, fe5fa92c, 2, 80808000, 0) + 1e8
> fee45c90 _ZN13WpiWebRequest7ExecuteEv (0, 4, fee91edc, 1ffb98, e9ba0, fe5faee4) + 1a44
> fee47bf8 _ZN16WpiSystemManager22ProcessLansaWebRequestEP21WpiWebServerConnectorPKcS3_ (0, 253038, e9ba0, 207749, ff000000, ff0000) + 6dc
> fee4a69c _ZN16WpiSystemManager17ProcessWebRequestEP21WpiWebServerConnector (ffffffff, 207740, 3, 1878e8, 80808080, 1010101) + 394
> fee5b0f8 ???????? (ffffffff, 7a400, fee91ef0, 2060a0, 9b344, 9)
> 0003c034 ap_run_handler (2060a0, 2, 4, 2060a0, 233d28, 235a40) + 3c
> 0003c4b0 ap_invoke_handler (2060a0, 7a400, 2060a0, 9bc00, 9bb78, 181b78) + b8
> 00053f84 ap_process_request (2060a0, 2, 4, 2060a0, 233d28, 235a40) + 160
> 00051124 ap_process_http_connection (233d30, 2, 0, 9bc00, 9bb78, 181b78) + 10c
> 000426f0 ap_run_process_connection (233d30, 233a80, 233a80, 82, 233d28, 235a40) + 3c
> 00068444 worker_thread (13d7f0, 2, 0, 9bc00, 9bc00, 8) + 20c
> ff1fe5d4 dummy_worker (13d7f0, fe5fc000, 0, 0, ff1fe5c8, 1) + c
> fefc6d4c _lwp_start (0, 0, 0, 0, 0, 0)
Whatever third-party modue this is, it seems hung and is preventing
this child process from gracefully exiting (via e.g.
MaxRequestsPerchild, graceful restart, or MaxSpareThreads)
if they're not making any progress, you might try forcibly killing them.
--
Eric Covener
covener@gmail.com
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Strange problem - Apache httpd 2.2.11 on Solaris
10 sparc
Posted by Martijn Moret <ma...@mmoret.com>.
Van: "Eric Covener" <co...@gmail.com>
Aan: users@httpd.apache.org
Verzonden: Dinsdag 11 augustus 2009 13:17:23 GMT +01:00 Amsterdam / Berlijn / Bern / Rome / Stockholm / Wenen
Onderwerp: Re: [users@httpd] Strange problem - Apache httpd 2.2.11 on Solaris 10 sparc
Can you find the processes using high cpu and post the unique parts of
pstack output?
--
Eric Covener
covener@gmail.com
Hi,
Here's the pstack output of a process consuming high cpu.
# cat pstack.28625
28625: /usr/local/apache2/bestelxl/bin/httpd -f /usr/local/apache2/bestelxl/c
----------------- lwp# 1 / thread# 1 --------------------
fefcaa84 lwp_wait (5, ffbff644)
fefc391c _thrp_join (5, 0, ffbff6ac, 1, ffbff644, ff043800) + 34
ff1fe70c apr_thread_join (ffbff72c, 13d7f0, 2, 0, ff043800, 856a0) + c
00068964 join_workers (8, 187d90, 684f4, 13d5a8, 0, 1) + ec
00068cb8 child_main (2, 67844, 0, ff158000, ff043800, ff332a00) + 270
00068e98 make_child (9b000, 2, 0, 9c000, 9a800, 9bc00) + 128
000696c0 ap_mpm_run (feb80178, 16, 0, 16, 1, 40) + 738
00029490 main (a6b60, 99000, 9b000, 9b000, 9b000, 0) + 76c
00028764 _start (0, 0, 0, 0, 0, 0) + 5c
----------------- lwp# 5 / thread# 5 --------------------
fee5c4fc _ZN27WpiWebServerApacheConnector4ReadEPcPm (253038, 26f898, fe5fa92c, fee45c90, ff03e3c0, ff0485b8) + 8
fee5c2fc _ZN27WpiWebServerApacheConnector7ReadAllEPPcPm (253038, 26f898, fe5fa92c, 2, 80808000, 0) + 1e8
fee45c90 _ZN13WpiWebRequest7ExecuteEv (0, 4, fee91edc, 1ffb98, e9ba0, fe5faee4) + 1a44
fee47bf8 _ZN16WpiSystemManager22ProcessLansaWebRequestEP21WpiWebServerConnectorPKcS3_ (0, 253038, e9ba0, 207749, ff000000, ff0000) + 6dc
fee4a69c _ZN16WpiSystemManager17ProcessWebRequestEP21WpiWebServerConnector (ffffffff, 207740, 3, 1878e8, 80808080, 1010101) + 394
fee5b0f8 ???????? (ffffffff, 7a400, fee91ef0, 2060a0, 9b344, 9)
0003c034 ap_run_handler (2060a0, 2, 4, 2060a0, 233d28, 235a40) + 3c
0003c4b0 ap_invoke_handler (2060a0, 7a400, 2060a0, 9bc00, 9bb78, 181b78) + b8
00053f84 ap_process_request (2060a0, 2, 4, 2060a0, 233d28, 235a40) + 160
00051124 ap_process_http_connection (233d30, 2, 0, 9bc00, 9bb78, 181b78) + 10c
000426f0 ap_run_process_connection (233d30, 233a80, 233a80, 82, 233d28, 235a40) + 3c
00068444 worker_thread (13d7f0, 2, 0, 9bc00, 9bc00, 8) + 20c
ff1fe5d4 dummy_worker (13d7f0, fe5fc000, 0, 0, ff1fe5c8, 1) + c
fefc6d4c _lwp_start (0, 0, 0, 0, 0, 0)
----------------- lwp# 6 / thread# 6 --------------------
ff1fe5c8 dummy_worker(), exit value = 0x00000000
** zombie (exited, not detached, not yet joined) **
----------------- lwp# 7 / thread# 7 --------------------
fee5c54c _ZN27WpiWebServerApacheConnector4ReadEPcPm (1fd338, 26eeb8, fe3fa92c, fee45c90, ff03e3c0, ff0485b8) + 58
fee5c2fc _ZN27WpiWebServerApacheConnector7ReadAllEPPcPm (1fd338, 26eeb8, fe3fa92c, 2, 80808000, 0) + 1e8
fee45c90 _ZN13WpiWebRequest7ExecuteEv (0, 4, fee91edc, e9b58, e9bd0, fe3faee4) + 1a44
fee47bf8 _ZN16WpiSystemManager22ProcessLansaWebRequestEP21WpiWebServerConnectorPKcS3_ (0, 1fd338, e9bd0, 2a7b11, ff000000, ff0000) + 6dc
fee4a69c _ZN16WpiSystemManager17ProcessWebRequestEP21WpiWebServerConnector (ffffffff, 2a7b08, 3, 1878e8, 80808080, 1010101) + 394
fee5b0f8 ???????? (ffffffff, 7a400, fee91ef0, 2a6468, 9b344, 9)
0003c034 ap_run_handler (2a6468, 4, 4, 2a6468, 21ad20, 2a0410) + 3c
0003c4b0 ap_invoke_handler (2a6468, 7a400, 2a6468, 9bc00, 9bb78, 181b78) + b8
00053f84 ap_process_request (2a6468, 4, 4, 2a6468, 21ad20, 2a0410) + 160
00051124 ap_process_http_connection (21ad28, 2, 0, 9bc00, 9bb78, 181b78) + 10c
000426f0 ap_run_process_connection (21ad28, 21aa78, 21aa78, 84, 21ad20, 2a0410) + 3c
00068444 worker_thread (13d830, 2, 0, 9bc00, 9bc00, 10) + 20c
ff1fe5d4 dummy_worker (13d830, fe3fc000, 0, 0, ff1fe5c8, 1) + c
fefc6d4c _lwp_start (0, 0, 0, 0, 0, 0)
----------------- lwp# 8 / thread# 8 --------------------
ff1fe5c8 dummy_worker(), exit value = 0x00000000
** zombie (exited, not detached, not yet joined) **
----------------- lwp# 9 / thread# 9 --------------------
ff1fe5c8 dummy_worker(), exit value = 0x00000000
** zombie (exited, not detached, not yet joined) **
----------------- lwp# 10 / thread# 10 --------------------
fee5c4f4 _ZN27WpiWebServerApacheConnector4ReadEPcPm (233990, 26c250, fe0fa92c, 2, 80808000, 0)
fee45c90 _ZN13WpiWebRequest7ExecuteEv (0, 4, fee91edc, 1ffb38, e9c00, fe0faee4) + 1a44
fee47bf8 _ZN16WpiSystemManager22ProcessLansaWebRequestEP21WpiWebServerConnectorPKcS3_ (0, 233990, e9c00, 1b8b71, ff000000, ff0000) + 6dc
fee4a69c _ZN16WpiSystemManager17ProcessWebRequestEP21WpiWebServerConnector (ffffffff, 1b8b68, 3, 1878e8, 80808080, 1010101) + 394
fee5b0f8 ???????? (ffffffff, 7a400, fee91ef0, 1b74c8, 9b344, 9)
0003c034 ap_run_handler (1b74c8, 7, 4, 1b74c8, 1a7738, 1b3478) + 3c
0003c4b0 ap_invoke_handler (1b74c8, 7a400, 1b74c8, 9bc00, 9bb78, 181b78) + b8
00053f84 ap_process_request (1b74c8, 7, 4, 1b74c8, 1a7738, 1b3478) + 160
00051124 ap_process_http_connection (1a7740, 2, 0, 9bc00, 9bb78, 181b78) + 10c
000426f0 ap_run_process_connection (1a7740, 1a7490, 1a7490, 87, 1a7738, 1b3478) + 3c
00068444 worker_thread (13d890, 2, 0, 9bc00, 9bc00, 1c) + 20c
ff1fe5d4 dummy_worker (13d890, fe0fc000, 0, 0, ff1fe5c8, 1) + c
fefc6d4c _lwp_start (0, 0, 0, 0, 0, 0)
----------------- lwp# 11 / thread# 11 --------------------
ff1fe5c8 dummy_worker(), exit value = 0x00000000
** zombie (exited, not detached, not yet joined) **
----------------- lwp# 12 / thread# 12 --------------------
ff1fe5c8 dummy_worker(), exit value = 0x00000000
** zombie (exited, not detached, not yet joined) **
----------------- lwp# 13 / thread# 13 --------------------
ff1fe5c8 dummy_worker(), exit value = 0x00000000
** zombie (exited, not detached, not yet joined) **
----------------- lwp# 14 / thread# 14 --------------------
ff1fe5c8 dummy_worker(), exit value = 0x00000000
** zombie (exited, not detached, not yet joined) **
----------------- lwp# 15 / thread# 15 --------------------
ff1fe5c8 dummy_worker(), exit value = 0x00000000
** zombie (exited, not detached, not yet joined) **
----------------- lwp# 16 / thread# 16 --------------------
ff1fe5c8 dummy_worker(), exit value = 0x00000000
** zombie (exited, not detached, not yet joined) **
----------------- lwp# 17 / thread# 17 --------------------
ff1fe5c8 dummy_worker(), exit value = 0x00000000
** zombie (exited, not detached, not yet joined) **
----------------- lwp# 18 / thread# 18 --------------------
ff1fe5c8 dummy_worker(), exit value = 0x00000000
** zombie (exited, not detached, not yet joined) **
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Strange problem - Apache httpd 2.2.11 on Solaris 10
sparc
Posted by Eric Covener <co...@gmail.com>.
Can you find the processes using high cpu and post the unique parts of
pstack output?
--
Eric Covener
covener@gmail.com
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org