You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by Dave Barton <da...@comodo.com> on 2011/01/31 17:53:00 UTC
[users@httpd] Apache spawning "broken" sub-processes under load
Hi,
We've recently had a pretty large increase in the usage of one of our
Wordpress-powered sites. I'm seeing some very odd behaviour and I'm at a
loss as to how to investigate it further. Some of the apache processes
are spawning children with broken CPU usage and time stats (as shown by
ps auxf). I think it might be best just to show some examples;
root 3153 0.0 0.4 304688 12960 ? Ss 15:26 0:00
/usr/sbin/apache2 -D SSL -D PHP5 -D STATUS -d /usr/lib64/apache2 -f
/etc/apache2/httpd.conf -k start
apache 3155 0.0 0.1 215864 3816 ? S 15:26 0:00 \_
/usr/sbin/apache2 -D SSL -D PHP5 -D STATUS -d /usr/lib64/apache2 -f
/etc/apache2/httpd.conf -k start
apache 12135 129895172 4.0 542620 127208 ? Sl 16:33 21114987:41
\_ /usr/sbin/apache2 -D SSL -D PHP5 -D STATUS -d /usr/lib64/apache2 -f
/etc/apache2/httpd.conf -k start
apache 12353 10.6 4.0 729908 127676 ? Sl 16:34 1:20 \_
/usr/sbin/apache2 -D SSL -D PHP5 -D STATUS -d /usr/lib64/apache2 -f
/etc/apache2/httpd.conf -k start
apache 12875 11.1 3.8 773840 121116 ? Sl 16:36 1:09 \_
/usr/sbin/apache2 -D SSL -D PHP5 -D STATUS -d /usr/lib64/apache2 -f
/etc/apache2/httpd.conf -k start
apache 13222 9.1 4.1 845240 130568 ? Sl 16:38 0:49 \_
/usr/sbin/apache2 -D SSL -D PHP5 -D STATUS -d /usr/lib64/apache2 -f
/etc/apache2/httpd.conf -k start
apache 13445 10.5 3.6 725956 114972 ? Sl 16:39 0:50 \_
/usr/sbin/apache2 -D SSL -D PHP5 -D STATUS -d /usr/lib64/apache2 -f
/etc/apache2/httpd.conf -k start
apache 13582 9.4 3.5 792056 112884 ? Sl 16:39 0:42 \_
/usr/sbin/apache2 -D SSL -D PHP5 -D STATUS -d /usr/lib64/apache2 -f
/etc/apache2/httpd.conf -k start
apache 14148 9.2 3.4 724848 107920 ? Sl 16:42 0:27 \_
/usr/sbin/apache2 -D SSL -D PHP5 -D STATUS -d /usr/lib64/apache2 -f
/etc/apache2/httpd.conf -k start
apache 14253 290358163 3.4 709912 108308 ? Sl 16:42 21133796:09
\_ /usr/sbin/apache2 -D SSL -D PHP5 -D STATUS -d /usr/lib64/apache2 -f
/etc/apache2/httpd.conf -k start
apache 14287 431815 3.5 717512 110180 ? Sl 16:42 19215:48 \_
/usr/sbin/apache2 -D SSL -D PHP5 -D STATUS -d /usr/lib64/apache2 -f
/etc/apache2/httpd.conf -k start
apache 14356 9.5 3.5 722780 111908 ? Sl 16:43 0:24 \_
/usr/sbin/apache2 -D SSL -D PHP5 -D STATUS -d /usr/lib64/apache2 -f
/etc/apache2/httpd.conf -k start
apache 14392 324909060 3.9 735640 123796 ? Sl 16:43 21133795:59
\_ /usr/sbin/apache2 -D SSL -D PHP5 -D STATUS -d /usr/lib64/apache2 -f
/etc/apache2/httpd.conf -k start
apache 14418 9.6 4.0 781036 128604 ? Sl 16:43 0:21 \_
/usr/sbin/apache2 -D SSL -D PHP5 -D STATUS -d /usr/lib64/apache2 -f
/etc/apache2/httpd.conf -k start
apache 14952 11.8 2.8 709736 91072 ? Sl 16:46 0:07 \_
/usr/sbin/apache2 -D SSL -D PHP5 -D STATUS -d /usr/lib64/apache2 -f
/etc/apache2/httpd.conf -k start
apache 15294 8.4 2.5 761932 80584 ? Sl 16:46 0:02 \_
/usr/sbin/apache2 -D SSL -D PHP5 -D STATUS -d /usr/lib64/apache2 -f
/etc/apache2/httpd.conf -k start
apache 15329 90923337 2.3 695200 73448 ? Sl 16:46 21153009:43
\_ /usr/sbin/apache2 -D SSL -D PHP5 -D STATUS -d /usr/lib64/apache2 -f
/etc/apache2/httpd.conf -k start
apache 15364 7.8 2.3 683200 74408 ? Sl 16:46 0:01 \_
/usr/sbin/apache2 -D SSL -D PHP5 -D STATUS -d /usr/lib64/apache2 -f
/etc/apache2/httpd.conf -k start
apache 15371 9.3 2.3 694896 74564 ? Sl 16:46 0:02 \_
/usr/sbin/apache2 -D SSL -D PHP5 -D STATUS -d /usr/lib64/apache2 -f
/etc/apache2/httpd.conf -k start
apache 15423 15.8 2.3 755896 74888 ? Sl 16:47 0:02 \_
/usr/sbin/apache2 -D SSL -D PHP5 -D STATUS -d /usr/lib64/apache2 -f
/etc/apache2/httpd.conf -k start
The apache server-status page shows these up as processing requests
correctly (state K, C or R) but with improbably high values in the CPU
column.
If anyone has any ideas where I can start looking (or can just tell me
how to fix it!) it would be greatly appreciated.
Cheers
--
Dave Barton
Senior Systems Administrator
Comodo CA Ltd
Re: [users@httpd] Apache spawning "broken" sub-processes under load
Posted by Dave Barton <da...@comodo.com>.
Hi,
To answer the questions from the other email first;
>>Which version of WP?
>>Did you make -- recently -- any changes? Upgrade, install new plugins?
>>Did you traffic spontaneously increase?
It's an old version (2.7) but I'm not sure how much of an option an
upgrade is. I'll have to raise it with our developers.
>>Given the line-breaks it's *really* hard to read what you mean :-/
Sorry :(
>>Are you running this in a Zone/Container/Jail/Vz/blah
The WordPress installation is running as an Apache vhost on a Xen
virtual machine. The VM is only used for Apache and doesn't run anything
else (except stuff like ntpd and snmpd of course).
Other answers are in-line below. Thanks for trying to help me out! :)
Dave
On 31/01/11 18:28, Igor Galić wrote:
>
> ----- "Dave Barton" <da...@comodo.com> wrote:
>
>> Should really have included this;
>>
>> # apache2 -v
>> Server version: Apache/2.2.14 (Unix)
>> Server built: Jan 29 2010 12:34:06
>>
>> # apache2ctl modules
>> [Mon Jan 31 17:29:24 2011] [warn] module status_module is already
>> loaded, skipping
>> Loaded Modules:
>> core_module (static)
>> mpm_worker_module (static)
>> http_module (static)
>> so_module (static)
>> alias_module (shared)
>> auth_basic_module (shared)
>> authn_alias_module (shared)
>> authn_anon_module (shared)
>> authn_default_module (shared)
>> authn_file_module (shared)
>> authz_groupfile_module (shared)
>
> Do you really need all these modules?
> Take a look at: http://httpd.apache.org/docs/current/mod/
> To see which you can safely remove.
Looking into that now, thanks for the tip.
>
> [snip]
>> info_module (shared)
>>
>> # php -v
>> PHP 5.2.12-pl0-gentoo (cli) (built: Jan 26 2010 16:48:31)
>> Copyright (c) 1997-2009 The PHP Group
>> Zend Engine v2.2.0, Copyright (c) 1998-2009 Zend Technologies
>
> Are you running PHP with APC or a different accelerator? If not,
> you should a) consider it, and b) be aware that WP has some issues
> with these..
We're not running any accelerator with PHP. The traffic levels for this
site are generally so low that it's not an issue and they can be more
trouble than they are worth.
>
>> # uname -a
>> Linux mcpweb2 2.6.31-xen-r10 #1 SMP Sat Nov 27 14:17:17 UTC 2010
>> x86_64
>> Intel(R) Xeon(R) CPU E5530 @ 2.40GHz GenuineIntel GNU/Linux
>>
>> If anything else is needed, please let me know.
>>
>> Cheers
>>
>> Dave Barton
>> Senior Systems Administrator
>> Comodo CA Ltd
>
> i
>
Re: [users@httpd] Apache spawning "broken" sub-processes under load
Posted by Igor Galić <i....@brainsware.org>.
----- "Dave Barton" <da...@comodo.com> wrote:
> Should really have included this;
>
> # apache2 -v
> Server version: Apache/2.2.14 (Unix)
> Server built: Jan 29 2010 12:34:06
>
> # apache2ctl modules
> [Mon Jan 31 17:29:24 2011] [warn] module status_module is already
> loaded, skipping
> Loaded Modules:
> core_module (static)
> mpm_worker_module (static)
> http_module (static)
> so_module (static)
> alias_module (shared)
> auth_basic_module (shared)
> authn_alias_module (shared)
> authn_anon_module (shared)
> authn_default_module (shared)
> authn_file_module (shared)
> authz_groupfile_module (shared)
Do you really need all these modules?
Take a look at: http://httpd.apache.org/docs/current/mod/
To see which you can safely remove.
[snip]
> info_module (shared)
>
> # php -v
> PHP 5.2.12-pl0-gentoo (cli) (built: Jan 26 2010 16:48:31)
> Copyright (c) 1997-2009 The PHP Group
> Zend Engine v2.2.0, Copyright (c) 1998-2009 Zend Technologies
Are you running PHP with APC or a different accelerator? If not,
you should a) consider it, and b) be aware that WP has some issues
with these..
> # uname -a
> Linux mcpweb2 2.6.31-xen-r10 #1 SMP Sat Nov 27 14:17:17 UTC 2010
> x86_64
> Intel(R) Xeon(R) CPU E5530 @ 2.40GHz GenuineIntel GNU/Linux
>
> If anything else is needed, please let me know.
>
> Cheers
>
> Dave Barton
> Senior Systems Administrator
> Comodo CA Ltd
i
--
Igor Galić
Tel: +43 (0) 664 886 22 883
Mail: i.galic@brainsware.org
URL: http://brainsware.org/
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Apache spawning "broken" sub-processes under load
Posted by Dave Barton <da...@comodo.com>.
Should really have included this;
# apache2 -v
Server version: Apache/2.2.14 (Unix)
Server built: Jan 29 2010 12:34:06
# apache2ctl modules
[Mon Jan 31 17:29:24 2011] [warn] module status_module is already
loaded, skipping
Loaded Modules:
core_module (static)
mpm_worker_module (static)
http_module (static)
so_module (static)
alias_module (shared)
auth_basic_module (shared)
authn_alias_module (shared)
authn_anon_module (shared)
authn_default_module (shared)
authn_file_module (shared)
authz_groupfile_module (shared)
authz_host_module (shared)
authz_user_module (shared)
autoindex_module (shared)
cgid_module (shared)
dbd_module (shared)
deflate_module (shared)
dir_module (shared)
env_module (shared)
expires_module (shared)
filter_module (shared)
headers_module (shared)
include_module (shared)
log_config_module (shared)
logio_module (shared)
mime_module (shared)
mime_magic_module (shared)
negotiation_module (shared)
rewrite_module (shared)
setenvif_module (shared)
ssl_module (shared)
status_module (shared)
unique_id_module (shared)
usertrack_module (shared)
php5_module (shared)
info_module (shared)
# php -v
PHP 5.2.12-pl0-gentoo (cli) (built: Jan 26 2010 16:48:31)
Copyright (c) 1997-2009 The PHP Group
Zend Engine v2.2.0, Copyright (c) 1998-2009 Zend Technologies
# uname -a
Linux mcpweb2 2.6.31-xen-r10 #1 SMP Sat Nov 27 14:17:17 UTC 2010 x86_64
Intel(R) Xeon(R) CPU E5530 @ 2.40GHz GenuineIntel GNU/Linux
If anything else is needed, please let me know.
Cheers
Dave Barton
Senior Systems Administrator
Comodo CA Ltd
On 31/01/11 16:53, Dave Barton wrote:
> Hi,
>
> We've recently had a pretty large increase in the usage of one of our
> Wordpress-powered sites. I'm seeing some very odd behaviour and I'm at a
> loss as to how to investigate it further. Some of the apache processes
> are spawning children with broken CPU usage and time stats (as shown by
> ps auxf). I think it might be best just to show some examples;
>
> root 3153 0.0 0.4 304688 12960 ? Ss 15:26 0:00
> /usr/sbin/apache2 -D SSL -D PHP5 -D STATUS -d /usr/lib64/apache2 -f
> /etc/apache2/httpd.conf -k start
> apache 3155 0.0 0.1 215864 3816 ? S 15:26 0:00 \_
> /usr/sbin/apache2 -D SSL -D PHP5 -D STATUS -d /usr/lib64/apache2 -f
> /etc/apache2/httpd.conf -k start
> apache 12135 129895172 4.0 542620 127208 ? Sl 16:33 21114987:41
> \_ /usr/sbin/apache2 -D SSL -D PHP5 -D STATUS -d /usr/lib64/apache2 -f
> /etc/apache2/httpd.conf -k start
> apache 12353 10.6 4.0 729908 127676 ? Sl 16:34 1:20 \_
> /usr/sbin/apache2 -D SSL -D PHP5 -D STATUS -d /usr/lib64/apache2 -f
> /etc/apache2/httpd.conf -k start
> apache 12875 11.1 3.8 773840 121116 ? Sl 16:36 1:09 \_
> /usr/sbin/apache2 -D SSL -D PHP5 -D STATUS -d /usr/lib64/apache2 -f
> /etc/apache2/httpd.conf -k start
> apache 13222 9.1 4.1 845240 130568 ? Sl 16:38 0:49 \_
> /usr/sbin/apache2 -D SSL -D PHP5 -D STATUS -d /usr/lib64/apache2 -f
> /etc/apache2/httpd.conf -k start
> apache 13445 10.5 3.6 725956 114972 ? Sl 16:39 0:50 \_
> /usr/sbin/apache2 -D SSL -D PHP5 -D STATUS -d /usr/lib64/apache2 -f
> /etc/apache2/httpd.conf -k start
> apache 13582 9.4 3.5 792056 112884 ? Sl 16:39 0:42 \_
> /usr/sbin/apache2 -D SSL -D PHP5 -D STATUS -d /usr/lib64/apache2 -f
> /etc/apache2/httpd.conf -k start
> apache 14148 9.2 3.4 724848 107920 ? Sl 16:42 0:27 \_
> /usr/sbin/apache2 -D SSL -D PHP5 -D STATUS -d /usr/lib64/apache2 -f
> /etc/apache2/httpd.conf -k start
> apache 14253 290358163 3.4 709912 108308 ? Sl 16:42 21133796:09
> \_ /usr/sbin/apache2 -D SSL -D PHP5 -D STATUS -d /usr/lib64/apache2 -f
> /etc/apache2/httpd.conf -k start
> apache 14287 431815 3.5 717512 110180 ? Sl 16:42 19215:48 \_
> /usr/sbin/apache2 -D SSL -D PHP5 -D STATUS -d /usr/lib64/apache2 -f
> /etc/apache2/httpd.conf -k start
> apache 14356 9.5 3.5 722780 111908 ? Sl 16:43 0:24 \_
> /usr/sbin/apache2 -D SSL -D PHP5 -D STATUS -d /usr/lib64/apache2 -f
> /etc/apache2/httpd.conf -k start
> apache 14392 324909060 3.9 735640 123796 ? Sl 16:43 21133795:59
> \_ /usr/sbin/apache2 -D SSL -D PHP5 -D STATUS -d /usr/lib64/apache2 -f
> /etc/apache2/httpd.conf -k start
> apache 14418 9.6 4.0 781036 128604 ? Sl 16:43 0:21 \_
> /usr/sbin/apache2 -D SSL -D PHP5 -D STATUS -d /usr/lib64/apache2 -f
> /etc/apache2/httpd.conf -k start
> apache 14952 11.8 2.8 709736 91072 ? Sl 16:46 0:07 \_
> /usr/sbin/apache2 -D SSL -D PHP5 -D STATUS -d /usr/lib64/apache2 -f
> /etc/apache2/httpd.conf -k start
> apache 15294 8.4 2.5 761932 80584 ? Sl 16:46 0:02 \_
> /usr/sbin/apache2 -D SSL -D PHP5 -D STATUS -d /usr/lib64/apache2 -f
> /etc/apache2/httpd.conf -k start
> apache 15329 90923337 2.3 695200 73448 ? Sl 16:46 21153009:43
> \_ /usr/sbin/apache2 -D SSL -D PHP5 -D STATUS -d /usr/lib64/apache2 -f
> /etc/apache2/httpd.conf -k start
> apache 15364 7.8 2.3 683200 74408 ? Sl 16:46 0:01 \_
> /usr/sbin/apache2 -D SSL -D PHP5 -D STATUS -d /usr/lib64/apache2 -f
> /etc/apache2/httpd.conf -k start
> apache 15371 9.3 2.3 694896 74564 ? Sl 16:46 0:02 \_
> /usr/sbin/apache2 -D SSL -D PHP5 -D STATUS -d /usr/lib64/apache2 -f
> /etc/apache2/httpd.conf -k start
> apache 15423 15.8 2.3 755896 74888 ? Sl 16:47 0:02 \_
> /usr/sbin/apache2 -D SSL -D PHP5 -D STATUS -d /usr/lib64/apache2 -f
> /etc/apache2/httpd.conf -k start
>
> The apache server-status page shows these up as processing requests
> correctly (state K, C or R) but with improbably high values in the CPU
> column.
>
> If anyone has any ideas where I can start looking (or can just tell me
> how to fix it!) it would be greatly appreciated.
>
> Cheers