You are viewing a plain text version of this content. The canonical link for it is here.
Posted to bugs@httpd.apache.org by bu...@apache.org on 2011/04/14 11:23:12 UTC
DO NOT REPLY [Bug 51062] New: httpd child process consumes CPU
https://issues.apache.org/bugzilla/show_bug.cgi?id=51062
Summary: httpd child process consumes CPU
Product: Apache httpd-2
Version: 2.2.15
Platform: PC
OS/Version: Linux
Status: NEW
Severity: normal
Priority: P2
Component: Core
AssignedTo: bugs@httpd.apache.org
ReportedBy: KKostalkowicz@ivmx.pl
Httpd is used as a reverse proxy, load is not significantly high.
Actual behaviour: one of httpd child processes is running (for 14 hours)
Expected behaviour: all child processes should be sleeping (having run at most
several seconds)
Got two traces by attaching to the faulty process via gdb.
* trace 1
#0 apr_bucket_free (mem=0x8fa9768) at buckets/apr_buckets_alloc.c:187
#1 0x001589c8 in apr_brigade_cleanup (data=0x8f5c3d0)
at buckets/apr_brigade.c:44
#2 0x001589fd in brigade_cleanup (data=0x8f5c3d0) at buckets/apr_brigade.c:34
#3 0x008870ed in run_cleanups (cref=0x8f5b230) at memory/unix/apr_pools.c:2308
#4 0x00887ee7 in apr_pool_clear (pool=0x8f5b220)
at memory/unix/apr_pools.c:737
#5 0x080aa186 in child_main (child_num_arg=<value optimized out>)
at prefork.c:554
#6 0x080aa5f7 in make_child (s=0x8e509a8, slot=16) at prefork.c:758
#7 0x080aaf50 in ap_mpm_run (_pconf=0x8e49600, plog=0x8e856f0, s=0x8e509a8)
at prefork.c:893
#8 0x08069aa5 in main (argc=149190136, argv=0x8f59218) at main.c:740
* trace 2
#0 0x001589b6 in apr_brigade_cleanup (data=0x8f5c3d0)
at buckets/apr_brigade.c:44
#1 0x001589fd in brigade_cleanup (data=0x8f5c3d0) at buckets/apr_brigade.c:34
#2 0x008870ed in run_cleanups (cref=0x8f5b230) at memory/unix/apr_pools.c:2308
#3 0x00887ee7 in apr_pool_clear (pool=0x8f5b220)
at memory/unix/apr_pools.c:737
#4 0x080aa186 in child_main (child_num_arg=<value optimized out>)
at prefork.c:554
#5 0x080aa5f7 in make_child (s=0x8e509a8, slot=16) at prefork.c:758
#6 0x080aaf50 in ap_mpm_run (_pconf=0x8e49600, plog=0x8e856f0, s=0x8e509a8)
at prefork.c:893
#7 0x08069aa5 in main (argc=149190136, argv=0x8f59218) at main.c:740
* conclusion
Process got stuck in brigade_cleanup(), continually invoking
apr_brigade_cleanup()
Possibly related issues: 46785, 50335, 34288.
I can provide more output from gdb, just give me the input.
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 51062] httpd child process consumes CPU
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=51062
--- Comment #9 from Nick Kew <ni...@webthing.com> 2011-05-13 12:12:09 UTC ---
Trunk patch in r1102687
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 51062] httpd child process consumes CPU
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=51062
--- Comment #6 from Krzysztof Kostałkowicz <KK...@ivmx.pl> 2011-04-15 06:44:09 EDT ---
The process loops trying do dispose a corrupted list.
Don't ask me where the corruption occurred.
Below some output from gdb with my notes.
---
apr_brigade_cleanup (data=0x8f5c3d0) at buckets/apr_brigade.c:42
42 while (!APR_BRIGADE_EMPTY(b)) {
# *b
(gdb) print *((apr_bucket_brigade *)data)
$3 = {p = 0x8f5b220, list = {next = 0x8fa9768, prev = 0x8fa9768},
bucket_alloc = 0x8f9f670}
# b
(gdb) print ((apr_bucket_brigade *)data)
$4 = (struct apr_bucket_brigade *) 0x8f5c3d0
# list head
(gdb) print ((apr_bucket_brigade *)data)->list
$5 = {next = 0x8fa9768, prev = 0x8fa9768}
# address of list head
(gdb) print &((apr_bucket_brigade *)data)->list
$24 = (struct apr_bucket_list *) 0x8f5c3d4
# first element of b
(gdb) print (((apr_bucket_brigade *)data)->list)->next
$6 = (struct apr_bucket *) 0x8fa9768
# value of first element of b
(gdb) print *(((apr_bucket_brigade *)data)->list)->next
$7 = {link = {next = 0x8f75c34, prev = 0x8f75c34}, type = 0x16c0e0,
length = 4294967295, start = -1, data = 0x8f75608,
free = 0x80662f8 <ap...@plt>, list = 0x8fa9698}
# second element of b
(gdb) print (((apr_bucket_brigade *)data)->list)->next->link->next
$8 = (struct apr_bucket * volatile) 0x8f75c34
# value of second element of b
(gdb) print *(((apr_bucket_brigade *)data)->list)->next->link->next
$9 = {link = {next = 0x8f75c34, prev = 0x8f75c34}, type = 0x8fa9698,
length = 150427552, start = 6062569827032112, data = 0x8066538,
free = 0x8f755c8, list = 0x8f75c54}
# sentinel for b (computed with APR_RING_SENTINEL)
(gdb) print (struct apr_bucket *)((char *)(&(&((apr_bucket_brigade
*)data)->list)->next))
$23 = (struct apr_bucket *) 0x8f5c3d4
# it's not (but should be) possible to reach sentinel of b from b
# while loops because the list is not empty
# ie. APR_RING_FIRST((hp)) != APR_RING_SENTINEL((hp), elem, link)
# apr_bucket_free(void *mem) goes into if_true branch
# it does not change the data structure (the assignments do not change
anything)
# if check_not_already_free(node_header_t *node) was called
# it would abort during first loop iteration
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 51062] httpd child process consumes CPU
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=51062
--- Comment #3 from Krzysztof Kostałkowicz <KK...@ivmx.pl> 2011-04-14 07:30:59 EDT ---
Fortunately, the process is still running. Here comes the version. I'll try to
get the traces later.
Server version: Apache/2.2.15 (Unix)
Server built: Jun 18 2010 17:28:27
Server's Module Magic Number: 20051115:24
Server loaded: APR 1.4.2, APR-Util 1.3.9
Compiled using: APR 1.4.2, APR-Util 1.3.9
Architecture: 32-bit
Server MPM: Prefork
threaded: no
forked: yes (variable process count)
Server compiled with....
-D APACHE_MPM_DIR="server/mpm/prefork"
-D APR_HAS_SENDFILE
-D APR_HAS_MMAP
-D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
-D APR_USE_SYSVSEM_SERIALIZE
-D APR_USE_PTHREAD_SERIALIZE
-D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
-D APR_HAS_OTHER_CHILD
-D AP_HAVE_RELIABLE_PIPED_LOGS
-D DYNAMIC_MODULE_LIMIT=128
-D HTTPD_ROOT="/usr/local/apache2"
-D SUEXEC_BIN="/usr/local/apache2/bin/suexec"
-D DEFAULT_PIDLOG="logs/httpd.pid"
-D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
-D DEFAULT_LOCKFILE="logs/accept.lock"
-D DEFAULT_ERRORLOG="logs/error_log"
-D AP_TYPES_CONFIG_FILE="conf/mime.types"
-D SERVER_CONFIG_FILE="conf/httpd.conf"
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 51062] httpd child process consumes CPU
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=51062
--- Comment #1 from Nick Kew <ni...@webthing.com> 2011-04-14 06:28:00 EDT ---
That's an interesting traceback. If you could get an execution trace (like
dtrace, ptrace, strace) it could hopefully tell us more detail if it is indeed
looping.
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 51062] httpd child process consumes CPU
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=51062
Krzysztof Kostałkowicz <KK...@ivmx.pl> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEEDINFO |NEW
--- Comment #4 from Krzysztof Kostałkowicz <KK...@ivmx.pl> 2011-04-14 09:54:46 EDT ---
# strace -p 6063
Process 6063 attached - interrupt to quit
(no syscalls...)
(pressed ^C)
Process 6063 detached
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 51062] httpd child process consumes CPU
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=51062
--- Comment #8 from Nick Kew <ni...@webthing.com> 2011-05-12 00:27:13 UTC ---
Hmm, don't know why it picked up APR 0.9.0 as version ....
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 51062] httpd child process consumes CPU
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=51062
Nick Kew <ni...@webthing.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |NEEDINFO
--- Comment #2 from Nick Kew <ni...@webthing.com> 2011-04-14 06:29:08 EDT ---
Also I should ask, what APR and APR-UTIL versions are you using?
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 51062] httpd child process consumes CPU
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=51062
Nick Kew <ni...@webthing.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Component|Core |APR-util
Version|2.2.15 |HEAD
Product|Apache httpd-2 |APR
--- Comment #7 from Nick Kew <ni...@webthing.com> 2011-05-12 00:25:41 UTC ---
Nice work - thanks!
I see trunk APR has an update that looks as if it should serve a similar
purpose to your suggestion of check_not_already_free:
APR_BRIGADE_CHECK_CONSISTENCY(b);
is inserted before the while() loop in apr_brigade_cleanup.
If noone comes up with a better selection, I'll add an explicit test to abort
the loop and return APR_EGENERAL if the bucket pointer repeats within the loop.
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 51062] httpd child process consumes CPU
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=51062
--- Comment #5 from Krzysztof Kostałkowicz <KK...@ivmx.pl> 2011-04-14 10:02:58 EDT ---
# ltrace -p 6063
apr_bucket_free(0x8fa9768, 0x8f5b220, 0xbff8a5d8, 0x887e0c, 0x8f5b230)
= 0x8fa9758
apr_bucket_destroy_noop(0x8f75608, 0x8f5b220, 0xbff8a5d8, 0x887e0c, 0x8f5b230)
= 0x8f75608
apr_bucket_free(0x8fa9768, 0x8f5b220, 0xbff8a5d8, 0x887e0c, 0x8f5b230)
= 0x8fa9758
apr_bucket_destroy_noop(0x8f75608, 0x8f5b220, 0xbff8a5d8, 0x887e0c, 0x8f5b230)
= 0x8f75608
apr_bucket_free(0x8fa9768, 0x8f5b220, 0xbff8a5d8, 0x887e0c, 0x8f5b230)
= 0x8fa9758
apr_bucket_destroy_noop(0x8f75608, 0x8f5b220, 0xbff8a5d8, 0x887e0c, 0x8f5b230)
= 0x8f75608
apr_bucket_free(0x8fa9768, 0x8f5b220, 0xbff8a5d8, 0x887e0c, 0x8f5b230)
= 0x8fa9758
apr_bucket_destroy_noop(0x8f75608, 0x8f5b220, 0xbff8a5d8, 0x887e0c, 0x8f5b230)
= 0x8f75608
apr_bucket_free(0x8fa9768, 0x8f5b220, 0x887e0c, 0x8f5b230 <unfinished ...>
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org