You are viewing a plain text version of this content. The canonical link for it is here.
Posted to bugs@httpd.apache.org by bu...@apache.org on 2007/10/02 12:54:41 UTC
DO NOT REPLY [Bug 43533] New: - Frequent crashes in mod_include's bndm()
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG�
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
<http://issues.apache.org/bugzilla/show_bug.cgi?id=43533>.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND�
INSERTED IN THE BUG DATABASE.
http://issues.apache.org/bugzilla/show_bug.cgi?id=43533
Summary: Frequent crashes in mod_include's bndm()
Product: Apache httpd-2
Version: 2.0.61
Platform: Macintosh
OS/Version: Mac OS X 10.4
Status: NEW
Severity: normal
Priority: P2
Component: mod_include
AssignedTo: bugs@httpd.apache.org
ReportedBy: noah@hd.se
Over the past years Apache has crashed several times every hour.
I've been running Apache 2 since 2.0.40 and now we're up at 2.0.61 and still
experiencing these crashes so I guess it's about time I report this bug.
Our site heavily relies on SSI, in particular <!--#include virtual=..-->
(they're often nested aswell), and mod_rewrite. Output is compressed with
mod_deflate. Often the crashdumps looks like this:
crashdump[26541]: crashdump[26541]: Thread 20 Crashed:
crashdump[26541]: 0 httpd 0x000047a0 bndm + 120
(mod_include.c:317)
crashdump[26541]: 1 httpd 0x0000a100 find_start_sequence
+ 136 (mod_include.c:2388)
crashdump[26541]: 2 httpd 0x0000bad0 send_parsed_content
+ 1288 (mod_include.c:3054)
crashdump[26541]: 3 httpd 0x0000dbf8 includes_filter +
1468 (mod_include.c:3591)
crashdump[26541]: 4 httpd 0x00068e30 ap_pass_brigade +
236 (util_filter.c:512)
crashdump[26541]: 5 httpd 0x0004a5c8 default_handler +
1888 (core.c:3648)
crashdump[26541]: 6 httpd 0x00060038 ap_run_handler + 136
(config.c:152)
crashdump[26541]: 7 httpd 0x00060d8c ap_invoke_handler +
424 (config.c:364)
crashdump[26541]: 8 httpd 0x00051f98 ap_run_sub_req + 104
(request.c:1855)
crashdump[26541]: 9 httpd 0x00005a68 handle_include + 968
(mod_include.c:782)
crashdump[26541]: 10 httpd 0x0000cc80 send_parsed_content
+ 5816 (mod_include.c:3309)
crashdump[26541]: 11 httpd 0x0000dbf8 includes_filter +
1468 (mod_include.c:3591)
crashdump[26541]: 12 httpd 0x00068e30 ap_pass_brigade +
236 (util_filter.c:512)
crashdump[26541]: 13 httpd 0x0004a5c8 default_handler +
1888 (core.c:3648)
crashdump[26541]: 14 httpd 0x00060038 ap_run_handler + 136
(config.c:152)
crashdump[26541]: 15 httpd 0x00060d8c ap_invoke_handler +
424 (config.c:364)
crashdump[26541]: 16 httpd 0x0001d980 ap_process_request +
148 (http_request.c:249)
crashdump[26541]: 17 httpd 0x000145a4
ap_process_http_connection + 136 (http_core.c:255)
crashdump[26541]: 18 httpd 0x0006bcdc
ap_run_process_connection + 136 (connection.c:43)
crashdump[26541]: 19 httpd 0x0006c318
ap_process_connection + 132 (connection.c:178)
crashdump[26541]: 20 httpd 0x0003b408 process_socket + 196
(worker.c:523)
crashdump[26541]: 21 httpd 0x0003befc worker_thread + 552
(worker.c:843)
crashdump[26541]: 22 libapr-0.0.dylib 0x003796f0 dummy_worker + 68
(thread.c:105)
crashdump[26541]: 23 libSystem.B.dylib 0x9002bd08 _pthread_body + 96
--------------------------
crashdump[4103]: crashdump[4103]: Thread 15 Crashed:
crashdump[4103]: 0 httpd 0x000047a0 bndm + 120
(mod_include.c:317)
crashdump[4103]: 1 httpd 0x0000a100 find_start_sequence
+ 136 (mod_include.c:2388)
crashdump[4103]: 2 httpd 0x0000bad0 send_parsed_content
+ 1288 (mod_include.c:3054)
crashdump[4103]: 3 httpd 0x0000dbf8 includes_filter +
1468 (mod_include.c:3591)
crashdump[4103]: 4 httpd 0x00068e30 ap_pass_brigade +
236 (util_filter.c:512)
crashdump[4103]: 5 httpd 0x0004a5c8 default_handler +
1888 (core.c:3648)
crashdump[4103]: 6 httpd 0x00060038 ap_run_handler + 136
(config.c:152)
crashdump[4103]: 7 httpd 0x00060d8c ap_invoke_handler +
424 (config.c:364)
crashdump[4103]: 8 httpd 0x00051f98 ap_run_sub_req + 104
(request.c:1855)
crashdump[4103]: 9 httpd 0x00005a68 handle_include + 968
(mod_include.c:782)
crashdump[4103]: 10 httpd 0x0000cc80 send_parsed_content
+ 5816 (mod_include.c:3309)
crashdump[4103]: 11 httpd 0x0000dbf8 includes_filter +
1468 (mod_include.c:3591)
crashdump[4103]: 12 httpd 0x00068e30 ap_pass_brigade +
236 (util_filter.c:512)
crashdump[4103]: 13 httpd 0x0004a5c8 default_handler +
1888 (core.c:3648)
crashdump[4103]: 14 httpd 0x00060038 ap_run_handler + 136
(config.c:152)
crashdump[4103]: 15 httpd 0x00060d8c ap_invoke_handler +
424 (config.c:364)
crashdump[4103]: 16 httpd 0x0001d980 ap_process_request +
148 (http_request.c:249)
crashdump[4103]: 17 httpd 0x000145a4
ap_process_http_connection + 136 (http_core.c:255)
crashdump[4103]: 18 httpd 0x0006bcdc
ap_run_process_connection + 136 (connection.c:43)
crashdump[4103]: 19 httpd 0x0006c318
ap_process_connection + 132 (connection.c:178)
crashdump[4103]: 20 httpd 0x0003b408 process_socket + 196
(worker.c:523)
crashdump[4103]: 21 httpd 0x0003befc worker_thread + 552
(worker.c:843)
crashdump[4103]: 22 libapr-0.0.dylib 0x003796f0 dummy_worker + 68
(thread.c:105)
crashdump[4103]: 23 libSystem.B.dylib 0x9002bd08 _pthread_body + 96
------------------
I saw someone else having a similar problem here. There was an open bug about
random crashes in crc32 and bndm, but with Apache 2.2.
I've tried to log the PID along with the HTTP requests into a logfile and
grepping out the PID after the crash but I guess the thread crashes before the
log is written since I'm not finding anything interesting there.
I've not been able to reproduce this but it keeps happening several times each
hour. Since we've got a fair share of traffic to this host I can't really gdb
Apache and wait for a crash and see what's going on.
It might be a memory corruption since we're getting crashes in other places
aswell. It's not likely to be a hardware problem since the box have been
replaced two times already (for performance reasons) and the crashes have
continued on the new hardware aswell.
Here's one of the other crashes we're getting quite frequently but not as
frequently as with bndm():
crashdump[14604]: crashdump[14604]: Thread 7 Crashed:
crashdump[14604]: 0 httpd 0x0000a6c0 find_directive + 284
(mod_include.c:2552)
crashdump[14604]: 1 httpd 0x0000bf14 send_parsed_content
+ 2380 (mod_include.c:3118)
crashdump[14604]: 2 httpd 0x0000dbf8 includes_filter +
1468 (mod_include.c:3591)
crashdump[14604]: 3 httpd 0x00068e30 ap_pass_brigade +
236 (util_filter.c:512)
crashdump[14604]: 4 httpd 0x0004a5c8 default_handler +
1888 (core.c:3648)
crashdump[14604]: 5 httpd 0x00060038 ap_run_handler + 136
(config.c:152)
crashdump[14604]: 6 httpd 0x00060d8c ap_invoke_handler +
424 (config.c:364)
crashdump[14604]: 7 httpd 0x00051f98 ap_run_sub_req + 104
(request.c:1855)
crashdump[14604]: 8 httpd 0x00005a68 handle_include + 968
(mod_include.c:782)
crashdump[14604]: 9 httpd 0x0000cc80 send_parsed_content
+ 5816 (mod_include.c:3309)
crashdump[14604]: 10 httpd 0x0000dbf8 includes_filter +
1468 (mod_include.c:3591)
crashdump[14604]: 11 httpd 0x00068e30 ap_pass_brigade +
236 (util_filter.c:512)
crashdump[14604]: 12 httpd 0x0004a5c8 default_handler +
1888 (core.c:3648)
crashdump[14604]: 13 httpd 0x00060038 ap_run_handler + 136
(config.c:152)
crashdump[14604]: 14 httpd 0x00060d8c ap_invoke_handler +
424 (config.c:364)
crashdump[14604]: 15 httpd 0x00051f98 ap_run_sub_req + 104
(request.c:1855)
crashdump[14604]: 16 httpd 0x00005a68 handle_include + 968
(mod_include.c:782)
crashdump[14604]: 17 httpd 0x0000cc80 send_parsed_content
+ 5816 (mod_include.c:3309)
crashdump[14604]: 18 httpd 0x0000dbf8 includes_filter +
1468 (mod_include.c:3591)
crashdump[14604]: 19 httpd 0x00068e30 ap_pass_brigade +
236 (util_filter.c:512)
crashdump[14604]: 20 httpd 0x0004a5c8 default_handler +
1888 (core.c:3648)
crashdump[14604]: 21 httpd 0x00060038 ap_run_handler + 136
(config.c:152)
crashdump[14604]: 22 httpd 0x00060d8c ap_invoke_handler +
424 (config.c:364)
crashdump[14604]: 23 httpd 0x0001d980 ap_process_request +
148 (http_request.c:249)
crashdump[14604]: 24 httpd 0x000145a4
ap_process_http_connection + 136 (http_core.c:255)
crashdump[14604]: 25 httpd 0x0006bcdc
ap_run_process_connection + 136 (connection.c:43)
crashdump[14604]: 26 httpd 0x0006c318
ap_process_connection + 132 (connection.c:178)
crashdump[14604]: 27 httpd 0x0003b408 process_socket + 196
(worker.c:523)
crashdump[14604]: 28 httpd 0x0003befc worker_thread + 552
(worker.c:843)
crashdump[14604]: 29 libapr-0.0.dylib 0x003796f0 dummy_worker + 68
(thread.c:105)
------
This problem has occured on Mac OS X 10.3 and up to 10.4.10, both Server
versions. We've been running a broad list of different versions of Apache 2,
from .40 to .61.
This problem occurs with the threaded worker.
Does anyone have any idea what's going on here?
--
Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 43533] - Frequent crashes in mod_include's bndm()
Posted by bu...@apache.org.
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG�
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
<http://issues.apache.org/bugzilla/show_bug.cgi?id=43533>.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND�
INSERTED IN THE BUG DATABASE.
http://issues.apache.org/bugzilla/show_bug.cgi?id=43533
jorton@redhat.com changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |RESOLVED
Resolution| |DUPLICATE
------- Additional Comments From jorton@redhat.com 2007-10-11 06:26 -------
I recently worked with a user to track down an issue with identical backtraces -
the root cause was PR 36780; (optimistically?) marking as a duplicate.
*** This bug has been marked as a duplicate of 36780 ***
--
Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 43533] - Frequent crashes in mod_include's bndm()
Posted by bu...@apache.org.
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG�
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
<http://issues.apache.org/bugzilla/show_bug.cgi?id=43533>.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND�
INSERTED IN THE BUG DATABASE.
http://issues.apache.org/bugzilla/show_bug.cgi?id=43533
------- Additional Comments From noah@hd.se 2007-10-12 15:09 -------
This is what Mac OS X's crashdump process have to say.
Thread 2 Crashed:
0 httpd 0x00004760 bndm + 120 (mod_include.c:317)
1 httpd 0x0000a0c0 find_start_sequence + 136
(mod_include.c:2388)
2 httpd 0x0000ba90 send_parsed_content + 1288
(mod_include.c:3054)
3 httpd 0x0000dbb8 includes_filter + 1468
(mod_include.c:3591)
4 httpd 0x00068e20 ap_pass_brigade + 236 (util_filter.c:512)
5 httpd 0x0004a588 default_handler + 1888 (core.c:3648)
6 httpd 0x00060028 ap_run_handler + 136 (config.c:152)
7 httpd 0x00060d7c ap_invoke_handler + 424 (config.c:364)
8 httpd 0x00051f58 ap_run_sub_req + 104 (request.c:1855)
9 httpd 0x00005a28 handle_include + 968 (mod_include.c:782)
10 httpd 0x0000cc40 send_parsed_content + 5816
(mod_include.c:3309)
11 httpd 0x0000dbb8 includes_filter + 1468
(mod_include.c:3591)
12 httpd 0x00068e20 ap_pass_brigade + 236 (util_filter.c:512)
13 httpd 0x0004a588 default_handler + 1888 (core.c:3648)
14 httpd 0x00060028 ap_run_handler + 136 (config.c:152)
15 httpd 0x00060d7c ap_invoke_handler + 424 (config.c:364)
16 httpd 0x0001d940 ap_process_request + 148
(http_request.c:249)
17 httpd 0x00014564 ap_process_http_connection + 136
(http_core.c:255)
18 httpd 0x0006bcdc ap_run_process_connection + 136
(connection.c:43)
19 httpd 0x0006c318 ap_process_connection + 132
(connection.c:178)
20 httpd 0x0003b3c8 process_socket + 196 (worker.c:523)
21 httpd 0x0003bebc worker_thread + 552 (worker.c:843)
22 libapr-0.0.dylib 0x003796f0 dummy_worker + 68 (thread.c:105)
23 libSystem.B.dylib 0x9002bd08 _pthread_body + 96
Now, if I'm running GDB, I'm getting this.
# gdb -q /opt/FrontA/apache/bin/httpd core.9148
Reading symbols for shared libraries ........... done
#0 0x900148a8 in read ()
(gdb) source /data/FrontA/software/httpd-2.0.61/.gdbinit
(gdb) thread 2
[Switching to thread 2 (core thread 1)]
#0 0x9001f88c in select ()
(gdb) bt
#0 0x9001f88c in select ()
#1 0x003851ec in ?? ()
#2 0x00386728 in ?? ()
#3 0x00372e2c in ?? ()
#4 0x001af150 in ?? ()
#5 0x001aba20 in ?? ()
#6 0x0004ac48 in core_input_filter (f=0x133d060, b=0x11d4bd0,
mode=AP_MODE_GETLINE, block=APR_BLOCK_READ, readbytes=0) at core.c:3802
#7 0x00068d04 in ap_get_brigade (next=0x133d060, bb=0x11d4bd0,
mode=AP_MODE_GETLINE, block=APR_BLOCK_READ, readbytes=0) at util_filter.c:475
#8 0x0004a83c in net_time_filter (f=0x137ff70, b=0x11d4bd0,
mode=AP_MODE_GETLINE, block=APR_BLOCK_READ, readbytes=0) at core.c:3705
#9 0x00068d04 in ap_get_brigade (next=0x137ff70, bb=0x11d4bd0,
mode=AP_MODE_GETLINE, block=APR_BLOCK_READ, readbytes=0) at util_filter.c:475
#10 0x0005b51c in ap_rgetline_core (s=0x1327c58, n=8192, read=0xf0101b40,
r=0x1327c40, fold=0, bb=0x11d4bd0) at protocol.c:230
#11 0x0005beac in read_request_line (r=0x1327c40, bb=0x11d4bd0) at protocol.c:586
#12 0x0005cb70 in ap_read_request (conn=0x133ce10) at protocol.c:877
#13 0x00014648 in ap_process_http_connection () at apr_xml.c:737
#14 0x0006bcdc in ap_run_process_connection (c=0x133ce10) at connection.c:43
#15 0x0006c318 in ap_process_connection (c=0x133ce10, csd=0x1321200) at
connection.c:176
#16 0x0003b3c8 in process_socket (p=0x11973a0, sock=0x1321200, my_child_num=7,
my_thread_num=0, bucket_alloc=0x185fc18) at worker.c:522
#17 0x0003bebc in worker_thread (thd=0x416c60, dummy=0x469760) at worker.c:842
#18 0x003796f0 in ?? ()
#19 0x9002bd08 in _pthread_body ()
GDB and crashdump sees different addresses for "0x00014648 in
ap_process_http_connection". Is this normal?
After that it seems that the stackframes differs wildly, at least in my eyes.
gdb -v says
GNU gdb 6.1-20040303 (Apple version gdb-384) (Mon Mar 21 00:05:26 GMT 2005)
Apache was compiled using:
./configure --enable-so --enable-mods-shared=auth-anon auth-digest file-cache
cache disk-cache mem-cache deflate mime-magic expires headers use
rtrack unique-id proxy proxy-http ssl vhost-alias cgi info speling rewrite
--with-mpm=prefork --mandir=/usr/share/man --prefix=/opt/FrontA/apache-2
.0.61-prefork --enable-pool-debug
Five years back or so when I last used GDB I recall it had issues with threads,
is that still so?
Maybe I should try to see if the problem occurs with MPM prefork unless you've
got some input on this thing?
I tried doing a bunch of backtraces on other core files aswell but their
backtraces all look like the one above.
--
Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 43533] - Frequent crashes in mod_include's bndm()
Posted by bu...@apache.org.
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG�
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
<http://issues.apache.org/bugzilla/show_bug.cgi?id=43533>.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND�
INSERTED IN THE BUG DATABASE.
http://issues.apache.org/bugzilla/show_bug.cgi?id=43533
------- Additional Comments From noah@hd.se 2007-10-12 02:13 -------
(In reply to comment #2)
> Reporter says the patch for bug 36780 didn't help, so re-opening.
>
> Noah:
>
> 1) can you reproduce this with prefork rather than worker?
I'll try.
Is it crucial for the debugging part or just interesting to know whether it
might be a threading related problem or not?
> 2) can you get a coredump and do some debugging with gdb?
>
> What is needed out of the core dump is:
>
> # gdb /path/to/httpd core.dump
> (gdb) source /patch/to/httpd/src/.gdbinit
>
> then select a frame with a apr_bucket_brigade * in scope, e.g.
> send_parsed_content, e.g. "up 2" from that first "Thread 20", and do:
>
> (gdb) dump_brigade bb
>
> to dump the contents of the brigade.
I've enabled coredumps and I'll see if I can find some time to look into it.
--
Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 43533] - Frequent crashes in mod_include's bndm()
Posted by bu...@apache.org.
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG�
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
<http://issues.apache.org/bugzilla/show_bug.cgi?id=43533>.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND�
INSERTED IN THE BUG DATABASE.
http://issues.apache.org/bugzilla/show_bug.cgi?id=43533
jorton@redhat.com changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|RESOLVED |REOPENED
Resolution|DUPLICATE |
------- Additional Comments From jorton@redhat.com 2007-10-12 01:28 -------
Reporter says the patch for bug 36780 didn't help, so re-opening.
Noah:
1) can you reproduce this with prefork rather than worker?
2) can you get a coredump and do some debugging with gdb?
What is needed out of the core dump is:
# gdb /path/to/httpd core.dump
(gdb) source /patch/to/httpd/src/.gdbinit
then select a frame with a apr_bucket_brigade * in scope, e.g.
send_parsed_content, e.g. "up 2" from that first "Thread 20", and do:
(gdb) dump_brigade bb
to dump the contents of the brigade.
--
Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org