coredump backtraces

[Cliff Woolley <cl...@yahoo.com>]

---Brian Behlendorf said:
> Speaking of which, there are 5 corefiles in /usr/local/apache2/corefiles
> from the current process if anyone wants to look at them.  Looks like we
> segfault once every couple of hours or so, so this is really scraping the
> bottom of the barrel.  =)

Four of these (httpd.core.[1-3,5]) are a segfault when clearing a pool.  The
fault occurs when running dir_cleanup().  We've been seeing this for a while but
haven't fixed it yet.

The backtrace looks like this:
(gdb) bt
#0  0x28165964 in isatty () from /usr/lib/libc.so.4
#1  0x28165d49 in free () from /usr/lib/libc.so.4
#2  0x281523b2 in closedir () from /usr/lib/libc.so.4
#3  0x808cd48 in dir_cleanup (thedir=0x81d71f4) at dir.c:68
#4  0x808875a in run_cleanups (c=0x81d7ecc) at apr_pools.c:625
#5  0x8088857 in apr_clear_pool (a=0x81dc00c) at apr_pools.c:729
#6  0x80888b3 in apr_pool_destroy (a=0x81dc00c) at apr_pools.c:758
#7  0x806d5cc in ap_process_http_connection (c=0x81310ec) at connection.c:253
#8  0x806d398 in ap_run_process_connection (c=0x81310ec) at connection.c:82
#9  0x806d4fc in ap_process_connection (c=0x81310ec) at connection.c:216
#10 0x806446e in child_main (child_num_arg=90) at prefork.c:796
#11 0x806459d in make_child (s=0x80b4514, slot=90) at prefork.c:869
#12 0x8064795 in perform_idle_server_maintenance () at prefork.c:1010
#13 0x8064ab1 in ap_mpm_run (_pconf=0x80b400c, plog=0x80de00c, s=0x80b4514)
    at prefork.c:1180
#14 0x8068f0c in main (argc=1, argv=0xbfbffa10) at main.c:428
#15 0x8058af5 in _start ()


httpd.core.4 is a failed bucket assert:

(gdb) bt
#0  0x2812a3c8 in kill () from /usr/lib/libc.so.4
#1  0x2816609e in abort () from /usr/lib/libc.so.4
#2  0x80683ee in ap_log_assert (
    szExp=0x80964e0 "(((&(req_cfg->bb)->list))->next == (struct apr_bucket
*)((char *)((&(req_cfg->bb)->list)) - ((size_t)(&((struct apr_bucket *)0)->
link))))", szFile=0x80962e0 "http_protocol.c", nLine=2722) at log.c:562
#3  0x8060004 in ap_setup_client_block (r=0x81ca03c, read_policy=2)
    at http_protocol.c:2722
#4  0x80602f5 in ap_discard_request_body (r=0x81ca03c) at http_protocol.c:2868
#5  0x805bb4e in default_handler (r=0x81ca03c) at http_core.c:2975
#6  0x8065658 in ap_run_handler (r=0x81ca03c) at config.c:131
#7  0x80659d3 in ap_invoke_handler (r=0x81ca03c) at config.c:300
#8  0x8063278 in process_request_internal (r=0x81ca03c) at http_request.c:1367
#9  0x8063327 in ap_process_request (r=0x81ca03c) at http_request.c:1406
#10 0x806d58b in ap_process_http_connection (c=0x81310ec) at connection.c:244
#11 0x806d398 in ap_run_process_connection (c=0x81310ec) at connection.c:82
#12 0x806d4fc in ap_process_connection (c=0x81310ec) at connection.c:216
#13 0x806446e in child_main (child_num_arg=88) at prefork.c:796
#14 0x806459d in make_child (s=0x80b4514, slot=88) at prefork.c:869
#15 0x8064795 in perform_idle_server_maintenance () at prefork.c:1010
#16 0x8064ab1 in ap_mpm_run (_pconf=0x80b400c, plog=0x80de00c, s=0x80b4514)
    at prefork.c:1180
#17 0x8068f0c in main (argc=1, argv=0xbfbffa10) at main.c:428
#18 0x8058af5 in _start ()


I'll look into these, but if anybody else has any thoughts on them, feel free to
look as well.

--Cliff

[Issue #2 Update] RE: coredump backtraces

[Cliff Woolley <cl...@yahoo.com>]

> httpd.core.4 is a failed bucket assert:
> (gdb) bt
> #2  0x80683ee in ap_log_assert (
>     szExp=0x80964e0 "(((&(req_cfg->bb)->list))->next == (struct apr_bucket
> *)((char *)((&(req_cfg->bb)->list)) - ((size_t)(&((struct apr_bucket *)0)->
> link))))", szFile=0x80962e0 "http_protocol.c", nLine=2722) at log.c:562
> #3  0x8060004 in ap_setup_client_block (r=0x81ca03c, read_policy=2)
>     at http_protocol.c:2722
> #4  0x80602f5 in ap_discard_request_body (r=0x81ca03c) at http_protocol.c:2868
> #5  0x805bb4e in default_handler (r=0x81ca03c) at http_core.c:2975

Just wanted to update you all on this.

I spent some time yesterday investigating this issue with gdb.  The assert just
checks to make sure that req_cfg->bb is an empty brigade, which, obviously, it
wasn't.  More specifically, it turns out that the brigade had one bucket in it:
a zero-length heap bucket.  So somewhere there's a way to return (presumably
from ap_getline()) without cleaning out req_cfg->bb first.  The heap bucket
itself is kind of interesting... it's not just a newly-created zero-length
bucket, because the start and end pointers both reference byte 58 of the data,
implying that the bucket originally had 58 bytes in it which were then consumed.
Unsurprisingly, those 58 bytes are "GET /images/apache_pb.gif HTTP/1.0\r\nHost:
www.apache.org\r\n".

I hope to be able to track this down in ap_getline(), but if anyone has ideas
right off, they'd be welcome.  If anyone wants to see a dump of the relevant
data structures from the corefile, let me know and I'll forward a transcript of
my gdb session to you.

--Cliff