You are viewing a plain text version of this content. The canonical link for it is here.
Posted to bugs@httpd.apache.org by bu...@apache.org on 2007/03/22 18:56:08 UTC
DO NOT REPLY [Bug 41930] New: - Bus error core dump in memcpy - apr_brigade_write
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG�
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
<http://issues.apache.org/bugzilla/show_bug.cgi?id=41930>.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND�
INSERTED IN THE BUG DATABASE.
http://issues.apache.org/bugzilla/show_bug.cgi?id=41930
Summary: Bus error core dump in memcpy - apr_brigade_write
Product: Apache httpd-2
Version: 2.0.59
Platform: Other
OS/Version: Linux
Status: NEW
Severity: normal
Priority: P2
Component: All
AssignedTo: bugs@httpd.apache.org
ReportedBy: vmferr@gmail.com
Hi everyone!
My team is trying to track down what seems to be a random core dump on a large
production site at work. It appears to be a memory corruption problem somewhere.
Checking the core files with gdb I find apr_brigade_write() called with
pointers to invalid memory locations.
Another type of core dump that usually happens to us, is the one that was
previously discussed by Paul Linder (bug n.39738). Although I think they are
not the same problem, the cause may be the same: invalid memory locations.
It was reproduced not only with Apache 2.0.59, but with Apache 2.0.53 as well.
What follow are the datails of the environment where this core dump took place:
[cmc@madarrcc33 apache_2.0.53_debug]$ uname -a
Linux madarrcc33.indra.es 2.6.10-1.771_FC2smp #1 SMP Mon Mar 28 01:10:51 EST
2005 i686 i686 i386 GNU/Linux
[cmc@madarrcc33 bin]$ cat /proc/meminfo | grep Mem
MemTotal: 3895676 kB
MemFree: 499028 kB
[cmc@madarrcc33 bin]$ ./httpd -l
Compiled in modules:
core.c
mod_access.c
mod_auth.c
mod_include.c
mod_log_config.c
mod_env.c
mod_setenvif.c
worker.c
http_core.c
mod_mime.c
mod_status.c
mod_autoindex.c
mod_asis.c
mod_cgid.c
mod_negotiation.c
mod_dir.c
mod_imap.c
mod_actions.c
mod_userdir.c
mod_alias.c
mod_so.c
And this is a sample backtrace of the controversial thread:
[...]
Thread 9 (process 10826):
#0 0x001077a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
No symbol table info available.
#1 0x00147276 in kill () from /lib/tls/libc.so.6
No symbol table info available.
#2 0x08089052 in sig_coredump (sig=10760) at mpm_common.c:956
No locals.
#3 <signal handler called>
No symbol table info available.
#4 0x00187ff5 in memcpy () from /lib/tls/libc.so.6
No symbol table info available.
#5 0x0085fe66 in apr_brigade_write (b=0x9300298, flush=0, ctx=0x0,
str=0x8f15d000 <Address 0x8f15d000 out of bounds>,
nbyte=1585) at apr_brigade.c:417
e = (apr_bucket *) 0x9303485
remaining = 2400571392
buf =
0x9303485 "/cmc/entorno/intranetwl8/docroot/INT/componentes/INT_Pla081_CoePerson
as/0,0,61000_0_0&glo_entorno=dev&glo_portal=INT&idSeccion1=57129&idSeccion2=5713
0&idSeccion3=57131&idSeccion4=57132,00.html"
#6 0x08091021 in core_output_filter (f=0x92febc0, b=0x9320ad8) at core.c:4033
d = (apr_bucket *) 0x9302168
rv = 154149224
more = (apr_bucket_brigade *) 0x0
c = (conn_rec *) 0x92fe828
net = (core_net_rec *) 0x92feb98
ctx = (core_output_filter_ctx_t *) 0x92fecd0
eblock = APR_NONBLOCK_READ
input_pool = (apr_pool_t *) 0x9326fb8
#7 0x080899f7 in ap_pass_brigade (next=0x11db, bb=0x318) at util_filter.c:512
e = (apr_bucket *) 0x9300c78
#8 0x08068730 in chunk_filter (f=0x93087e8, b=0x9328e70) at http_core.c:218
hdr_len = 154143864
bytes = 2
eos = (apr_bucket *) 0x93008b8
flush = (apr_bucket *) 0x0
chunk_hdr = "2\r\n\000�\2100\t\000\000\000\000\220S�\000hG1\t"
c = (conn_rec *) 0x92fe828
more = (apr_bucket_brigade *) 0x0
e = (apr_bucket *) 0x9300c28
rv = 154143864
#9 0x080899f7 in ap_pass_brigade (next=0x11db, bb=0x318) at util_filter.c:512
e = (apr_bucket *) 0x9300c78
#10 0x0808baa6 in ap_content_length_filter (f=0x9327c58, b=0x9328e70) at
protocol.c:1232
r = (request_rec *) 0x9326ff0
ctx = (struct content_length_ctx *) 0x9328ec8
e = (apr_bucket *) 0x93008b8
eos = 1
eblock = APR_NONBLOCK_READ
#11 0x080899f7 in ap_pass_brigade (next=0x11db, bb=0x318) at util_filter.c:512
e = (apr_bucket *) 0x9300c78
#12 0x08064d95 in send_parsed_content (f=0x93288a8, bb=0x9328a68) at
mod_include.c:3388
data = 0x0
len = 2
release = 151270848
newb = (apr_bucket *) 0x9328a6c
store = (char **) 0xaded68b8
store_len = (apr_size_t *) 0x9302760
index = 154307176
ctx = (ssi_ctx_t *) 0x92fec00
r = (request_rec *) 0x9326ff0
b = (apr_bucket *) 0x93008b8
pass_bb = (apr_bucket_brigade *) 0x9328e70
rv = 0
magic = 0x93288a8 "�\023�\b"
#13 0x080899f7 in ap_pass_brigade (next=0x11db, bb=0x318) at util_filter.c:512
e = (apr_bucket *) 0x9300c78
#14 0x08090351 in default_handler (r=0x9326ff0) at core.c:3610
req_cfg = (core_request_config *) 0x9300c78
c = (conn_rec *) 0x92fe828
bb = (apr_bucket_brigade *) 0x9328a68
e = (apr_bucket *) 0x9300708
d = (core_dir_config *) 0x9328158
errstatus = 154307180
fd = (apr_file_t *) 0x9328970
status = 154143864
bld_content_md5 = 154142472
#15 0x0807eb2e in ap_run_handler (r=0x9326ff0) at config.c:152
pHook = (ap_LINK_handler_t *) 0x9300c78
n = 8
rv = 154143864
#16 0x0807f042 in ap_invoke_handler (r=0x9326ff0) at config.c:364
new_handler = 0x11db <Address 0x11db out of bounds>
p2 = 0x9300c78 "\234\0020\t\234\0020\t �\206"
handler = 0x9027d38 "text/html"
result = 154300400
old_handler = 0x0
#17 0x0806c7f3 in ap_process_request (r=0x9326ff0) at http_request.c:249
access_status = 4571
#18 0x080688d1 in ap_process_http_connection (c=0x92fe828) at http_core.c:251
r = (request_rec *) 0x9326ff0
csd_set = 0
csd = (apr_socket_t *) 0x0
#19 0x08087caa in ap_run_process_connection (c=0x92fe828) at connection.c:43
pHook = (ap_LINK_process_connection_t *) 0x9300c78
n = 0
rv = 154143864
#20 0x0807bd09 in process_socket (p=0x92fe700, sock=0x92fe738,
my_child_num=4571, my_thread_num=154143864,
bucket_alloc=0x9300708) at worker.c:521
current_conn = (conn_rec *) 0x92fe828
conn_id = 154134568
csd = 16
sbh = (ap_sb_handle_t *) 0x92fe820
#21 0x0807c352 in worker_thread (thd=0x904b5e0, dummy=0x9300c78) at worker.c:835
process_slot = 1
thread_slot = 25
csd = (apr_socket_t *) 0x92fe738
bucket_alloc = (apr_bucket_alloc_t *) 0x9300c78
last_ptrans = (apr_pool_t *) 0x0
ptrans = (apr_pool_t *) 0x92fe700
rv = 154143864
is_idle = 1
#22 0x00cc0148 in dummy_worker (opaque=0x9300c78) at thread.c:105
No locals.
#23 0x0035a98c in start_thread () from /lib/tls/libpthread.so.0
No symbol table info available.
#24 0x001db7da in clone () from /lib/tls/libc.so.6
No symbol table info available.
[...]
--
Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 41930] - Bus error core dump in memcpy - apr_brigade_write
Posted by bu...@apache.org.
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG�
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
<http://issues.apache.org/bugzilla/show_bug.cgi?id=41930>.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND�
INSERTED IN THE BUG DATABASE.
http://issues.apache.org/bugzilla/show_bug.cgi?id=41930
------- Additional Comments From jorton@redhat.com 2007-03-23 05:35 -------
SIGBUS (rather than SIGSEGV) on Linux usually only happens when an mmaped file
is e.g. truncated out from under the server. Do you have files in this site
which may be modified in-place whilst being served?
I'd first try "EnableMMap off" to see if this goes away.
--
Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 41930] - Bus error core dump in memcpy - apr_brigade_write
Posted by bu...@apache.org.
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG�
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
<http://issues.apache.org/bugzilla/show_bug.cgi?id=41930>.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND�
INSERTED IN THE BUG DATABASE.
http://issues.apache.org/bugzilla/show_bug.cgi?id=41930
vmferr@gmail.com changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEEDINFO |RESOLVED
Resolution| |LATER
------- Additional Comments From vmferr@gmail.com 2007-07-13 06:03 -------
Thanks to everyone!
I'm sorry I have said nothing for three months.
"Do you have files in this site which may be modified in-place whilst being
served?" --> Yes, we have. Most of the request to Apache ask for SSI pages -
that have to be completed before being sent to the client-. What's more,
sometimes we save them to disk -by way of a cache memory-.
The directiva "EnableMMAP" was in fact enabled. Disabling it, this behaviour
seems to have disappeared.
We keep having other troubles but I think they are not related to this thread.
Let me not to close this thread for the time being, until we are sure this
behaviour is not appearing again.
--
Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 41930] - Bus error core dump in memcpy - apr_brigade_write
Posted by bu...@apache.org.
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG�
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
<http://issues.apache.org/bugzilla/show_bug.cgi?id=41930>.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND�
INSERTED IN THE BUG DATABASE.
http://issues.apache.org/bugzilla/show_bug.cgi?id=41930
rpluem@apache.org changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |NEEDINFO
------- Additional Comments From rpluem@apache.org 2007-03-23 15:13 -------
(In reply to comment #2)
> (gdb) frame 6
> #6 0x08091c7d in core_output_filter (f=0x8c5b7a0, b=0xb23c0a48) at core.c:4103
> 4103 apr_brigade_write(temp_brig, NULL,
> NULL, str, n);
>
> (gdb) dump_brigade b
> dump of brigade 0xb23c0a48
> | type (address) | length | data addr | contents | rc
> --------------------------------------------------------------------------------
> 0 | MMAP (0x08c5ffe0) | 1037 | 0x08c604e0 | [Cannot access memory at
> address 0xaaf1e000
I agree with Joe here: Looks like a problem with an MMAPed file. So please try
"EnableMMap off" as Joe suggested.
Where do you have the files stored that are delivered by httpd (local disk / NFS
/ something else)?
Have you changed these files while the crash happened?
--
Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 41930] - Bus error core dump in memcpy - apr_brigade_write
Posted by bu...@apache.org.
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG�
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
<http://issues.apache.org/bugzilla/show_bug.cgi?id=41930>.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND�
INSERTED IN THE BUG DATABASE.
http://issues.apache.org/bugzilla/show_bug.cgi?id=41930
vmferr@gmail.com changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEEDINFO |NEW
------- Additional Comments From vmferr@gmail.com 2007-03-23 05:19 -------
Using the .gdbinit file delivered with the sources of Apache 2.0.59, what
follows is the information you requested:
(gdb) frame 5
#5 0x00cc8b12 in apr_brigade_write (b=0x8c5ccd8, flush=0, ctx=0x0,
str=0xaaf1e000 <Address 0xaaf1e000 out of bounds>, nbyte=1037) at
apr_brigade.c:417
417 memcpy(buf, str, nbyte);
(gdb) dump_brigade b
dump of brigade 0x8c5ccd8
| type (address) | length | data addr | contents | rc
--------------------------------------------------------------------------------
0 | HEAP (0x08c60030) | 108 | 0x08c601c0 | [4d~~<!-- VERSION ...] | 1
end of brigade
(gdb) dump_bucket e
bucket=Cannot access memory at address 0xa200a0a
(gdb) frame 6
#6 0x08091c7d in core_output_filter (f=0x8c5b7a0, b=0xb23c0a48) at core.c:4103
4103 apr_brigade_write(temp_brig, NULL,
NULL, str, n);
(gdb) dump_brigade b
dump of brigade 0xb23c0a48
| type (address) | length | data addr | contents | rc
--------------------------------------------------------------------------------
0 | MMAP (0x08c5ffe0) | 1037 | 0x08c604e0 | [Cannot access memory at
address 0xaaf1e000
(gdb) dump_bucket temp
No symbol "temp" in current context.
(gdb) dump_bucket e
No symbol "e" in current context.
In advance, thank you for your help!
--
Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 41930] - Bus error core dump in memcpy - apr_brigade_write
Posted by bu...@apache.org.
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG�
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
<http://issues.apache.org/bugzilla/show_bug.cgi?id=41930>.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND�
INSERTED IN THE BUG DATABASE.
http://issues.apache.org/bugzilla/show_bug.cgi?id=41930
rpluem@apache.org changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |NEEDINFO
------- Additional Comments From rpluem@apache.org 2007-03-22 13:35 -------
(In reply to comment #0)
Please ensure that you have the gdb macros loaded that are defined in .gdbinit
that is delivered with the sources of httpd (see
http://httpd.apache.org/dev/debugging.html)
> #4 0x00187ff5 in memcpy () from /lib/tls/libc.so.6
> No symbol table info available.
> #5 0x0085fe66 in apr_brigade_write (b=0x9300298, flush=0, ctx=0x0,
> str=0x8f15d000 <Address 0x8f15d000 out of bounds>,
> nbyte=1585) at apr_brigade.c:417
> e = (apr_bucket *) 0x9303485
> remaining = 2400571392
That looks far too large for me. This could be an indicator for an overflow here.
Please issue:
frame 5
dump_brigade b
dump_bucket e
> buf =
> 0x9303485 "/cmc/entorno/intranetwl8/docroot/INT/componentes/INT_Pla081_CoePerson
> as/0,0,61000_0_0&glo_entorno=dev&glo_portal=INT&idSeccion1=57129&idSeccion2=5713
> 0&idSeccion3=57131&idSeccion4=57132,00.html"
> #6 0x08091021 in core_output_filter (f=0x92febc0, b=0x9320ad8) at core.c:4033
> d = (apr_bucket *) 0x9302168
> rv = 154149224
> more = (apr_bucket_brigade *) 0x0
> c = (conn_rec *) 0x92fe828
> net = (core_net_rec *) 0x92feb98
> ctx = (core_output_filter_ctx_t *) 0x92fecd0
> eblock = APR_NONBLOCK_READ
> input_pool = (apr_pool_t *) 0x9326fb8
Please issue
frame 6
dump_brigade b
dump_bucket temp
dump_bucket e
--
Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org