You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Mazhar Shaikh <ma...@gmail.com> on 2016/12/14 11:28:36 UTC
kafka_2.11-0.9.0.1 crash with java coredump
Hi All,
I am using kafka_2.11-0.9.0.1 with java version "1.7.0_51".
On random days kafka process stops (crashes) with a java coredump file as
below.
(gdb) bt
#0 0x00007f33059f70d5 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f33059fa83b in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007f33049ae405 in os::abort(bool) () from
/opt/jdk1.7.0_51/jre/lib/amd64/server/libjvm.so
#3 0x00007f3304b2d347 in VMError::report_and_die() () from
/opt/jdk1.7.0_51/jre/lib/amd64/server/libjvm.so
#4 0x00007f3304b2d8de in crash_handler(int, siginfo*, void*) () from
/opt/jdk1.7.0_51/jre/lib/amd64/server/libjvm.so
#5 <signal handler called>
#6 0x00007f33046b92f5 in
G1BlockOffsetArray::forward_to_block_containing_addr_slow(HeapWord*,
HeapWord*, void const*) () from
/opt/jdk1.7.0_51/jre/lib/amd64/server/libjvm.so
#7 0x00007f33049a60f0 in os::print_location(outputStream*, long, bool) ()
from /opt/jdk1.7.0_51/jre/lib/amd64/server/libjvm.so
#8 0x00007f33049b2678 in os::print_register_info(outputStream*, void*) ()
from /opt/jdk1.7.0_51/jre/lib/amd64/server/libjvm.so
#9 0x00007f3304b2b94b in VMError::report(outputStream*) () from
/opt/jdk1.7.0_51/jre/lib/amd64/server/libjvm.so
#10 0x00007f3304b2cf4a in VMError::report_and_die() () from
/opt/jdk1.7.0_51/jre/lib/amd64/server/libjvm.so
#11 0x00007f33049b2d8f in JVM_handle_linux_signal () from
/opt/jdk1.7.0_51/jre/lib/amd64/server/libjvm.so
#12 <signal handler called>
#13 0x00007f32ffbc64bf in ?? ()
#14 0x00000000ca57b708 in ?? ()
#15 0x00007f32fae97928 in ?? ()
#16 0x00000000bf2f05e8 in ?? ()
#17 0xffffffffffffffff in ?? ()
#18 0x00000000c3b27610 in ?? ()
#19 0x00000000bed92898 in ?? ()
#20 0x00000000e269aac8 in ?? ()
#21 0x0000000000000000 in ?? ()
Can anyone suggest a solution to overcome this issue.
Thank you.
Mazhar Shaikh.
Re: kafka_2.11-0.9.0.1 crash with java coredump
Posted by Ismael Juma <is...@juma.me.uk>.
Hi Mazhar,
That looks like a G1 GC bug. Lots of bug in G1 have been fixed since JDK 7
update 51. I highly recommend that you upgrade to a more recent JDK (JDK 8
recommended). Alternatively you could try the CMS GC.
Ismael
On Wed, Dec 14, 2016 at 3:28 AM, Mazhar Shaikh <ma...@gmail.com>
wrote:
> Hi All,
>
> I am using kafka_2.11-0.9.0.1 with java version "1.7.0_51".
>
> On random days kafka process stops (crashes) with a java coredump file as
> below.
>
> (gdb) bt
> #0 0x00007f33059f70d5 in raise () from /lib/x86_64-linux-gnu/libc.so.6
> #1 0x00007f33059fa83b in abort () from /lib/x86_64-linux-gnu/libc.so.6
> #2 0x00007f33049ae405 in os::abort(bool) () from
> /opt/jdk1.7.0_51/jre/lib/amd64/server/libjvm.so
> #3 0x00007f3304b2d347 in VMError::report_and_die() () from
> /opt/jdk1.7.0_51/jre/lib/amd64/server/libjvm.so
> #4 0x00007f3304b2d8de in crash_handler(int, siginfo*, void*) () from
> /opt/jdk1.7.0_51/jre/lib/amd64/server/libjvm.so
> #5 <signal handler called>
> #6 0x00007f33046b92f5 in
> G1BlockOffsetArray::forward_to_block_containing_addr_slow(HeapWord*,
> HeapWord*, void const*) () from
> /opt/jdk1.7.0_51/jre/lib/amd64/server/libjvm.so
> #7 0x00007f33049a60f0 in os::print_location(outputStream*, long, bool) ()
> from /opt/jdk1.7.0_51/jre/lib/amd64/server/libjvm.so
> #8 0x00007f33049b2678 in os::print_register_info(outputStream*, void*) ()
> from /opt/jdk1.7.0_51/jre/lib/amd64/server/libjvm.so
> #9 0x00007f3304b2b94b in VMError::report(outputStream*) () from
> /opt/jdk1.7.0_51/jre/lib/amd64/server/libjvm.so
> #10 0x00007f3304b2cf4a in VMError::report_and_die() () from
> /opt/jdk1.7.0_51/jre/lib/amd64/server/libjvm.so
> #11 0x00007f33049b2d8f in JVM_handle_linux_signal () from
> /opt/jdk1.7.0_51/jre/lib/amd64/server/libjvm.so
> #12 <signal handler called>
> #13 0x00007f32ffbc64bf in ?? ()
> #14 0x00000000ca57b708 in ?? ()
> #15 0x00007f32fae97928 in ?? ()
> #16 0x00000000bf2f05e8 in ?? ()
> #17 0xffffffffffffffff in ?? ()
> #18 0x00000000c3b27610 in ?? ()
> #19 0x00000000bed92898 in ?? ()
> #20 0x00000000e269aac8 in ?? ()
> #21 0x0000000000000000 in ?? ()
>
>
> Can anyone suggest a solution to overcome this issue.
>
> Thank you.
>
> Mazhar Shaikh.
>
Re: kafka_2.11-0.9.0.1 crash with java coredump
Posted by "Schumann,Robert" <ro...@ebay.com>.
I would suggest trying a recent java version first, if I read about this one: http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2014-December/011534.html
Cheers
Robert
--
Robert Schumann | Lead DevOps Engineer | mobile.de GmbH
T: + 49. 30. 8109. 7219
M: +49.151. 5504. 8246
F: +49. 30. 8109. 7131
roschumann@ebay.com
www.mobile.de
Marktplatz 1 | 14532 Europarc Dreilinden | Germany
Geschaeftsfuehrer: Malte Krueger
HRB Nr.: 18517 P, AG Potsdam
Sitz der Gesellschaft: Kleinmachnow
On 14/12/2016, 19:50, "Apurva Mehta" <ap...@confluent.io> wrote:
>I would suggest creating a JIRA and describing in detail what was going on
>in the cluster when this happened, and posting the associated broker /
>state change / controller logs.
>
>Thanks,
>Apurva
>
>On Wed, Dec 14, 2016 at 3:28 AM, Mazhar Shaikh <ma...@gmail.com>
>wrote:
>
>> Hi All,
>>
>> I am using kafka_2.11-0.9.0.1 with java version "1.7.0_51".
>>
>> On random days kafka process stops (crashes) with a java coredump file as
>> below.
>>
>> (gdb) bt
>> #0 0x00007f33059f70d5 in raise () from /lib/x86_64-linux-gnu/libc.so.6
>> #1 0x00007f33059fa83b in abort () from /lib/x86_64-linux-gnu/libc.so.6
>> #2 0x00007f33049ae405 in os::abort(bool) () from
>> /opt/jdk1.7.0_51/jre/lib/amd64/server/libjvm.so
>> #3 0x00007f3304b2d347 in VMError::report_and_die() () from
>> /opt/jdk1.7.0_51/jre/lib/amd64/server/libjvm.so
>> #4 0x00007f3304b2d8de in crash_handler(int, siginfo*, void*) () from
>> /opt/jdk1.7.0_51/jre/lib/amd64/server/libjvm.so
>> #5 <signal handler called>
>> #6 0x00007f33046b92f5 in
>> G1BlockOffsetArray::forward_to_block_containing_addr_slow(HeapWord*,
>> HeapWord*, void const*) () from
>> /opt/jdk1.7.0_51/jre/lib/amd64/server/libjvm.so
>> #7 0x00007f33049a60f0 in os::print_location(outputStream*, long, bool) ()
>> from /opt/jdk1.7.0_51/jre/lib/amd64/server/libjvm.so
>> #8 0x00007f33049b2678 in os::print_register_info(outputStream*, void*) ()
>> from /opt/jdk1.7.0_51/jre/lib/amd64/server/libjvm.so
>> #9 0x00007f3304b2b94b in VMError::report(outputStream*) () from
>> /opt/jdk1.7.0_51/jre/lib/amd64/server/libjvm.so
>> #10 0x00007f3304b2cf4a in VMError::report_and_die() () from
>> /opt/jdk1.7.0_51/jre/lib/amd64/server/libjvm.so
>> #11 0x00007f33049b2d8f in JVM_handle_linux_signal () from
>> /opt/jdk1.7.0_51/jre/lib/amd64/server/libjvm.so
>> #12 <signal handler called>
>> #13 0x00007f32ffbc64bf in ?? ()
>> #14 0x00000000ca57b708 in ?? ()
>> #15 0x00007f32fae97928 in ?? ()
>> #16 0x00000000bf2f05e8 in ?? ()
>> #17 0xffffffffffffffff in ?? ()
>> #18 0x00000000c3b27610 in ?? ()
>> #19 0x00000000bed92898 in ?? ()
>> #20 0x00000000e269aac8 in ?? ()
>> #21 0x0000000000000000 in ?? ()
>>
>>
>> Can anyone suggest a solution to overcome this issue.
>>
>> Thank you.
>>
>> Mazhar Shaikh.
>>
Re: kafka_2.11-0.9.0.1 crash with java coredump
Posted by "Schumann,Robert" <ro...@ebay.com>.
I would suggest trying a recent java version first, if I read about this one: http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2014-December/011534.html
Cheers
Robert
--
Robert Schumann | Lead DevOps Engineer | mobile.de GmbH
T: + 49. 30. 8109. 7219
M: +49.151. 5504. 8246
F: +49. 30. 8109. 7131
roschumann@ebay.com
www.mobile.de
Marktplatz 1 | 14532 Europarc Dreilinden | Germany
Geschaeftsfuehrer: Malte Krueger
HRB Nr.: 18517 P, AG Potsdam
Sitz der Gesellschaft: Kleinmachnow
On 14/12/2016, 19:50, "Apurva Mehta" <ap...@confluent.io> wrote:
>I would suggest creating a JIRA and describing in detail what was going on
>in the cluster when this happened, and posting the associated broker /
>state change / controller logs.
>
>Thanks,
>Apurva
>
>On Wed, Dec 14, 2016 at 3:28 AM, Mazhar Shaikh <ma...@gmail.com>
>wrote:
>
>> Hi All,
>>
>> I am using kafka_2.11-0.9.0.1 with java version "1.7.0_51".
>>
>> On random days kafka process stops (crashes) with a java coredump file as
>> below.
>>
>> (gdb) bt
>> #0 0x00007f33059f70d5 in raise () from /lib/x86_64-linux-gnu/libc.so.6
>> #1 0x00007f33059fa83b in abort () from /lib/x86_64-linux-gnu/libc.so.6
>> #2 0x00007f33049ae405 in os::abort(bool) () from
>> /opt/jdk1.7.0_51/jre/lib/amd64/server/libjvm.so
>> #3 0x00007f3304b2d347 in VMError::report_and_die() () from
>> /opt/jdk1.7.0_51/jre/lib/amd64/server/libjvm.so
>> #4 0x00007f3304b2d8de in crash_handler(int, siginfo*, void*) () from
>> /opt/jdk1.7.0_51/jre/lib/amd64/server/libjvm.so
>> #5 <signal handler called>
>> #6 0x00007f33046b92f5 in
>> G1BlockOffsetArray::forward_to_block_containing_addr_slow(HeapWord*,
>> HeapWord*, void const*) () from
>> /opt/jdk1.7.0_51/jre/lib/amd64/server/libjvm.so
>> #7 0x00007f33049a60f0 in os::print_location(outputStream*, long, bool) ()
>> from /opt/jdk1.7.0_51/jre/lib/amd64/server/libjvm.so
>> #8 0x00007f33049b2678 in os::print_register_info(outputStream*, void*) ()
>> from /opt/jdk1.7.0_51/jre/lib/amd64/server/libjvm.so
>> #9 0x00007f3304b2b94b in VMError::report(outputStream*) () from
>> /opt/jdk1.7.0_51/jre/lib/amd64/server/libjvm.so
>> #10 0x00007f3304b2cf4a in VMError::report_and_die() () from
>> /opt/jdk1.7.0_51/jre/lib/amd64/server/libjvm.so
>> #11 0x00007f33049b2d8f in JVM_handle_linux_signal () from
>> /opt/jdk1.7.0_51/jre/lib/amd64/server/libjvm.so
>> #12 <signal handler called>
>> #13 0x00007f32ffbc64bf in ?? ()
>> #14 0x00000000ca57b708 in ?? ()
>> #15 0x00007f32fae97928 in ?? ()
>> #16 0x00000000bf2f05e8 in ?? ()
>> #17 0xffffffffffffffff in ?? ()
>> #18 0x00000000c3b27610 in ?? ()
>> #19 0x00000000bed92898 in ?? ()
>> #20 0x00000000e269aac8 in ?? ()
>> #21 0x0000000000000000 in ?? ()
>>
>>
>> Can anyone suggest a solution to overcome this issue.
>>
>> Thank you.
>>
>> Mazhar Shaikh.
>>
Re: kafka_2.11-0.9.0.1 crash with java coredump
Posted by Apurva Mehta <ap...@confluent.io>.
I would suggest creating a JIRA and describing in detail what was going on
in the cluster when this happened, and posting the associated broker /
state change / controller logs.
Thanks,
Apurva
On Wed, Dec 14, 2016 at 3:28 AM, Mazhar Shaikh <ma...@gmail.com>
wrote:
> Hi All,
>
> I am using kafka_2.11-0.9.0.1 with java version "1.7.0_51".
>
> On random days kafka process stops (crashes) with a java coredump file as
> below.
>
> (gdb) bt
> #0 0x00007f33059f70d5 in raise () from /lib/x86_64-linux-gnu/libc.so.6
> #1 0x00007f33059fa83b in abort () from /lib/x86_64-linux-gnu/libc.so.6
> #2 0x00007f33049ae405 in os::abort(bool) () from
> /opt/jdk1.7.0_51/jre/lib/amd64/server/libjvm.so
> #3 0x00007f3304b2d347 in VMError::report_and_die() () from
> /opt/jdk1.7.0_51/jre/lib/amd64/server/libjvm.so
> #4 0x00007f3304b2d8de in crash_handler(int, siginfo*, void*) () from
> /opt/jdk1.7.0_51/jre/lib/amd64/server/libjvm.so
> #5 <signal handler called>
> #6 0x00007f33046b92f5 in
> G1BlockOffsetArray::forward_to_block_containing_addr_slow(HeapWord*,
> HeapWord*, void const*) () from
> /opt/jdk1.7.0_51/jre/lib/amd64/server/libjvm.so
> #7 0x00007f33049a60f0 in os::print_location(outputStream*, long, bool) ()
> from /opt/jdk1.7.0_51/jre/lib/amd64/server/libjvm.so
> #8 0x00007f33049b2678 in os::print_register_info(outputStream*, void*) ()
> from /opt/jdk1.7.0_51/jre/lib/amd64/server/libjvm.so
> #9 0x00007f3304b2b94b in VMError::report(outputStream*) () from
> /opt/jdk1.7.0_51/jre/lib/amd64/server/libjvm.so
> #10 0x00007f3304b2cf4a in VMError::report_and_die() () from
> /opt/jdk1.7.0_51/jre/lib/amd64/server/libjvm.so
> #11 0x00007f33049b2d8f in JVM_handle_linux_signal () from
> /opt/jdk1.7.0_51/jre/lib/amd64/server/libjvm.so
> #12 <signal handler called>
> #13 0x00007f32ffbc64bf in ?? ()
> #14 0x00000000ca57b708 in ?? ()
> #15 0x00007f32fae97928 in ?? ()
> #16 0x00000000bf2f05e8 in ?? ()
> #17 0xffffffffffffffff in ?? ()
> #18 0x00000000c3b27610 in ?? ()
> #19 0x00000000bed92898 in ?? ()
> #20 0x00000000e269aac8 in ?? ()
> #21 0x0000000000000000 in ?? ()
>
>
> Can anyone suggest a solution to overcome this issue.
>
> Thank you.
>
> Mazhar Shaikh.
>
Re: kafka_2.11-0.9.0.1 crash with java coredump
Posted by Apurva Mehta <ap...@confluent.io>.
I would suggest creating a JIRA and describing in detail what was going on
in the cluster when this happened, and posting the associated broker /
state change / controller logs.
Thanks,
Apurva
On Wed, Dec 14, 2016 at 3:28 AM, Mazhar Shaikh <ma...@gmail.com>
wrote:
> Hi All,
>
> I am using kafka_2.11-0.9.0.1 with java version "1.7.0_51".
>
> On random days kafka process stops (crashes) with a java coredump file as
> below.
>
> (gdb) bt
> #0 0x00007f33059f70d5 in raise () from /lib/x86_64-linux-gnu/libc.so.6
> #1 0x00007f33059fa83b in abort () from /lib/x86_64-linux-gnu/libc.so.6
> #2 0x00007f33049ae405 in os::abort(bool) () from
> /opt/jdk1.7.0_51/jre/lib/amd64/server/libjvm.so
> #3 0x00007f3304b2d347 in VMError::report_and_die() () from
> /opt/jdk1.7.0_51/jre/lib/amd64/server/libjvm.so
> #4 0x00007f3304b2d8de in crash_handler(int, siginfo*, void*) () from
> /opt/jdk1.7.0_51/jre/lib/amd64/server/libjvm.so
> #5 <signal handler called>
> #6 0x00007f33046b92f5 in
> G1BlockOffsetArray::forward_to_block_containing_addr_slow(HeapWord*,
> HeapWord*, void const*) () from
> /opt/jdk1.7.0_51/jre/lib/amd64/server/libjvm.so
> #7 0x00007f33049a60f0 in os::print_location(outputStream*, long, bool) ()
> from /opt/jdk1.7.0_51/jre/lib/amd64/server/libjvm.so
> #8 0x00007f33049b2678 in os::print_register_info(outputStream*, void*) ()
> from /opt/jdk1.7.0_51/jre/lib/amd64/server/libjvm.so
> #9 0x00007f3304b2b94b in VMError::report(outputStream*) () from
> /opt/jdk1.7.0_51/jre/lib/amd64/server/libjvm.so
> #10 0x00007f3304b2cf4a in VMError::report_and_die() () from
> /opt/jdk1.7.0_51/jre/lib/amd64/server/libjvm.so
> #11 0x00007f33049b2d8f in JVM_handle_linux_signal () from
> /opt/jdk1.7.0_51/jre/lib/amd64/server/libjvm.so
> #12 <signal handler called>
> #13 0x00007f32ffbc64bf in ?? ()
> #14 0x00000000ca57b708 in ?? ()
> #15 0x00007f32fae97928 in ?? ()
> #16 0x00000000bf2f05e8 in ?? ()
> #17 0xffffffffffffffff in ?? ()
> #18 0x00000000c3b27610 in ?? ()
> #19 0x00000000bed92898 in ?? ()
> #20 0x00000000e269aac8 in ?? ()
> #21 0x0000000000000000 in ?? ()
>
>
> Can anyone suggest a solution to overcome this issue.
>
> Thank you.
>
> Mazhar Shaikh.
>