You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Yang <te...@gmail.com> on 2011/09/08 03:21:02 UTC
SIGSEGV during compaction?
I started compaction using nodetool,
then always reproducibly, I get a SEGV in a code that I added to the
Cassandra code, which simply calls get_slice().
have you seen SEGV associated with compaction? anyone could suggest a
route on how to debug this?
I filed a bug on sun website, right now the only possible approach I
can try is to use another JDK
Thanks
Yang
Re: SIGSEGV during compaction?
Posted by Yang <te...@gmail.com>.
unfortunately tried java7, same
On Wed, Sep 7, 2011 at 6:22 PM, Yang <te...@gmail.com> wrote:
> some info in the debug file that JVM exported:
>
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> # SIGSEGV (0xb) at pc=0x00002aaaab37cbfa, pid=7236, tid=1179806016
> #
> # JRE version: 6.0_27-b07
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (20.2-b06 mixed mode
> linux-amd64 compressed oops)
> # Problematic frame:
> # J com.cgm.whisky.filter.WithinLimitIpFrequencyCap.isValid(Lorg/apache/avro/specific/SpecificRecord;)Lcom/cgm/whisky/EventsFilter$ValidityCode;
> #
> # If you would like to submit a bug report, please visit:
> # http://java.sun.com/webapps/bugreport/crash.jsp
> #
>
> --------------- T H R E A D ---------------
>
> Current thread (0x00002aaab80e2800): JavaThread "pool-3-thread-8"
> [_thread_in_Java, id=7669,
> stack(0x0000000046426000,0x0000000046527000)]
>
> siginfo:si_signo=SIGSEGV: si_errno=0, si_code=1 (SEGV_MAPERR),
> si_addr=0x00002aaabc000000
>
> Registers:
> RAX=0x00000007914355e8, RBX=0x000000000000058a,
> RCX=0x0000000791461b38, RDX=0x0000000000000000
> RSP=0x00000000465259f0, RBP=0x00000000f222b894,
> RSI=0x0000000791433f20, RDI=0x00002aaaab37ca60
> R8 =0x00000000d0931f61, R9 =0x00000000f2286ab2,
> R10=0x0000000000000000, R11=0x00002aaabc000000
> R12=0x0000000000000000, R13=0x00000000465259f0,
> R14=0x0000000000000002, R15=0x00002aaab80e2800
> RIP=0x00002aaaab37cbfa, EFLAGS=0x0000000000010202,
> CSGSFS=0x0100000000000033, ERR=0x0000000000000004
> TRAPNO=0x000000000000000e
>
> Top of Stack: (sp=0x00000000465259f0)
> 0x00000000465259f0: 000000068a828dc8 0000000791433f20
> 0x0000000046525a00: 000000079145ee60 000005890000058a
>
>
> On Wed, Sep 7, 2011 at 6:21 PM, Yang <te...@gmail.com> wrote:
>> I started compaction using nodetool,
>> then always reproducibly, I get a SEGV in a code that I added to the
>> Cassandra code, which simply calls get_slice().
>>
>> have you seen SEGV associated with compaction? anyone could suggest a
>> route on how to debug this?
>>
>> I filed a bug on sun website, right now the only possible approach I
>> can try is to use another JDK
>>
>>
>> Thanks
>> Yang
>>
>
Re: SIGSEGV during compaction?
Posted by Yang <te...@gmail.com>.
this is trunk.
sorry I did more tests, the -XX:-UseCompressedOops suggested by
Jonathan actually DOES solve the problem. my previous tries possibly
used the wrong scripts.
Thanks guys
Yang
On Thu, Sep 8, 2011 at 12:07 AM, Sylvain Lebresne <sy...@datastax.com> wrote:
> Are you using current trunk ? Or 0.8 ?
>
> Because if on trunk, a SIGSEGV could also be due to CASSANDRA-2521,
> if we happen to force the unmapping of a file but tries to access it afterwards
> (which shouldn't happen but ...).
>
> --
> Sylvain
>
> On Thu, Sep 8, 2011 at 7:36 AM, Yang <te...@gmail.com> wrote:
>> hmmmm, all other things remaining the same, I put jna.jar into classpath,
>> now it successfully completed a compaction without problems
>>
>> On Wed, Sep 7, 2011 at 10:06 PM, Yang <te...@gmail.com> wrote:
>>> thanks Jonathan.
>>>
>>> I tried openJdk too, same , filed bug to both Oracle and openJdk
>>>
>>>
>>> tried -XX:-UseCompressedOops , same SEGV
>>>
>>> Oracle bug site asks "does it appear with -server and -Xint", I tried
>>> these options, so far no SEGV yet, maybe slower, but haven't measured
>>> exactly
>>>
>>>
>>>
>>> On Wed, Sep 7, 2011 at 8:56 PM, Jonathan Ellis <jb...@gmail.com> wrote:
>>>> You should report a bug to Oracle.
>>>>
>>>> In the meantime you could try turning off compressed oops -- that's
>>>> been a source of a lot of GC bugs in the past.
>>>>
>>>> On Wed, Sep 7, 2011 at 8:22 PM, Yang <te...@gmail.com> wrote:
>>>>> some info in the debug file that JVM exported:
>>>>>
>>>>> #
>>>>> # A fatal error has been detected by the Java Runtime Environment:
>>>>> #
>>>>> # SIGSEGV (0xb) at pc=0x00002aaaab37cbfa, pid=7236, tid=1179806016
>>>>> #
>>>>> # JRE version: 6.0_27-b07
>>>>> # Java VM: Java HotSpot(TM) 64-Bit Server VM (20.2-b06 mixed mode
>>>>> linux-amd64 compressed oops)
>>>>> # Problematic frame:
>>>>> # J com.cgm.whisky.filter.WithinLimitIpFrequencyCap.isValid(Lorg/apache/avro/specific/SpecificRecord;)Lcom/cgm/whisky/EventsFilter$ValidityCode;
>>>>> #
>>>>> # If you would like to submit a bug report, please visit:
>>>>> # http://java.sun.com/webapps/bugreport/crash.jsp
>>>>> #
>>>>>
>>>>> --------------- T H R E A D ---------------
>>>>>
>>>>> Current thread (0x00002aaab80e2800): JavaThread "pool-3-thread-8"
>>>>> [_thread_in_Java, id=7669,
>>>>> stack(0x0000000046426000,0x0000000046527000)]
>>>>>
>>>>> siginfo:si_signo=SIGSEGV: si_errno=0, si_code=1 (SEGV_MAPERR),
>>>>> si_addr=0x00002aaabc000000
>>>>>
>>>>> Registers:
>>>>> RAX=0x00000007914355e8, RBX=0x000000000000058a,
>>>>> RCX=0x0000000791461b38, RDX=0x0000000000000000
>>>>> RSP=0x00000000465259f0, RBP=0x00000000f222b894,
>>>>> RSI=0x0000000791433f20, RDI=0x00002aaaab37ca60
>>>>> R8 =0x00000000d0931f61, R9 =0x00000000f2286ab2,
>>>>> R10=0x0000000000000000, R11=0x00002aaabc000000
>>>>> R12=0x0000000000000000, R13=0x00000000465259f0,
>>>>> R14=0x0000000000000002, R15=0x00002aaab80e2800
>>>>> RIP=0x00002aaaab37cbfa, EFLAGS=0x0000000000010202,
>>>>> CSGSFS=0x0100000000000033, ERR=0x0000000000000004
>>>>> TRAPNO=0x000000000000000e
>>>>>
>>>>> Top of Stack: (sp=0x00000000465259f0)
>>>>> 0x00000000465259f0: 000000068a828dc8 0000000791433f20
>>>>> 0x0000000046525a00: 000000079145ee60 000005890000058a
>>>>>
>>>>>
>>>>> On Wed, Sep 7, 2011 at 6:21 PM, Yang <te...@gmail.com> wrote:
>>>>>> I started compaction using nodetool,
>>>>>> then always reproducibly, I get a SEGV in a code that I added to the
>>>>>> Cassandra code, which simply calls get_slice().
>>>>>>
>>>>>> have you seen SEGV associated with compaction? anyone could suggest a
>>>>>> route on how to debug this?
>>>>>>
>>>>>> I filed a bug on sun website, right now the only possible approach I
>>>>>> can try is to use another JDK
>>>>>>
>>>>>>
>>>>>> Thanks
>>>>>> Yang
>>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Jonathan Ellis
>>>> Project Chair, Apache Cassandra
>>>> co-founder of DataStax, the source for professional Cassandra support
>>>> http://www.datastax.com
>>>>
>>>
>>
>
Re: SIGSEGV during compaction?
Posted by Yang <te...@gmail.com>.
thanks, it does seem possible that this is due to accessing unmapped buffer:
the following code will always generate SEGV.
import java.lang.reflect.Method;
import java.nio.*;
import java.nio.channels.FileChannel;
import java.io.*;
public class TestRandomAccess {
public static void main(String args[]) throws Exception {
new File("/tmp/blah").delete();
FileOutputStream os = new FileOutputStream("/tmp/blah");
for(int i=0;i<10000;i++) os.write( (byte)(i&0xff));
os.close();
RandomAccessFile f = new RandomAccessFile("/tmp/blah", "rw");
MappedByteBuffer buf =
f.getChannel().map(FileChannel.MapMode.READ_ONLY,0, 10000);
byte x = buf.get(1000);
System.out.println(x);
Method cleanerMethod =
Class.forName("sun.nio.ch.DirectBuffer").getMethod("cleaner");
Object cleaner = cleanerMethod.invoke(buf);
cleaner.getClass().getMethod("clean").invoke(cleaner);
buf.get(1000);
}
}
On Sun, Sep 11, 2011 at 3:35 PM, Jonathan Ellis <jb...@gmail.com> wrote:
> The problem would be because we're trying to access a mappedbytebuffer
> that has been unmapped, not because of OS deletion semantics.
>
> On Sun, Sep 11, 2011 at 2:12 PM, Yang <te...@gmail.com> wrote:
>> unfortunately it appeared again, I confirmed it appeared with the
>> -XX:-UseCompressedOops .
>>
>> so, if it's due to accessing unmapped file, the problem should
>> disappear if I use direct random access file, I'll try that.
>>
>> also if a process A (not only java) has a file mmapped, and another
>> process deletes the file, I thought Unix should keep the file open for
>> A, no ?? if this is true, then we shouldn't get a SEGV due to forcing
>> unmmap ????
>>
>>
>> Thanks
>> Yang
>> On Thu, Sep 8, 2011 at 12:07 AM, Sylvain Lebresne <sy...@datastax.com> wrote:
>>> Are you using current trunk ? Or 0.8 ?
>>>
>>> Because if on trunk, a SIGSEGV could also be due to CASSANDRA-2521,
>>> if we happen to force the unmapping of a file but tries to access it afterwards
>>> (which shouldn't happen but ...).
>>>
>>> --
>>> Sylvain
>>>
>>> On Thu, Sep 8, 2011 at 7:36 AM, Yang <te...@gmail.com> wrote:
>>>> hmmmm, all other things remaining the same, I put jna.jar into classpath,
>>>> now it successfully completed a compaction without problems
>>>>
>>>> On Wed, Sep 7, 2011 at 10:06 PM, Yang <te...@gmail.com> wrote:
>>>>> thanks Jonathan.
>>>>>
>>>>> I tried openJdk too, same , filed bug to both Oracle and openJdk
>>>>>
>>>>>
>>>>> tried -XX:-UseCompressedOops , same SEGV
>>>>>
>>>>> Oracle bug site asks "does it appear with -server and -Xint", I tried
>>>>> these options, so far no SEGV yet, maybe slower, but haven't measured
>>>>> exactly
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Sep 7, 2011 at 8:56 PM, Jonathan Ellis <jb...@gmail.com> wrote:
>>>>>> You should report a bug to Oracle.
>>>>>>
>>>>>> In the meantime you could try turning off compressed oops -- that's
>>>>>> been a source of a lot of GC bugs in the past.
>>>>>>
>>>>>> On Wed, Sep 7, 2011 at 8:22 PM, Yang <te...@gmail.com> wrote:
>>>>>>> some info in the debug file that JVM exported:
>>>>>>>
>>>>>>> #
>>>>>>> # A fatal error has been detected by the Java Runtime Environment:
>>>>>>> #
>>>>>>> # SIGSEGV (0xb) at pc=0x00002aaaab37cbfa, pid=7236, tid=1179806016
>>>>>>> #
>>>>>>> # JRE version: 6.0_27-b07
>>>>>>> # Java VM: Java HotSpot(TM) 64-Bit Server VM (20.2-b06 mixed mode
>>>>>>> linux-amd64 compressed oops)
>>>>>>> # Problematic frame:
>>>>>>> # J com.cgm.whisky.filter.WithinLimitIpFrequencyCap.isValid(Lorg/apache/avro/specific/SpecificRecord;)Lcom/cgm/whisky/EventsFilter$ValidityCode;
>>>>>>> #
>>>>>>> # If you would like to submit a bug report, please visit:
>>>>>>> # http://java.sun.com/webapps/bugreport/crash.jsp
>>>>>>> #
>>>>>>>
>>>>>>> --------------- T H R E A D ---------------
>>>>>>>
>>>>>>> Current thread (0x00002aaab80e2800): JavaThread "pool-3-thread-8"
>>>>>>> [_thread_in_Java, id=7669,
>>>>>>> stack(0x0000000046426000,0x0000000046527000)]
>>>>>>>
>>>>>>> siginfo:si_signo=SIGSEGV: si_errno=0, si_code=1 (SEGV_MAPERR),
>>>>>>> si_addr=0x00002aaabc000000
>>>>>>>
>>>>>>> Registers:
>>>>>>> RAX=0x00000007914355e8, RBX=0x000000000000058a,
>>>>>>> RCX=0x0000000791461b38, RDX=0x0000000000000000
>>>>>>> RSP=0x00000000465259f0, RBP=0x00000000f222b894,
>>>>>>> RSI=0x0000000791433f20, RDI=0x00002aaaab37ca60
>>>>>>> R8 =0x00000000d0931f61, R9 =0x00000000f2286ab2,
>>>>>>> R10=0x0000000000000000, R11=0x00002aaabc000000
>>>>>>> R12=0x0000000000000000, R13=0x00000000465259f0,
>>>>>>> R14=0x0000000000000002, R15=0x00002aaab80e2800
>>>>>>> RIP=0x00002aaaab37cbfa, EFLAGS=0x0000000000010202,
>>>>>>> CSGSFS=0x0100000000000033, ERR=0x0000000000000004
>>>>>>> TRAPNO=0x000000000000000e
>>>>>>>
>>>>>>> Top of Stack: (sp=0x00000000465259f0)
>>>>>>> 0x00000000465259f0: 000000068a828dc8 0000000791433f20
>>>>>>> 0x0000000046525a00: 000000079145ee60 000005890000058a
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Sep 7, 2011 at 6:21 PM, Yang <te...@gmail.com> wrote:
>>>>>>>> I started compaction using nodetool,
>>>>>>>> then always reproducibly, I get a SEGV in a code that I added to the
>>>>>>>> Cassandra code, which simply calls get_slice().
>>>>>>>>
>>>>>>>> have you seen SEGV associated with compaction? anyone could suggest a
>>>>>>>> route on how to debug this?
>>>>>>>>
>>>>>>>> I filed a bug on sun website, right now the only possible approach I
>>>>>>>> can try is to use another JDK
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>> Yang
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Jonathan Ellis
>>>>>> Project Chair, Apache Cassandra
>>>>>> co-founder of DataStax, the source for professional Cassandra support
>>>>>> http://www.datastax.com
>>>>>>
>>>>>
>>>>
>>>
>>
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com
>
Re: SIGSEGV during compaction?
Posted by Jonathan Ellis <jb...@gmail.com>.
The problem would be because we're trying to access a mappedbytebuffer
that has been unmapped, not because of OS deletion semantics.
On Sun, Sep 11, 2011 at 2:12 PM, Yang <te...@gmail.com> wrote:
> unfortunately it appeared again, I confirmed it appeared with the
> -XX:-UseCompressedOops .
>
> so, if it's due to accessing unmapped file, the problem should
> disappear if I use direct random access file, I'll try that.
>
> also if a process A (not only java) has a file mmapped, and another
> process deletes the file, I thought Unix should keep the file open for
> A, no ?? if this is true, then we shouldn't get a SEGV due to forcing
> unmmap ????
>
>
> Thanks
> Yang
> On Thu, Sep 8, 2011 at 12:07 AM, Sylvain Lebresne <sy...@datastax.com> wrote:
>> Are you using current trunk ? Or 0.8 ?
>>
>> Because if on trunk, a SIGSEGV could also be due to CASSANDRA-2521,
>> if we happen to force the unmapping of a file but tries to access it afterwards
>> (which shouldn't happen but ...).
>>
>> --
>> Sylvain
>>
>> On Thu, Sep 8, 2011 at 7:36 AM, Yang <te...@gmail.com> wrote:
>>> hmmmm, all other things remaining the same, I put jna.jar into classpath,
>>> now it successfully completed a compaction without problems
>>>
>>> On Wed, Sep 7, 2011 at 10:06 PM, Yang <te...@gmail.com> wrote:
>>>> thanks Jonathan.
>>>>
>>>> I tried openJdk too, same , filed bug to both Oracle and openJdk
>>>>
>>>>
>>>> tried -XX:-UseCompressedOops , same SEGV
>>>>
>>>> Oracle bug site asks "does it appear with -server and -Xint", I tried
>>>> these options, so far no SEGV yet, maybe slower, but haven't measured
>>>> exactly
>>>>
>>>>
>>>>
>>>> On Wed, Sep 7, 2011 at 8:56 PM, Jonathan Ellis <jb...@gmail.com> wrote:
>>>>> You should report a bug to Oracle.
>>>>>
>>>>> In the meantime you could try turning off compressed oops -- that's
>>>>> been a source of a lot of GC bugs in the past.
>>>>>
>>>>> On Wed, Sep 7, 2011 at 8:22 PM, Yang <te...@gmail.com> wrote:
>>>>>> some info in the debug file that JVM exported:
>>>>>>
>>>>>> #
>>>>>> # A fatal error has been detected by the Java Runtime Environment:
>>>>>> #
>>>>>> # SIGSEGV (0xb) at pc=0x00002aaaab37cbfa, pid=7236, tid=1179806016
>>>>>> #
>>>>>> # JRE version: 6.0_27-b07
>>>>>> # Java VM: Java HotSpot(TM) 64-Bit Server VM (20.2-b06 mixed mode
>>>>>> linux-amd64 compressed oops)
>>>>>> # Problematic frame:
>>>>>> # J com.cgm.whisky.filter.WithinLimitIpFrequencyCap.isValid(Lorg/apache/avro/specific/SpecificRecord;)Lcom/cgm/whisky/EventsFilter$ValidityCode;
>>>>>> #
>>>>>> # If you would like to submit a bug report, please visit:
>>>>>> # http://java.sun.com/webapps/bugreport/crash.jsp
>>>>>> #
>>>>>>
>>>>>> --------------- T H R E A D ---------------
>>>>>>
>>>>>> Current thread (0x00002aaab80e2800): JavaThread "pool-3-thread-8"
>>>>>> [_thread_in_Java, id=7669,
>>>>>> stack(0x0000000046426000,0x0000000046527000)]
>>>>>>
>>>>>> siginfo:si_signo=SIGSEGV: si_errno=0, si_code=1 (SEGV_MAPERR),
>>>>>> si_addr=0x00002aaabc000000
>>>>>>
>>>>>> Registers:
>>>>>> RAX=0x00000007914355e8, RBX=0x000000000000058a,
>>>>>> RCX=0x0000000791461b38, RDX=0x0000000000000000
>>>>>> RSP=0x00000000465259f0, RBP=0x00000000f222b894,
>>>>>> RSI=0x0000000791433f20, RDI=0x00002aaaab37ca60
>>>>>> R8 =0x00000000d0931f61, R9 =0x00000000f2286ab2,
>>>>>> R10=0x0000000000000000, R11=0x00002aaabc000000
>>>>>> R12=0x0000000000000000, R13=0x00000000465259f0,
>>>>>> R14=0x0000000000000002, R15=0x00002aaab80e2800
>>>>>> RIP=0x00002aaaab37cbfa, EFLAGS=0x0000000000010202,
>>>>>> CSGSFS=0x0100000000000033, ERR=0x0000000000000004
>>>>>> TRAPNO=0x000000000000000e
>>>>>>
>>>>>> Top of Stack: (sp=0x00000000465259f0)
>>>>>> 0x00000000465259f0: 000000068a828dc8 0000000791433f20
>>>>>> 0x0000000046525a00: 000000079145ee60 000005890000058a
>>>>>>
>>>>>>
>>>>>> On Wed, Sep 7, 2011 at 6:21 PM, Yang <te...@gmail.com> wrote:
>>>>>>> I started compaction using nodetool,
>>>>>>> then always reproducibly, I get a SEGV in a code that I added to the
>>>>>>> Cassandra code, which simply calls get_slice().
>>>>>>>
>>>>>>> have you seen SEGV associated with compaction? anyone could suggest a
>>>>>>> route on how to debug this?
>>>>>>>
>>>>>>> I filed a bug on sun website, right now the only possible approach I
>>>>>>> can try is to use another JDK
>>>>>>>
>>>>>>>
>>>>>>> Thanks
>>>>>>> Yang
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Jonathan Ellis
>>>>> Project Chair, Apache Cassandra
>>>>> co-founder of DataStax, the source for professional Cassandra support
>>>>> http://www.datastax.com
>>>>>
>>>>
>>>
>>
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com
Re: SIGSEGV during compaction?
Posted by Yang <te...@gmail.com>.
unfortunately it appeared again, I confirmed it appeared with the
-XX:-UseCompressedOops .
so, if it's due to accessing unmapped file, the problem should
disappear if I use direct random access file, I'll try that.
also if a process A (not only java) has a file mmapped, and another
process deletes the file, I thought Unix should keep the file open for
A, no ?? if this is true, then we shouldn't get a SEGV due to forcing
unmmap ????
Thanks
Yang
On Thu, Sep 8, 2011 at 12:07 AM, Sylvain Lebresne <sy...@datastax.com> wrote:
> Are you using current trunk ? Or 0.8 ?
>
> Because if on trunk, a SIGSEGV could also be due to CASSANDRA-2521,
> if we happen to force the unmapping of a file but tries to access it afterwards
> (which shouldn't happen but ...).
>
> --
> Sylvain
>
> On Thu, Sep 8, 2011 at 7:36 AM, Yang <te...@gmail.com> wrote:
>> hmmmm, all other things remaining the same, I put jna.jar into classpath,
>> now it successfully completed a compaction without problems
>>
>> On Wed, Sep 7, 2011 at 10:06 PM, Yang <te...@gmail.com> wrote:
>>> thanks Jonathan.
>>>
>>> I tried openJdk too, same , filed bug to both Oracle and openJdk
>>>
>>>
>>> tried -XX:-UseCompressedOops , same SEGV
>>>
>>> Oracle bug site asks "does it appear with -server and -Xint", I tried
>>> these options, so far no SEGV yet, maybe slower, but haven't measured
>>> exactly
>>>
>>>
>>>
>>> On Wed, Sep 7, 2011 at 8:56 PM, Jonathan Ellis <jb...@gmail.com> wrote:
>>>> You should report a bug to Oracle.
>>>>
>>>> In the meantime you could try turning off compressed oops -- that's
>>>> been a source of a lot of GC bugs in the past.
>>>>
>>>> On Wed, Sep 7, 2011 at 8:22 PM, Yang <te...@gmail.com> wrote:
>>>>> some info in the debug file that JVM exported:
>>>>>
>>>>> #
>>>>> # A fatal error has been detected by the Java Runtime Environment:
>>>>> #
>>>>> # SIGSEGV (0xb) at pc=0x00002aaaab37cbfa, pid=7236, tid=1179806016
>>>>> #
>>>>> # JRE version: 6.0_27-b07
>>>>> # Java VM: Java HotSpot(TM) 64-Bit Server VM (20.2-b06 mixed mode
>>>>> linux-amd64 compressed oops)
>>>>> # Problematic frame:
>>>>> # J com.cgm.whisky.filter.WithinLimitIpFrequencyCap.isValid(Lorg/apache/avro/specific/SpecificRecord;)Lcom/cgm/whisky/EventsFilter$ValidityCode;
>>>>> #
>>>>> # If you would like to submit a bug report, please visit:
>>>>> # http://java.sun.com/webapps/bugreport/crash.jsp
>>>>> #
>>>>>
>>>>> --------------- T H R E A D ---------------
>>>>>
>>>>> Current thread (0x00002aaab80e2800): JavaThread "pool-3-thread-8"
>>>>> [_thread_in_Java, id=7669,
>>>>> stack(0x0000000046426000,0x0000000046527000)]
>>>>>
>>>>> siginfo:si_signo=SIGSEGV: si_errno=0, si_code=1 (SEGV_MAPERR),
>>>>> si_addr=0x00002aaabc000000
>>>>>
>>>>> Registers:
>>>>> RAX=0x00000007914355e8, RBX=0x000000000000058a,
>>>>> RCX=0x0000000791461b38, RDX=0x0000000000000000
>>>>> RSP=0x00000000465259f0, RBP=0x00000000f222b894,
>>>>> RSI=0x0000000791433f20, RDI=0x00002aaaab37ca60
>>>>> R8 =0x00000000d0931f61, R9 =0x00000000f2286ab2,
>>>>> R10=0x0000000000000000, R11=0x00002aaabc000000
>>>>> R12=0x0000000000000000, R13=0x00000000465259f0,
>>>>> R14=0x0000000000000002, R15=0x00002aaab80e2800
>>>>> RIP=0x00002aaaab37cbfa, EFLAGS=0x0000000000010202,
>>>>> CSGSFS=0x0100000000000033, ERR=0x0000000000000004
>>>>> TRAPNO=0x000000000000000e
>>>>>
>>>>> Top of Stack: (sp=0x00000000465259f0)
>>>>> 0x00000000465259f0: 000000068a828dc8 0000000791433f20
>>>>> 0x0000000046525a00: 000000079145ee60 000005890000058a
>>>>>
>>>>>
>>>>> On Wed, Sep 7, 2011 at 6:21 PM, Yang <te...@gmail.com> wrote:
>>>>>> I started compaction using nodetool,
>>>>>> then always reproducibly, I get a SEGV in a code that I added to the
>>>>>> Cassandra code, which simply calls get_slice().
>>>>>>
>>>>>> have you seen SEGV associated with compaction? anyone could suggest a
>>>>>> route on how to debug this?
>>>>>>
>>>>>> I filed a bug on sun website, right now the only possible approach I
>>>>>> can try is to use another JDK
>>>>>>
>>>>>>
>>>>>> Thanks
>>>>>> Yang
>>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Jonathan Ellis
>>>> Project Chair, Apache Cassandra
>>>> co-founder of DataStax, the source for professional Cassandra support
>>>> http://www.datastax.com
>>>>
>>>
>>
>
Re: SIGSEGV during compaction?
Posted by Sylvain Lebresne <sy...@datastax.com>.
Are you using current trunk ? Or 0.8 ?
Because if on trunk, a SIGSEGV could also be due to CASSANDRA-2521,
if we happen to force the unmapping of a file but tries to access it afterwards
(which shouldn't happen but ...).
--
Sylvain
On Thu, Sep 8, 2011 at 7:36 AM, Yang <te...@gmail.com> wrote:
> hmmmm, all other things remaining the same, I put jna.jar into classpath,
> now it successfully completed a compaction without problems
>
> On Wed, Sep 7, 2011 at 10:06 PM, Yang <te...@gmail.com> wrote:
>> thanks Jonathan.
>>
>> I tried openJdk too, same , filed bug to both Oracle and openJdk
>>
>>
>> tried -XX:-UseCompressedOops , same SEGV
>>
>> Oracle bug site asks "does it appear with -server and -Xint", I tried
>> these options, so far no SEGV yet, maybe slower, but haven't measured
>> exactly
>>
>>
>>
>> On Wed, Sep 7, 2011 at 8:56 PM, Jonathan Ellis <jb...@gmail.com> wrote:
>>> You should report a bug to Oracle.
>>>
>>> In the meantime you could try turning off compressed oops -- that's
>>> been a source of a lot of GC bugs in the past.
>>>
>>> On Wed, Sep 7, 2011 at 8:22 PM, Yang <te...@gmail.com> wrote:
>>>> some info in the debug file that JVM exported:
>>>>
>>>> #
>>>> # A fatal error has been detected by the Java Runtime Environment:
>>>> #
>>>> # SIGSEGV (0xb) at pc=0x00002aaaab37cbfa, pid=7236, tid=1179806016
>>>> #
>>>> # JRE version: 6.0_27-b07
>>>> # Java VM: Java HotSpot(TM) 64-Bit Server VM (20.2-b06 mixed mode
>>>> linux-amd64 compressed oops)
>>>> # Problematic frame:
>>>> # J com.cgm.whisky.filter.WithinLimitIpFrequencyCap.isValid(Lorg/apache/avro/specific/SpecificRecord;)Lcom/cgm/whisky/EventsFilter$ValidityCode;
>>>> #
>>>> # If you would like to submit a bug report, please visit:
>>>> # http://java.sun.com/webapps/bugreport/crash.jsp
>>>> #
>>>>
>>>> --------------- T H R E A D ---------------
>>>>
>>>> Current thread (0x00002aaab80e2800): JavaThread "pool-3-thread-8"
>>>> [_thread_in_Java, id=7669,
>>>> stack(0x0000000046426000,0x0000000046527000)]
>>>>
>>>> siginfo:si_signo=SIGSEGV: si_errno=0, si_code=1 (SEGV_MAPERR),
>>>> si_addr=0x00002aaabc000000
>>>>
>>>> Registers:
>>>> RAX=0x00000007914355e8, RBX=0x000000000000058a,
>>>> RCX=0x0000000791461b38, RDX=0x0000000000000000
>>>> RSP=0x00000000465259f0, RBP=0x00000000f222b894,
>>>> RSI=0x0000000791433f20, RDI=0x00002aaaab37ca60
>>>> R8 =0x00000000d0931f61, R9 =0x00000000f2286ab2,
>>>> R10=0x0000000000000000, R11=0x00002aaabc000000
>>>> R12=0x0000000000000000, R13=0x00000000465259f0,
>>>> R14=0x0000000000000002, R15=0x00002aaab80e2800
>>>> RIP=0x00002aaaab37cbfa, EFLAGS=0x0000000000010202,
>>>> CSGSFS=0x0100000000000033, ERR=0x0000000000000004
>>>> TRAPNO=0x000000000000000e
>>>>
>>>> Top of Stack: (sp=0x00000000465259f0)
>>>> 0x00000000465259f0: 000000068a828dc8 0000000791433f20
>>>> 0x0000000046525a00: 000000079145ee60 000005890000058a
>>>>
>>>>
>>>> On Wed, Sep 7, 2011 at 6:21 PM, Yang <te...@gmail.com> wrote:
>>>>> I started compaction using nodetool,
>>>>> then always reproducibly, I get a SEGV in a code that I added to the
>>>>> Cassandra code, which simply calls get_slice().
>>>>>
>>>>> have you seen SEGV associated with compaction? anyone could suggest a
>>>>> route on how to debug this?
>>>>>
>>>>> I filed a bug on sun website, right now the only possible approach I
>>>>> can try is to use another JDK
>>>>>
>>>>>
>>>>> Thanks
>>>>> Yang
>>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Jonathan Ellis
>>> Project Chair, Apache Cassandra
>>> co-founder of DataStax, the source for professional Cassandra support
>>> http://www.datastax.com
>>>
>>
>
Re: SIGSEGV during compaction?
Posted by Yang <te...@gmail.com>.
hmmmm, all other things remaining the same, I put jna.jar into classpath,
now it successfully completed a compaction without problems
On Wed, Sep 7, 2011 at 10:06 PM, Yang <te...@gmail.com> wrote:
> thanks Jonathan.
>
> I tried openJdk too, same , filed bug to both Oracle and openJdk
>
>
> tried -XX:-UseCompressedOops , same SEGV
>
> Oracle bug site asks "does it appear with -server and -Xint", I tried
> these options, so far no SEGV yet, maybe slower, but haven't measured
> exactly
>
>
>
> On Wed, Sep 7, 2011 at 8:56 PM, Jonathan Ellis <jb...@gmail.com> wrote:
>> You should report a bug to Oracle.
>>
>> In the meantime you could try turning off compressed oops -- that's
>> been a source of a lot of GC bugs in the past.
>>
>> On Wed, Sep 7, 2011 at 8:22 PM, Yang <te...@gmail.com> wrote:
>>> some info in the debug file that JVM exported:
>>>
>>> #
>>> # A fatal error has been detected by the Java Runtime Environment:
>>> #
>>> # SIGSEGV (0xb) at pc=0x00002aaaab37cbfa, pid=7236, tid=1179806016
>>> #
>>> # JRE version: 6.0_27-b07
>>> # Java VM: Java HotSpot(TM) 64-Bit Server VM (20.2-b06 mixed mode
>>> linux-amd64 compressed oops)
>>> # Problematic frame:
>>> # J com.cgm.whisky.filter.WithinLimitIpFrequencyCap.isValid(Lorg/apache/avro/specific/SpecificRecord;)Lcom/cgm/whisky/EventsFilter$ValidityCode;
>>> #
>>> # If you would like to submit a bug report, please visit:
>>> # http://java.sun.com/webapps/bugreport/crash.jsp
>>> #
>>>
>>> --------------- T H R E A D ---------------
>>>
>>> Current thread (0x00002aaab80e2800): JavaThread "pool-3-thread-8"
>>> [_thread_in_Java, id=7669,
>>> stack(0x0000000046426000,0x0000000046527000)]
>>>
>>> siginfo:si_signo=SIGSEGV: si_errno=0, si_code=1 (SEGV_MAPERR),
>>> si_addr=0x00002aaabc000000
>>>
>>> Registers:
>>> RAX=0x00000007914355e8, RBX=0x000000000000058a,
>>> RCX=0x0000000791461b38, RDX=0x0000000000000000
>>> RSP=0x00000000465259f0, RBP=0x00000000f222b894,
>>> RSI=0x0000000791433f20, RDI=0x00002aaaab37ca60
>>> R8 =0x00000000d0931f61, R9 =0x00000000f2286ab2,
>>> R10=0x0000000000000000, R11=0x00002aaabc000000
>>> R12=0x0000000000000000, R13=0x00000000465259f0,
>>> R14=0x0000000000000002, R15=0x00002aaab80e2800
>>> RIP=0x00002aaaab37cbfa, EFLAGS=0x0000000000010202,
>>> CSGSFS=0x0100000000000033, ERR=0x0000000000000004
>>> TRAPNO=0x000000000000000e
>>>
>>> Top of Stack: (sp=0x00000000465259f0)
>>> 0x00000000465259f0: 000000068a828dc8 0000000791433f20
>>> 0x0000000046525a00: 000000079145ee60 000005890000058a
>>>
>>>
>>> On Wed, Sep 7, 2011 at 6:21 PM, Yang <te...@gmail.com> wrote:
>>>> I started compaction using nodetool,
>>>> then always reproducibly, I get a SEGV in a code that I added to the
>>>> Cassandra code, which simply calls get_slice().
>>>>
>>>> have you seen SEGV associated with compaction? anyone could suggest a
>>>> route on how to debug this?
>>>>
>>>> I filed a bug on sun website, right now the only possible approach I
>>>> can try is to use another JDK
>>>>
>>>>
>>>> Thanks
>>>> Yang
>>>>
>>>
>>
>>
>>
>> --
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder of DataStax, the source for professional Cassandra support
>> http://www.datastax.com
>>
>
Re: SIGSEGV during compaction?
Posted by Yang <te...@gmail.com>.
thanks Jonathan.
I tried openJdk too, same , filed bug to both Oracle and openJdk
tried -XX:-UseCompressedOops , same SEGV
Oracle bug site asks "does it appear with -server and -Xint", I tried
these options, so far no SEGV yet, maybe slower, but haven't measured
exactly
On Wed, Sep 7, 2011 at 8:56 PM, Jonathan Ellis <jb...@gmail.com> wrote:
> You should report a bug to Oracle.
>
> In the meantime you could try turning off compressed oops -- that's
> been a source of a lot of GC bugs in the past.
>
> On Wed, Sep 7, 2011 at 8:22 PM, Yang <te...@gmail.com> wrote:
>> some info in the debug file that JVM exported:
>>
>> #
>> # A fatal error has been detected by the Java Runtime Environment:
>> #
>> # SIGSEGV (0xb) at pc=0x00002aaaab37cbfa, pid=7236, tid=1179806016
>> #
>> # JRE version: 6.0_27-b07
>> # Java VM: Java HotSpot(TM) 64-Bit Server VM (20.2-b06 mixed mode
>> linux-amd64 compressed oops)
>> # Problematic frame:
>> # J com.cgm.whisky.filter.WithinLimitIpFrequencyCap.isValid(Lorg/apache/avro/specific/SpecificRecord;)Lcom/cgm/whisky/EventsFilter$ValidityCode;
>> #
>> # If you would like to submit a bug report, please visit:
>> # http://java.sun.com/webapps/bugreport/crash.jsp
>> #
>>
>> --------------- T H R E A D ---------------
>>
>> Current thread (0x00002aaab80e2800): JavaThread "pool-3-thread-8"
>> [_thread_in_Java, id=7669,
>> stack(0x0000000046426000,0x0000000046527000)]
>>
>> siginfo:si_signo=SIGSEGV: si_errno=0, si_code=1 (SEGV_MAPERR),
>> si_addr=0x00002aaabc000000
>>
>> Registers:
>> RAX=0x00000007914355e8, RBX=0x000000000000058a,
>> RCX=0x0000000791461b38, RDX=0x0000000000000000
>> RSP=0x00000000465259f0, RBP=0x00000000f222b894,
>> RSI=0x0000000791433f20, RDI=0x00002aaaab37ca60
>> R8 =0x00000000d0931f61, R9 =0x00000000f2286ab2,
>> R10=0x0000000000000000, R11=0x00002aaabc000000
>> R12=0x0000000000000000, R13=0x00000000465259f0,
>> R14=0x0000000000000002, R15=0x00002aaab80e2800
>> RIP=0x00002aaaab37cbfa, EFLAGS=0x0000000000010202,
>> CSGSFS=0x0100000000000033, ERR=0x0000000000000004
>> TRAPNO=0x000000000000000e
>>
>> Top of Stack: (sp=0x00000000465259f0)
>> 0x00000000465259f0: 000000068a828dc8 0000000791433f20
>> 0x0000000046525a00: 000000079145ee60 000005890000058a
>>
>>
>> On Wed, Sep 7, 2011 at 6:21 PM, Yang <te...@gmail.com> wrote:
>>> I started compaction using nodetool,
>>> then always reproducibly, I get a SEGV in a code that I added to the
>>> Cassandra code, which simply calls get_slice().
>>>
>>> have you seen SEGV associated with compaction? anyone could suggest a
>>> route on how to debug this?
>>>
>>> I filed a bug on sun website, right now the only possible approach I
>>> can try is to use another JDK
>>>
>>>
>>> Thanks
>>> Yang
>>>
>>
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com
>
Re: SIGSEGV during compaction?
Posted by Jonathan Ellis <jb...@gmail.com>.
You should report a bug to Oracle.
In the meantime you could try turning off compressed oops -- that's
been a source of a lot of GC bugs in the past.
On Wed, Sep 7, 2011 at 8:22 PM, Yang <te...@gmail.com> wrote:
> some info in the debug file that JVM exported:
>
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> # SIGSEGV (0xb) at pc=0x00002aaaab37cbfa, pid=7236, tid=1179806016
> #
> # JRE version: 6.0_27-b07
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (20.2-b06 mixed mode
> linux-amd64 compressed oops)
> # Problematic frame:
> # J com.cgm.whisky.filter.WithinLimitIpFrequencyCap.isValid(Lorg/apache/avro/specific/SpecificRecord;)Lcom/cgm/whisky/EventsFilter$ValidityCode;
> #
> # If you would like to submit a bug report, please visit:
> # http://java.sun.com/webapps/bugreport/crash.jsp
> #
>
> --------------- T H R E A D ---------------
>
> Current thread (0x00002aaab80e2800): JavaThread "pool-3-thread-8"
> [_thread_in_Java, id=7669,
> stack(0x0000000046426000,0x0000000046527000)]
>
> siginfo:si_signo=SIGSEGV: si_errno=0, si_code=1 (SEGV_MAPERR),
> si_addr=0x00002aaabc000000
>
> Registers:
> RAX=0x00000007914355e8, RBX=0x000000000000058a,
> RCX=0x0000000791461b38, RDX=0x0000000000000000
> RSP=0x00000000465259f0, RBP=0x00000000f222b894,
> RSI=0x0000000791433f20, RDI=0x00002aaaab37ca60
> R8 =0x00000000d0931f61, R9 =0x00000000f2286ab2,
> R10=0x0000000000000000, R11=0x00002aaabc000000
> R12=0x0000000000000000, R13=0x00000000465259f0,
> R14=0x0000000000000002, R15=0x00002aaab80e2800
> RIP=0x00002aaaab37cbfa, EFLAGS=0x0000000000010202,
> CSGSFS=0x0100000000000033, ERR=0x0000000000000004
> TRAPNO=0x000000000000000e
>
> Top of Stack: (sp=0x00000000465259f0)
> 0x00000000465259f0: 000000068a828dc8 0000000791433f20
> 0x0000000046525a00: 000000079145ee60 000005890000058a
>
>
> On Wed, Sep 7, 2011 at 6:21 PM, Yang <te...@gmail.com> wrote:
>> I started compaction using nodetool,
>> then always reproducibly, I get a SEGV in a code that I added to the
>> Cassandra code, which simply calls get_slice().
>>
>> have you seen SEGV associated with compaction? anyone could suggest a
>> route on how to debug this?
>>
>> I filed a bug on sun website, right now the only possible approach I
>> can try is to use another JDK
>>
>>
>> Thanks
>> Yang
>>
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com
Re: SIGSEGV during compaction?
Posted by Yang <te...@gmail.com>.
some info in the debug file that JVM exported:
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00002aaaab37cbfa, pid=7236, tid=1179806016
#
# JRE version: 6.0_27-b07
# Java VM: Java HotSpot(TM) 64-Bit Server VM (20.2-b06 mixed mode
linux-amd64 compressed oops)
# Problematic frame:
# J com.cgm.whisky.filter.WithinLimitIpFrequencyCap.isValid(Lorg/apache/avro/specific/SpecificRecord;)Lcom/cgm/whisky/EventsFilter$ValidityCode;
#
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
#
--------------- T H R E A D ---------------
Current thread (0x00002aaab80e2800): JavaThread "pool-3-thread-8"
[_thread_in_Java, id=7669,
stack(0x0000000046426000,0x0000000046527000)]
siginfo:si_signo=SIGSEGV: si_errno=0, si_code=1 (SEGV_MAPERR),
si_addr=0x00002aaabc000000
Registers:
RAX=0x00000007914355e8, RBX=0x000000000000058a,
RCX=0x0000000791461b38, RDX=0x0000000000000000
RSP=0x00000000465259f0, RBP=0x00000000f222b894,
RSI=0x0000000791433f20, RDI=0x00002aaaab37ca60
R8 =0x00000000d0931f61, R9 =0x00000000f2286ab2,
R10=0x0000000000000000, R11=0x00002aaabc000000
R12=0x0000000000000000, R13=0x00000000465259f0,
R14=0x0000000000000002, R15=0x00002aaab80e2800
RIP=0x00002aaaab37cbfa, EFLAGS=0x0000000000010202,
CSGSFS=0x0100000000000033, ERR=0x0000000000000004
TRAPNO=0x000000000000000e
Top of Stack: (sp=0x00000000465259f0)
0x00000000465259f0: 000000068a828dc8 0000000791433f20
0x0000000046525a00: 000000079145ee60 000005890000058a
On Wed, Sep 7, 2011 at 6:21 PM, Yang <te...@gmail.com> wrote:
> I started compaction using nodetool,
> then always reproducibly, I get a SEGV in a code that I added to the
> Cassandra code, which simply calls get_slice().
>
> have you seen SEGV associated with compaction? anyone could suggest a
> route on how to debug this?
>
> I filed a bug on sun website, right now the only possible approach I
> can try is to use another JDK
>
>
> Thanks
> Yang
>