You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Maciej Miklas <ma...@googlemail.com> on 2012/03/22 09:54:44 UTC
Cassandra - crash with “free() invalid pointer”
Hi *,
My Cassandra installation runs on flowing system:
- Linux with Kernel 2.6.32.22
- jna-3.3.0
- Java 1.7.0-b147
Sometimes we are getting following error:
*** glibc detected *** /var/opt/java1.7/bin/java: free(): invalid
pointer: 0x00007f66088a6000 ***
======= Backtrace: =========
/lib/libc.so.6[0x7f661d7099a8]
/lib/libc.so.6(cfree+0x76)[0x7f661d70bab6]
/lib64/ld-linux-x86-64.so.2(_dl_deallocate_tls+0x59)[0x7f661e02f349]
/lib/libpthread.so.0[0x7f661de09237]
/lib/libpthread.so.0[0x7f661de0931a]
/lib/libpthread.so.0[0x7f661de0a0bd]
/lib/libc.so.6(clone+0x6d)[0x7f661d76564d]
======= Memory map: ========
00400000-00401000 r-xp 00000000 68:07 537448203
/var/opt/jdk1.7.0/bin/java
00600000-00601000 rw-p 00000000 68:07 537448203
/var/opt/jdk1.7.0/bin/java
01bae000-01fd0000 rw-p 00000000 00:00 0 [heap]
01fd0000-15798000 rw-p 00000000 00:00 0 [heap]
40002000-40005000 ---p 00000000 00:00 0
40005000-40023000 rw-p 00000000 00:00 0
40030000-40033000 ---p 00000000 00:00 0
40033000-40051000 rw-p 00000000 00:00 0
Does anyone have similar problems? or maybe some hints?
Thanks,
Maciej
Re: Cassandra - crash with “free() invalid pointer”
Posted by Maciej Miklas <ma...@googlemail.com>.
thanks - I will try it
On Thu, Mar 22, 2012 at 10:15 PM, Ben Coverston
<be...@datastax.com>wrote:
> Use a version of the Java 6 runtime, Cassandra hasn't been tested at all
> with the Java 7 runtime.
>
>
> On Thu, Mar 22, 2012 at 1:27 PM, Benoit Perroud <be...@noisette.ch>wrote:
>
>> Sounds like a race condition in the off heap caching while calling
>> Unsafe.free().
>>
>> Do you use cache ? What is your use case when you encounter this error
>> ? Are you able to reproduce it ?
>>
>>
>> 2012/3/22 Maciej Miklas <ma...@googlemail.com>:
>> > Hi *,
>> >
>> > My Cassandra installation runs on flowing system:
>> >
>> > Linux with Kernel 2.6.32.22
>> > jna-3.3.0
>> > Java 1.7.0-b147
>> >
>> > Sometimes we are getting following error:
>> >
>> > *** glibc detected *** /var/opt/java1.7/bin/java: free(): invalid
>> pointer:
>> > 0x00007f66088a6000 ***
>> > ======= Backtrace: =========
>> > /lib/libc.so.6[0x7f661d7099a8]
>> > /lib/libc.so.6(cfree+0x76)[0x7f661d70bab6]
>> > /lib64/ld-linux-x86-64.so.2(_dl_deallocate_tls+0x59)[0x7f661e02f349]
>> > /lib/libpthread.so.0[0x7f661de09237]
>> > /lib/libpthread.so.0[0x7f661de0931a]
>> > /lib/libpthread.so.0[0x7f661de0a0bd]
>> > /lib/libc.so.6(clone+0x6d)[0x7f661d76564d]
>> > ======= Memory map: ========
>> > 00400000-00401000 r-xp 00000000 68:07 537448203
>> > /var/opt/jdk1.7.0/bin/java
>> > 00600000-00601000 rw-p 00000000 68:07 537448203
>> > /var/opt/jdk1.7.0/bin/java
>> > 01bae000-01fd0000 rw-p 00000000 00:00 0
>> > [heap]
>> > 01fd0000-15798000 rw-p 00000000 00:00 0
>> > [heap]
>> > 40002000-40005000 ---p 00000000 00:00 0
>> > 40005000-40023000 rw-p 00000000 00:00 0
>> > 40030000-40033000 ---p 00000000 00:00 0
>> > 40033000-40051000 rw-p 00000000 00:00 0
>> >
>> > Does anyone have similar problems? or maybe some hints?
>> >
>> > Thanks,
>> > Maciej
>>
>>
>>
>> --
>> sent from my Nokia 3210
>>
>
>
>
> --
> Ben Coverston
> DataStax -- The Apache Cassandra Company
>
>
Re: Cassandra - crash with “free() invalid pointer”
Posted by Ben Coverston <be...@datastax.com>.
Use a version of the Java 6 runtime, Cassandra hasn't been tested at all
with the Java 7 runtime.
On Thu, Mar 22, 2012 at 1:27 PM, Benoit Perroud <be...@noisette.ch> wrote:
> Sounds like a race condition in the off heap caching while calling
> Unsafe.free().
>
> Do you use cache ? What is your use case when you encounter this error
> ? Are you able to reproduce it ?
>
>
> 2012/3/22 Maciej Miklas <ma...@googlemail.com>:
> > Hi *,
> >
> > My Cassandra installation runs on flowing system:
> >
> > Linux with Kernel 2.6.32.22
> > jna-3.3.0
> > Java 1.7.0-b147
> >
> > Sometimes we are getting following error:
> >
> > *** glibc detected *** /var/opt/java1.7/bin/java: free(): invalid
> pointer:
> > 0x00007f66088a6000 ***
> > ======= Backtrace: =========
> > /lib/libc.so.6[0x7f661d7099a8]
> > /lib/libc.so.6(cfree+0x76)[0x7f661d70bab6]
> > /lib64/ld-linux-x86-64.so.2(_dl_deallocate_tls+0x59)[0x7f661e02f349]
> > /lib/libpthread.so.0[0x7f661de09237]
> > /lib/libpthread.so.0[0x7f661de0931a]
> > /lib/libpthread.so.0[0x7f661de0a0bd]
> > /lib/libc.so.6(clone+0x6d)[0x7f661d76564d]
> > ======= Memory map: ========
> > 00400000-00401000 r-xp 00000000 68:07 537448203
> > /var/opt/jdk1.7.0/bin/java
> > 00600000-00601000 rw-p 00000000 68:07 537448203
> > /var/opt/jdk1.7.0/bin/java
> > 01bae000-01fd0000 rw-p 00000000 00:00 0
> > [heap]
> > 01fd0000-15798000 rw-p 00000000 00:00 0
> > [heap]
> > 40002000-40005000 ---p 00000000 00:00 0
> > 40005000-40023000 rw-p 00000000 00:00 0
> > 40030000-40033000 ---p 00000000 00:00 0
> > 40033000-40051000 rw-p 00000000 00:00 0
> >
> > Does anyone have similar problems? or maybe some hints?
> >
> > Thanks,
> > Maciej
>
>
>
> --
> sent from my Nokia 3210
>
--
Ben Coverston
DataStax -- The Apache Cassandra Company
Re: Cassandra - crash with “free() invalid pointer”
Posted by Vijay <vi...@gmail.com>.
Can you send us the stack trace which you can find in the hs_err_pid*.log?
is the system memory all used up (free)? any errors in the logs just before
the crash?
Regards,
</VJ>
On Mon, Mar 26, 2012 at 12:35 AM, Maciej Miklas
<ma...@googlemail.com>wrote:
> I have row cache - it's about 20GB big in this case.
> The problem can be reproduced with our load test - we are using 20 reader
> threads on single Cassandra node.
>
> I will retest it with Java 6 - still it looks to me like JNA problem and
> JDK in this case should not matter, but we will see.
>
>
> On Thu, Mar 22, 2012 at 8:27 PM, Benoit Perroud <be...@noisette.ch>wrote:
>
>> Sounds like a race condition in the off heap caching while calling
>> Unsafe.free().
>>
>> Do you use cache ? What is your use case when you encounter this error
>> ? Are you able to reproduce it ?
>>
>>
>> 2012/3/22 Maciej Miklas <ma...@googlemail.com>:
>> > Hi *,
>> >
>> > My Cassandra installation runs on flowing system:
>> >
>> > Linux with Kernel 2.6.32.22
>> > jna-3.3.0
>> > Java 1.7.0-b147
>> >
>> > Sometimes we are getting following error:
>> >
>> > *** glibc detected *** /var/opt/java1.7/bin/java: free(): invalid
>> pointer:
>> > 0x00007f66088a6000 ***
>> > ======= Backtrace: =========
>> > /lib/libc.so.6[0x7f661d7099a8]
>> > /lib/libc.so.6(cfree+0x76)[0x7f661d70bab6]
>> > /lib64/ld-linux-x86-64.so.2(_dl_deallocate_tls+0x59)[0x7f661e02f349]
>> > /lib/libpthread.so.0[0x7f661de09237]
>> > /lib/libpthread.so.0[0x7f661de0931a]
>> > /lib/libpthread.so.0[0x7f661de0a0bd]
>> > /lib/libc.so.6(clone+0x6d)[0x7f661d76564d]
>> > ======= Memory map: ========
>> > 00400000-00401000 r-xp 00000000 68:07 537448203
>> > /var/opt/jdk1.7.0/bin/java
>> > 00600000-00601000 rw-p 00000000 68:07 537448203
>> > /var/opt/jdk1.7.0/bin/java
>> > 01bae000-01fd0000 rw-p 00000000 00:00 0
>> > [heap]
>> > 01fd0000-15798000 rw-p 00000000 00:00 0
>> > [heap]
>> > 40002000-40005000 ---p 00000000 00:00 0
>> > 40005000-40023000 rw-p 00000000 00:00 0
>> > 40030000-40033000 ---p 00000000 00:00 0
>> > 40033000-40051000 rw-p 00000000 00:00 0
>> >
>> > Does anyone have similar problems? or maybe some hints?
>> >
>> > Thanks,
>> > Maciej
>>
>>
>>
>> --
>> sent from my Nokia 3210
>>
>
>
Re: Cassandra - crash with “free() invalid pointer”
Posted by Maciej Miklas <ma...@googlemail.com>.
I have row cache - it's about 20GB big in this case.
The problem can be reproduced with our load test - we are using 20 reader
threads on single Cassandra node.
I will retest it with Java 6 - still it looks to me like JNA problem and
JDK in this case should not matter, but we will see.
On Thu, Mar 22, 2012 at 8:27 PM, Benoit Perroud <be...@noisette.ch> wrote:
> Sounds like a race condition in the off heap caching while calling
> Unsafe.free().
>
> Do you use cache ? What is your use case when you encounter this error
> ? Are you able to reproduce it ?
>
>
> 2012/3/22 Maciej Miklas <ma...@googlemail.com>:
> > Hi *,
> >
> > My Cassandra installation runs on flowing system:
> >
> > Linux with Kernel 2.6.32.22
> > jna-3.3.0
> > Java 1.7.0-b147
> >
> > Sometimes we are getting following error:
> >
> > *** glibc detected *** /var/opt/java1.7/bin/java: free(): invalid
> pointer:
> > 0x00007f66088a6000 ***
> > ======= Backtrace: =========
> > /lib/libc.so.6[0x7f661d7099a8]
> > /lib/libc.so.6(cfree+0x76)[0x7f661d70bab6]
> > /lib64/ld-linux-x86-64.so.2(_dl_deallocate_tls+0x59)[0x7f661e02f349]
> > /lib/libpthread.so.0[0x7f661de09237]
> > /lib/libpthread.so.0[0x7f661de0931a]
> > /lib/libpthread.so.0[0x7f661de0a0bd]
> > /lib/libc.so.6(clone+0x6d)[0x7f661d76564d]
> > ======= Memory map: ========
> > 00400000-00401000 r-xp 00000000 68:07 537448203
> > /var/opt/jdk1.7.0/bin/java
> > 00600000-00601000 rw-p 00000000 68:07 537448203
> > /var/opt/jdk1.7.0/bin/java
> > 01bae000-01fd0000 rw-p 00000000 00:00 0
> > [heap]
> > 01fd0000-15798000 rw-p 00000000 00:00 0
> > [heap]
> > 40002000-40005000 ---p 00000000 00:00 0
> > 40005000-40023000 rw-p 00000000 00:00 0
> > 40030000-40033000 ---p 00000000 00:00 0
> > 40033000-40051000 rw-p 00000000 00:00 0
> >
> > Does anyone have similar problems? or maybe some hints?
> >
> > Thanks,
> > Maciej
>
>
>
> --
> sent from my Nokia 3210
>
Re: Cassandra - crash with “free() invalid pointer”
Posted by Benoit Perroud <be...@noisette.ch>.
Sounds like a race condition in the off heap caching while calling
Unsafe.free().
Do you use cache ? What is your use case when you encounter this error
? Are you able to reproduce it ?
2012/3/22 Maciej Miklas <ma...@googlemail.com>:
> Hi *,
>
> My Cassandra installation runs on flowing system:
>
> Linux with Kernel 2.6.32.22
> jna-3.3.0
> Java 1.7.0-b147
>
> Sometimes we are getting following error:
>
> *** glibc detected *** /var/opt/java1.7/bin/java: free(): invalid pointer:
> 0x00007f66088a6000 ***
> ======= Backtrace: =========
> /lib/libc.so.6[0x7f661d7099a8]
> /lib/libc.so.6(cfree+0x76)[0x7f661d70bab6]
> /lib64/ld-linux-x86-64.so.2(_dl_deallocate_tls+0x59)[0x7f661e02f349]
> /lib/libpthread.so.0[0x7f661de09237]
> /lib/libpthread.so.0[0x7f661de0931a]
> /lib/libpthread.so.0[0x7f661de0a0bd]
> /lib/libc.so.6(clone+0x6d)[0x7f661d76564d]
> ======= Memory map: ========
> 00400000-00401000 r-xp 00000000 68:07 537448203
> /var/opt/jdk1.7.0/bin/java
> 00600000-00601000 rw-p 00000000 68:07 537448203
> /var/opt/jdk1.7.0/bin/java
> 01bae000-01fd0000 rw-p 00000000 00:00 0
> [heap]
> 01fd0000-15798000 rw-p 00000000 00:00 0
> [heap]
> 40002000-40005000 ---p 00000000 00:00 0
> 40005000-40023000 rw-p 00000000 00:00 0
> 40030000-40033000 ---p 00000000 00:00 0
> 40033000-40051000 rw-p 00000000 00:00 0
>
> Does anyone have similar problems? or maybe some hints?
>
> Thanks,
> Maciej
--
sent from my Nokia 3210