You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Amlan Roy <am...@cleartrip.com> on 2015/03/30 22:12:41 UTC
nodetool cleanup error
Hi,
I have added new nodes to an existing cluster and ran the “nodetool cleanup”. I am getting the following error. Wanted to know if there is any solution to it.
Regards,
Amlan
Error occurred during cleanup
java.util.concurrent.ExecutionException: java.lang.AssertionError: Memory was freed
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:188)
at org.apache.cassandra.db.compaction.CompactionManager.performAllSSTableOperation(CompactionManager.java:234)
at org.apache.cassandra.db.compaction.CompactionManager.performCleanup(CompactionManager.java:272)
at org.apache.cassandra.db.ColumnFamilyStore.forceCleanup(ColumnFamilyStore.java:1115)
at org.apache.cassandra.service.StorageService.forceKeyspaceCleanup(StorageService.java:2177)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:75)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:279)
at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1487)
at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:97)
at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1328)
at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1420)
at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:848)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
at sun.rmi.transport.Transport$1.run(Transport.java:177)
at sun.rmi.transport.Transport$1.run(Transport.java:174)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:556)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:811)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:670)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.AssertionError: Memory was freed
at org.apache.cassandra.io.util.Memory.checkPosition(Memory.java:259)
at org.apache.cassandra.io.util.Memory.getInt(Memory.java:211)
at org.apache.cassandra.io.sstable.IndexSummary.getIndex(IndexSummary.java:79)
at org.apache.cassandra.io.sstable.IndexSummary.getKey(IndexSummary.java:84)
at org.apache.cassandra.io.sstable.IndexSummary.binarySearch(IndexSummary.java:58)
at org.apache.cassandra.io.sstable.SSTableReader.getIndexScanPosition(SSTableReader.java:602)
at org.apache.cassandra.io.sstable.SSTableReader.getPosition(SSTableReader.java:947)
at org.apache.cassandra.io.sstable.SSTableReader.getPosition(SSTableReader.java:910)
at org.apache.cassandra.io.sstable.SSTableReader.getPositionsForRanges(SSTableReader.java:819)
at org.apache.cassandra.db.ColumnFamilyStore.getExpectedCompactedFileSize(ColumnFamilyStore.java:1088)
at org.apache.cassandra.db.compaction.CompactionManager.doCleanupCompaction(CompactionManager.java:564)
at org.apache.cassandra.db.compaction.CompactionManager.access$400(CompactionManager.java:63)
at org.apache.cassandra.db.compaction.CompactionManager$5.perform(CompactionManager.java:281)
at org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:225)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
Re: nodetool cleanup error
Posted by Jeff Ferland <jb...@tubularlabs.com>.
Code problem that was patched in https://issues.apache.org/jira/browse/CASSANDRA-8716 <https://issues.apache.org/jira/browse/CASSANDRA-8716>. Upgrade to 2.0.13
> On Mar 30, 2015, at 1:12 PM, Amlan Roy <am...@cleartrip.com> wrote:
>
> Hi,
>
> I have added new nodes to an existing cluster and ran the “nodetool cleanup”. I am getting the following error. Wanted to know if there is any solution to it.
>
> Regards,
> Amlan
>
> Error occurred during cleanup
> java.util.concurrent.ExecutionException: java.lang.AssertionError: Memory was freed
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:188)
> at org.apache.cassandra.db.compaction.CompactionManager.performAllSSTableOperation(CompactionManager.java:234)
> at org.apache.cassandra.db.compaction.CompactionManager.performCleanup(CompactionManager.java:272)
> at org.apache.cassandra.db.ColumnFamilyStore.forceCleanup(ColumnFamilyStore.java:1115)
> at org.apache.cassandra.service.StorageService.forceKeyspaceCleanup(StorageService.java:2177)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:75)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:279)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1487)
> at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:97)
> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1328)
> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1420)
> at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:848)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> at sun.rmi.transport.Transport$1.run(Transport.java:177)
> at sun.rmi.transport.Transport$1.run(Transport.java:174)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:556)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:811)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:670)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.AssertionError: Memory was freed
> at org.apache.cassandra.io.util.Memory.checkPosition(Memory.java:259)
> at org.apache.cassandra.io.util.Memory.getInt(Memory.java:211)
> at org.apache.cassandra.io.sstable.IndexSummary.getIndex(IndexSummary.java:79)
> at org.apache.cassandra.io.sstable.IndexSummary.getKey(IndexSummary.java:84)
> at org.apache.cassandra.io.sstable.IndexSummary.binarySearch(IndexSummary.java:58)
> at org.apache.cassandra.io.sstable.SSTableReader.getIndexScanPosition(SSTableReader.java:602)
> at org.apache.cassandra.io.sstable.SSTableReader.getPosition(SSTableReader.java:947)
> at org.apache.cassandra.io.sstable.SSTableReader.getPosition(SSTableReader.java:910)
> at org.apache.cassandra.io.sstable.SSTableReader.getPositionsForRanges(SSTableReader.java:819)
> at org.apache.cassandra.db.ColumnFamilyStore.getExpectedCompactedFileSize(ColumnFamilyStore.java:1088)
> at org.apache.cassandra.db.compaction.CompactionManager.doCleanupCompaction(CompactionManager.java:564)
> at org.apache.cassandra.db.compaction.CompactionManager.access$400(CompactionManager.java:63)
> at org.apache.cassandra.db.compaction.CompactionManager$5.perform(CompactionManager.java:281)
> at org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:225)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
Re: nodetool cleanup error
Posted by Marcus Eriksson <kr...@gmail.com>.
It should work on 2.0.13. If it fails with that assertion, you should just
retry. If that does not work, and you can reproduce this, please file a
ticket
/Marcus
On Tue, Mar 31, 2015 at 9:33 AM, Amlan Roy <am...@cleartrip.com> wrote:
> Hi,
>
> Thanks for the reply. Since nodetool cleanup is not working even after
> upgrading to 2.0.13, is it recommended to go to an older version (2.0.11
> for example, with 2.0.12 also it did not work). Is there any other way of
> cleaning data from existing nodes after adding a new node.
>
> Regards,
> Amlan
>
> On 31-Mar-2015, at 5:00 am, Yuki Morishita <mo...@gmail.com> wrote:
>
> > Looks like the issue is
> https://issues.apache.org/jira/browse/CASSANDRA-9070.
> >
> > On Mon, Mar 30, 2015 at 6:25 PM, Robert Coli <rc...@eventbrite.com>
> wrote:
> >> On Mon, Mar 30, 2015 at 4:21 PM, Amlan Roy <am...@cleartrip.com>
> wrote:
> >>>
> >>> Thanks for the reply. I have upgraded to 2.0.13. Now I get the
> following
> >>> error.
> >>
> >>
> >> If cleanup is still excepting for you on 2.0.13 with some sstables you
> have,
> >> I would strongly consider :
> >>
> >> 1) file a JIRA (http://issues.apache.org) and attach / offer the
> sstables
> >> for debugging
> >> 2) let the list know the JIRA id of the ticket
> >>
> >> =Rob
> >>
> >
> >
> >
> > --
> > Yuki Morishita
> > t:yukim (http://twitter.com/yukim)
>
>
Re: nodetool cleanup error
Posted by Amlan Roy <am...@cleartrip.com>.
Hi,
Thanks for the reply. Since nodetool cleanup is not working even after upgrading to 2.0.13, is it recommended to go to an older version (2.0.11 for example, with 2.0.12 also it did not work). Is there any other way of cleaning data from existing nodes after adding a new node.
Regards,
Amlan
On 31-Mar-2015, at 5:00 am, Yuki Morishita <mo...@gmail.com> wrote:
> Looks like the issue is https://issues.apache.org/jira/browse/CASSANDRA-9070.
>
> On Mon, Mar 30, 2015 at 6:25 PM, Robert Coli <rc...@eventbrite.com> wrote:
>> On Mon, Mar 30, 2015 at 4:21 PM, Amlan Roy <am...@cleartrip.com> wrote:
>>>
>>> Thanks for the reply. I have upgraded to 2.0.13. Now I get the following
>>> error.
>>
>>
>> If cleanup is still excepting for you on 2.0.13 with some sstables you have,
>> I would strongly consider :
>>
>> 1) file a JIRA (http://issues.apache.org) and attach / offer the sstables
>> for debugging
>> 2) let the list know the JIRA id of the ticket
>>
>> =Rob
>>
>
>
>
> --
> Yuki Morishita
> t:yukim (http://twitter.com/yukim)
Re: nodetool cleanup error
Posted by Yuki Morishita <mo...@gmail.com>.
Looks like the issue is https://issues.apache.org/jira/browse/CASSANDRA-9070.
On Mon, Mar 30, 2015 at 6:25 PM, Robert Coli <rc...@eventbrite.com> wrote:
> On Mon, Mar 30, 2015 at 4:21 PM, Amlan Roy <am...@cleartrip.com> wrote:
>>
>> Thanks for the reply. I have upgraded to 2.0.13. Now I get the following
>> error.
>
>
> If cleanup is still excepting for you on 2.0.13 with some sstables you have,
> I would strongly consider :
>
> 1) file a JIRA (http://issues.apache.org) and attach / offer the sstables
> for debugging
> 2) let the list know the JIRA id of the ticket
>
> =Rob
>
--
Yuki Morishita
t:yukim (http://twitter.com/yukim)
Re: nodetool cleanup error
Posted by Robert Coli <rc...@eventbrite.com>.
On Mon, Mar 30, 2015 at 4:21 PM, Amlan Roy <am...@cleartrip.com> wrote:
> Thanks for the reply. I have upgraded to 2.0.13. Now I get the following
> error.
>
If cleanup is still excepting for you on 2.0.13 with some sstables you
have, I would strongly consider :
1) file a JIRA (http://issues.apache.org) and attach / offer the sstables
for debugging
2) let the list know the JIRA id of the ticket
=Rob
Re: nodetool cleanup error
Posted by Amlan Roy <am...@cleartrip.com>.
Hi,
Thanks for the reply. I have upgraded to 2.0.13. Now I get the following error.
Regards,
Amlan
Exception in thread "main" java.lang.AssertionError: [SSTableReader(path='/data/1/cassandra/data/xxx/xxx/xxx.db'), SSTableReader(path='/data/1/cassandra/data/xxx/xxx/xxx.db')]
at org.apache.cassandra.db.ColumnFamilyStore$13.call(ColumnFamilyStore.java:2176)
at org.apache.cassandra.db.ColumnFamilyStore$13.call(ColumnFamilyStore.java:2173)
at org.apache.cassandra.db.ColumnFamilyStore.runWithCompactionsDisabled(ColumnFamilyStore.java:2155)
at org.apache.cassandra.db.ColumnFamilyStore.markAllCompacting(ColumnFamilyStore.java:2186)
at org.apache.cassandra.db.compaction.CompactionManager.performAllSSTableOperation(CompactionManager.java:215)
at org.apache.cassandra.db.compaction.CompactionManager.performCleanup(CompactionManager.java:272)
at org.apache.cassandra.db.ColumnFamilyStore.forceCleanup(ColumnFamilyStore.java:1117)
at org.apache.cassandra.service.StorageService.forceKeyspaceCleanup(StorageService.java:2177)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:75)
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:279)
at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1487)
at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:97)
at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1328)
at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1420)
at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:848)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
at sun.rmi.transport.Transport$1.run(Transport.java:177)
at sun.rmi.transport.Transport$1.run(Transport.java:174)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:556)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:811)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:670)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
On 31-Mar-2015, at 1:50 am, Duncan Sands <du...@gmail.com> wrote:
> Hi Amlan,
>
> On 30/03/15 22:12, Amlan Roy wrote:
>> Hi,
>>
>> I have added new nodes to an existing cluster and ran the “nodetool cleanup”. I
>> am getting the following error. Wanted to know if there is any solution to it.
>>
>> Regards,
>> Amlan
>>
>> Error occurred during cleanup
>> java.util.concurrent.ExecutionException: java.lang.AssertionError: Memory was freed
>
> this is fixed in 2.0.13.
>
> Best wishes, Duncan.
Re: nodetool cleanup error
Posted by Duncan Sands <du...@gmail.com>.
Hi Amlan,
On 30/03/15 22:12, Amlan Roy wrote:
> Hi,
>
> I have added new nodes to an existing cluster and ran the “nodetool cleanup”. I
> am getting the following error. Wanted to know if there is any solution to it.
>
> Regards,
> Amlan
>
> Error occurred during cleanup
> java.util.concurrent.ExecutionException: java.lang.AssertionError: Memory was freed
this is fixed in 2.0.13.
Best wishes, Duncan.