You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Yaniv Kunda (JIRA)" <ji...@apache.org> on 2011/02/24 16:33:38 UTC
[jira] Created: (CASSANDRA-2240) nodetool scrub hangs or throws an
exception
nodetool scrub hangs or throws an exception
-------------------------------------------
Key: CASSANDRA-2240
URL: https://issues.apache.org/jira/browse/CASSANDRA-2240
Project: Cassandra
Issue Type: Bug
Components: Tools
Environment: using build #314 from hudson
Reporter: Yaniv Kunda
trying to run nodetool scrub hung or (only happened one time) threw the following exception:
Error occured while scrubbing keyspace <mykeyspace>
java.util.concurrent.ExecutionException: java.lang.AssertionError
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
at java.util.concurrent.FutureTask.get(FutureTask.java:111)
at org.apache.cassandra.db.CompactionManager.performScrub(CompactionManager.java:203)
at org.apache.cassandra.db.ColumnFamilyStore.scrub(ColumnFamilyStore.java:934)
at org.apache.cassandra.service.StorageService.scrub(StorageService.java:1247)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
at sun.rmi.transport.Transport$1.run(Transport.java:177)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
Caused by: java.lang.AssertionError
at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62)
at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627)
at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:541)
at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
... 3 more
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Issue Comment Edited: (CASSANDRA-2240) nodetool scrub hangs
or throws an exception
Posted by "Yaniv Kunda (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12998902#comment-12998902 ]
Yaniv Kunda edited comment on CASSANDRA-2240 at 2/24/11 3:42 PM:
-----------------------------------------------------------------
attached the tables that can be used to reproduce the hang
was (Author: kunda):
these are the tables that can be used to reproduce the hang
> nodetool scrub hangs or throws an exception
> -------------------------------------------
>
> Key: CASSANDRA-2240
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2240
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Environment: using build #314 from hudson
> Reporter: Yaniv Kunda
> Attachments: test-0.6.x-tables.tar.gz
>
>
> trying to run nodetool scrub hung or (only happened one time) threw the following exception:
> Error occured while scrubbing keyspace <mykeyspace>
> java.util.concurrent.ExecutionException: java.lang.AssertionError
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> at org.apache.cassandra.db.CompactionManager.performScrub(CompactionManager.java:203)
> at org.apache.cassandra.db.ColumnFamilyStore.scrub(ColumnFamilyStore.java:934)
> at org.apache.cassandra.service.StorageService.scrub(StorageService.java:1247)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
> at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
> at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> at sun.rmi.transport.Transport$1.run(Transport.java:177)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
> Caused by: java.lang.AssertionError
> at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62)
> at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:541)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> ... 3 more
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-2240) nodetool scrub hangs or throws an
exception
Posted by "Yaniv Kunda (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yaniv Kunda updated CASSANDRA-2240:
-----------------------------------
Description:
trying to run nodetool scrub hung or (only happened one time) threw the following exception:
ERROR [CompactionExecutor:1] 2011-02-28 10:26:26,620 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
java.lang.AssertionError
at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62)
at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627)
at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:538)
at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
was:
trying to run nodetool scrub hung or (only happened one time) threw the following exception:
Error occured while scrubbing keyspace <mykeyspace>
java.util.concurrent.ExecutionException: java.lang.AssertionError
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
at java.util.concurrent.FutureTask.get(FutureTask.java:111)
at org.apache.cassandra.db.CompactionManager.performScrub(CompactionManager.java:203)
at org.apache.cassandra.db.ColumnFamilyStore.scrub(ColumnFamilyStore.java:934)
at org.apache.cassandra.service.StorageService.scrub(StorageService.java:1247)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
at sun.rmi.transport.Transport$1.run(Transport.java:177)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
Caused by: java.lang.AssertionError
at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62)
at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627)
at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:541)
at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
... 3 more
> nodetool scrub hangs or throws an exception
> -------------------------------------------
>
> Key: CASSANDRA-2240
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2240
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.7.3
> Environment: using build #314 from hudson
> Reporter: Yaniv Kunda
> Assignee: Jonathan Ellis
> Fix For: 0.7.4
>
> Attachments: exception2.txt, jstack1.txt, signatureBuckets-f-104.tar.gz, system.log.2.gz, system.log.gz, test-0.6.x-tables.tar.gz, userChannelFilter-f-210.tar.gz
>
>
> trying to run nodetool scrub hung or (only happened one time) threw the following exception:
> ERROR [CompactionExecutor:1] 2011-02-28 10:26:26,620 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.lang.AssertionError
> at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62)
> at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:538)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2240) nodetool scrub hangs or throws
an exception
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13000522#comment-13000522 ]
Jonathan Ellis commented on CASSANDRA-2240:
-------------------------------------------
Looked at userChannelFilter-f-210.tar.gz. Data file does not match index even a little. Scrub can't help there.
> nodetool scrub hangs or throws an exception
> -------------------------------------------
>
> Key: CASSANDRA-2240
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2240
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.7.3
> Environment: using build #314 from hudson
> Reporter: Yaniv Kunda
> Assignee: Jonathan Ellis
> Fix For: 0.7.4
>
> Attachments: exception2.txt, jstack1.txt, signatureBuckets-f-104.tar.gz, system.log.2.gz, system.log.gz, test-0.6.x-tables.tar.gz, userChannelFilter-f-210.tar.gz
>
>
> trying to run nodetool scrub hung or (only happened one time) threw the following exception:
> ERROR [CompactionExecutor:1] 2011-02-28 10:26:26,620 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.lang.AssertionError
> at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62)
> at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:538)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Issue Comment Edited: (CASSANDRA-2240) nodetool scrub hangs
or throws an exception
Posted by "Yaniv Kunda (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12999022#comment-12999022 ]
Yaniv Kunda edited comment on CASSANDRA-2240 at 2/24/11 8:22 PM:
-----------------------------------------------------------------
I now restarted the server again, and two things happened:
1) performing a scrub on the example CF and a bunch of other CFs succeeded without hanging
2) performing a scrub on a different CF resulting in the following stack trace (on different retries):
{code}
ERROR [CompactionExecutor:1] 2011-02-24 22:02:30,329 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
java.io.IOError: java.io.EOFException
at org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:113)
at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:549)
at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
Caused by: java.io.EOFException
at java.io.RandomAccessFile.readInt(RandomAccessFile.java:776)
at org.apache.cassandra.io.sstable.IndexHelper.skipBloomFilter(IndexHelper.java:47)
at org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:104)
... 8 more
{code}
was (Author: kunda):
I now restarted the server again, and two things happened:
1) performing a scrub on the example CF and a bunch of other CFs succeeded without hanging
2) performing a scrub on a different CF resulting in the following stack trace:
{code}
ERROR [CompactionExecutor:1] 2011-02-24 22:02:30,329 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
java.io.IOError: java.io.EOFException
at org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:113)
at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:549)
at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
Caused by: java.io.EOFException
at java.io.RandomAccessFile.readInt(RandomAccessFile.java:776)
at org.apache.cassandra.io.sstable.IndexHelper.skipBloomFilter(IndexHelper.java:47)
at org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:104)
... 8 more
{code}
> nodetool scrub hangs or throws an exception
> -------------------------------------------
>
> Key: CASSANDRA-2240
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2240
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Environment: using build #314 from hudson
> Reporter: Yaniv Kunda
> Assignee: Jonathan Ellis
> Attachments: test-0.6.x-tables.tar.gz
>
>
> trying to run nodetool scrub hung or (only happened one time) threw the following exception:
> Error occured while scrubbing keyspace <mykeyspace>
> java.util.concurrent.ExecutionException: java.lang.AssertionError
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> at org.apache.cassandra.db.CompactionManager.performScrub(CompactionManager.java:203)
> at org.apache.cassandra.db.ColumnFamilyStore.scrub(ColumnFamilyStore.java:934)
> at org.apache.cassandra.service.StorageService.scrub(StorageService.java:1247)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
> at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
> at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> at sun.rmi.transport.Transport$1.run(Transport.java:177)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
> Caused by: java.lang.AssertionError
> at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62)
> at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:541)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> ... 3 more
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2240) nodetool scrub hangs or throws
an exception
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13000898#comment-13000898 ]
Jonathan Ellis commented on CASSANDRA-2240:
-------------------------------------------
bq. v4 is there, you named it 2240.txt
Oops -- that was the original patch, not actually v4. Must have been tired.
> nodetool scrub hangs or throws an exception
> -------------------------------------------
>
> Key: CASSANDRA-2240
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2240
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.7.3
> Environment: using build #314 from hudson
> Reporter: Yaniv Kunda
> Assignee: Jonathan Ellis
> Fix For: 0.7.4
>
> Attachments: 2240-v2.txt, 2240-v3.txt, 2240-v5.patch, 2240-v6.txt, 2240.txt, 2240.txt, exception2.txt, jstack1.txt, signatureBuckets-f-104.tar.gz, system.log.2.gz, system.log.gz, test-0.6.x-tables.tar.gz, userChannelFilter-f-210.tar.gz
>
>
> trying to run nodetool scrub hung or (only happened one time) threw the following exception:
> ERROR [CompactionExecutor:1] 2011-02-28 10:26:26,620 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.lang.AssertionError
> at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62)
> at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:538)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2240) nodetool scrub hangs or throws
an exception
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13000894#comment-13000894 ]
Jonathan Ellis commented on CASSANDRA-2240:
-------------------------------------------
I also backed out the read-directly-from-file change in IndexHelper. Will create a new ticket for that.
> nodetool scrub hangs or throws an exception
> -------------------------------------------
>
> Key: CASSANDRA-2240
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2240
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.7.3
> Environment: using build #314 from hudson
> Reporter: Yaniv Kunda
> Assignee: Jonathan Ellis
> Fix For: 0.7.4
>
> Attachments: 2240-v2.txt, 2240-v3.txt, 2240-v5.patch, 2240-v6.txt, 2240.txt, 2240.txt, exception2.txt, jstack1.txt, signatureBuckets-f-104.tar.gz, system.log.2.gz, system.log.gz, test-0.6.x-tables.tar.gz, userChannelFilter-f-210.tar.gz
>
>
> trying to run nodetool scrub hung or (only happened one time) threw the following exception:
> ERROR [CompactionExecutor:1] 2011-02-28 10:26:26,620 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.lang.AssertionError
> at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62)
> at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:538)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2240) nodetool scrub hangs or throws
an exception
Posted by "Sébastien Giroux (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13000696#comment-13000696 ]
Sébastien Giroux commented on CASSANDRA-2240:
---------------------------------------------
Looking at the data directory, it looks like it was worked as excepted before the exception happened. The tmp SStable is 1.0GB... Looks like we are getting close!
> nodetool scrub hangs or throws an exception
> -------------------------------------------
>
> Key: CASSANDRA-2240
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2240
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.7.3
> Environment: using build #314 from hudson
> Reporter: Yaniv Kunda
> Assignee: Jonathan Ellis
> Fix For: 0.7.4
>
> Attachments: 2240-v2.txt, 2240.txt, exception2.txt, jstack1.txt, signatureBuckets-f-104.tar.gz, system.log.2.gz, system.log.gz, test-0.6.x-tables.tar.gz, userChannelFilter-f-210.tar.gz
>
>
> trying to run nodetool scrub hung or (only happened one time) threw the following exception:
> ERROR [CompactionExecutor:1] 2011-02-28 10:26:26,620 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.lang.AssertionError
> at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62)
> at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:538)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2240) nodetool scrub hangs or throws
an exception
Posted by "Sébastien Giroux (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13000920#comment-13000920 ]
Sébastien Giroux commented on CASSANDRA-2240:
---------------------------------------------
I'm testing it as we speak!
Log are flooded with this:
INFO 15:31:15,783 Invalid bloom filter in SSTableReader(path='/var/lib/cassandra/data/Wajam/WebsiteWajams-f-164-Data.db'); will rebuild it
INFO 15:31:15,783 Invalid bloom filter in SSTableReader(path='/var/lib/cassandra/data/Wajam/WebsiteWajams-f-164-Data.db'); will rebuild it
INFO 15:31:15,783 Invalid bloom filter in SSTableReader(path='/var/lib/cassandra/data/Wajam/WebsiteWajams-f-164-Data.db'); will rebuild it
INFO 15:31:15,783 Invalid bloom filter in SSTableReader(path='/var/lib/cassandra/data/Wajam/WebsiteWajams-f-164-Data.db'); will rebuild it
I'm not sure why it's scrubbing "WebsiteWajams" CF as I asked for "Wajam" (nodetool -h 127.0.0.1 scrub Wajam Wajam) ... Either way, this sstable is messed up too so that's fine.
Hopefully it won't just do like v4 that ran all night and was still echoing invalid bloom filter for the same sstable in the log this morning, Looks like it was stuck in a infinite loop or something.
It looks like it's scrubbing another sstable now so that's already an improvment.
Will keep you guy updated!
> nodetool scrub hangs or throws an exception
> -------------------------------------------
>
> Key: CASSANDRA-2240
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2240
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.7.3
> Environment: using build #314 from hudson
> Reporter: Yaniv Kunda
> Assignee: Jonathan Ellis
> Fix For: 0.7.4
>
> Attachments: 2240-v2.txt, 2240-v3.txt, 2240-v5.patch, 2240-v6.txt, 2240.txt, 2240.txt, exception2.txt, jstack1.txt, signatureBuckets-f-104.tar.gz, system.log.2.gz, system.log.gz, test-0.6.x-tables.tar.gz, userChannelFilter-f-210.tar.gz
>
>
> trying to run nodetool scrub hung or (only happened one time) threw the following exception:
> ERROR [CompactionExecutor:1] 2011-02-28 10:26:26,620 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.lang.AssertionError
> at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62)
> at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:538)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-2240) nodetool scrub hangs or throws an
exception
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-2240:
--------------------------------------
Attachment: 2240.txt
Patch to make scrub less crashy. It still can't magically fix severely corrupted files like the exhibits here, though.
> nodetool scrub hangs or throws an exception
> -------------------------------------------
>
> Key: CASSANDRA-2240
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2240
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.7.3
> Environment: using build #314 from hudson
> Reporter: Yaniv Kunda
> Assignee: Jonathan Ellis
> Fix For: 0.7.4
>
> Attachments: 2240.txt, exception2.txt, jstack1.txt, signatureBuckets-f-104.tar.gz, system.log.2.gz, system.log.gz, test-0.6.x-tables.tar.gz, userChannelFilter-f-210.tar.gz
>
>
> trying to run nodetool scrub hung or (only happened one time) threw the following exception:
> ERROR [CompactionExecutor:1] 2011-02-28 10:26:26,620 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.lang.AssertionError
> at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62)
> at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:538)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-2240) nodetool scrub hangs or throws an
exception
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-2240:
--------------------------------------
Affects Version/s: 0.7.3
Fix Version/s: 0.7.3
Couldn't reproduce by scrubbing sstables produced by 0.6 stress.py
> nodetool scrub hangs or throws an exception
> -------------------------------------------
>
> Key: CASSANDRA-2240
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2240
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.7.3
> Environment: using build #314 from hudson
> Reporter: Yaniv Kunda
> Assignee: Jonathan Ellis
> Fix For: 0.7.3
>
> Attachments: test-0.6.x-tables.tar.gz
>
>
> trying to run nodetool scrub hung or (only happened one time) threw the following exception:
> Error occured while scrubbing keyspace <mykeyspace>
> java.util.concurrent.ExecutionException: java.lang.AssertionError
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> at org.apache.cassandra.db.CompactionManager.performScrub(CompactionManager.java:203)
> at org.apache.cassandra.db.ColumnFamilyStore.scrub(ColumnFamilyStore.java:934)
> at org.apache.cassandra.service.StorageService.scrub(StorageService.java:1247)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
> at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
> at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> at sun.rmi.transport.Transport$1.run(Transport.java:177)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
> Caused by: java.lang.AssertionError
> at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62)
> at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:541)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> ... 3 more
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2240) nodetool scrub hangs or throws
an exception
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13000944#comment-13000944 ]
Jonathan Ellis commented on CASSANDRA-2240:
-------------------------------------------
Yes, we'll get this into 0.7.3.
> nodetool scrub hangs or throws an exception
> -------------------------------------------
>
> Key: CASSANDRA-2240
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2240
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.7.3
> Environment: using build #314 from hudson
> Reporter: Yaniv Kunda
> Assignee: Jonathan Ellis
> Fix For: 0.7.4
>
> Attachments: 2240-v2.txt, 2240-v3.txt, 2240-v5.patch, 2240-v6.txt, 2240.txt, 2240.txt, exception2.txt, jstack1.txt, signatureBuckets-f-104.tar.gz, system.log.2.gz, system.log.gz, test-0.6.x-tables.tar.gz, userChannelFilter-f-210.tar.gz
>
>
> trying to run nodetool scrub hung or (only happened one time) threw the following exception:
> ERROR [CompactionExecutor:1] 2011-02-28 10:26:26,620 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.lang.AssertionError
> at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62)
> at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:538)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2240) nodetool scrub hangs or throws
an exception
Posted by "Yaniv Kunda (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12999921#comment-12999921 ]
Yaniv Kunda commented on CASSANDRA-2240:
----------------------------------------
After performing more tests, I realized why the problem could not be reproduced - the scrub process hang on a different CF, and afterwards any scrub operation would hang, until the server is restarted.
I was able to narrow down the problem to a specific sstable - I will soon post it along the stack trace.
> nodetool scrub hangs or throws an exception
> -------------------------------------------
>
> Key: CASSANDRA-2240
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2240
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.7.3
> Environment: using build #314 from hudson
> Reporter: Yaniv Kunda
> Assignee: Jonathan Ellis
> Fix For: 0.7.3
>
> Attachments: test-0.6.x-tables.tar.gz
>
>
> trying to run nodetool scrub hung or (only happened one time) threw the following exception:
> Error occured while scrubbing keyspace <mykeyspace>
> java.util.concurrent.ExecutionException: java.lang.AssertionError
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> at org.apache.cassandra.db.CompactionManager.performScrub(CompactionManager.java:203)
> at org.apache.cassandra.db.ColumnFamilyStore.scrub(ColumnFamilyStore.java:934)
> at org.apache.cassandra.service.StorageService.scrub(StorageService.java:1247)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
> at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
> at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> at sun.rmi.transport.Transport$1.run(Transport.java:177)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
> Caused by: java.lang.AssertionError
> at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62)
> at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:541)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> ... 3 more
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2240) nodetool scrub hangs or throws
an exception
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13000602#comment-13000602 ]
Jonathan Ellis commented on CASSANDRA-2240:
-------------------------------------------
Sebastien, go ahead and try scrub after rebuilding w/ the newest patch.
But if your example is like the others, you're seeing something other than the CASSANDRA-2211/CASSANDRA-2216 corruption that scrub is intended to deal with.
So far the only suggestion I have to track that down is to start over and run with snapshot_before_compaction turned on in cassandra.yaml, so when a corrupt sstable is generated we will know where it came from.
> nodetool scrub hangs or throws an exception
> -------------------------------------------
>
> Key: CASSANDRA-2240
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2240
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.7.3
> Environment: using build #314 from hudson
> Reporter: Yaniv Kunda
> Assignee: Jonathan Ellis
> Fix For: 0.7.4
>
> Attachments: 2240.txt, exception2.txt, jstack1.txt, signatureBuckets-f-104.tar.gz, system.log.2.gz, system.log.gz, test-0.6.x-tables.tar.gz, userChannelFilter-f-210.tar.gz
>
>
> trying to run nodetool scrub hung or (only happened one time) threw the following exception:
> ERROR [CompactionExecutor:1] 2011-02-28 10:26:26,620 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.lang.AssertionError
> at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62)
> at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:538)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-2240) nodetool scrub hangs or throws an
exception
Posted by "Yaniv Kunda (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yaniv Kunda updated CASSANDRA-2240:
-----------------------------------
Attachment: system.log.2.gz
another system log, this time leading to a vanilla java.io.EOFException
> nodetool scrub hangs or throws an exception
> -------------------------------------------
>
> Key: CASSANDRA-2240
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2240
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.7.3
> Environment: using build #314 from hudson
> Reporter: Yaniv Kunda
> Assignee: Jonathan Ellis
> Fix For: 0.7.4
>
> Attachments: exception2.txt, jstack1.txt, signatureBuckets-f-104.tar.gz, system.log.2.gz, system.log.gz, test-0.6.x-tables.tar.gz, userChannelFilter-f-210.tar.gz
>
>
> trying to run nodetool scrub hung or (only happened one time) threw the following exception:
> Error occured while scrubbing keyspace <mykeyspace>
> java.util.concurrent.ExecutionException: java.lang.AssertionError
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> at org.apache.cassandra.db.CompactionManager.performScrub(CompactionManager.java:203)
> at org.apache.cassandra.db.ColumnFamilyStore.scrub(ColumnFamilyStore.java:934)
> at org.apache.cassandra.service.StorageService.scrub(StorageService.java:1247)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
> at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
> at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> at sun.rmi.transport.Transport$1.run(Transport.java:177)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
> Caused by: java.lang.AssertionError
> at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62)
> at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:541)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> ... 3 more
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2240) nodetool scrub hangs or throws
an exception
Posted by "Sébastien Giroux (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13000048#comment-13000048 ]
Sébastien Giroux commented on CASSANDRA-2240:
---------------------------------------------
We are running into the same issue using #325 from Hudson. Exception everytime.
Everything started when we upgraded from 0.6.8 to 0.7.2. One of the node (17 nodes total) started to be slow, we realized one CF wasn't compacting and throw exception (Error in ThreadPoolExecutor java.io.IOError: org.apache.cassandra.db.ColumnSerializer$CorruptColumnException: invalid column name length 0).
So I gave the new scrub function a try and but looks like it doesnt work. We really need to have this fixed!
Thank you!
> nodetool scrub hangs or throws an exception
> -------------------------------------------
>
> Key: CASSANDRA-2240
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2240
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.7.3
> Environment: using build #314 from hudson
> Reporter: Yaniv Kunda
> Assignee: Jonathan Ellis
> Fix For: 0.7.3
>
> Attachments: exception2.txt, jstack1.txt, signatureBuckets-f-104.tar.gz, test-0.6.x-tables.tar.gz, userChannelFilter-f-210.tar.gz
>
>
> trying to run nodetool scrub hung or (only happened one time) threw the following exception:
> Error occured while scrubbing keyspace <mykeyspace>
> java.util.concurrent.ExecutionException: java.lang.AssertionError
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> at org.apache.cassandra.db.CompactionManager.performScrub(CompactionManager.java:203)
> at org.apache.cassandra.db.ColumnFamilyStore.scrub(ColumnFamilyStore.java:934)
> at org.apache.cassandra.service.StorageService.scrub(StorageService.java:1247)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
> at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
> at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> at sun.rmi.transport.Transport$1.run(Transport.java:177)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
> Caused by: java.lang.AssertionError
> at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62)
> at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:541)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> ... 3 more
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2240) nodetool scrub hangs or throws
an exception
Posted by "Yaniv Kunda (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12999006#comment-12999006 ]
Yaniv Kunda commented on CASSANDRA-2240:
----------------------------------------
I didn't have time to apply the patch yet - but I will next week.
Regarding the example CF - as [~slebresne] commented in CASSANDRA-2217, it was indeed created in Cassandra 0.6.5 -
so the cli cannot be used to create it ;)
However, here is KS xml element that defined it in storage-conf.xml:
{code:xml}
<Keyspace Name="generalUtils">
<ColumnFamily Name="userActionUtilsKey" CompareWith="UTF8Type" />
<ColumnFamily Name="facebookShowIds" CompareWith="UTF8Type" />
<ReplicaPlacementStrategy>org.apache.cassandra.locator.RackUnawareStrategy</ReplicaPlacementStrategy>
<ReplicationFactor>1</ReplicationFactor>
<EndPointSnitch>org.apache.cassandra.locator.EndPointSnitch</EndPointSnitch>
</Keyspace>
{code}
> nodetool scrub hangs or throws an exception
> -------------------------------------------
>
> Key: CASSANDRA-2240
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2240
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Environment: using build #314 from hudson
> Reporter: Yaniv Kunda
> Assignee: Jonathan Ellis
> Attachments: test-0.6.x-tables.tar.gz
>
>
> trying to run nodetool scrub hung or (only happened one time) threw the following exception:
> Error occured while scrubbing keyspace <mykeyspace>
> java.util.concurrent.ExecutionException: java.lang.AssertionError
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> at org.apache.cassandra.db.CompactionManager.performScrub(CompactionManager.java:203)
> at org.apache.cassandra.db.ColumnFamilyStore.scrub(ColumnFamilyStore.java:934)
> at org.apache.cassandra.service.StorageService.scrub(StorageService.java:1247)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
> at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
> at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> at sun.rmi.transport.Transport$1.run(Transport.java:177)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
> Caused by: java.lang.AssertionError
> at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62)
> at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:541)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> ... 3 more
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2240) nodetool scrub hangs or throws
an exception
Posted by "Sébastien Giroux (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13000923#comment-13000923 ]
Sébastien Giroux commented on CASSANDRA-2240:
---------------------------------------------
INFO 15:42:52,690 Scrub of SSTableReader(path='/var/lib/cassandra/data/Wajam/WebsiteWajams-f-155-Data.db') complete: 3271113 in new sstable
"3271113 in new sstable"... I'm not sure it's 3271113 what... might want to specify! :)
I think it's the first time I see this line tho so it's good news. Is there any way the "invalid bloom filter, will rebuild it" line can only appear once per sstable ? I'm guessing its generating useless IO writing about 1000 of those lines/second in the log!
Good work guys, very appreciated :)
> nodetool scrub hangs or throws an exception
> -------------------------------------------
>
> Key: CASSANDRA-2240
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2240
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.7.3
> Environment: using build #314 from hudson
> Reporter: Yaniv Kunda
> Assignee: Jonathan Ellis
> Fix For: 0.7.4
>
> Attachments: 2240-v2.txt, 2240-v3.txt, 2240-v5.patch, 2240-v6.txt, 2240.txt, 2240.txt, exception2.txt, jstack1.txt, signatureBuckets-f-104.tar.gz, system.log.2.gz, system.log.gz, test-0.6.x-tables.tar.gz, userChannelFilter-f-210.tar.gz
>
>
> trying to run nodetool scrub hung or (only happened one time) threw the following exception:
> ERROR [CompactionExecutor:1] 2011-02-28 10:26:26,620 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.lang.AssertionError
> at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62)
> at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:538)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2240) nodetool scrub hangs or throws
an exception
Posted by "Sébastien Giroux (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13000873#comment-13000873 ]
Sébastien Giroux commented on CASSANDRA-2240:
---------------------------------------------
Thank you Sylvain
Unfortunatly, i'm unable to apply v5 patch for some reason, is it a svn or git patch ?
> nodetool scrub hangs or throws an exception
> -------------------------------------------
>
> Key: CASSANDRA-2240
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2240
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.7.3
> Environment: using build #314 from hudson
> Reporter: Yaniv Kunda
> Assignee: Jonathan Ellis
> Fix For: 0.7.4
>
> Attachments: 2240-v2.txt, 2240-v3.txt, 2240-v5.patch, 2240.txt, 2240.txt, exception2.txt, jstack1.txt, signatureBuckets-f-104.tar.gz, system.log.2.gz, system.log.gz, test-0.6.x-tables.tar.gz, userChannelFilter-f-210.tar.gz
>
>
> trying to run nodetool scrub hung or (only happened one time) threw the following exception:
> ERROR [CompactionExecutor:1] 2011-02-28 10:26:26,620 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.lang.AssertionError
> at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62)
> at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:538)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2240) nodetool scrub hangs or throws
an exception
Posted by "Sébastien Giroux (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13000931#comment-13000931 ]
Sébastien Giroux commented on CASSANDRA-2240:
---------------------------------------------
I did see a few exceptions every now and then. Not sure if you need to do anything about this. Sorry for all these posts, hopefully it help :):
224574- INFO 16:06:40,744 Retrying from row index; data is 245642090 bytes starting at 42
224657- WARN 16:06:40,744 Retry failed too. Skipping to next row (retry's stacktrace follows)
224745-java.io.IOError: java.io.EOFException
224783- at org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:117)
224884: at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:610)
224966- at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
225050- at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
225131- at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
225202- at java.util.concurrent.FutureTask.run(FutureTask.java:166)
225263- at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
225347- at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
225431- at java.lang.Thread.run(Thread.java:636)
225473-Caused by: java.io.EOFException
225505- at java.io.DataInputStream.readInt(DataInputStream.java:392)
225567- at org.apache.cassandra.utils.BloomFilterSerializer.deserialize(BloomFilterSerializer.java:47)
> nodetool scrub hangs or throws an exception
> -------------------------------------------
>
> Key: CASSANDRA-2240
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2240
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.7.3
> Environment: using build #314 from hudson
> Reporter: Yaniv Kunda
> Assignee: Jonathan Ellis
> Fix For: 0.7.4
>
> Attachments: 2240-v2.txt, 2240-v3.txt, 2240-v5.patch, 2240-v6.txt, 2240.txt, 2240.txt, exception2.txt, jstack1.txt, signatureBuckets-f-104.tar.gz, system.log.2.gz, system.log.gz, test-0.6.x-tables.tar.gz, userChannelFilter-f-210.tar.gz
>
>
> trying to run nodetool scrub hung or (only happened one time) threw the following exception:
> ERROR [CompactionExecutor:1] 2011-02-28 10:26:26,620 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.lang.AssertionError
> at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62)
> at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:538)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2240) nodetool scrub hangs or throws
an exception
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12999003#comment-12999003 ]
Jonathan Ellis commented on CASSANDRA-2240:
-------------------------------------------
What do you see when you apply the patch for CASSANDRA-2241?
What create column family statement should I give the cli, to create this userActionUtilsKey CF?
> nodetool scrub hangs or throws an exception
> -------------------------------------------
>
> Key: CASSANDRA-2240
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2240
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Environment: using build #314 from hudson
> Reporter: Yaniv Kunda
> Assignee: Jonathan Ellis
> Attachments: test-0.6.x-tables.tar.gz
>
>
> trying to run nodetool scrub hung or (only happened one time) threw the following exception:
> Error occured while scrubbing keyspace <mykeyspace>
> java.util.concurrent.ExecutionException: java.lang.AssertionError
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> at org.apache.cassandra.db.CompactionManager.performScrub(CompactionManager.java:203)
> at org.apache.cassandra.db.ColumnFamilyStore.scrub(ColumnFamilyStore.java:934)
> at org.apache.cassandra.service.StorageService.scrub(StorageService.java:1247)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
> at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
> at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> at sun.rmi.transport.Transport$1.run(Transport.java:177)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
> Caused by: java.lang.AssertionError
> at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62)
> at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:541)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> ... 3 more
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2240) nodetool scrub hangs or throws
an exception
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13000935#comment-13000935 ]
Jonathan Ellis commented on CASSANDRA-2240:
-------------------------------------------
bq. I did see a few exceptions every now and then
Did you see any "Error reading index file" lines before that? That's the only way I can think that it would come up with such a strange row size.
> nodetool scrub hangs or throws an exception
> -------------------------------------------
>
> Key: CASSANDRA-2240
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2240
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.7.3
> Environment: using build #314 from hudson
> Reporter: Yaniv Kunda
> Assignee: Jonathan Ellis
> Fix For: 0.7.4
>
> Attachments: 2240-v2.txt, 2240-v3.txt, 2240-v5.patch, 2240-v6.txt, 2240.txt, 2240.txt, exception2.txt, jstack1.txt, signatureBuckets-f-104.tar.gz, system.log.2.gz, system.log.gz, test-0.6.x-tables.tar.gz, userChannelFilter-f-210.tar.gz
>
>
> trying to run nodetool scrub hung or (only happened one time) threw the following exception:
> ERROR [CompactionExecutor:1] 2011-02-28 10:26:26,620 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.lang.AssertionError
> at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62)
> at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:538)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-2240) nodetool scrub hangs or throws an
exception
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-2240:
--------------------------------------
Attachment: 2240-v3.txt
v3 adds support for the index file ending before the data file
> nodetool scrub hangs or throws an exception
> -------------------------------------------
>
> Key: CASSANDRA-2240
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2240
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.7.3
> Environment: using build #314 from hudson
> Reporter: Yaniv Kunda
> Assignee: Jonathan Ellis
> Fix For: 0.7.4
>
> Attachments: 2240-v2.txt, 2240-v3.txt, 2240.txt, exception2.txt, jstack1.txt, signatureBuckets-f-104.tar.gz, system.log.2.gz, system.log.gz, test-0.6.x-tables.tar.gz, userChannelFilter-f-210.tar.gz
>
>
> trying to run nodetool scrub hung or (only happened one time) threw the following exception:
> ERROR [CompactionExecutor:1] 2011-02-28 10:26:26,620 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.lang.AssertionError
> at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62)
> at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:538)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Issue Comment Edited: (CASSANDRA-2240) nodetool scrub hangs
or throws an exception
Posted by "Sébastien Giroux (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13000895#comment-13000895 ]
Sébastien Giroux edited comment on CASSANDRA-2240 at 3/1/11 2:48 PM:
---------------------------------------------------------------------
Yep, that worked. So it's currently running with your patch, and so far I get no output from scrub after running for like 5 mins, I usually get the "Scrubbing SSTABLE" in the first min so I'm not sure if that's normal. There is the "tmp" files in the data directory but looks like they are stuck at 0 byte.
Oh, looks like I will try v6 now then!
Jonathan, v4 is there, you named it 2240.txt :)
was (Author: wajam):
Yep, that worked. So it's currently running with your patch, and so far I get no output from scrub after running for like 5 mins, I usually get the "Scrubbing SSTABLE" in the first min so I'm not sure if that's normal. There is the "tmp" files in the data directory but looks like they are stuck at 0 byte.
> nodetool scrub hangs or throws an exception
> -------------------------------------------
>
> Key: CASSANDRA-2240
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2240
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.7.3
> Environment: using build #314 from hudson
> Reporter: Yaniv Kunda
> Assignee: Jonathan Ellis
> Fix For: 0.7.4
>
> Attachments: 2240-v2.txt, 2240-v3.txt, 2240-v5.patch, 2240-v6.txt, 2240.txt, 2240.txt, exception2.txt, jstack1.txt, signatureBuckets-f-104.tar.gz, system.log.2.gz, system.log.gz, test-0.6.x-tables.tar.gz, userChannelFilter-f-210.tar.gz
>
>
> trying to run nodetool scrub hung or (only happened one time) threw the following exception:
> ERROR [CompactionExecutor:1] 2011-02-28 10:26:26,620 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.lang.AssertionError
> at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62)
> at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:538)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Resolved: (CASSANDRA-2240) nodetool scrub hangs or throws an
exception
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis resolved CASSANDRA-2240.
---------------------------------------
Resolution: Fixed
Reviewer: slebresne
committed w/ update to goodRows message and moving the BF/row header messages to debug level
thanks!
> nodetool scrub hangs or throws an exception
> -------------------------------------------
>
> Key: CASSANDRA-2240
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2240
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.7.3
> Environment: using build #314 from hudson
> Reporter: Yaniv Kunda
> Assignee: Jonathan Ellis
> Fix For: 0.7.4
>
> Attachments: 2240-v2.txt, 2240-v3.txt, 2240-v5.patch, 2240-v6.txt, 2240.txt, 2240.txt, exception2.txt, jstack1.txt, signatureBuckets-f-104.tar.gz, system.log.2.gz, system.log.gz, test-0.6.x-tables.tar.gz, userChannelFilter-f-210.tar.gz
>
>
> trying to run nodetool scrub hung or (only happened one time) threw the following exception:
> ERROR [CompactionExecutor:1] 2011-02-28 10:26:26,620 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.lang.AssertionError
> at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62)
> at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:538)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-2240) nodetool scrub hangs or throws an
exception
Posted by "Yaniv Kunda (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yaniv Kunda updated CASSANDRA-2240:
-----------------------------------
Attachment: test-0.6.x-tables.tar.gz
these are the tables that can be used to reproduce the hang
> nodetool scrub hangs or throws an exception
> -------------------------------------------
>
> Key: CASSANDRA-2240
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2240
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Environment: using build #314 from hudson
> Reporter: Yaniv Kunda
> Attachments: test-0.6.x-tables.tar.gz
>
>
> trying to run nodetool scrub hung or (only happened one time) threw the following exception:
> Error occured while scrubbing keyspace <mykeyspace>
> java.util.concurrent.ExecutionException: java.lang.AssertionError
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> at org.apache.cassandra.db.CompactionManager.performScrub(CompactionManager.java:203)
> at org.apache.cassandra.db.ColumnFamilyStore.scrub(ColumnFamilyStore.java:934)
> at org.apache.cassandra.service.StorageService.scrub(StorageService.java:1247)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
> at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
> at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> at sun.rmi.transport.Transport$1.run(Transport.java:177)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
> Caused by: java.lang.AssertionError
> at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62)
> at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:541)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> ... 3 more
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2240) nodetool scrub hangs or throws
an exception
Posted by "Sébastien Giroux (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13000060#comment-13000060 ]
Sébastien Giroux commented on CASSANDRA-2240:
---------------------------------------------
I tried to activate debug log but since this is a live node, it seems there is WAY too much going on. Here is the INFO log:
ERROR 23:10:35,105 Fatal exception in thread Thread[CompactionExecutor:1,1,main]
java.io.IOError: org.apache.cassandra.db.ColumnSerializer$CorruptColumnException: invalid column name length 0
at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:246)
at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:262)
at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:223)
at java.util.concurrent.ConcurrentSkipListMap.buildFromSorted(ConcurrentSkipListMap.java:1521)
at java.util.concurrent.ConcurrentSkipListMap.<init>(ConcurrentSkipListMap.java:1471)
at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:366)
at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:314)
at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:129)
at org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:172)
at org.apache.cassandra.io.PrecompactedRow.<init>(PrecompactedRow.java:78)
at org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIterator.java:139)
at org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:108)
at org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:43)
at org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
at org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
at org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
at org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:448)
at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:123)
at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:93)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
Caused by: org.apache.cassandra.db.ColumnSerializer$CorruptColumnException: invalid column name length 0
at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:68)
at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:242)
... 25 more
INFO 23:10:35,106 Scrubbing SSTableReader(path='/var/lib/cassandra/data/Wajam/Comment-f-710-Data.db')
ERROR 23:10:37,489 Fatal exception in thread Thread[CompactionExecutor:1,1,main]
java.io.EOFException
at java.io.RandomAccessFile.readFully(RandomAccessFile.java:416)
at java.io.RandomAccessFile.readFully(RandomAccessFile.java:394)
at org.apache.cassandra.io.util.BufferedRandomAccessFile.readBytes(BufferedRandomAccessFile.java:268)
at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:310)
at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:284)
at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:540)
at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
And the exception that is output by nodetool scrub:
Error occured while scrubbing keyspace <keyspacename>
java.util.concurrent.ExecutionException: java.io.EOFException
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
at java.util.concurrent.FutureTask.get(FutureTask.java:111)
at org.apache.cassandra.db.CompactionManager.performScrub(CompactionManager.java:203)
at org.apache.cassandra.db.ColumnFamilyStore.scrub(ColumnFamilyStore.java:963)
at org.apache.cassandra.service.StorageService.scrub(StorageService.java:1247)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
at sun.rmi.transport.Transport$1.run(Transport.java:177)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
Caused by: java.io.EOFException
at java.io.RandomAccessFile.readFully(RandomAccessFile.java:416)
at java.io.RandomAccessFile.readFully(RandomAccessFile.java:394)
at org.apache.cassandra.io.util.BufferedRandomAccessFile.readBytes(BufferedRandomAccessFile.java:268)
at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:310)
at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:284)
at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:540)
at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
... 3 more
> nodetool scrub hangs or throws an exception
> -------------------------------------------
>
> Key: CASSANDRA-2240
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2240
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.7.3
> Environment: using build #314 from hudson
> Reporter: Yaniv Kunda
> Assignee: Jonathan Ellis
> Fix For: 0.7.3
>
> Attachments: exception2.txt, jstack1.txt, signatureBuckets-f-104.tar.gz, test-0.6.x-tables.tar.gz, userChannelFilter-f-210.tar.gz
>
>
> trying to run nodetool scrub hung or (only happened one time) threw the following exception:
> Error occured while scrubbing keyspace <mykeyspace>
> java.util.concurrent.ExecutionException: java.lang.AssertionError
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> at org.apache.cassandra.db.CompactionManager.performScrub(CompactionManager.java:203)
> at org.apache.cassandra.db.ColumnFamilyStore.scrub(ColumnFamilyStore.java:934)
> at org.apache.cassandra.service.StorageService.scrub(StorageService.java:1247)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
> at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
> at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> at sun.rmi.transport.Transport$1.run(Transport.java:177)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
> Caused by: java.lang.AssertionError
> at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62)
> at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:541)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> ... 3 more
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Issue Comment Edited: (CASSANDRA-2240) nodetool scrub hangs
or throws an exception
Posted by "Sébastien Giroux (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13000920#comment-13000920 ]
Sébastien Giroux edited comment on CASSANDRA-2240 at 3/1/11 3:36 PM:
---------------------------------------------------------------------
I'm testing it as we speak!
Log are flooded with this:
INFO 15:31:15,783 Invalid bloom filter in SSTableReader(path='/var/lib/cassandra/data/Wajam/WebsiteWajams-f-164-Data.db'); will rebuild it
INFO 15:31:15,783 Invalid bloom filter in SSTableReader(path='/var/lib/cassandra/data/Wajam/WebsiteWajams-f-164-Data.db'); will rebuild it
INFO 15:31:15,783 Invalid bloom filter in SSTableReader(path='/var/lib/cassandra/data/Wajam/WebsiteWajams-f-164-Data.db'); will rebuild it
INFO 15:31:15,783 Invalid bloom filter in SSTableReader(path='/var/lib/cassandra/data/Wajam/WebsiteWajams-f-164-Data.db'); will rebuild it
I'm not sure why it's scrubbing "WebsiteWajams" CF as I asked for "Wajam" (nodetool -h 127.0.0.1 scrub Wajam Wajam) ... Either way, this sstable is messed up too so that's fine.
Hopefully it won't just do like v4 that ran all night and was still echoing invalid bloom filter for the same sstable in the log this morning, Looks like it was stuck in a infinite loop or something.
It looks like it's scrubbing another sstable now so that's already an improvment.
Will keep you guys updated!
was (Author: wajam):
I'm testing it as we speak!
Log are flooded with this:
INFO 15:31:15,783 Invalid bloom filter in SSTableReader(path='/var/lib/cassandra/data/Wajam/WebsiteWajams-f-164-Data.db'); will rebuild it
INFO 15:31:15,783 Invalid bloom filter in SSTableReader(path='/var/lib/cassandra/data/Wajam/WebsiteWajams-f-164-Data.db'); will rebuild it
INFO 15:31:15,783 Invalid bloom filter in SSTableReader(path='/var/lib/cassandra/data/Wajam/WebsiteWajams-f-164-Data.db'); will rebuild it
INFO 15:31:15,783 Invalid bloom filter in SSTableReader(path='/var/lib/cassandra/data/Wajam/WebsiteWajams-f-164-Data.db'); will rebuild it
I'm not sure why it's scrubbing "WebsiteWajams" CF as I asked for "Wajam" (nodetool -h 127.0.0.1 scrub Wajam Wajam) ... Either way, this sstable is messed up too so that's fine.
Hopefully it won't just do like v4 that ran all night and was still echoing invalid bloom filter for the same sstable in the log this morning, Looks like it was stuck in a infinite loop or something.
It looks like it's scrubbing another sstable now so that's already an improvment.
Will keep you guy updated!
> nodetool scrub hangs or throws an exception
> -------------------------------------------
>
> Key: CASSANDRA-2240
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2240
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.7.3
> Environment: using build #314 from hudson
> Reporter: Yaniv Kunda
> Assignee: Jonathan Ellis
> Fix For: 0.7.4
>
> Attachments: 2240-v2.txt, 2240-v3.txt, 2240-v5.patch, 2240-v6.txt, 2240.txt, 2240.txt, exception2.txt, jstack1.txt, signatureBuckets-f-104.tar.gz, system.log.2.gz, system.log.gz, test-0.6.x-tables.tar.gz, userChannelFilter-f-210.tar.gz
>
>
> trying to run nodetool scrub hung or (only happened one time) threw the following exception:
> ERROR [CompactionExecutor:1] 2011-02-28 10:26:26,620 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.lang.AssertionError
> at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62)
> at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:538)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-2240) nodetool scrub hangs or throws an
exception
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sylvain Lebresne updated CASSANDRA-2240:
----------------------------------------
Attachment: 2240-v5.patch
Stepping in while Jonathan takes some well deserved rest and attaching v5 (based on last version attached). This makes the following changes:
* In doScrub(), move the first indexFile.readLong() out of the assert.
* Fix computation of dataStartFromIndex (was missing th 4 or 8 bytes for the data size).
* IndexHelper.defreezeBloomFilter don't leave the file pointer after the bloomFilter for new BF since it reads directly from the file (instead of reading the bytes at once and deserializing from that). Correct this.
* Log if a row has been correctly read the first time but index start and size are different from data start and size (since index should then be manually rebuilt).
* Do a retry if dataStart == dataStartFromIndex but dataSize != dataSizeFromIndex (in case the row size only has been corrupted).
Lastly, a minor remarks: the patch removes a flush in BF.serialize(). Maybe this belongs to another ticket ?
> nodetool scrub hangs or throws an exception
> -------------------------------------------
>
> Key: CASSANDRA-2240
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2240
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.7.3
> Environment: using build #314 from hudson
> Reporter: Yaniv Kunda
> Assignee: Jonathan Ellis
> Fix For: 0.7.4
>
> Attachments: 2240-v2.txt, 2240-v3.txt, 2240-v5.patch, 2240.txt, 2240.txt, exception2.txt, jstack1.txt, signatureBuckets-f-104.tar.gz, system.log.2.gz, system.log.gz, test-0.6.x-tables.tar.gz, userChannelFilter-f-210.tar.gz
>
>
> trying to run nodetool scrub hung or (only happened one time) threw the following exception:
> ERROR [CompactionExecutor:1] 2011-02-28 10:26:26,620 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.lang.AssertionError
> at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62)
> at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:538)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-2240) nodetool scrub hangs or throws an
exception
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-2240:
--------------------------------------
Attachment: 2240-v6.txt
v6 applies Sylvain's fixes to v4 (which mysteriously disappeared from jira)
> nodetool scrub hangs or throws an exception
> -------------------------------------------
>
> Key: CASSANDRA-2240
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2240
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.7.3
> Environment: using build #314 from hudson
> Reporter: Yaniv Kunda
> Assignee: Jonathan Ellis
> Fix For: 0.7.4
>
> Attachments: 2240-v2.txt, 2240-v3.txt, 2240-v5.patch, 2240-v6.txt, 2240.txt, 2240.txt, exception2.txt, jstack1.txt, signatureBuckets-f-104.tar.gz, system.log.2.gz, system.log.gz, test-0.6.x-tables.tar.gz, userChannelFilter-f-210.tar.gz
>
>
> trying to run nodetool scrub hung or (only happened one time) threw the following exception:
> ERROR [CompactionExecutor:1] 2011-02-28 10:26:26,620 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.lang.AssertionError
> at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62)
> at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:538)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-2240) nodetool scrub hangs or throws an
exception
Posted by "Yaniv Kunda (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yaniv Kunda updated CASSANDRA-2240:
-----------------------------------
Attachment: signatureBuckets-f-104.tar.gz
jstack1.txt
Here is the sstable that hangs the scrub process - ran on 0.7 build #325
> nodetool scrub hangs or throws an exception
> -------------------------------------------
>
> Key: CASSANDRA-2240
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2240
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.7.3
> Environment: using build #314 from hudson
> Reporter: Yaniv Kunda
> Assignee: Jonathan Ellis
> Fix For: 0.7.3
>
> Attachments: jstack1.txt, signatureBuckets-f-104.tar.gz, test-0.6.x-tables.tar.gz
>
>
> trying to run nodetool scrub hung or (only happened one time) threw the following exception:
> Error occured while scrubbing keyspace <mykeyspace>
> java.util.concurrent.ExecutionException: java.lang.AssertionError
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> at org.apache.cassandra.db.CompactionManager.performScrub(CompactionManager.java:203)
> at org.apache.cassandra.db.ColumnFamilyStore.scrub(ColumnFamilyStore.java:934)
> at org.apache.cassandra.service.StorageService.scrub(StorageService.java:1247)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
> at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
> at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> at sun.rmi.transport.Transport$1.run(Transport.java:177)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
> Caused by: java.lang.AssertionError
> at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62)
> at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:541)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> ... 3 more
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2240) nodetool scrub hangs or throws
an exception
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12999007#comment-12999007 ]
Jonathan Ellis commented on CASSANDRA-2240:
-------------------------------------------
When I create a keyspace and CF w/ default settings and scrub it, I get
{noformat}
INFO 13:49:17,594 Scrubbing SSTableReader(path='/var/lib/cassandra/data/Keyspace1/userActionUtilsKey-9-Data.db')
INFO 13:49:17,856 Scrub of SSTableReader(path='/var/lib/cassandra/data/Keyspace1/userActionUtilsKey-9-Data.db') complete
{noformat}
> nodetool scrub hangs or throws an exception
> -------------------------------------------
>
> Key: CASSANDRA-2240
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2240
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Environment: using build #314 from hudson
> Reporter: Yaniv Kunda
> Assignee: Jonathan Ellis
> Attachments: test-0.6.x-tables.tar.gz
>
>
> trying to run nodetool scrub hung or (only happened one time) threw the following exception:
> Error occured while scrubbing keyspace <mykeyspace>
> java.util.concurrent.ExecutionException: java.lang.AssertionError
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> at org.apache.cassandra.db.CompactionManager.performScrub(CompactionManager.java:203)
> at org.apache.cassandra.db.ColumnFamilyStore.scrub(ColumnFamilyStore.java:934)
> at org.apache.cassandra.service.StorageService.scrub(StorageService.java:1247)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
> at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
> at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> at sun.rmi.transport.Transport$1.run(Transport.java:177)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
> Caused by: java.lang.AssertionError
> at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62)
> at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:541)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> ... 3 more
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2240) nodetool scrub hangs or throws
an exception
Posted by "Yaniv Kunda (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12999933#comment-12999933 ]
Yaniv Kunda commented on CASSANDRA-2240:
----------------------------------------
Update: restarted server and rerun scrub on the attached signatureBuckets CF, this time did not hung but gave the following error:
INFO [CompactionExecutor:1] 2011-02-27 13:21:27,928 CompactionManager.java (line 511) Scrubbing SSTableReader(path='/vm1/cassandraDB/data/userSimilarity/signatureBuckets-f-104-Data.db')
INFO [CompactionExecutor:1] 2011-02-27 13:21:28,430 SSTableIdentityIterator.java (line 90) Invalid bloom filter in SSTableReader(path='/vm1/cassandraDB/data/userSimilarity/signatureBuckets-f-104-Data.db'); will rebuild it
INFO [CompactionExecutor:1] 2011-02-27 13:21:28,430 SSTableIdentityIterator.java (line 99) Invalid row summary in SSTableReader(path='/vm1/cassandraDB/data/userSimilarity/signatureBuckets-f-104-Data.db'); will rebuild it
ERROR [CompactionExecutor:1] 2011-02-27 13:21:28,434 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
java.io.IOError: java.io.EOFException: attempted to skip 758940931 bytes but only skipped 1400349
at org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:113)
at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:548)
at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
Caused by: java.io.EOFException: attempted to skip 758940931 bytes but only skipped 1400349
at org.apache.cassandra.io.sstable.IndexHelper.skipBloomFilter(IndexHelper.java:51)
at org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:104)
... 8 more
> nodetool scrub hangs or throws an exception
> -------------------------------------------
>
> Key: CASSANDRA-2240
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2240
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.7.3
> Environment: using build #314 from hudson
> Reporter: Yaniv Kunda
> Assignee: Jonathan Ellis
> Fix For: 0.7.3
>
> Attachments: exception2.txt, jstack1.txt, signatureBuckets-f-104.tar.gz, test-0.6.x-tables.tar.gz, userChannelFilter-f-210.tar.gz
>
>
> trying to run nodetool scrub hung or (only happened one time) threw the following exception:
> Error occured while scrubbing keyspace <mykeyspace>
> java.util.concurrent.ExecutionException: java.lang.AssertionError
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> at org.apache.cassandra.db.CompactionManager.performScrub(CompactionManager.java:203)
> at org.apache.cassandra.db.ColumnFamilyStore.scrub(ColumnFamilyStore.java:934)
> at org.apache.cassandra.service.StorageService.scrub(StorageService.java:1247)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
> at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
> at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> at sun.rmi.transport.Transport$1.run(Transport.java:177)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
> Caused by: java.lang.AssertionError
> at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62)
> at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:541)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> ... 3 more
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2240) nodetool scrub hangs or throws
an exception
Posted by "Yaniv Kunda (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12999021#comment-12999021 ]
Yaniv Kunda commented on CASSANDRA-2240:
----------------------------------------
The previously mentioned tons of "Invalid bloom filter in SSTableReader" logs ended with a single stacktrace:
{code}
ERROR [CompactionExecutor:1] 2011-02-24 11:36:19,268 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
java.lang.AssertionError
at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62)
at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627)
at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:541)
at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
{code}
followed by what appears to be standard flushing/switching logs.
> nodetool scrub hangs or throws an exception
> -------------------------------------------
>
> Key: CASSANDRA-2240
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2240
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Environment: using build #314 from hudson
> Reporter: Yaniv Kunda
> Assignee: Jonathan Ellis
> Attachments: test-0.6.x-tables.tar.gz
>
>
> trying to run nodetool scrub hung or (only happened one time) threw the following exception:
> Error occured while scrubbing keyspace <mykeyspace>
> java.util.concurrent.ExecutionException: java.lang.AssertionError
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> at org.apache.cassandra.db.CompactionManager.performScrub(CompactionManager.java:203)
> at org.apache.cassandra.db.ColumnFamilyStore.scrub(ColumnFamilyStore.java:934)
> at org.apache.cassandra.service.StorageService.scrub(StorageService.java:1247)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
> at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
> at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> at sun.rmi.transport.Transport$1.run(Transport.java:177)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
> Caused by: java.lang.AssertionError
> at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62)
> at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:541)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> ... 3 more
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Issue Comment Edited: (CASSANDRA-2240) nodetool scrub hangs
or throws an exception
Posted by "Sébastien Giroux (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13000696#comment-13000696 ]
Sébastien Giroux edited comment on CASSANDRA-2240 at 3/1/11 4:23 AM:
---------------------------------------------------------------------
Looking at the data directory, it looks like it was working as excepted before the exception happened. The tmp SStable is 1.0GB... Looks like we are getting close!
was (Author: wajam):
Looking at the data directory, it looks like it was worked as excepted before the exception happened. The tmp SStable is 1.0GB... Looks like we are getting close!
> nodetool scrub hangs or throws an exception
> -------------------------------------------
>
> Key: CASSANDRA-2240
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2240
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.7.3
> Environment: using build #314 from hudson
> Reporter: Yaniv Kunda
> Assignee: Jonathan Ellis
> Fix For: 0.7.4
>
> Attachments: 2240-v2.txt, 2240.txt, exception2.txt, jstack1.txt, signatureBuckets-f-104.tar.gz, system.log.2.gz, system.log.gz, test-0.6.x-tables.tar.gz, userChannelFilter-f-210.tar.gz
>
>
> trying to run nodetool scrub hung or (only happened one time) threw the following exception:
> ERROR [CompactionExecutor:1] 2011-02-28 10:26:26,620 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.lang.AssertionError
> at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62)
> at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:538)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2240) nodetool scrub hangs or throws
an exception
Posted by "Sébastien Giroux (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13000720#comment-13000720 ]
Sébastien Giroux commented on CASSANDRA-2240:
---------------------------------------------
Now I'm getting this:
ERROR 04:58:53,338 Error reading index file. Scrub does not (yet) know how to recover from corrupt index files; you can try rebuilding it offline. See http://www.mail-archive.com/user@cassandra.apache.org/msg03325.html
ERROR 04:58:53,338 Fatal exception in thread Thread[CompactionExecutor:1,1,main]
java.lang.RuntimeException: java.io.EOFException
at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:565)
at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
Caused by: java.io.EOFException
at java.io.RandomAccessFile.readFully(RandomAccessFile.java:416)
at java.io.RandomAccessFile.readFully(RandomAccessFile.java:394)
at org.apache.cassandra.io.util.BufferedRandomAccessFile.readBytes(BufferedRandomAccessFile.java:268)
at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:310)
at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:284)
at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:559)
... 7 more
And
root@WajamCassandra12:/usr/share/cassandra# nodetool -h 127.0.0.1 scrub Wajam Wajam
Error occured while scrubbing keyspace Wajam
java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.io.EOFException
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
at java.util.concurrent.FutureTask.get(FutureTask.java:111)
at org.apache.cassandra.db.CompactionManager.performScrub(CompactionManager.java:204)
at org.apache.cassandra.db.ColumnFamilyStore.scrub(ColumnFamilyStore.java:963)
at org.apache.cassandra.service.StorageService.scrub(StorageService.java:1247)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
at sun.rmi.transport.Transport$1.run(Transport.java:177)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
Caused by: java.lang.RuntimeException: java.io.EOFException
at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:565)
at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
... 3 more
Caused by: java.io.EOFException
at java.io.RandomAccessFile.readFully(RandomAccessFile.java:416)
at java.io.RandomAccessFile.readFully(RandomAccessFile.java:394)
at org.apache.cassandra.io.util.BufferedRandomAccessFile.readBytes(BufferedRandomAccessFile.java:268)
at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:310)
at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:284)
at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:559)
... 7 more
> nodetool scrub hangs or throws an exception
> -------------------------------------------
>
> Key: CASSANDRA-2240
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2240
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.7.3
> Environment: using build #314 from hudson
> Reporter: Yaniv Kunda
> Assignee: Jonathan Ellis
> Fix For: 0.7.4
>
> Attachments: 2240-v2.txt, 2240-v3.txt, 2240.txt, exception2.txt, jstack1.txt, signatureBuckets-f-104.tar.gz, system.log.2.gz, system.log.gz, test-0.6.x-tables.tar.gz, userChannelFilter-f-210.tar.gz
>
>
> trying to run nodetool scrub hung or (only happened one time) threw the following exception:
> ERROR [CompactionExecutor:1] 2011-02-28 10:26:26,620 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.lang.AssertionError
> at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62)
> at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:538)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Reopened: (CASSANDRA-2240) nodetool scrub hangs or throws an
exception
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis reopened CASSANDRA-2240:
---------------------------------------
Assignee: Jonathan Ellis
... on closer inspection it does have colon-delimited keys, but scrub doesn't see them.
> nodetool scrub hangs or throws an exception
> -------------------------------------------
>
> Key: CASSANDRA-2240
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2240
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Environment: using build #314 from hudson
> Reporter: Yaniv Kunda
> Assignee: Jonathan Ellis
> Attachments: test-0.6.x-tables.tar.gz
>
>
> trying to run nodetool scrub hung or (only happened one time) threw the following exception:
> Error occured while scrubbing keyspace <mykeyspace>
> java.util.concurrent.ExecutionException: java.lang.AssertionError
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> at org.apache.cassandra.db.CompactionManager.performScrub(CompactionManager.java:203)
> at org.apache.cassandra.db.ColumnFamilyStore.scrub(ColumnFamilyStore.java:934)
> at org.apache.cassandra.service.StorageService.scrub(StorageService.java:1247)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
> at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
> at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> at sun.rmi.transport.Transport$1.run(Transport.java:177)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
> Caused by: java.lang.AssertionError
> at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62)
> at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:541)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> ... 3 more
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2240) nodetool scrub hangs or throws
an exception
Posted by "Yaniv Kunda (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12999020#comment-12999020 ]
Yaniv Kunda commented on CASSANDRA-2240:
----------------------------------------
I had to go back 26 x 20MiB logs consisting mostly of lines such as:
{code} INFO [CompactionExecutor:1] 2011-02-24 11:18:10,262 SSTableIdentityIterator.java (line 90) Invalid bloom filter in SSTableReader(path='/vm1/cassandraDB/data/Keyspace2/ruleGroup-f-243-Data.db'); will rebuild it {code}
which were preceded by a bunch of NegativeArraySizeExceptions, which were after the initialization log:
{code}
INFO [main] 2011-02-24 11:08:43,595 AbstractCassandraDaemon.java (line 77) Logging initialized
INFO [main] 2011-02-24 11:08:43,605 AbstractCassandraDaemon.java (line 97) Heap size: 8330936320/8331984896
INFO [main] 2011-02-24 11:08:43,606 CLibrary.java (line 61) JNA not found. Native methods will be disabled.
INFO [main] 2011-02-24 11:08:43,613 DatabaseDescriptor.java (line 121) Loading settings from file:/usr/local/apache-cassandra-2011-02-24_02-21-51/conf/cassandra.yaml
INFO [main] 2011-02-24 11:08:43,821 DatabaseDescriptor.java (line 181) DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
INFO [main] 2011-02-24 11:08:43,893 SSTable.java (line 147) Deleted /vm1/cassandraDB/data/system/LocationInfo-f-210
INFO [main] 2011-02-24 11:08:43,894 SSTable.java (line 147) Deleted /vm1/cassandraDB/data/system/LocationInfo-f-212
INFO [main] 2011-02-24 11:08:43,894 SSTable.java (line 147) Deleted /vm1/cassandraDB/data/system/LocationInfo-f-211
INFO [main] 2011-02-24 11:08:43,894 SSTable.java (line 147) Deleted /vm1/cassandraDB/data/system/LocationInfo-f-209
INFO [main] 2011-02-24 11:08:43,919 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/system/IndexInfo-f-5
INFO [main] 2011-02-24 11:08:43,937 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/system/Schema-f-89
INFO [main] 2011-02-24 11:08:43,945 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/system/Migrations-f-85
INFO [main] 2011-02-24 11:08:43,947 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/system/Migrations-f-87
INFO [main] 2011-02-24 11:08:43,948 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/system/Migrations-f-86
INFO [main] 2011-02-24 11:08:43,951 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/system/LocationInfo-f-213
INFO [main] 2011-02-24 11:08:43,953 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/system/LocationInfo-f-214
INFO [main] 2011-02-24 11:08:43,982 DatabaseDescriptor.java (line 461) Loading schema version b6fdb590-3e9e-11e0-8d0e-34b74a661156
WARN [main] 2011-02-24 11:08:44,128 DatabaseDescriptor.java (line 493) Schema definitions were defined both locally and in cassandra.yaml. Definitions in cassandra.yaml were ignored.
INFO [main] 2011-02-24 11:08:44,188 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/Keyspace2/recommenders-f-176
INFO [main] 2011-02-24 11:08:44,191 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/Keyspace2/recommenders-f-177
INFO [main] 2011-02-24 11:08:44,192 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/Keyspace2/recommenders-f-178
INFO [main] 2011-02-24 11:08:44,197 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/Keyspace2/businessRule-f-200
INFO [main] 2011-02-24 11:08:44,204 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/Keyspace2/recommendTo-f-92
INFO [main] 2011-02-24 11:08:44,205 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/Keyspace2/recommendTo-f-93
INFO [main] 2011-02-24 11:08:44,209 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/Keyspace2/followers-f-175
INFO [main] 2011-02-24 11:08:44,213 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/Keyspace2/names-f-165
INFO [main] 2011-02-24 11:08:44,218 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/Keyspace2/ruleGroup-f-244
INFO [main] 2011-02-24 11:08:44,218 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/Keyspace2/ruleGroup-f-243
INFO [main] 2011-02-24 11:08:44,225 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/Keyspace2/recommendFrom-f-109
INFO [main] 2011-02-24 11:08:44,233 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/Keyspace2/channels-f-2866
INFO [main] 2011-02-24 11:08:44,262 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/Keyspace2/channels-f-2867
INFO [main] 2011-02-24 11:08:44,286 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/Keyspace2/recommendAgain-f-1425
INFO [main] 2011-02-24 11:08:44,293 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/Keyspace1/userProfile-f-816
INFO [main] 2011-02-24 11:08:44,294 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/Keyspace1/userProfile-f-815
INFO [main] 2011-02-24 11:08:44,301 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/Keyspace1/showData-f-6153
INFO [main] 2011-02-24 11:08:44,302 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/Keyspace1/showData-f-6039
INFO [main] 2011-02-24 11:08:44,443 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/Keyspace1/showData-f-6130
INFO [main] 2011-02-24 11:08:44,498 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/Keyspace1/showData-f-6040
INFO [main] 2011-02-24 11:08:44,591 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/Keyspace1/showData-f-6143
INFO [main] 2011-02-24 11:08:44,608 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/Keyspace1/showData-f-6036
INFO [main] 2011-02-24 11:08:44,676 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/Keyspace1/showData-f-6035
INFO [main] 2011-02-24 11:08:44,852 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/Keyspace1/showData-f-6038
INFO [main] 2011-02-24 11:08:44,953 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/Keyspace1/showData-f-6134
INFO [main] 2011-02-24 11:08:44,960 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/Keyspace1/showData-f-6114
INFO [main] 2011-02-24 11:08:44,976 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/Keyspace1/showData-f-6096
INFO [main] 2011-02-24 11:08:45,040 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/Keyspace1/showData-f-6147
INFO [main] 2011-02-24 11:08:45,048 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/Keyspace1/showData-f-6152
INFO [main] 2011-02-24 11:08:45,062 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/Keyspace1/cloudData-f-5990
INFO [main] 2011-02-24 11:08:45,087 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/Keyspace1/cloudData-f-6063
INFO [main] 2011-02-24 11:08:45,109 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/Keyspace1/cloudData-f-6109
INFO [main] 2011-02-24 11:08:45,110 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/Keyspace1/cloudData-f-5991
INFO [main] 2011-02-24 11:08:45,136 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/Keyspace1/cloudData-f-6108
INFO [main] 2011-02-24 11:08:45,154 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/Keyspace1/userChannelFilter-f-207
INFO [main] 2011-02-24 11:08:45,154 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/Keyspace1/userChannelFilter-f-208
INFO [main] 2011-02-24 11:08:45,162 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/Keyspace1/popularCloudIds-f-203
INFO [main] 2011-02-24 11:08:45,166 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/Keyspace1/userActions-f-1160
INFO [main] 2011-02-24 11:08:45,177 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/Keyspace1/headends-f-5793
INFO [main] 2011-02-24 11:08:45,195 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/Keyspace1/headends-f-5801
INFO [main] 2011-02-24 11:08:45,196 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/Keyspace1/headends-f-5761
INFO [main] 2011-02-24 11:08:45,202 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/Keyspace1/headends-f-5800
INFO [main] 2011-02-24 11:08:45,213 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/Keyspace1/userActionLikes-f-10
INFO [main] 2011-02-24 11:08:45,218 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/Keyspace1/userAttributes-f-2517
INFO [main] 2011-02-24 11:08:45,219 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/Keyspace1/userAttributes-f-2533
INFO [main] 2011-02-24 11:08:45,221 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/Keyspace1/userAttributes-f-2532
INFO [main] 2011-02-24 11:08:45,230 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/userSimilarity/signatureBuckets-102
INFO [main] 2011-02-24 11:08:45,607 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/userSimilarity/signatureBuckets-103
INFO [main] 2011-02-24 11:08:45,799 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/userSimilarity/signatureBuckets-88
INFO [main] 2011-02-24 11:08:46,319 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/userSimilarity/signatureBuckets-93
INFO [main] 2011-02-24 11:08:46,846 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/userSimilarity/signatureBuckets-63
INFO [main] 2011-02-24 11:08:49,879 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/userSimilarity/signatureBuckets-83
INFO [main] 2011-02-24 11:08:51,340 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/userSimilarity/userSignatures-22
INFO [main] 2011-02-24 11:08:51,351 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/userSimilarity/userSignatures-23
INFO [main] 2011-02-24 11:08:51,355 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/userSimilarity/userSignatures-17
INFO [main] 2011-02-24 11:08:51,385 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/userSimilarity/userSignatures-24
INFO [main] 2011-02-24 11:08:51,390 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/userSimilarity/signatureFunctions-5
INFO [main] 2011-02-24 11:08:51,394 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/generalUtils/userActionUtilsKey-9
INFO [main] 2011-02-24 11:08:51,396 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/generalUtils/facebookShowIds-f-37
INFO [main] 2011-02-24 11:08:51,398 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/userSimilarity2/signatureBuckets-6
INFO [main] 2011-02-24 11:08:51,527 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/userSimilarity2/signatureBuckets-5
INFO [main] 2011-02-24 11:08:51,903 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/userSimilarity2/userSignatures-1
INFO [main] 2011-02-24 11:08:52,006 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/userSimilarity2/userSignatures-e-3
INFO [main] 2011-02-24 11:08:52,009 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/userSimilarity2/userSignatures-2
INFO [main] 2011-02-24 11:08:52,013 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/userSimilarity2/signatureFunctions-2
INFO [main] 2011-02-24 11:08:52,013 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/userSimilarity2/signatureFunctions-1
INFO [main] 2011-02-24 11:08:52,014 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/userSimilarity2/signatureFunctions-e-3
INFO [main] 2011-02-24 11:08:52,019 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/realTimeSocial/facebookActions-f-186
INFO [main] 2011-02-24 11:08:52,019 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/realTimeSocial/facebookActions-f-185
INFO [main] 2011-02-24 11:08:52,022 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/realTimeSocial/watchingUsers-f-477
INFO [main] 2011-02-24 11:08:52,038 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/realTimeSocial/showState-e-35
INFO [main] 2011-02-24 11:08:52,039 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/realTimeSocial/showState-e-37
INFO [main] 2011-02-24 11:08:52,039 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/realTimeSocial/showState-e-36
INFO [main] 2011-02-24 11:08:52,042 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/realTimeSocial/watchingUserHistory-f-462
INFO [main] 2011-02-24 11:08:52,042 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/realTimeSocial/watchingUserHistory-f-461
INFO [main] 2011-02-24 11:08:52,043 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/realTimeSocial/watchingUserHistory-f-463
INFO [main] 2011-02-24 11:08:52,045 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/chats/chatUsers-f-129
INFO [main] 2011-02-24 11:08:52,046 SSTableReader.java (line 154) Opening /vm1/cassandraDB/data/chats/chatHistory-f-129
INFO [main] 2011-02-24 11:08:52,055 CommitLogSegment.java (line 50) Creating new commitlog segment /vm1/cassandraDB/commitlog/CommitLog-1298538532055.log
INFO [main] 2011-02-24 11:08:52,061 CommitLog.java (line 155) Replaying /vm1/cassandraDB/commitlog/CommitLog-1298537386811.log
INFO [main] 2011-02-24 11:08:52,073 CommitLog.java (line 311) Finished reading /vm1/cassandraDB/commitlog/CommitLog-1298537386811.log
INFO [main] 2011-02-24 11:08:52,074 ColumnFamilyStore.java (line 666) switching in a fresh Memtable for recommendAgain at CommitLogContext(file='/vm1/cassandraDB/commitlog/CommitLog-1298538532055.log', position=0)
INFO [main] 2011-02-24 11:08:52,076 ColumnFamilyStore.java (line 977) Enqueuing flush of Memtable-recommendAgain@205025794(636 bytes, 7 operations)
INFO [FlushWriter:1] 2011-02-24 11:08:52,077 Memtable.java (line 157) Writing Memtable-recommendAgain@205025794(636 bytes, 7 operations)
INFO [FlushWriter:1] 2011-02-24 11:08:52,113 Memtable.java (line 164) Completed flushing /vm1/cassandraDB/data/Keyspace2/recommendAgain-f-1426-Data.db (859 bytes)
INFO [main] 2011-02-24 11:08:52,118 CommitLog.java (line 163) Log replay complete
INFO [main] 2011-02-24 11:08:52,131 StorageService.java (line 354) Cassandra version: 2011-02-24_02-21-51
INFO [main] 2011-02-24 11:08:52,131 StorageService.java (line 355) Thrift API version: 19.4.0
INFO [main] 2011-02-24 11:08:52,132 StorageService.java (line 368) Loading persisted ring state
INFO [main] 2011-02-24 11:08:52,132 StorageService.java (line 404) Starting up server gossip
INFO [main] 2011-02-24 11:08:52,137 ColumnFamilyStore.java (line 666) switching in a fresh Memtable for LocationInfo at CommitLogContext(file='/vm1/cassandraDB/commitlog/CommitLog-1298538532055.log', position=148)
INFO [main] 2011-02-24 11:08:52,137 ColumnFamilyStore.java (line 977) Enqueuing flush of Memtable-LocationInfo@1068086436(29 bytes, 1 operations)
INFO [FlushWriter:1] 2011-02-24 11:08:52,138 Memtable.java (line 157) Writing Memtable-LocationInfo@1068086436(29 bytes, 1 operations)
INFO [FlushWriter:1] 2011-02-24 11:08:52,178 Memtable.java (line 164) Completed flushing /vm1/cassandraDB/data/system/LocationInfo-f-215-Data.db (80 bytes)
INFO [main] 2011-02-24 11:08:52,198 StorageService.java (line 468) Using saved token 51653040247566871911249877869558549493
INFO [main] 2011-02-24 11:08:52,199 ColumnFamilyStore.java (line 666) switching in a fresh Memtable for LocationInfo at CommitLogContext(file='/vm1/cassandraDB/commitlog/CommitLog-1298538532055.log', position=444)
INFO [main] 2011-02-24 11:08:52,199 ColumnFamilyStore.java (line 977) Enqueuing flush of Memtable-LocationInfo@832074392(53 bytes, 2 operations)
INFO [FlushWriter:1] 2011-02-24 11:08:52,199 Memtable.java (line 157) Writing Memtable-LocationInfo@832074392(53 bytes, 2 operations)
INFO [FlushWriter:1] 2011-02-24 11:08:52,230 Memtable.java (line 164) Completed flushing /vm1/cassandraDB/data/system/LocationInfo-f-216-Data.db (163 bytes)
INFO [CompactionExecutor:1] 2011-02-24 11:08:52,231 CompactionManager.java (line 395) Compacting [SSTableReader(path='/vm1/cassandraDB/data/system/LocationInfo-f-213-Data.db'),SSTableReader(path='/vm1/cassandraDB/data/system/LocationInfo-f-214-Data.db'),SSTableReader(path='/vm1/cassandraDB/data/system/LocationInfo-f-215-Data.db'),SSTableReader(path='/vm1/cassandraDB/data/system/LocationInfo-f-216-Data.db')]
INFO [main] 2011-02-24 11:08:52,233 Mx4jTool.java (line 72) Will not load MX4J, mx4j-tools.jar is not in the classpath
INFO [CompactionExecutor:1] 2011-02-24 11:08:52,292 CompactionManager.java (line 482) Compacted to /vm1/cassandraDB/data/system/LocationInfo-tmp-f-217-Data.db. 851 to 445 (~52% of original) bytes for 3 keys. Time: 61ms.
INFO [main] 2011-02-24 11:08:52,296 CassandraDaemon.java (line 112) Binding thrift service to /0.0.0.0:9160
INFO [main] 2011-02-24 11:08:52,298 CassandraDaemon.java (line 126) Using TFastFramedTransport with a max frame size of 15728640 bytes.
INFO [Thread-3] 2011-02-24 11:08:52,300 CassandraDaemon.java (line 153) Listening for thrift clients...
{code}
> nodetool scrub hangs or throws an exception
> -------------------------------------------
>
> Key: CASSANDRA-2240
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2240
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Environment: using build #314 from hudson
> Reporter: Yaniv Kunda
> Assignee: Jonathan Ellis
> Attachments: test-0.6.x-tables.tar.gz
>
>
> trying to run nodetool scrub hung or (only happened one time) threw the following exception:
> Error occured while scrubbing keyspace <mykeyspace>
> java.util.concurrent.ExecutionException: java.lang.AssertionError
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> at org.apache.cassandra.db.CompactionManager.performScrub(CompactionManager.java:203)
> at org.apache.cassandra.db.ColumnFamilyStore.scrub(ColumnFamilyStore.java:934)
> at org.apache.cassandra.service.StorageService.scrub(StorageService.java:1247)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
> at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
> at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> at sun.rmi.transport.Transport$1.run(Transport.java:177)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
> Caused by: java.lang.AssertionError
> at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62)
> at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:541)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> ... 3 more
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2240) nodetool scrub hangs or throws
an exception
Posted by "Sébastien Giroux (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13000718#comment-13000718 ]
Sébastien Giroux commented on CASSANDRA-2240:
---------------------------------------------
I will try v3 soon and let you know. While looking at the patch, I found a typo:
logger.warn("No valid rows found while scrubbing " + sstable + "; it is marked for deltion now. If you want to attempt manual recovery, you can find a copy in the pre-scrub snapshot");
Should be "deletion" instead of "deltion"... no big deal :)
> nodetool scrub hangs or throws an exception
> -------------------------------------------
>
> Key: CASSANDRA-2240
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2240
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.7.3
> Environment: using build #314 from hudson
> Reporter: Yaniv Kunda
> Assignee: Jonathan Ellis
> Fix For: 0.7.4
>
> Attachments: 2240-v2.txt, 2240-v3.txt, 2240.txt, exception2.txt, jstack1.txt, signatureBuckets-f-104.tar.gz, system.log.2.gz, system.log.gz, test-0.6.x-tables.tar.gz, userChannelFilter-f-210.tar.gz
>
>
> trying to run nodetool scrub hung or (only happened one time) threw the following exception:
> ERROR [CompactionExecutor:1] 2011-02-28 10:26:26,620 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.lang.AssertionError
> at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62)
> at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:538)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-2240) nodetool scrub hangs or throws an
exception
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-2240:
--------------------------------------
Attachment: 2240-v2.txt
2211/2216 would leave valid row keys written, which is where yours is corrupt. That is why I think this is something else.
Here is a v2 though that will catch the error and try the next row from the index.
> nodetool scrub hangs or throws an exception
> -------------------------------------------
>
> Key: CASSANDRA-2240
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2240
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.7.3
> Environment: using build #314 from hudson
> Reporter: Yaniv Kunda
> Assignee: Jonathan Ellis
> Fix For: 0.7.4
>
> Attachments: 2240-v2.txt, 2240.txt, exception2.txt, jstack1.txt, signatureBuckets-f-104.tar.gz, system.log.2.gz, system.log.gz, test-0.6.x-tables.tar.gz, userChannelFilter-f-210.tar.gz
>
>
> trying to run nodetool scrub hung or (only happened one time) threw the following exception:
> ERROR [CompactionExecutor:1] 2011-02-28 10:26:26,620 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.lang.AssertionError
> at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62)
> at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:538)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-2240) nodetool scrub hangs or throws an
exception
Posted by "Yaniv Kunda (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yaniv Kunda updated CASSANDRA-2240:
-----------------------------------
Attachment: system.log.gz
Attached a system log leading up to a "java.io.EOFException: attempted to skip X bytes but only skipped Y" error during a scrub of a very big (probably uncompactable) sstable
> nodetool scrub hangs or throws an exception
> -------------------------------------------
>
> Key: CASSANDRA-2240
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2240
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.7.3
> Environment: using build #314 from hudson
> Reporter: Yaniv Kunda
> Assignee: Jonathan Ellis
> Fix For: 0.7.4
>
> Attachments: exception2.txt, jstack1.txt, signatureBuckets-f-104.tar.gz, system.log.gz, test-0.6.x-tables.tar.gz, userChannelFilter-f-210.tar.gz
>
>
> trying to run nodetool scrub hung or (only happened one time) threw the following exception:
> Error occured while scrubbing keyspace <mykeyspace>
> java.util.concurrent.ExecutionException: java.lang.AssertionError
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> at org.apache.cassandra.db.CompactionManager.performScrub(CompactionManager.java:203)
> at org.apache.cassandra.db.ColumnFamilyStore.scrub(ColumnFamilyStore.java:934)
> at org.apache.cassandra.service.StorageService.scrub(StorageService.java:1247)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
> at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
> at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> at sun.rmi.transport.Transport$1.run(Transport.java:177)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
> Caused by: java.lang.AssertionError
> at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62)
> at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:541)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> ... 3 more
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2240) nodetool scrub hangs or throws
an exception
Posted by "Yaniv Kunda (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13001088#comment-13001088 ]
Yaniv Kunda commented on CASSANDRA-2240:
----------------------------------------
Thank you so much, my sstables are finally clean!
> nodetool scrub hangs or throws an exception
> -------------------------------------------
>
> Key: CASSANDRA-2240
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2240
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.7.3
> Environment: using build #314 from hudson
> Reporter: Yaniv Kunda
> Assignee: Jonathan Ellis
> Fix For: 0.7.3
>
> Attachments: 2240-v2.txt, 2240-v3.txt, 2240-v5.patch, 2240-v6.txt, 2240.txt, 2240.txt, exception2.txt, jstack1.txt, signatureBuckets-f-104.tar.gz, system.log.2.gz, system.log.gz, test-0.6.x-tables.tar.gz, userChannelFilter-f-210.tar.gz
>
>
> trying to run nodetool scrub hung or (only happened one time) threw the following exception:
> ERROR [CompactionExecutor:1] 2011-02-28 10:26:26,620 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.lang.AssertionError
> at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62)
> at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:538)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2240) nodetool scrub hangs or throws
an exception
Posted by "Sébastien Giroux (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13000691#comment-13000691 ]
Sébastien Giroux commented on CASSANDRA-2240:
---------------------------------------------
Now it seems to hang for a while on a 1.4GB sstable. Eventually I get this exception...
ERROR 03:56:29,413 Fatal exception in thread Thread[CompactionExecutor:1,1,main]
java.lang.NullPointerException
at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:569)
at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
> nodetool scrub hangs or throws an exception
> -------------------------------------------
>
> Key: CASSANDRA-2240
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2240
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.7.3
> Environment: using build #314 from hudson
> Reporter: Yaniv Kunda
> Assignee: Jonathan Ellis
> Fix For: 0.7.4
>
> Attachments: 2240-v2.txt, 2240.txt, exception2.txt, jstack1.txt, signatureBuckets-f-104.tar.gz, system.log.2.gz, system.log.gz, test-0.6.x-tables.tar.gz, userChannelFilter-f-210.tar.gz
>
>
> trying to run nodetool scrub hung or (only happened one time) threw the following exception:
> ERROR [CompactionExecutor:1] 2011-02-28 10:26:26,620 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.lang.AssertionError
> at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62)
> at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:538)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2240) nodetool scrub hangs or throws
an exception
Posted by "Sébastien Giroux (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13000538#comment-13000538 ]
Sébastien Giroux commented on CASSANDRA-2240:
---------------------------------------------
I have a 71MB SS table that scrub fail to fix so maybe you would be interested in having it ? There is private data in there so I could upload it somewhere so only you can download it. Let me know how we can work together.
Thank you!
> nodetool scrub hangs or throws an exception
> -------------------------------------------
>
> Key: CASSANDRA-2240
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2240
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.7.3
> Environment: using build #314 from hudson
> Reporter: Yaniv Kunda
> Assignee: Jonathan Ellis
> Fix For: 0.7.4
>
> Attachments: exception2.txt, jstack1.txt, signatureBuckets-f-104.tar.gz, system.log.2.gz, system.log.gz, test-0.6.x-tables.tar.gz, userChannelFilter-f-210.tar.gz
>
>
> trying to run nodetool scrub hung or (only happened one time) threw the following exception:
> ERROR [CompactionExecutor:1] 2011-02-28 10:26:26,620 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.lang.AssertionError
> at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62)
> at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:538)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2240) nodetool scrub hangs or throws
an exception
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12999038#comment-12999038 ]
Jonathan Ellis commented on CASSANDRA-2240:
-------------------------------------------
I really can't help with just a stacktrace and none of the logs leading up to it.
Remember that scrub snapshots before it does its thing, so it's easy to restore the pre-scrubbed versions and try again.
> nodetool scrub hangs or throws an exception
> -------------------------------------------
>
> Key: CASSANDRA-2240
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2240
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Environment: using build #314 from hudson
> Reporter: Yaniv Kunda
> Assignee: Jonathan Ellis
> Attachments: test-0.6.x-tables.tar.gz
>
>
> trying to run nodetool scrub hung or (only happened one time) threw the following exception:
> Error occured while scrubbing keyspace <mykeyspace>
> java.util.concurrent.ExecutionException: java.lang.AssertionError
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> at org.apache.cassandra.db.CompactionManager.performScrub(CompactionManager.java:203)
> at org.apache.cassandra.db.ColumnFamilyStore.scrub(ColumnFamilyStore.java:934)
> at org.apache.cassandra.service.StorageService.scrub(StorageService.java:1247)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
> at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
> at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> at sun.rmi.transport.Transport$1.run(Transport.java:177)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
> Caused by: java.lang.AssertionError
> at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62)
> at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:541)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> ... 3 more
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Issue Comment Edited: (CASSANDRA-2240) nodetool scrub hangs
or throws an exception
Posted by "Yaniv Kunda (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12999022#comment-12999022 ]
Yaniv Kunda edited comment on CASSANDRA-2240 at 2/24/11 8:29 PM:
-----------------------------------------------------------------
I now restarted the server again, and two things happened:
1) performing a scrub on the example CF and a bunch of other CFs succeeded without hanging
2) performing a scrub on a different CF resulting in the following stack trace (on different retries):
{code}
ERROR [CompactionExecutor:1] 2011-02-24 22:02:30,329 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
java.io.IOError: java.io.EOFException
at org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:113)
at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:549)
at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
Caused by: java.io.EOFException
at java.io.RandomAccessFile.readInt(RandomAccessFile.java:776)
at org.apache.cassandra.io.sstable.IndexHelper.skipBloomFilter(IndexHelper.java:47)
at org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:104)
... 8 more
{code}
3) and on another CF:
{code}
ERROR [CompactionExecutor:1] 2011-02-24 22:28:28,002 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
java.lang.AssertionError
at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62)
at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627)
at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:541)
at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
{code}
was (Author: kunda):
I now restarted the server again, and two things happened:
1) performing a scrub on the example CF and a bunch of other CFs succeeded without hanging
2) performing a scrub on a different CF resulting in the following stack trace (on different retries):
{code}
ERROR [CompactionExecutor:1] 2011-02-24 22:02:30,329 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
java.io.IOError: java.io.EOFException
at org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:113)
at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:549)
at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
Caused by: java.io.EOFException
at java.io.RandomAccessFile.readInt(RandomAccessFile.java:776)
at org.apache.cassandra.io.sstable.IndexHelper.skipBloomFilter(IndexHelper.java:47)
at org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:104)
... 8 more
{code}
> nodetool scrub hangs or throws an exception
> -------------------------------------------
>
> Key: CASSANDRA-2240
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2240
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Environment: using build #314 from hudson
> Reporter: Yaniv Kunda
> Assignee: Jonathan Ellis
> Attachments: test-0.6.x-tables.tar.gz
>
>
> trying to run nodetool scrub hung or (only happened one time) threw the following exception:
> Error occured while scrubbing keyspace <mykeyspace>
> java.util.concurrent.ExecutionException: java.lang.AssertionError
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> at org.apache.cassandra.db.CompactionManager.performScrub(CompactionManager.java:203)
> at org.apache.cassandra.db.ColumnFamilyStore.scrub(ColumnFamilyStore.java:934)
> at org.apache.cassandra.service.StorageService.scrub(StorageService.java:1247)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
> at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
> at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> at sun.rmi.transport.Transport$1.run(Transport.java:177)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
> Caused by: java.lang.AssertionError
> at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62)
> at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:541)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> ... 3 more
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2240) nodetool scrub hangs or throws
an exception
Posted by "Sébastien Giroux (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13000938#comment-13000938 ]
Sébastien Giroux commented on CASSANDRA-2240:
---------------------------------------------
Yep I did see error reading index file before that.
Any chance to include this in 0.7.3 release ? I'm sure it would help a ton of people :)
> nodetool scrub hangs or throws an exception
> -------------------------------------------
>
> Key: CASSANDRA-2240
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2240
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.7.3
> Environment: using build #314 from hudson
> Reporter: Yaniv Kunda
> Assignee: Jonathan Ellis
> Fix For: 0.7.4
>
> Attachments: 2240-v2.txt, 2240-v3.txt, 2240-v5.patch, 2240-v6.txt, 2240.txt, 2240.txt, exception2.txt, jstack1.txt, signatureBuckets-f-104.tar.gz, system.log.2.gz, system.log.gz, test-0.6.x-tables.tar.gz, userChannelFilter-f-210.tar.gz
>
>
> trying to run nodetool scrub hung or (only happened one time) threw the following exception:
> ERROR [CompactionExecutor:1] 2011-02-28 10:26:26,620 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.lang.AssertionError
> at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62)
> at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:538)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-2240) nodetool scrub hangs or throws an
exception
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-2240:
--------------------------------------
Remaining Estimate: 16h
Original Estimate: 16h
> nodetool scrub hangs or throws an exception
> -------------------------------------------
>
> Key: CASSANDRA-2240
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2240
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.7.3
> Environment: using build #314 from hudson
> Reporter: Yaniv Kunda
> Assignee: Jonathan Ellis
> Fix For: 0.7.3
>
> Attachments: 2240-v2.txt, 2240-v3.txt, 2240-v5.patch, 2240-v6.txt, 2240.txt, 2240.txt, exception2.txt, jstack1.txt, signatureBuckets-f-104.tar.gz, system.log.2.gz, system.log.gz, test-0.6.x-tables.tar.gz, userChannelFilter-f-210.tar.gz
>
> Original Estimate: 16h
> Remaining Estimate: 16h
>
> trying to run nodetool scrub hung or (only happened one time) threw the following exception:
> ERROR [CompactionExecutor:1] 2011-02-28 10:26:26,620 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.lang.AssertionError
> at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62)
> at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:538)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2240) nodetool scrub hangs or throws
an exception
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12998900#comment-12998900 ]
Jonathan Ellis commented on CASSANDRA-2240:
-------------------------------------------
what is going on in the compactionmanager when it's "hung?" (use jstack)
> nodetool scrub hangs or throws an exception
> -------------------------------------------
>
> Key: CASSANDRA-2240
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2240
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Environment: using build #314 from hudson
> Reporter: Yaniv Kunda
>
> trying to run nodetool scrub hung or (only happened one time) threw the following exception:
> Error occured while scrubbing keyspace <mykeyspace>
> java.util.concurrent.ExecutionException: java.lang.AssertionError
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> at org.apache.cassandra.db.CompactionManager.performScrub(CompactionManager.java:203)
> at org.apache.cassandra.db.ColumnFamilyStore.scrub(ColumnFamilyStore.java:934)
> at org.apache.cassandra.service.StorageService.scrub(StorageService.java:1247)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
> at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
> at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> at sun.rmi.transport.Transport$1.run(Transport.java:177)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
> Caused by: java.lang.AssertionError
> at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62)
> at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:541)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> ... 3 more
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2240) nodetool scrub hangs or throws
an exception
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13000953#comment-13000953 ]
Hudson commented on CASSANDRA-2240:
-----------------------------------
Integrated in Cassandra-0.7 #336 (See [https://hudson.apache.org/hudson/job/Cassandra-0.7/336/])
make nodetool scrub more robust
patch by jbellis and slebresne; tested by Sébastien Giroux for CASSANDRA-2240
> nodetool scrub hangs or throws an exception
> -------------------------------------------
>
> Key: CASSANDRA-2240
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2240
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.7.3
> Environment: using build #314 from hudson
> Reporter: Yaniv Kunda
> Assignee: Jonathan Ellis
> Fix For: 0.7.3
>
> Attachments: 2240-v2.txt, 2240-v3.txt, 2240-v5.patch, 2240-v6.txt, 2240.txt, 2240.txt, exception2.txt, jstack1.txt, signatureBuckets-f-104.tar.gz, system.log.2.gz, system.log.gz, test-0.6.x-tables.tar.gz, userChannelFilter-f-210.tar.gz
>
>
> trying to run nodetool scrub hung or (only happened one time) threw the following exception:
> ERROR [CompactionExecutor:1] 2011-02-28 10:26:26,620 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.lang.AssertionError
> at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62)
> at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:538)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2240) nodetool scrub hangs or throws
an exception
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13000916#comment-13000916 ]
Sylvain Lebresne commented on CASSANDRA-2240:
---------------------------------------------
Alright, v6 looks good, +1. Though, we may want to wait to see if it works alright for Sebastien too.
> nodetool scrub hangs or throws an exception
> -------------------------------------------
>
> Key: CASSANDRA-2240
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2240
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.7.3
> Environment: using build #314 from hudson
> Reporter: Yaniv Kunda
> Assignee: Jonathan Ellis
> Fix For: 0.7.4
>
> Attachments: 2240-v2.txt, 2240-v3.txt, 2240-v5.patch, 2240-v6.txt, 2240.txt, 2240.txt, exception2.txt, jstack1.txt, signatureBuckets-f-104.tar.gz, system.log.2.gz, system.log.gz, test-0.6.x-tables.tar.gz, userChannelFilter-f-210.tar.gz
>
>
> trying to run nodetool scrub hung or (only happened one time) threw the following exception:
> ERROR [CompactionExecutor:1] 2011-02-28 10:26:26,620 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.lang.AssertionError
> at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62)
> at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:538)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2240) nodetool scrub hangs or throws
an exception
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13000880#comment-13000880 ]
Sylvain Lebresne commented on CASSANDRA-2240:
---------------------------------------------
It's a git patch, git apply or 'patch -p1 -i 2240-v5.patch' should do the trick (it does here). It's based on current cassandra-0.7 branch.
> nodetool scrub hangs or throws an exception
> -------------------------------------------
>
> Key: CASSANDRA-2240
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2240
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.7.3
> Environment: using build #314 from hudson
> Reporter: Yaniv Kunda
> Assignee: Jonathan Ellis
> Fix For: 0.7.4
>
> Attachments: 2240-v2.txt, 2240-v3.txt, 2240-v5.patch, 2240.txt, 2240.txt, exception2.txt, jstack1.txt, signatureBuckets-f-104.tar.gz, system.log.2.gz, system.log.gz, test-0.6.x-tables.tar.gz, userChannelFilter-f-210.tar.gz
>
>
> trying to run nodetool scrub hung or (only happened one time) threw the following exception:
> ERROR [CompactionExecutor:1] 2011-02-28 10:26:26,620 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.lang.AssertionError
> at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62)
> at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:538)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2240) nodetool scrub hangs or throws
an exception
Posted by "Sébastien Giroux (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13000635#comment-13000635 ]
Sébastien Giroux commented on CASSANDRA-2240:
---------------------------------------------
Jonathan,
I'm pretty positive I'm running into CASSANDRA-2216.
Applied the patch and still having the same exception, maybe this is related to CASSANDRA-2256 ?
ERROR 00:51:55,214 Fatal exception in thread Thread[CompactionExecutor:1,1,main]
java.io.EOFException
at java.io.RandomAccessFile.readFully(RandomAccessFile.java:416)
at java.io.RandomAccessFile.readFully(RandomAccessFile.java:394)
at org.apache.cassandra.io.util.BufferedRandomAccessFile.readBytes(BufferedRandomAccessFile.java:268)
at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:310)
at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:284)
at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:541)
at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
> nodetool scrub hangs or throws an exception
> -------------------------------------------
>
> Key: CASSANDRA-2240
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2240
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.7.3
> Environment: using build #314 from hudson
> Reporter: Yaniv Kunda
> Assignee: Jonathan Ellis
> Fix For: 0.7.4
>
> Attachments: 2240.txt, exception2.txt, jstack1.txt, signatureBuckets-f-104.tar.gz, system.log.2.gz, system.log.gz, test-0.6.x-tables.tar.gz, userChannelFilter-f-210.tar.gz
>
>
> trying to run nodetool scrub hung or (only happened one time) threw the following exception:
> ERROR [CompactionExecutor:1] 2011-02-28 10:26:26,620 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.lang.AssertionError
> at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62)
> at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:538)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-2240) nodetool scrub hangs or throws an
exception
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-2240:
--------------------------------------
Fix Version/s: (was: 0.7.4)
0.7.3
> nodetool scrub hangs or throws an exception
> -------------------------------------------
>
> Key: CASSANDRA-2240
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2240
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.7.3
> Environment: using build #314 from hudson
> Reporter: Yaniv Kunda
> Assignee: Jonathan Ellis
> Fix For: 0.7.3
>
> Attachments: 2240-v2.txt, 2240-v3.txt, 2240-v5.patch, 2240-v6.txt, 2240.txt, 2240.txt, exception2.txt, jstack1.txt, signatureBuckets-f-104.tar.gz, system.log.2.gz, system.log.gz, test-0.6.x-tables.tar.gz, userChannelFilter-f-210.tar.gz
>
>
> trying to run nodetool scrub hung or (only happened one time) threw the following exception:
> ERROR [CompactionExecutor:1] 2011-02-28 10:26:26,620 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.lang.AssertionError
> at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62)
> at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:538)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2240) nodetool scrub hangs or throws
an exception
Posted by "Yaniv Kunda (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12999022#comment-12999022 ]
Yaniv Kunda commented on CASSANDRA-2240:
----------------------------------------
I now restarted the server again, and two things happened:
1) performing a scrub on the example CF and a bunch of other CFs succeeded without hanging
2) performing a scrub on a different CF resulting in the following stack trace:
{code}
ERROR [CompactionExecutor:1] 2011-02-24 22:02:30,329 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
java.io.IOError: java.io.EOFException
at org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:113)
at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:549)
at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
Caused by: java.io.EOFException
at java.io.RandomAccessFile.readInt(RandomAccessFile.java:776)
at org.apache.cassandra.io.sstable.IndexHelper.skipBloomFilter(IndexHelper.java:47)
at org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:104)
... 8 more
{code}
> nodetool scrub hangs or throws an exception
> -------------------------------------------
>
> Key: CASSANDRA-2240
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2240
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Environment: using build #314 from hudson
> Reporter: Yaniv Kunda
> Assignee: Jonathan Ellis
> Attachments: test-0.6.x-tables.tar.gz
>
>
> trying to run nodetool scrub hung or (only happened one time) threw the following exception:
> Error occured while scrubbing keyspace <mykeyspace>
> java.util.concurrent.ExecutionException: java.lang.AssertionError
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> at org.apache.cassandra.db.CompactionManager.performScrub(CompactionManager.java:203)
> at org.apache.cassandra.db.ColumnFamilyStore.scrub(ColumnFamilyStore.java:934)
> at org.apache.cassandra.service.StorageService.scrub(StorageService.java:1247)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
> at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
> at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> at sun.rmi.transport.Transport$1.run(Transport.java:177)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
> Caused by: java.lang.AssertionError
> at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62)
> at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:541)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> ... 3 more
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Resolved: (CASSANDRA-2240) nodetool scrub hangs or throws an
exception
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis resolved CASSANDRA-2240.
---------------------------------------
Resolution: Won't Fix
This is not a valid sstable. It claims (from its lack of version string) that it contains encoded row keys, meaning <token>:<key>, but it actually does not. scrub can't help you with that.
> nodetool scrub hangs or throws an exception
> -------------------------------------------
>
> Key: CASSANDRA-2240
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2240
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Environment: using build #314 from hudson
> Reporter: Yaniv Kunda
> Attachments: test-0.6.x-tables.tar.gz
>
>
> trying to run nodetool scrub hung or (only happened one time) threw the following exception:
> Error occured while scrubbing keyspace <mykeyspace>
> java.util.concurrent.ExecutionException: java.lang.AssertionError
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> at org.apache.cassandra.db.CompactionManager.performScrub(CompactionManager.java:203)
> at org.apache.cassandra.db.ColumnFamilyStore.scrub(ColumnFamilyStore.java:934)
> at org.apache.cassandra.service.StorageService.scrub(StorageService.java:1247)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
> at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
> at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> at sun.rmi.transport.Transport$1.run(Transport.java:177)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
> Caused by: java.lang.AssertionError
> at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62)
> at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:541)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> ... 3 more
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2240) nodetool scrub hangs or throws
an exception
Posted by "Sébastien Giroux (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13000732#comment-13000732 ]
Sébastien Giroux commented on CASSANDRA-2240:
---------------------------------------------
Scrub is running with v4 right now, we will see what happen, no exception so far. There is these things being displayed about fifty millions time in the log tho:) :
INFO 05:33:29,521 Invalid row summary in SSTableReader(path='/var/lib/cassandra/data/Wajam/WebsiteWajams-f-146-Data.db'); will rebuild it
INFO 05:33:29,548 Invalid bloom filter in SSTableReader(path='/var/lib/cassandra/data/Wajam/WebsiteWajams-f-146-Data.db'); will rebuild it
INFO 05:33:29,859 Invalid row summary in SSTableReader(path='/var/lib/cassandra/data/Wajam/WebsiteWajams-f-146-Data.db'); will rebuild it
INFO 05:33:29,882 Invalid bloom filter in SSTableReader(path='/var/lib/cassandra/data/Wajam/WebsiteWajams-f-146-Data.db'); will rebuild it
INFO 05:33:31,223 Invalid row summary in SSTableReader(path='/var/lib/cassandra/data/Wajam/WebsiteWajams-f-146-Data.db'); will rebuild it
INFO 05:33:31,249 Invalid bloom filter in SSTableReader(path='/var/lib/cassandra/data/Wajam/WebsiteWajams-f-146-Data.db'); will rebuild it
INFO 05:33:31,644 Invalid row summary in SSTableReader(path='/var/lib/cassandra/data/Wajam/WebsiteWajams-f-146-Data.db'); will rebuild it
INFO 05:33:31,678 Invalid bloom filter in SSTableReader(path='/var/lib/cassandra/data/Wajam/WebsiteWajams-f-146-Data.db'); will rebuild it
INFO 05:33:32,102 Invalid row summary in SSTableReader(path='/var/lib/cassandra/data/Wajam/WebsiteWajams-f-146-Data.db'); will rebuild it
INFO 05:33:32,129 Invalid bloom filter in SSTableReader(path='/var/lib/cassandra/data/Wajam/WebsiteWajams-f-146-Data.db'); will rebuild it
INFO 05:33:32,393 Invalid row summary in SSTableReader(path='/var/lib/cassandra/data/Wajam/WebsiteWajams-f-146-Data.db'); will rebuild it
INFO 05:33:32,425 Invalid bloom filter in SSTableReader(path='/var/lib/cassandra/data/Wajam/WebsiteWajams-f-146-Data.db'); will rebuild it
INFO 05:33:32,713 Invalid row summary in SSTableReader(path='/var/lib/cassandra/data/Wajam/WebsiteWajams-f-146-Data.db'); will rebuild it
INFO 05:33:32,734 Invalid bloom filter in SSTableReader(path='/var/lib/cassandra/data/Wajam/WebsiteWajams-f-146-Data.db'); will rebuild it
INFO 05:33:33,004 Invalid row summary in SSTableReader(path='/var/lib/cassandra/data/Wajam/WebsiteWajams-f-146-Data.db'); will rebuild it
INFO 05:33:33,031 Invalid bloom filter in SSTableReader(path='/var/lib/cassandra/data/Wajam/WebsiteWajams-f-146-Data.db'); will rebuild it
INFO 05:33:33,305 Invalid row summary in SSTableReader(path='/var/lib/cassandra/data/Wajam/WebsiteWajams-f-146-Data.db'); will rebuild it
INFO 05:33:33,331 Invalid bloom filter in SSTableReader(path='/var/lib/cassandra/data/Wajam/WebsiteWajams-f-146-Data.db'); will rebuild it
INFO 05:33:33,599 Invalid row summary in SSTableReader(path='/var/lib/cassandra/data/Wajam/WebsiteWajams-f-146-Data.db'); will rebuild it
INFO 05:33:33,623 Invalid bloom filter in SSTableReader(path='/var/lib/cassandra/data/Wajam/WebsiteWajams-f-146-Data.db'); will rebuild it
INFO 05:33:33,897 Invalid row summary in SSTableReader(path='/var/lib/cassandra/data/Wajam/WebsiteWajams-f-146-Data.db'); will rebuild it
INFO 05:33:33,917 Invalid bloom filter in SSTableReader(path='/var/lib/cassandra/data/Wajam/WebsiteWajams-f-146-Data.db'); will rebuild it
INFO 05:33:34,330 Invalid row summary in SSTableReader(path='/var/lib/cassandra/data/Wajam/WebsiteWajams-f-146-Data.db'); will rebuild it
INFO 05:33:34,375 Invalid bloom filter in SSTableReader(path='/var/lib/cassandra/data/Wajam/WebsiteWajams-f-146-Data.db'); will rebuild it
INFO 05:33:35,504 Invalid row summary in SSTableReader(path='/var/lib/cassandra/data/Wajam/WebsiteWajams-f-146-Data.db'); will rebuild it
INFO 05:33:35,525 Invalid bloom filter in SSTableReader(path='/var/lib/cassandra/data/Wajam/WebsiteWajams-f-146-Data.db'); will rebuild it
INFO 05:33:35,933 Invalid row summary in SSTableReader(path='/var/lib/cassandra/data/Wajam/WebsiteWajams-f-146-Data.db'); will rebuild it
Either way, as long as it fix my ss table I will be happy :) I will let you know in the morning if there is any more issue
Thank you!
> nodetool scrub hangs or throws an exception
> -------------------------------------------
>
> Key: CASSANDRA-2240
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2240
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.7.3
> Environment: using build #314 from hudson
> Reporter: Yaniv Kunda
> Assignee: Jonathan Ellis
> Fix For: 0.7.4
>
> Attachments: 2240-v2.txt, 2240-v3.txt, 2240.txt, 2240.txt, exception2.txt, jstack1.txt, signatureBuckets-f-104.tar.gz, system.log.2.gz, system.log.gz, test-0.6.x-tables.tar.gz, userChannelFilter-f-210.tar.gz
>
>
> trying to run nodetool scrub hung or (only happened one time) threw the following exception:
> ERROR [CompactionExecutor:1] 2011-02-28 10:26:26,620 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.lang.AssertionError
> at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62)
> at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:538)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2240) nodetool scrub hangs or throws
an exception
Posted by "Yaniv Kunda (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12999943#comment-12999943 ]
Yaniv Kunda commented on CASSANDRA-2240:
----------------------------------------
Repeating the process now results in a different failure on a different and very big (>500MB) sstable of the same signatureBuckets CF:
ERROR [CompactionExecutor:1] 2011-02-27 13:26:01,307 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
java.lang.AssertionError
at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62)
at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627)
at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:538)
at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
I think it has something to do with [~jbellis]'s comment about missing colon delimiter -
if scrub cannot solve this, is there any way to fix this problem?
> nodetool scrub hangs or throws an exception
> -------------------------------------------
>
> Key: CASSANDRA-2240
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2240
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.7.3
> Environment: using build #314 from hudson
> Reporter: Yaniv Kunda
> Assignee: Jonathan Ellis
> Fix For: 0.7.3
>
> Attachments: exception2.txt, jstack1.txt, signatureBuckets-f-104.tar.gz, test-0.6.x-tables.tar.gz, userChannelFilter-f-210.tar.gz
>
>
> trying to run nodetool scrub hung or (only happened one time) threw the following exception:
> Error occured while scrubbing keyspace <mykeyspace>
> java.util.concurrent.ExecutionException: java.lang.AssertionError
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> at org.apache.cassandra.db.CompactionManager.performScrub(CompactionManager.java:203)
> at org.apache.cassandra.db.ColumnFamilyStore.scrub(ColumnFamilyStore.java:934)
> at org.apache.cassandra.service.StorageService.scrub(StorageService.java:1247)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
> at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
> at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> at sun.rmi.transport.Transport$1.run(Transport.java:177)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
> Caused by: java.lang.AssertionError
> at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62)
> at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:541)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> ... 3 more
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-2240) nodetool scrub hangs or throws an
exception
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-2240:
--------------------------------------
Attachment: 2240.txt
v4 attached. (might be my last for the night; getting late here. but I'll check back first thing in the morning)
> nodetool scrub hangs or throws an exception
> -------------------------------------------
>
> Key: CASSANDRA-2240
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2240
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.7.3
> Environment: using build #314 from hudson
> Reporter: Yaniv Kunda
> Assignee: Jonathan Ellis
> Fix For: 0.7.4
>
> Attachments: 2240-v2.txt, 2240-v3.txt, 2240.txt, 2240.txt, exception2.txt, jstack1.txt, signatureBuckets-f-104.tar.gz, system.log.2.gz, system.log.gz, test-0.6.x-tables.tar.gz, userChannelFilter-f-210.tar.gz
>
>
> trying to run nodetool scrub hung or (only happened one time) threw the following exception:
> ERROR [CompactionExecutor:1] 2011-02-28 10:26:26,620 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.lang.AssertionError
> at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62)
> at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:538)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Issue Comment Edited: (CASSANDRA-2240) nodetool scrub hangs
or throws an exception
Posted by "Sébastien Giroux (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13000923#comment-13000923 ]
Sébastien Giroux edited comment on CASSANDRA-2240 at 3/1/11 3:50 PM:
---------------------------------------------------------------------
INFO 15:42:52,690 Scrub of SSTableReader(path='/var/lib/cassandra/data/Wajam/WebsiteWajams-f-155-Data.db') complete: 3271113 in new sstable
"3271113 in new sstable"... I'm not sure it's 3271113 what... Looks like "good rows" according to the patch, might want to specify! :)
I think it's the first time I see this line tho so it's good news. Is there any way the "invalid bloom filter, will rebuild it" line can only appear once per sstable ? I'm guessing its generating useless IO writing about 1000 of those lines/second in the log!
This should be definitly included in 0.7.3 which hopefully will be released soon!
Good work guys, very appreciated :)
was (Author: wajam):
INFO 15:42:52,690 Scrub of SSTableReader(path='/var/lib/cassandra/data/Wajam/WebsiteWajams-f-155-Data.db') complete: 3271113 in new sstable
"3271113 in new sstable"... I'm not sure it's 3271113 what... might want to specify! :)
I think it's the first time I see this line tho so it's good news. Is there any way the "invalid bloom filter, will rebuild it" line can only appear once per sstable ? I'm guessing its generating useless IO writing about 1000 of those lines/second in the log!
Good work guys, very appreciated :)
> nodetool scrub hangs or throws an exception
> -------------------------------------------
>
> Key: CASSANDRA-2240
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2240
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.7.3
> Environment: using build #314 from hudson
> Reporter: Yaniv Kunda
> Assignee: Jonathan Ellis
> Fix For: 0.7.4
>
> Attachments: 2240-v2.txt, 2240-v3.txt, 2240-v5.patch, 2240-v6.txt, 2240.txt, 2240.txt, exception2.txt, jstack1.txt, signatureBuckets-f-104.tar.gz, system.log.2.gz, system.log.gz, test-0.6.x-tables.tar.gz, userChannelFilter-f-210.tar.gz
>
>
> trying to run nodetool scrub hung or (only happened one time) threw the following exception:
> ERROR [CompactionExecutor:1] 2011-02-28 10:26:26,620 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.lang.AssertionError
> at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62)
> at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:538)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2240) nodetool scrub hangs or throws
an exception
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12999005#comment-12999005 ]
Jonathan Ellis commented on CASSANDRA-2240:
-------------------------------------------
also, can you attach the system.log from when you started cassandra? we're trying to figure out why it's using the new-version BloomFilterSerializer, when it should be using LegacyBloomFilterSerializer.
> nodetool scrub hangs or throws an exception
> -------------------------------------------
>
> Key: CASSANDRA-2240
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2240
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Environment: using build #314 from hudson
> Reporter: Yaniv Kunda
> Assignee: Jonathan Ellis
> Attachments: test-0.6.x-tables.tar.gz
>
>
> trying to run nodetool scrub hung or (only happened one time) threw the following exception:
> Error occured while scrubbing keyspace <mykeyspace>
> java.util.concurrent.ExecutionException: java.lang.AssertionError
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> at org.apache.cassandra.db.CompactionManager.performScrub(CompactionManager.java:203)
> at org.apache.cassandra.db.ColumnFamilyStore.scrub(ColumnFamilyStore.java:934)
> at org.apache.cassandra.service.StorageService.scrub(StorageService.java:1247)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
> at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
> at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> at sun.rmi.transport.Transport$1.run(Transport.java:177)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
> Caused by: java.lang.AssertionError
> at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62)
> at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:541)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> ... 3 more
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2240) nodetool scrub hangs or throws
an exception
Posted by "Yaniv Kunda (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12998916#comment-12998916 ]
Yaniv Kunda commented on CASSANDRA-2240:
----------------------------------------
Here's the trace I got (narrowed down):
"CompactionExecutor:1" prio=10 tid=0x000000001eb67800 nid=0x7fdb runnable [0x0000000040dc4000]
java.lang.Thread.State: RUNNABLE
at java.io.DataInputStream.readFully(DataInputStream.java:195)
at java.io.DataInputStream.readLong(DataInputStream.java:416)
at org.apache.cassandra.utils.BloomFilterSerializer.deserialize(BloomFilterSerializer.java:51)
at org.apache.cassandra.utils.BloomFilterSerializer.deserialize(BloomFilterSerializer.java:30)
at org.apache.cassandra.io.sstable.IndexHelper.defreezeBloomFilter(IndexHelper.java:108)
at org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:86)
at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:549)
at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- <0x00002aaabd5f71c8> (a java.util.concurrent.ThreadPoolExecutor$Worker)
- <0x00002aaabd6cd008> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
> nodetool scrub hangs or throws an exception
> -------------------------------------------
>
> Key: CASSANDRA-2240
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2240
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Environment: using build #314 from hudson
> Reporter: Yaniv Kunda
> Assignee: Jonathan Ellis
> Attachments: test-0.6.x-tables.tar.gz
>
>
> trying to run nodetool scrub hung or (only happened one time) threw the following exception:
> Error occured while scrubbing keyspace <mykeyspace>
> java.util.concurrent.ExecutionException: java.lang.AssertionError
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> at org.apache.cassandra.db.CompactionManager.performScrub(CompactionManager.java:203)
> at org.apache.cassandra.db.ColumnFamilyStore.scrub(ColumnFamilyStore.java:934)
> at org.apache.cassandra.service.StorageService.scrub(StorageService.java:1247)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
> at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
> at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> at sun.rmi.transport.Transport$1.run(Transport.java:177)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
> Caused by: java.lang.AssertionError
> at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62)
> at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:541)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> ... 3 more
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-2240) nodetool scrub hangs or throws an
exception
Posted by "Yaniv Kunda (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yaniv Kunda updated CASSANDRA-2240:
-----------------------------------
Attachment: userChannelFilter-f-210.tar.gz
exception2.txt
And here is another sstable that doesn't hang but throws an exception
> nodetool scrub hangs or throws an exception
> -------------------------------------------
>
> Key: CASSANDRA-2240
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2240
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.7.3
> Environment: using build #314 from hudson
> Reporter: Yaniv Kunda
> Assignee: Jonathan Ellis
> Fix For: 0.7.3
>
> Attachments: exception2.txt, jstack1.txt, signatureBuckets-f-104.tar.gz, test-0.6.x-tables.tar.gz, userChannelFilter-f-210.tar.gz
>
>
> trying to run nodetool scrub hung or (only happened one time) threw the following exception:
> Error occured while scrubbing keyspace <mykeyspace>
> java.util.concurrent.ExecutionException: java.lang.AssertionError
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> at org.apache.cassandra.db.CompactionManager.performScrub(CompactionManager.java:203)
> at org.apache.cassandra.db.ColumnFamilyStore.scrub(ColumnFamilyStore.java:934)
> at org.apache.cassandra.service.StorageService.scrub(StorageService.java:1247)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
> at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
> at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> at sun.rmi.transport.Transport$1.run(Transport.java:177)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
> Caused by: java.lang.AssertionError
> at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62)
> at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:541)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> ... 3 more
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2240) nodetool scrub hangs or throws
an exception
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13000057#comment-13000057 ]
Jonathan Ellis commented on CASSANDRA-2240:
-------------------------------------------
can someone post a log file from a failed scrub attempt? ideally with log level debug but I'll take info if that's what you have.
> nodetool scrub hangs or throws an exception
> -------------------------------------------
>
> Key: CASSANDRA-2240
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2240
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.7.3
> Environment: using build #314 from hudson
> Reporter: Yaniv Kunda
> Assignee: Jonathan Ellis
> Fix For: 0.7.3
>
> Attachments: exception2.txt, jstack1.txt, signatureBuckets-f-104.tar.gz, test-0.6.x-tables.tar.gz, userChannelFilter-f-210.tar.gz
>
>
> trying to run nodetool scrub hung or (only happened one time) threw the following exception:
> Error occured while scrubbing keyspace <mykeyspace>
> java.util.concurrent.ExecutionException: java.lang.AssertionError
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> at org.apache.cassandra.db.CompactionManager.performScrub(CompactionManager.java:203)
> at org.apache.cassandra.db.ColumnFamilyStore.scrub(ColumnFamilyStore.java:934)
> at org.apache.cassandra.service.StorageService.scrub(StorageService.java:1247)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
> at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
> at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> at sun.rmi.transport.Transport$1.run(Transport.java:177)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
> Caused by: java.lang.AssertionError
> at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62)
> at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:541)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> ... 3 more
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2240) nodetool scrub hangs or throws
an exception
Posted by "Sébastien Giroux (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13000895#comment-13000895 ]
Sébastien Giroux commented on CASSANDRA-2240:
---------------------------------------------
Yep, that worked. So it's currently running with your patch, and so far I get no output from scrub after running for like 5 mins, I usually get the "Scrubbing SSTABLE" in the first min so I'm not sure if that's normal. There is the "tmp" files in the data directory but looks like they are stuck at 0 byte.
> nodetool scrub hangs or throws an exception
> -------------------------------------------
>
> Key: CASSANDRA-2240
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2240
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.7.3
> Environment: using build #314 from hudson
> Reporter: Yaniv Kunda
> Assignee: Jonathan Ellis
> Fix For: 0.7.4
>
> Attachments: 2240-v2.txt, 2240-v3.txt, 2240-v5.patch, 2240-v6.txt, 2240.txt, 2240.txt, exception2.txt, jstack1.txt, signatureBuckets-f-104.tar.gz, system.log.2.gz, system.log.gz, test-0.6.x-tables.tar.gz, userChannelFilter-f-210.tar.gz
>
>
> trying to run nodetool scrub hung or (only happened one time) threw the following exception:
> ERROR [CompactionExecutor:1] 2011-02-28 10:26:26,620 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.lang.AssertionError
> at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62)
> at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:538)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira