You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Anthony Molinaro (JIRA)" <ji...@apache.org> on 2010/07/19 20:10:02 UTC
[jira] Created: (CASSANDRA-1300) Assert in RandomPartitioner causes
unending compaction
Assert in RandomPartitioner causes unending compaction
------------------------------------------------------
Key: CASSANDRA-1300
URL: https://issues.apache.org/jira/browse/CASSANDRA-1300
Project: Cassandra
Issue Type: Bug
Affects Versions: 0.6.3
Reporter: Anthony Molinaro
I have a set of 4 sstables which equal about 600GB of data, whenever they are compacted after about 2 days I get this exception
ERROR [COMPACTION-POOL:1] 2010-07-17 19:09:49,784 CassandraDaemon.java (line 83) Uncaught exception in thread Thread[COMPACTION-POOL:1,5,main]
java.util.concurrent.ExecutionException: java.lang.StringIndexOutOfBoundsException: String index out of range: -1
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
at java.util.concurrent.FutureTask.get(FutureTask.java:83)
at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(Debug+gableThreadPoolExecutor.java:86)
at org.apache.cassandra.db.CompactionManager$CompactionExecutor.afterExecute(CompactionManager.java:577)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)Caused by: java.lang.StringIndexOutOfBoundsException: String index out of range: -1
at java.lang.String.substring(String.java:1938)
at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:50)
at org.apache.cassandra.io.IteratingRow.<init>(IteratingRow.java:48)
at org.apache.cassandra.io.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:136)
at org.apache.cassandra.io.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:113)
at org.apache.cassandra.io.SSTableScanner.next(SSTableScanner.java:105)
at org.apache.cassandra.io.SSTableScanner.next(SSTableScanner.java:34)
at org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:284)
at org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326)
at org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230)
at org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:68)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:135)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:130)
at org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
at org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
at org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:295)
at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:102)
at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:83)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
... 2 more
After this expection, the compaction starts over and then crashes again. This led to several thousand outstanding sstables hanging out waiting to be compacted.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1300) Assert in RandomPartitioner
causes unending compaction
Posted by "Anthony Molinaro (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12890004#action_12890004 ]
Anthony Molinaro commented on CASSANDRA-1300:
---------------------------------------------
So looking at the other bug it seems like the way to detect a bad sstable is to run sstablekeys and see if there is an exception thrown?
If so, I did that on the 4 -Data.db files and got no exceptions. However, it looks like the -Index.db and -Filter.db files are also sstables, so should I run on each of them? Also, I assume its then okay to just delete the bad sstable (ie, if I find a bad -Index.db delete it, then repair), or do I need to delete all 3 files (ie, if 3-Index.db is bad, do I delete 3-Data.db, 3-Filter.db and 3-Index.db)?
> Assert in RandomPartitioner causes unending compaction
> ------------------------------------------------------
>
> Key: CASSANDRA-1300
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1300
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.6.3
> Reporter: Anthony Molinaro
>
> I have a set of 4 sstables which equal about 600GB of data, whenever they are compacted after about 2 days I get this exception
> ERROR [COMPACTION-POOL:1] 2010-07-17 19:09:49,784 CassandraDaemon.java (line 83) Uncaught exception in thread Thread[COMPACTION-POOL:1,5,main]
> java.util.concurrent.ExecutionException: java.lang.StringIndexOutOfBoundsException: String index out of range: -1
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
> at java.util.concurrent.FutureTask.get(FutureTask.java:83)
> at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(Debug+gableThreadPoolExecutor.java:86)
> at org.apache.cassandra.db.CompactionManager$CompactionExecutor.afterExecute(CompactionManager.java:577)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)Caused by: java.lang.StringIndexOutOfBoundsException: String index out of range: -1
> at java.lang.String.substring(String.java:1938)
> at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:50)
> at org.apache.cassandra.io.IteratingRow.<init>(IteratingRow.java:48)
> at org.apache.cassandra.io.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:136)
> at org.apache.cassandra.io.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:113)
> at org.apache.cassandra.io.SSTableScanner.next(SSTableScanner.java:105)
> at org.apache.cassandra.io.SSTableScanner.next(SSTableScanner.java:34)
> at org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:284)
> at org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326)
> at org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230)
> at org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:68)
> at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:135)
> at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:130)
> at org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
> at org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
> at org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:295)
> at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:102)
> at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:83)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> ... 2 more
> After this expection, the compaction starts over and then crashes again. This led to several thousand outstanding sstables hanging out waiting to be compacted.
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1300) Assert in RandomPartitioner
causes unending compaction
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12890037#action_12890037 ]
Jonathan Ellis commented on CASSANDRA-1300:
-------------------------------------------
sstable2json is what you use to test the data file
> Assert in RandomPartitioner causes unending compaction
> ------------------------------------------------------
>
> Key: CASSANDRA-1300
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1300
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.6.3
> Reporter: Anthony Molinaro
>
> I have a set of 4 sstables which equal about 600GB of data, whenever they are compacted after about 2 days I get this exception
> ERROR [COMPACTION-POOL:1] 2010-07-17 19:09:49,784 CassandraDaemon.java (line 83) Uncaught exception in thread Thread[COMPACTION-POOL:1,5,main]
> java.util.concurrent.ExecutionException: java.lang.StringIndexOutOfBoundsException: String index out of range: -1
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
> at java.util.concurrent.FutureTask.get(FutureTask.java:83)
> at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(Debug+gableThreadPoolExecutor.java:86)
> at org.apache.cassandra.db.CompactionManager$CompactionExecutor.afterExecute(CompactionManager.java:577)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)Caused by: java.lang.StringIndexOutOfBoundsException: String index out of range: -1
> at java.lang.String.substring(String.java:1938)
> at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:50)
> at org.apache.cassandra.io.IteratingRow.<init>(IteratingRow.java:48)
> at org.apache.cassandra.io.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:136)
> at org.apache.cassandra.io.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:113)
> at org.apache.cassandra.io.SSTableScanner.next(SSTableScanner.java:105)
> at org.apache.cassandra.io.SSTableScanner.next(SSTableScanner.java:34)
> at org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:284)
> at org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326)
> at org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230)
> at org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:68)
> at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:135)
> at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:130)
> at org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
> at org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
> at org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:295)
> at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:102)
> at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:83)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> ... 2 more
> After this expection, the compaction starts over and then crashes again. This led to several thousand outstanding sstables hanging out waiting to be compacted.
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (CASSANDRA-1300) Assert in RandomPartitioner
causes unending compaction
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis resolved CASSANDRA-1300.
---------------------------------------
Resolution: Duplicate
Duplicate of CASSANDRA-1170.
Short version: delete the bad sstable and repair.
> Assert in RandomPartitioner causes unending compaction
> ------------------------------------------------------
>
> Key: CASSANDRA-1300
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1300
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.6.3
> Reporter: Anthony Molinaro
>
> I have a set of 4 sstables which equal about 600GB of data, whenever they are compacted after about 2 days I get this exception
> ERROR [COMPACTION-POOL:1] 2010-07-17 19:09:49,784 CassandraDaemon.java (line 83) Uncaught exception in thread Thread[COMPACTION-POOL:1,5,main]
> java.util.concurrent.ExecutionException: java.lang.StringIndexOutOfBoundsException: String index out of range: -1
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
> at java.util.concurrent.FutureTask.get(FutureTask.java:83)
> at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(Debug+gableThreadPoolExecutor.java:86)
> at org.apache.cassandra.db.CompactionManager$CompactionExecutor.afterExecute(CompactionManager.java:577)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)Caused by: java.lang.StringIndexOutOfBoundsException: String index out of range: -1
> at java.lang.String.substring(String.java:1938)
> at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:50)
> at org.apache.cassandra.io.IteratingRow.<init>(IteratingRow.java:48)
> at org.apache.cassandra.io.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:136)
> at org.apache.cassandra.io.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:113)
> at org.apache.cassandra.io.SSTableScanner.next(SSTableScanner.java:105)
> at org.apache.cassandra.io.SSTableScanner.next(SSTableScanner.java:34)
> at org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:284)
> at org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326)
> at org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230)
> at org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:68)
> at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:135)
> at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:130)
> at org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
> at org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
> at org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:295)
> at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:102)
> at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:83)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> ... 2 more
> After this expection, the compaction starts over and then crashes again. This led to several thousand outstanding sstables hanging out waiting to be compacted.
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.