Bad SSTable killing compaction?

[Anthony Molinaro <an...@alumni.caltech.edu>]

Hi,

  I'm seeing an error on one of my nodes after it spends about 2 days
compacting, after which it starts over on the same compaction and
after another 2 days fails again.  This causes this node to get horribly
behind on compaction.  I've gotten it to clear up a bunch of files by
upping the minimum compaction threshold so it skips this file for a
while, but babysitting this node is no fun.  This has been happening
for a while, but I hoped the upgrade from 0.5.1 to 0.6.3 would fix
it.  Anyway, is this a known problem and are there any work arounds?

Here's the exception

ERROR [COMPACTION-POOL:1] 2010-07-17 19:09:49,784 CassandraDaemon.java (line 83) Uncaught exception in thread Thread[COMPACTION-POOL:1,5,main]
java.util.concurrent.ExecutionException: java.lang.StringIndexOutOfBoundsException: String index out of range: -1
        at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
        at java.util.concurrent.FutureTask.get(FutureTask.java:83)
        at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:86)        
        at org.apache.cassandra.db.CompactionManager$CompactionExecutor.afterExecute(CompactionManager.java:577)        
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888)        
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)        
        at java.lang.Thread.run(Thread.java:619)Caused by: java.lang.StringIndexOutOfBoundsException: String index out of range: -1        
        at java.lang.String.substring(String.java:1938)
        at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:50)
        at org.apache.cassandra.io.IteratingRow.<init>(IteratingRow.java:48)   
        at org.apache.cassandra.io.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:136)
        at org.apache.cassandra.io.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:113)
        at org.apache.cassandra.io.SSTableScanner.next(SSTableScanner.java:105)
        at org.apache.cassandra.io.SSTableScanner.next(SSTableScanner.java:34)
        at org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:284)
        at org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326)
        at org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230)
        at org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:68)
        at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:135)
        at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:130)
        at org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
        at org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
        at org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:295)
        at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:102)
        at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:83)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        ... 2 more

Thanks,

-Anthony

-- 
------------------------------------------------------------------------
Anthony Molinaro                           <an...@alumni.caltech.edu>