1.0 compaction exceptions

[Terry Cumaranatunge <cu...@gmail.com>]

Hello,

I'm seeng the following exception during compaction. This is happening on
Linux (RHEL 6) running Cassandra 1.0. This particular instance is with
Leveled compaction, but we have seen this error in the past with SizeTiered
compaction as well. We are not using compression.

Any ideas on what may be causing this problem? How do you cleanup this
error? Compaction for this CF is stuck because it aborts every time it sees
this error.

I've attached more of the surrounding system.log file as well in case it
helps. I can also provide the sstable file that it is having problems with,
but it is 2.7MB when compressed.

ERROR [CompactionExecutor:6] 2011-10-23 17:37:28,826
AbstractCassandraDaemon.java (line 133) Fatal excepti
on in thread Thread[CompactionExecutor:6,1,main]
java.io.IOError: java.io.IOException: dataSize of 3763098337272659968
starting at 9684826 would be larger
than file /var/lib/cassandra/data/MSA/modseq-h-841-Data.db length 10486277
        at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:153
)
        at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:85)
        at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:69)
        at
org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:177
)
        at
org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:142
)
        at
org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:134)
        at
org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:37)
        at
org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:147)
        at
org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:124)
        at
org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:98)
        at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
        at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
        at
com.google.common.collect.Iterators$7.computeNext(Iterators.java:614)
        at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
        at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
        at
org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:179)
        at
org.apache.cassandra.db.compaction.LeveledCompactionTask.execute(LeveledCompactionTask.java:47)
        at
org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:131)
        at
org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:114)
        at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: dataSize of 3763098337272659968 starting at
9684826 would be larger than f
ile /var/lib/cassandra/data/MSA/modseq-h-841-Data.db length 10486277
        at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:114