Which SSTable caused CorruptSSTableException?

[Ike Walker <ik...@flite.com>]

Running Cassandra 1.2.9 in AWS with a 12 host cluster, I am getting lots of
CorruptSSTableException in system.log on one of my hosts.

Is it possible to find out which SSTable(s) is/are corrupt?

I'm currently running "nodetool scrub" on the relevant host, but that
doesn't seem like an efficient way to fix the problem (if it fixes it at
all).

Here's an example error:

ERROR [ReplicateOnWriteStage:1070] 2014-02-25 17:43:03,518
CassandraDaemon.java (line 192) Exception in thread
Thread[ReplicateOnWriteStage:1070,5,main]
java.lang.RuntimeException:
org.apache.cassandra.io.sstable.CorruptSSTableException:
java.io.EOFException
    at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1597)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:724)
Caused by: org.apache.cassandra.io.sstable.CorruptSSTableException:
java.io.EOFException
    at
org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNamesIterator.java:65)
    at
org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:81)
    at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:68)
    at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:272)
    at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65)
    at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1391)
    at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1214)
    at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1126)
    at org.apache.cassandra.db.Table.getRow(Table.java:347)
    at
org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:64)
    at
org.apache.cassandra.db.CounterMutation.makeReplicationMutation(CounterMutation.java:90)
    at
org.apache.cassandra.service.StorageProxy$7$1.runMayThrow(StorageProxy.java:772)
    at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1593)
    ... 3 more
Caused by: java.io.EOFException
    at java.io.RandomAccessFile.readFully(RandomAccessFile.java:416)
    at java.io.RandomAccessFile.readFully(RandomAccessFile.java:394)
    at
org.apache.cassandra.io.util.RandomAccessReader.readBytes(RandomAccessReader.java:380)
    at
org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:392)
    at
org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:371)
    at
org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:116)
    at
org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNamesIterator.java:60)
    ... 15 more

Re: Which SSTable caused CorruptSSTableException?

[Robert Coli <rc...@eventbrite.com>]

On Tue, Feb 25, 2014 at 9:48 AM, Ike Walker <ik...@flite.com> wrote:

> Running Cassandra 1.2.9 in AWS with a 12 host cluster, I am getting lots
> of CorruptSSTableException in system.log on one of my hosts.
>
> Is it possible to find out which SSTable(s) is/are corrupt?
>

This feature has been requested a few times. I suggest looking for a JIRA
and filing a feature request if one does not exist. If you do, please reply
and let the list know in order to close the circle for googlers. :)

In the interim, if you try to dump the files (to /dev/null?) via
sstable2jason (or sstablekeys in the case of a corrupt index file) you can
determine which file is corrupt. You can also attempt to correlate which
compaction thread is excepting with the files that it was trying to
compact, to narrow down the search.

=Rob