You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Karl Hiramoto <ka...@hiramoto.org> on 2011/03/08 16:30:09 UTC

nodetool repair hung in 0.7.3

I never saw this before upgrading to 0.7.3 but now I do nodetool repair 
and it sits there for hours.  Previously it took about 20 minutes per 
node (about 10GB of data per node).

I had some OOM crashes, but haven't seen them since I increased the heap 
size and decreased the key cache.


In the nodes logs I see:


  INFO [manual-repair-dbf777bf-12ff-4172-8d7f-a4ed6d9ccc6a] 2011-03-08 
15:15:41,856 AntiEntropyService.java (line 752) No neighbors to repair 
with: manual-repair-dbf777bf-12ff-4172-8d7f-a4ed6d9ccc6a completed.
  INFO [manual-repair-0d4e4e0d-0641-4e25-8e20-baabbce344dd] 2011-03-08 
15:15:41,862 AntiEntropyService.java (line 770) Waiting for repair 
requests: [#<TreeRequest 
manual-repair-0d4e4e0d-0641-4e25-8e20-baabbce344dd, /10.224.4.10, 
(OfferRequest,Offer)>, #<TreeRequest 
manual-repair-0d4e4e0d-0641-4e25-8e20-baabbce344dd, /10.224.4.15, 
(OfferRequest,Offer)>]
  INFO [CompactionExecutor:1] 2011-03-08 15:15:41,862 
ColumnFamilyStore.java (line 695) switching in a fresh Memtable for 
Offer at 
CommitLogContext(file='/home/cassandra/commitlog/CommitLog-1299596902726.log', 
position=110125920)
  INFO [CompactionExecutor:1] 2011-03-08 15:15:41,862 
ColumnFamilyStore.java (line 1006) Enqueuing flush of 
Memtable-Offer@185502190(41184536 bytes, 109934 operations)
  INFO [FlushWriter:1] 2011-03-08 15:15:41,862 Memtable.java (line 157) 
Writing Memtable-Offer@185502190(41184536 bytes, 109934 operations)
  INFO [FlushWriter:1] 2011-03-08 15:15:42,334 Memtable.java (line 164) 
Completed flushing 
/home/cassandra/data/OfferRequest/Offer-f-5616-Data.db (38695861 bytes)
ERROR [CompactionExecutor:1] 2011-03-08 15:15:43,835 
PrecompactedRow.java (line 82) Skipping row 
DecoratedKey(85070595676245833572375607193993401386, 
383236424d374449494a47) in 
/home/cassandra/data/OfferRequest/Offer-f-5571-Data.db
java.io.EOFException
         at java.io.RandomAccessFile.readFully(RandomAccessFile.java:416)
         at java.io.RandomAccessFile.readFully(RandomAccessFile.java:394)
         at 
org.apache.cassandra.io.util.BufferedRandomAccessFile.readBytes(BufferedRandomAccessFile.java:268)
         at 
org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:310)
         at 
org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:267)
         at 
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:94)
         at 
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
         at 
org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:129)
         at 
org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:176)
         at 
org.apache.cassandra.io.PrecompactedRow.<init>(PrecompactedRow.java:78)
         at 
org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIterator.java:139)
         at 
org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:108)
         at 
org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:43)
         at 
org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
         at 
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
         at 
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
         at 
org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
         at 
org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
         at 
org.apache.cassandra.db.CompactionManager.doValidationCompaction(CompactionManager.java:803)
         at 
org.apache.cassandra.db.CompactionManager.access$800(CompactionManager.java:56)
         at 
org.apache.cassandra.db.CompactionManager$6.call(CompactionManager.java:358)
         at 
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
         at java.util.concurrent.FutureTask.run(FutureTask.java:166)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
         at java.lang.Thread.run(Thread.java:636)
ERROR [CompactionExecutor:1] 2011-03-08 15:15:43,837 
PrecompactedRow.java (line 82) Skipping row 
DecoratedKey(85070596351011236998883565851954247305, 
4332383230413943313234) in 
/home/cassandra/data/OfferRequest/Offer-f-5571-Data.db
org.apache.cassandra.db.ColumnSerializer$CorruptColumnException: invalid 
column name length 0
         at 
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:68)
         at 
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
         at 
org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:129)
         at 
org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:176)
         at 
org.apache.cassandra.io.PrecompactedRow.<init>(PrecompactedRow.java:78)
         at 
org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIterator.java:139)
         at 
org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:108)
         at 
org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:43)
         at 
org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
         at 
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
         at 
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
         at 
org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
         at 
org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
         at 
org.apache.cassandra.db.CompactionManager.doValidationCompaction(CompactionManager.java:803)
         at 
org.apache.cassandra.db.CompactionManager.access$800(CompactionManager.java:56)
         at 
org.apache.cassandra.db.CompactionManager$6.call(CompactionManager.java:358)
         at 
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
         at java.util.concurrent.FutureTask.run(FutureTask.java:166)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
         at java.lang.Thread.run(Thread.java:636)
ERROR [CompactionExecutor:1] 2011-03-08 15:15:43,841 
PrecompactedRow.java (line 82) Skipping row 
DecoratedKey(85070619433259740737929970212513100751, 
314d414244334c374d4c4e) in 
/home/cassandra/data/OfferRequest/Offer-f-5571-Data.db
org.apache.cassandra.db.ColumnSerializer$CorruptColumnException: invalid 
column name length 0
         at 
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:68)
         at 
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
         at 
org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:129)
         at 
org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:176)
         at 
org.apache.cassandra.io.PrecompactedRow.<init>(PrecompactedRow.java:78)
         at 
org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIterator.java:139)
         at 
org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:108)
         at 
org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:43)
         at 
org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
         at 
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
         at 
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
         at 
org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
         at 
org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
         at 
org.apache.cassandra.db.CompactionManager.doValidationCompaction(CompactionManager.java:803)
         at 
org.apache.cassandra.db.CompactionManager.access$800(CompactionManager.java:56)
         at 
org.apache.cassandra.db.CompactionManager$6.call(CompactionManager.java:358)
         at 
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
         at java.util.concurrent.FutureTask.run(FutureTask.java:166)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
         at java.lang.Thread.run(Thread.java:636)
ERROR [CompactionExecutor:1] 2011-03-08 15:15:43,843 
AbstractCassandraDaemon.java (line 114) Fatal exception in thread 
Thread[CompactionExecutor:1,1,main]
java.io.IOError: java.io.EOFException
         at 
org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:117)
         at 
org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:67)
         at 
org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:179)
         at 
org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:144)
         at 
org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:136)
         at 
org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:39)
         at 
org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:284)
         at 
org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326)
         at 
org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230)
         at 
org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:68)
         at 
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
         at 
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
         at 
org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
         at 
org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
         at 
org.apache.cassandra.db.CompactionManager.doValidationCompaction(CompactionManager.java:803)
         at 
org.apache.cassandra.db.CompactionManager.access$800(CompactionManager.java:56)
         at 
org.apache.cassandra.db.CompactionManager$6.call(CompactionManager.java:358)
         at 
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
         at java.util.concurrent.FutureTask.run(FutureTask.java:166)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
         at java.lang.Thread.run(Thread.java:636)
Caused by: java.io.EOFException
         at 
org.apache.cassandra.io.sstable.IndexHelper.skipIndex(IndexHelper.java:65)
         at 
org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:109)
         ... 21 more
  INFO [CompactionExecutor:1] 2011-03-08 15:15:43,844 
CompactionManager.java (line 396) Compacting 
[SSTableReader(path='/home/cassandra/data/OfferRequest/Offer-f-5565-Data.db'),SSTableReader(path='/home/cassandra/data/OfferRequest/Offer-f-5569-Data.db'),SSTableReader(path='/home/cassandra/data/OfferRequest/Offer-f-5571-Data.db'),SSTableReader(path='/home/cassandra/data/OfferRequest/Offer-f-5590-Data.db')]
ERROR [CompactionExecutor:1] 2011-03-08 15:15:45,179 
PrecompactedRow.java (line 82) Skipping row 
DecoratedKey(85070595676245833572375607193993401386, 
383236424d374449494a47) in 
/home/cassandra/data/OfferRequest/Offer-f-5571-Data.db
java.io.EOFException
         at java.io.RandomAccessFile.readFully(RandomAccessFile.java:416)
         at java.io.RandomAccessFile.readFully(RandomAccessFile.java:394)
         at 
org.apache.cassandra.io.util.BufferedRandomAccessFile.readBytes(BufferedRandomAccessFile.java:268)
         at 
org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:310)
         at 
org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:267)
         at 
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:94)
         at 
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
         at 
org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:129)
         at 
org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:176)
         at 
org.apache.cassandra.io.PrecompactedRow.<init>(PrecompactedRow.java:78)
         at 
org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIterator.java:139)
         at 
org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:108)
         at 
org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:43)
         at 
org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
         at 
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
         at 
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
         at 
org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
         at 
org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
         at 
org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:449)
         at 
org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:124)
         at 
org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:94)
         at 
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
         at java.util.concurrent.FutureTask.run(FutureTask.java:166)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
         at java.lang.Thread.run(Thread.java:636)
ERROR [CompactionExecutor:1] 2011-03-08 15:15:45,180 
PrecompactedRow.java (line 82) Skipping row 
DecoratedKey(85070596351011236998883565851954247305, 
4332383230413943313234) in 
/home/cassandra/data/OfferRequest/Offer-f-5571-Data.db
org.apache.cassandra.db.ColumnSerializer$CorruptColumnException: invalid 
column name length 0
         at 
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:68)
         at 
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
         at 
org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:129)
         at 
org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:176)
         at 
org.apache.cassandra.io.PrecompactedRow.<init>(PrecompactedRow.java:78)
         at 
org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIterator.java:139)
         at 
org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:108)
         at 
org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:43)
         at 
org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
         at 
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
         at 
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
         at 
org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
         at 
org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
         at 
org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:449)
         at 
org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:124)
         at 
org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:94)
         at 
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
         at java.util.concurrent.FutureTask.run(FutureTask.java:166)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
         at java.lang.Thread.run(Thread.java:636)
ERROR [CompactionExecutor:1] 2011-03-08 15:15:45,184 
PrecompactedRow.java (line 82) Skipping row 
DecoratedKey(85070619433259740737929970212513100751, 
314d414244334c374d4c4e) in 
/home/cassandra/data/OfferRequest/Offer-f-5571-Data.db
org.apache.cassandra.db.ColumnSerializer$CorruptColumnException: invalid 
column name length 0
         at 
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:68)
         at 
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
         at 
org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:129)
         at 
org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:176)
         at 
org.apache.cassandra.io.PrecompactedRow.<init>(PrecompactedRow.java:78)
         at 
org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIterator.java:139)
         at 
org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:108)
         at 
org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:43)
         at 
org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
         at 
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
         at 
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
         at 
org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
         at 
org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
         at 
org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:449)
         at 
org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:124)
         at 
org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:94)
         at 
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
         at java.util.concurrent.FutureTask.run(FutureTask.java:166)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
         at java.lang.Thread.run(Thread.java:636)
ERROR [CompactionExecutor:1] 2011-03-08 15:15:45,186 
AbstractCassandraDaemon.java (line 114) Fatal exception in thread 
Thread[CompactionExecutor:1,1,main]
java.io.IOError: java.io.EOFException
         at 
org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:117)
         at 
org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:67)
         at 
org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:179)
         at 
org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:144)
         at 
org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:136)
         at 
org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:39)
         at 
org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:284)
         at 
org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326)
         at 
org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230)
         at 
org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:68)
         at 
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
         at 
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
         at 
org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
         at 
org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
         at 
org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:449)
         at 
org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:124)
         at 
org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:94)
         at 
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
         at java.util.concurrent.FutureTask.run(FutureTask.java:166)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
         at java.lang.Thread.run(Thread.java:636)
Caused by: java.io.EOFException
         at 
org.apache.cassandra.io.sstable.IndexHelper.skipIndex(IndexHelper.java:65)
         at 
org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:109)
         ... 21 more

Re: nodetool repair hung in 0.7.3

Posted by Sylvain Lebresne <sy...@datastax.com>.
I suspect you are in the case of
https://issues.apache.org/jira/browse/CASSANDRA-2290.

That is some neighbor node died or was unable to perform its part of the
repair. You can always
retry making sure all node are and stay alive to see if it is the former
one. But seeing the
other exception in your log, there something wrong deserializing your rows,
so this may be
what is messing up with repair.

But there is another thread on those exceptions, right?

--
Sylvain


On Tue, Mar 8, 2011 at 4:38 PM, Karl Hiramoto <ka...@hiramoto.org> wrote:

> On 08/03/2011 16:34, Sylvain Lebresne wrote:
>
>> I just saw repair hang here too, it's actually very easy to reproduce. I'm
>> looking at it right now.
>>
>> --
>>
>
> Thanks.   Should i bump  GCGraceSeconds  since i can no longer repair?  I
> tried repair on 3 nodes of a 6 node cluster and they all hang.
>
>
> Would downgrading to 0.7.2  be the best option?  Any potential problems
> caused by downgrade?
>
>
> --
> Karl
>
>
>

Re: nodetool repair hung in 0.7.3

Posted by Karl Hiramoto <ka...@hiramoto.org>.
On 08/03/2011 16:34, Sylvain Lebresne wrote:
> I just saw repair hang here too, it's actually very easy to reproduce. 
> I'm looking at it right now.
>
> --

Thanks.   Should i bump  GCGraceSeconds  since i can no longer repair?  
I tried repair on 3 nodes of a 6 node cluster and they all hang.


Would downgrading to 0.7.2  be the best option?  Any potential problems 
caused by downgrade?


--
Karl

Re: nodetool repair hung in 0.7.3

Posted by Sylvain Lebresne <sy...@datastax.com>.
I just saw repair hang here too, it's actually very easy to reproduce. I'm
looking at it right now.

--
Sylvain

On Tue, Mar 8, 2011 at 4:30 PM, Karl Hiramoto <ka...@hiramoto.org> wrote:

> I never saw this before upgrading to 0.7.3 but now I do nodetool repair and
> it sits there for hours.  Previously it took about 20 minutes per node
> (about 10GB of data per node).
>
> I had some OOM crashes, but haven't seen them since I increased the heap
> size and decreased the key cache.
>
>
> In the nodes logs I see:
>
>
>  INFO [manual-repair-dbf777bf-12ff-4172-8d7f-a4ed6d9ccc6a] 2011-03-08
> 15:15:41,856 AntiEntropyService.java (line 752) No neighbors to repair with:
> manual-repair-dbf777bf-12ff-4172-8d7f-a4ed6d9ccc6a completed.
>  INFO [manual-repair-0d4e4e0d-0641-4e25-8e20-baabbce344dd] 2011-03-08
> 15:15:41,862 AntiEntropyService.java (line 770) Waiting for repair requests:
> [#<TreeRequest manual-repair-0d4e4e0d-0641-4e25-8e20-baabbce344dd, /
> 10.224.4.10, (OfferRequest,Offer)>, #<TreeRequest
> manual-repair-0d4e4e0d-0641-4e25-8e20-baabbce344dd, /10.224.4.15,
> (OfferRequest,Offer)>]
>  INFO [CompactionExecutor:1] 2011-03-08 15:15:41,862 ColumnFamilyStore.java
> (line 695) switching in a fresh Memtable for Offer at
> CommitLogContext(file='/home/cassandra/commitlog/CommitLog-1299596902726.log',
> position=110125920)
>  INFO [CompactionExecutor:1] 2011-03-08 15:15:41,862 ColumnFamilyStore.java
> (line 1006) Enqueuing flush of Memtable-Offer@185502190(41184536 bytes,
> 109934 operations)
>  INFO [FlushWriter:1] 2011-03-08 15:15:41,862 Memtable.java (line 157)
> Writing Memtable-Offer@185502190(41184536 bytes, 109934 operations)
>  INFO [FlushWriter:1] 2011-03-08 15:15:42,334 Memtable.java (line 164)
> Completed flushing /home/cassandra/data/OfferRequest/Offer-f-5616-Data.db
> (38695861 bytes)
> ERROR [CompactionExecutor:1] 2011-03-08 15:15:43,835 PrecompactedRow.java
> (line 82) Skipping row DecoratedKey(85070595676245833572375607193993401386,
> 383236424d374449494a47) in
> /home/cassandra/data/OfferRequest/Offer-f-5571-Data.db
> java.io.EOFException
>        at java.io.RandomAccessFile.readFully(RandomAccessFile.java:416)
>        at java.io.RandomAccessFile.readFully(RandomAccessFile.java:394)
>        at
> org.apache.cassandra.io.util.BufferedRandomAccessFile.readBytes(BufferedRandomAccessFile.java:268)
>        at
> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:310)
>        at
> org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:267)
>        at
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:94)
>        at
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
>        at
> org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:129)
>        at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:176)
>        at
> org.apache.cassandra.io.PrecompactedRow.<init>(PrecompactedRow.java:78)
>        at
> org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIterator.java:139)
>        at
> org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:108)
>        at
> org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:43)
>        at
> org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
>        at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
>        at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
>        at
> org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
>        at
> org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
>        at
> org.apache.cassandra.db.CompactionManager.doValidationCompaction(CompactionManager.java:803)
>        at
> org.apache.cassandra.db.CompactionManager.access$800(CompactionManager.java:56)
>        at
> org.apache.cassandra.db.CompactionManager$6.call(CompactionManager.java:358)
>        at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>        at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>        at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>        at java.lang.Thread.run(Thread.java:636)
> ERROR [CompactionExecutor:1] 2011-03-08 15:15:43,837 PrecompactedRow.java
> (line 82) Skipping row DecoratedKey(85070596351011236998883565851954247305,
> 4332383230413943313234) in
> /home/cassandra/data/OfferRequest/Offer-f-5571-Data.db
> org.apache.cassandra.db.ColumnSerializer$CorruptColumnException: invalid
> column name length 0
>        at
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:68)
>        at
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
>        at
> org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:129)
>        at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:176)
>        at
> org.apache.cassandra.io.PrecompactedRow.<init>(PrecompactedRow.java:78)
>        at
> org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIterator.java:139)
>        at
> org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:108)
>        at
> org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:43)
>        at
> org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
>        at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
>        at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
>        at
> org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
>        at
> org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
>        at
> org.apache.cassandra.db.CompactionManager.doValidationCompaction(CompactionManager.java:803)
>        at
> org.apache.cassandra.db.CompactionManager.access$800(CompactionManager.java:56)
>        at
> org.apache.cassandra.db.CompactionManager$6.call(CompactionManager.java:358)
>        at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>        at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>        at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>        at java.lang.Thread.run(Thread.java:636)
> ERROR [CompactionExecutor:1] 2011-03-08 15:15:43,841 PrecompactedRow.java
> (line 82) Skipping row DecoratedKey(85070619433259740737929970212513100751,
> 314d414244334c374d4c4e) in
> /home/cassandra/data/OfferRequest/Offer-f-5571-Data.db
> org.apache.cassandra.db.ColumnSerializer$CorruptColumnException: invalid
> column name length 0
>        at
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:68)
>        at
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
>        at
> org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:129)
>        at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:176)
>        at
> org.apache.cassandra.io.PrecompactedRow.<init>(PrecompactedRow.java:78)
>        at
> org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIterator.java:139)
>        at
> org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:108)
>        at
> org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:43)
>        at
> org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
>        at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
>        at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
>        at
> org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
>        at
> org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
>        at
> org.apache.cassandra.db.CompactionManager.doValidationCompaction(CompactionManager.java:803)
>        at
> org.apache.cassandra.db.CompactionManager.access$800(CompactionManager.java:56)
>        at
> org.apache.cassandra.db.CompactionManager$6.call(CompactionManager.java:358)
>        at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>        at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>        at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>        at java.lang.Thread.run(Thread.java:636)
> ERROR [CompactionExecutor:1] 2011-03-08 15:15:43,843
> AbstractCassandraDaemon.java (line 114) Fatal exception in thread
> Thread[CompactionExecutor:1,1,main]
> java.io.IOError: java.io.EOFException
>        at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:117)
>        at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:67)
>        at
> org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:179)
>        at
> org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:144)
>        at
> org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:136)
>        at
> org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:39)
>        at
> org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:284)
>        at
> org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326)
>        at
> org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230)
>        at
> org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:68)
>        at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
>        at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
>        at
> org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
>        at
> org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
>        at
> org.apache.cassandra.db.CompactionManager.doValidationCompaction(CompactionManager.java:803)
>        at
> org.apache.cassandra.db.CompactionManager.access$800(CompactionManager.java:56)
>        at
> org.apache.cassandra.db.CompactionManager$6.call(CompactionManager.java:358)
>        at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>        at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>        at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>        at java.lang.Thread.run(Thread.java:636)
> Caused by: java.io.EOFException
>        at
> org.apache.cassandra.io.sstable.IndexHelper.skipIndex(IndexHelper.java:65)
>        at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:109)
>        ... 21 more
>  INFO [CompactionExecutor:1] 2011-03-08 15:15:43,844 CompactionManager.java
> (line 396) Compacting
> [SSTableReader(path='/home/cassandra/data/OfferRequest/Offer-f-5565-Data.db'),SSTableReader(path='/home/cassandra/data/OfferRequest/Offer-f-5569-Data.db'),SSTableReader(path='/home/cassandra/data/OfferRequest/Offer-f-5571-Data.db'),SSTableReader(path='/home/cassandra/data/OfferRequest/Offer-f-5590-Data.db')]
> ERROR [CompactionExecutor:1] 2011-03-08 15:15:45,179 PrecompactedRow.java
> (line 82) Skipping row DecoratedKey(85070595676245833572375607193993401386,
> 383236424d374449494a47) in
> /home/cassandra/data/OfferRequest/Offer-f-5571-Data.db
> java.io.EOFException
>        at java.io.RandomAccessFile.readFully(RandomAccessFile.java:416)
>        at java.io.RandomAccessFile.readFully(RandomAccessFile.java:394)
>        at
> org.apache.cassandra.io.util.BufferedRandomAccessFile.readBytes(BufferedRandomAccessFile.java:268)
>        at
> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:310)
>        at
> org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:267)
>        at
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:94)
>        at
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
>        at
> org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:129)
>        at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:176)
>        at
> org.apache.cassandra.io.PrecompactedRow.<init>(PrecompactedRow.java:78)
>        at
> org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIterator.java:139)
>        at
> org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:108)
>        at
> org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:43)
>        at
> org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
>        at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
>        at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
>        at
> org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
>        at
> org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
>        at
> org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:449)
>        at
> org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:124)
>        at
> org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:94)
>        at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>        at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>        at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>        at java.lang.Thread.run(Thread.java:636)
> ERROR [CompactionExecutor:1] 2011-03-08 15:15:45,180 PrecompactedRow.java
> (line 82) Skipping row DecoratedKey(85070596351011236998883565851954247305,
> 4332383230413943313234) in
> /home/cassandra/data/OfferRequest/Offer-f-5571-Data.db
> org.apache.cassandra.db.ColumnSerializer$CorruptColumnException: invalid
> column name length 0
>        at
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:68)
>        at
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
>        at
> org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:129)
>        at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:176)
>        at
> org.apache.cassandra.io.PrecompactedRow.<init>(PrecompactedRow.java:78)
>        at
> org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIterator.java:139)
>        at
> org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:108)
>        at
> org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:43)
>        at
> org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
>        at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
>        at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
>        at
> org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
>        at
> org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
>        at
> org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:449)
>        at
> org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:124)
>        at
> org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:94)
>        at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>        at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>        at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>        at java.lang.Thread.run(Thread.java:636)
> ERROR [CompactionExecutor:1] 2011-03-08 15:15:45,184 PrecompactedRow.java
> (line 82) Skipping row DecoratedKey(85070619433259740737929970212513100751,
> 314d414244334c374d4c4e) in
> /home/cassandra/data/OfferRequest/Offer-f-5571-Data.db
> org.apache.cassandra.db.ColumnSerializer$CorruptColumnException: invalid
> column name length 0
>        at
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:68)
>        at
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
>        at
> org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:129)
>        at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:176)
>        at
> org.apache.cassandra.io.PrecompactedRow.<init>(PrecompactedRow.java:78)
>        at
> org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIterator.java:139)
>        at
> org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:108)
>        at
> org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:43)
>        at
> org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
>        at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
>        at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
>        at
> org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
>        at
> org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
>        at
> org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:449)
>        at
> org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:124)
>        at
> org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:94)
>        at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>        at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>        at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>        at java.lang.Thread.run(Thread.java:636)
> ERROR [CompactionExecutor:1] 2011-03-08 15:15:45,186
> AbstractCassandraDaemon.java (line 114) Fatal exception in thread
> Thread[CompactionExecutor:1,1,main]
> java.io.IOError: java.io.EOFException
>        at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:117)
>        at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:67)
>        at
> org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:179)
>        at
> org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:144)
>        at
> org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:136)
>        at
> org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:39)
>        at
> org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:284)
>        at
> org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326)
>        at
> org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230)
>        at
> org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:68)
>        at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
>        at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
>        at
> org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
>        at
> org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
>        at
> org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:449)
>        at
> org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:124)
>        at
> org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:94)
>        at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>        at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>        at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>        at java.lang.Thread.run(Thread.java:636)
> Caused by: java.io.EOFException
>        at
> org.apache.cassandra.io.sstable.IndexHelper.skipIndex(IndexHelper.java:65)
>        at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:109)
>        ... 21 more
>