You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Madalina Matei <ma...@gmail.com> on 2013/08/07 17:00:48 UTC
Can't perform repair on a 1.1.5 cassandra node -SSTable corrupted
Hi,
I have a 5 nodes cassandra (version 1.1.5) ring, RF=2, CL- READ/Write =1.
After a node went down without any error reported in OS syslog or Cassandra
syslog i decided to perform a repair.
Each time i run a nodetool repair I get this error:
INFO [FlushWriter:5] 2013-08-07 11:09:26,770 Memtable.java (line 305)
Completed flushing /data/<data>-298-Data.db (18694 bytes) for commitlog
position ReplayPosition(segmentId=1375867548785, position=199)
ERROR [Thrift:286] 2013-08-07 11:10:04,448 CustomTThreadPoolServer.java
(line 204) Error occurred during processing of message.
java.lang.RuntimeException: error reading 1 of 1
at
org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:83)
at
org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:39)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:116)
at
org.apache.cassandra.utils.MergeIterator$OneToOne.computeNext(MergeIterator.java:203)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
at
org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:117)
at
org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:140)
at
org.apache.cassandra.db.RowIteratorFactory$2.getReduced(RowIteratorFactory.java:107)
at
org.apache.cassandra.db.RowIteratorFactory$2.getReduced(RowIteratorFactory.java:80)
at
org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:118)
at
org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:101)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
at
org.apache.cassandra.db.ColumnFamilyStore$2.computeNext(ColumnFamilyStore.java:1381)
at
org.apache.cassandra.db.ColumnFamilyStore$2.computeNext(ColumnFamilyStore.java:1377)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
at
org.apache.cassandra.db.ColumnFamilyStore.filter(ColumnFamilyStore.java:1454)
at
org.apache.cassandra.db.ColumnFamilyStore.getRangeSlice(ColumnFamilyStore.java:1433)
at
org.apache.cassandra.service.RangeSliceVerbHandler.executeLocally(RangeSliceVerbHandler.java:50)
at
org.apache.cassandra.service.StorageProxy.getRangeSlice(StorageProxy.java:870)
at
org.apache.cassandra.thrift.CassandraServer.get_range_slices(CassandraServer.java:691)
at
org.apache.cassandra.thrift.Cassandra$Processor$get_range_slices.getResult(Cassandra.java:3008)
at
org.apache.cassandra.thrift.Cassandra$Processor$get_range_slices.getResult(Cassandra.java:2996)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)
at
org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:186)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.io.IOException: FAILED_TO_UNCOMPRESS(5)
at org.xerial.snappy.SnappyNative.throw_error(SnappyNative.java:78)
at org.xerial.snappy.SnappyNative.rawUncompress(Native Method)
at org.xerial.snappy.Snappy.rawUncompress(Snappy.java:391)
at
org.apache.cassandra.io.compress.SnappyCompressor.uncompress(SnappyCompressor.java:94)
at
org.apache.cassandra.io.compress.CompressedRandomAccessReader.decompressChunk(CompressedRandomAccessReader.java:91)
at
org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:77)
at
org.apache.cassandra.io.util.RandomAccessReader.read(RandomAccessReader.java:302)
at java.io.RandomAccessFile.readFully(RandomAccessFile.java:381)
at java.io.RandomAccessFile.readFully(RandomAccessFile.java:361)
at
org.apache.cassandra.io.util.RandomAccessReader.readBytes(RandomAccessReader.java:324)
at
org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:398)
at
org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:380)
at
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:88)
at
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:83)
at
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:73)
at
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:37)
at
org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:79)
... 32 more
I performed first a sstablescrub and i got the same error.
The problem is that my map-reduce job fails each time throwing in this
specific node syslog an error complaining about the same SSTable.
ERROR [Thrift:50] 2013-08-07 09:30:40,103 CustomTThreadPoolServer.java
(line 204) Error occurred during processing of message.
java.lang.RuntimeException: SSTableScanner(file=/data/<data>-298-Data.db-
chunk length 65536, data length 773855911.
sstable=SSTableReader(path='/data/<data>-298-Data.db') exhausted=false)
failed to provide next columns from
KeyScanningIterator(finishedAt:105054114)
at
org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:198)
at
org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:151)
at
org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:143)
at
org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:38)
at
org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:149)
at
org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:126)
at
org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:100)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
at
org.apache.cassandra.db.ColumnFamilyStore$2.computeNext(ColumnFamilyStore.java:1381)
at
org.apache.cassandra.db.ColumnFamilyStore$2.computeNext(ColumnFamilyStore.java:1377)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
at
org.apache.cassandra.db.ColumnFamilyStore.filter(ColumnFamilyStore.java:1454)
at
org.apache.cassandra.db.ColumnFamilyStore.getRangeSlice(ColumnFamilyStore.java:1433)
at
org.apache.cassandra.service.RangeSliceVerbHandler.executeLocally(RangeSliceVerbHandler.java:50)
at
org.apache.cassandra.service.StorageProxy.getRangeSlice(StorageProxy.java:870)
at
org.apache.cassandra.thrift.CassandraServer.get_range_slices(CassandraServer.java:691)
at
org.apache.cassandra.thrift.Cassandra$Processor$get_range_slices.getResult(Cassandra.java:3008)
at
org.apache.cassandra.thrift.Cassandra$Processor$get_range_slices.getResult(Cassandra.java:2996)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)
at
org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:186)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.io.IOException: FAILED_TO_UNCOMPRESS(5)
at org.xerial.snappy.SnappyNative.throw_error(SnappyNative.java:78)
at org.xerial.snappy.SnappyNative.rawUncompress(Native Method)
at org.xerial.snappy.Snappy.rawUncompress(Snappy.java:391)
at
org.apache.cassandra.io.compress.SnappyCompressor.uncompress(SnappyCompressor.java:94)
at
org.apache.cassandra.io.compress.CompressedRandomAccessReader.decompressChunk(CompressedRandomAccessReader.java:91)
at
org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:77)
at
org.apache.cassandra.io.util.RandomAccessReader.read(RandomAccessReader.java:302)
at java.io.RandomAccessFile.readFully(RandomAccessFile.java:381)
at java.io.RandomAccessFile.readFully(RandomAccessFile.java:361)
at
org.apache.cassandra.io.util.RandomAccessReader.readBytes(RandomAccessReader.java:324)
at
org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:398)
at
org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:380)
at
org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:178)
Is there any way to fix that specific SSTable or any other recommendations?
Thanks,
Madalina
Re: Can't perform repair on a 1.1.5 cassandra node -SSTable corrupted
Posted by Ondřej Černoš <ce...@gmail.com>.
Hello,
please see these issues:
https://issues.apache.org/jira/browse/CASSANDRA-5686 and
https://issues.apache.org/jira/browse/CASSANDRA-5391 if you hit any of them.
regards,
ondrej cernos
On Wed, Aug 7, 2013 at 5:00 PM, Madalina Matei <ma...@gmail.com>wrote:
> Hi,
>
> I have a 5 nodes cassandra (version 1.1.5) ring, RF=2, CL- READ/Write =1.
> After a node went down without any error reported in OS syslog or Cassandra
> syslog i decided to perform a repair.
>
> Each time i run a nodetool repair I get this error:
>
> INFO [FlushWriter:5] 2013-08-07 11:09:26,770 Memtable.java (line 305)
> Completed flushing /data/<data>-298-Data.db (18694 bytes) for commitlog
> position ReplayPosition(segmentId=1375867548785, position=199)
> ERROR [Thrift:286] 2013-08-07 11:10:04,448 CustomTThreadPoolServer.java
> (line 204) Error occurred during processing of message.
> java.lang.RuntimeException: error reading 1 of 1
> at
> org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:83)
> at
> org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:39)
> at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
> at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
> at
> org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:116)
> at
> org.apache.cassandra.utils.MergeIterator$OneToOne.computeNext(MergeIterator.java:203)
> at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
> at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
> at
> org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:117)
> at
> org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:140)
> at
> org.apache.cassandra.db.RowIteratorFactory$2.getReduced(RowIteratorFactory.java:107)
> at
> org.apache.cassandra.db.RowIteratorFactory$2.getReduced(RowIteratorFactory.java:80)
> at
> org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:118)
> at
> org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:101)
> at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
> at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
> at
> org.apache.cassandra.db.ColumnFamilyStore$2.computeNext(ColumnFamilyStore.java:1381)
> at
> org.apache.cassandra.db.ColumnFamilyStore$2.computeNext(ColumnFamilyStore.java:1377)
> at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
> at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
> at
> org.apache.cassandra.db.ColumnFamilyStore.filter(ColumnFamilyStore.java:1454)
> at
> org.apache.cassandra.db.ColumnFamilyStore.getRangeSlice(ColumnFamilyStore.java:1433)
> at
> org.apache.cassandra.service.RangeSliceVerbHandler.executeLocally(RangeSliceVerbHandler.java:50)
> at
> org.apache.cassandra.service.StorageProxy.getRangeSlice(StorageProxy.java:870)
> at
> org.apache.cassandra.thrift.CassandraServer.get_range_slices(CassandraServer.java:691)
> at
> org.apache.cassandra.thrift.Cassandra$Processor$get_range_slices.getResult(Cassandra.java:3008)
> at
> org.apache.cassandra.thrift.Cassandra$Processor$get_range_slices.getResult(Cassandra.java:2996)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)
> at
> org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:186)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.io.IOException: FAILED_TO_UNCOMPRESS(5)
> at org.xerial.snappy.SnappyNative.throw_error(SnappyNative.java:78)
> at org.xerial.snappy.SnappyNative.rawUncompress(Native Method)
> at org.xerial.snappy.Snappy.rawUncompress(Snappy.java:391)
> at
> org.apache.cassandra.io.compress.SnappyCompressor.uncompress(SnappyCompressor.java:94)
> at
> org.apache.cassandra.io.compress.CompressedRandomAccessReader.decompressChunk(CompressedRandomAccessReader.java:91)
> at
> org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:77)
> at
> org.apache.cassandra.io.util.RandomAccessReader.read(RandomAccessReader.java:302)
> at java.io.RandomAccessFile.readFully(RandomAccessFile.java:381)
> at java.io.RandomAccessFile.readFully(RandomAccessFile.java:361)
> at
> org.apache.cassandra.io.util.RandomAccessReader.readBytes(RandomAccessReader.java:324)
> at
> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:398)
> at
> org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:380)
> at
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:88)
> at
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:83)
> at
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:73)
> at
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:37)
> at
> org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:79)
> ... 32 more
>
>
> I performed first a sstablescrub and i got the same error.
>
> The problem is that my map-reduce job fails each time throwing in this
> specific node syslog an error complaining about the same SSTable.
>
> ERROR [Thrift:50] 2013-08-07 09:30:40,103 CustomTThreadPoolServer.java
> (line 204) Error occurred during processing of message.
> java.lang.RuntimeException: SSTableScanner(file=/data/<data>-298-Data.db-
> chunk length 65536, data length 773855911.
> sstable=SSTableReader(path='/data/<data>-298-Data.db') exhausted=false)
> failed to provide next columns from
> KeyScanningIterator(finishedAt:105054114)
> at
> org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:198)
> at
> org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:151)
> at
> org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:143)
> at
> org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:38)
> at
> org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:149)
> at
> org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:126)
> at
> org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:100)
> at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
> at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
> at
> org.apache.cassandra.db.ColumnFamilyStore$2.computeNext(ColumnFamilyStore.java:1381)
> at
> org.apache.cassandra.db.ColumnFamilyStore$2.computeNext(ColumnFamilyStore.java:1377)
> at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
> at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
> at
> org.apache.cassandra.db.ColumnFamilyStore.filter(ColumnFamilyStore.java:1454)
> at
> org.apache.cassandra.db.ColumnFamilyStore.getRangeSlice(ColumnFamilyStore.java:1433)
> at
> org.apache.cassandra.service.RangeSliceVerbHandler.executeLocally(RangeSliceVerbHandler.java:50)
> at
> org.apache.cassandra.service.StorageProxy.getRangeSlice(StorageProxy.java:870)
> at
> org.apache.cassandra.thrift.CassandraServer.get_range_slices(CassandraServer.java:691)
> at
> org.apache.cassandra.thrift.Cassandra$Processor$get_range_slices.getResult(Cassandra.java:3008)
> at
> org.apache.cassandra.thrift.Cassandra$Processor$get_range_slices.getResult(Cassandra.java:2996)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)
> at
> org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:186)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.io.IOException: FAILED_TO_UNCOMPRESS(5)
> at org.xerial.snappy.SnappyNative.throw_error(SnappyNative.java:78)
> at org.xerial.snappy.SnappyNative.rawUncompress(Native Method)
> at org.xerial.snappy.Snappy.rawUncompress(Snappy.java:391)
> at
> org.apache.cassandra.io.compress.SnappyCompressor.uncompress(SnappyCompressor.java:94)
> at
> org.apache.cassandra.io.compress.CompressedRandomAccessReader.decompressChunk(CompressedRandomAccessReader.java:91)
> at
> org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:77)
> at
> org.apache.cassandra.io.util.RandomAccessReader.read(RandomAccessReader.java:302)
> at java.io.RandomAccessFile.readFully(RandomAccessFile.java:381)
> at java.io.RandomAccessFile.readFully(RandomAccessFile.java:361)
> at
> org.apache.cassandra.io.util.RandomAccessReader.readBytes(RandomAccessReader.java:324)
> at
> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:398)
> at
> org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:380)
> at
> org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:178)
>
>
> Is there any way to fix that specific SSTable or any other recommendations?
>
> Thanks,
> Madalina
>
>