[jira] [Commented] (CASSANDRA-6646) Disk Failure Policy ignores CorruptBlockException

["sankalp kohli (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-6646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13935373#comment-13935373 ] 

sankalp kohli commented on CASSANDRA-6646:
------------------------------------------

Here is the stack trace
ERROR [ReadStage:2257] xxxxx CassandraDaemon.java (line 191) Exception in thread Thread[ReadStage:2257,5,main]
org.apache.cassandra.io.sstable.CorruptSSTableException: org.apache.cassandra.io.compress.CorruptBlockException: (/xx/data/Cassandra/xxxxx/xxxxx/xxxxxxx-166-Data.db): corruption detected, chunk at 148883342 of length 23271.
       at org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:89)
       at org.apache.cassandra.io.util.RandomAccessReader.seek(RandomAccessReader.java:312)
       at org.apache.cassandra.io.util.PoolingSegmentedFile.getSegment(PoolingSegmentedFile.java:42)
       at org.apache.cassandra.io.sstable.SSTableReader.getFileDataInput(SSTableReader.java:1048)
       at org.apache.cassandra.db.columniterator.SimpleSliceReader.<init>(SimpleSliceReader.java:63)
       at org.apache.cassandra.db.columniterator.SSTableSliceIterator.createReader(SSTableSliceIterator.java:68)
       at org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SSTableSliceIterator.java:44)
       at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:104)
       at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:68)
       at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:272)
       at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65)
       at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1391)
       at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
       at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1123)
       at org.apache.cassandra.db.Table.getRow(Table.java:347)
       at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:70)
       at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:44)
       at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:56)
       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
       at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.cassandra.io.compress.CorruptBlockException: (xxxx-Data.db): corruption detected, chunk at 148883342 of length 23271.
       at org.apache.cassandra.io.compress.CompressedRandomAccessReader.decompressChunk(CompressedRandomAccessReader.java:120)
       at org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:85)
       ... 20 more


> Disk Failure Policy ignores CorruptBlockException 
> --------------------------------------------------
>
>                 Key: CASSANDRA-6646
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-6646
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: sankalp kohli
>            Assignee: Marcus Eriksson
>            Priority: Minor
>             Fix For: 2.0.7, 2.1 beta2
>
>         Attachments: 0001-make-CorruptSSTableException-an-FSError-to-be-able-t.patch
>
>
> If Cassandra is using compression and has a bad drive or stable, it will throw an CorruptBlockException. 
> Disk Failure Policy only works if it is an FSError and does not work for IOExceptions like this. 
> We need to better handle such exceptions as it causes nodes to not respond to the co-ordinator causing the client to timeout. 



--
This message was sent by Atlassian JIRA
(v6.2#6252)