You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Benjamin Schrauwen (JIRA)" <ji...@apache.org> on 2011/08/20 04:49:27 UTC
[jira] [Created] (CASSANDRA-3065) Major file corruption after
running nodetool cleanup
Major file corruption after running nodetool cleanup
----------------------------------------------------
Key: CASSANDRA-3065
URL: https://issues.apache.org/jira/browse/CASSANDRA-3065
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 0.8.3
Reporter: Benjamin Schrauwen
After running nodetool cleanup on two of the nodes in my 4 node cluster, almost all SSTables on those those machine got corrupted. I am not able to read them anymore with sstable2json, and the cassandra daemon is repetitively throwing:
ERROR [ReadStage:11] 2011-08-20 04:44:46,846 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[ReadStage:11,5,main]
java.lang.RuntimeException: java.lang.IndexOutOfBoundsException
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.IndexOutOfBoundsException
at java.nio.Buffer.checkIndex(Buffer.java:514)
at java.nio.DirectByteBuffer.get(DirectByteBuffer.java:209)
at org.apache.cassandra.io.util.MappedFileDataInput.read(MappedFileDataInput.java:104)
at java.io.InputStream.read(InputStream.java:154)
at org.apache.cassandra.io.util.AbstractDataInput.readInt(AbstractDataInput.java:196)
at org.apache.cassandra.io.sstable.IndexHelper.skipIndex(IndexHelper.java:61)
at org.apache.cassandra.db.columniterator.SimpleSliceReader.<init>(SimpleSliceReader.java:58)
at org.apache.cassandra.db.columniterator.SSTableSliceIterator.createReader(SSTableSliceIterator.java:91)
at org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SSTableSliceIterator.java:67)
at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:66)
at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:80)
at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1314)
at org.apache.cassandra.db.ColumnFamilyStore.cacheRow(ColumnFamilyStore.java:1181)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1221)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1168)
at org.apache.cassandra.db.Table.getRow(Table.java:385)
at org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:58)
at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:641)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3065) Major file corruption after
running nodetool cleanup
Posted by "Benjamin Schrauwen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13088194#comment-13088194 ]
Benjamin Schrauwen commented on CASSANDRA-3065:
-----------------------------------------------
Running completely through nodetool scrub for a column family also doesn't fix the corrupt files. After the scrub I still cannot read the files with sstable2json.
> Major file corruption after running nodetool cleanup
> ----------------------------------------------------
>
> Key: CASSANDRA-3065
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3065
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.3
> Reporter: Benjamin Schrauwen
>
> After running nodetool cleanup on two of the nodes in my 4 node cluster, almost all SSTables on those those machine got corrupted. I am not able to read them anymore with sstable2json, and the cassandra daemon is repetitively throwing:
> ERROR [ReadStage:11] 2011-08-20 04:44:46,846 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[ReadStage:11,5,main]
> java.lang.RuntimeException: java.lang.IndexOutOfBoundsException
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IndexOutOfBoundsException
> at java.nio.Buffer.checkIndex(Buffer.java:514)
> at java.nio.DirectByteBuffer.get(DirectByteBuffer.java:209)
> at org.apache.cassandra.io.util.MappedFileDataInput.read(MappedFileDataInput.java:104)
> at java.io.InputStream.read(InputStream.java:154)
> at org.apache.cassandra.io.util.AbstractDataInput.readInt(AbstractDataInput.java:196)
> at org.apache.cassandra.io.sstable.IndexHelper.skipIndex(IndexHelper.java:61)
> at org.apache.cassandra.db.columniterator.SimpleSliceReader.<init>(SimpleSliceReader.java:58)
> at org.apache.cassandra.db.columniterator.SSTableSliceIterator.createReader(SSTableSliceIterator.java:91)
> at org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SSTableSliceIterator.java:67)
> at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:66)
> at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:80)
> at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1314)
> at org.apache.cassandra.db.ColumnFamilyStore.cacheRow(ColumnFamilyStore.java:1181)
> at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1221)
> at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1168)
> at org.apache.cassandra.db.Table.getRow(Table.java:385)
> at org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:58)
> at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:641)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3065) Major file corruption after
running nodetool cleanup
Posted by "Benjamin Schrauwen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13088192#comment-13088192 ]
Benjamin Schrauwen commented on CASSANDRA-3065:
-----------------------------------------------
I have a zipped corrupt SSTable that I could send you, but it is 25MB. Any way to get it to you?
> Major file corruption after running nodetool cleanup
> ----------------------------------------------------
>
> Key: CASSANDRA-3065
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3065
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.3
> Reporter: Benjamin Schrauwen
>
> After running nodetool cleanup on two of the nodes in my 4 node cluster, almost all SSTables on those those machine got corrupted. I am not able to read them anymore with sstable2json, and the cassandra daemon is repetitively throwing:
> ERROR [ReadStage:11] 2011-08-20 04:44:46,846 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[ReadStage:11,5,main]
> java.lang.RuntimeException: java.lang.IndexOutOfBoundsException
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IndexOutOfBoundsException
> at java.nio.Buffer.checkIndex(Buffer.java:514)
> at java.nio.DirectByteBuffer.get(DirectByteBuffer.java:209)
> at org.apache.cassandra.io.util.MappedFileDataInput.read(MappedFileDataInput.java:104)
> at java.io.InputStream.read(InputStream.java:154)
> at org.apache.cassandra.io.util.AbstractDataInput.readInt(AbstractDataInput.java:196)
> at org.apache.cassandra.io.sstable.IndexHelper.skipIndex(IndexHelper.java:61)
> at org.apache.cassandra.db.columniterator.SimpleSliceReader.<init>(SimpleSliceReader.java:58)
> at org.apache.cassandra.db.columniterator.SSTableSliceIterator.createReader(SSTableSliceIterator.java:91)
> at org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SSTableSliceIterator.java:67)
> at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:66)
> at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:80)
> at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1314)
> at org.apache.cassandra.db.ColumnFamilyStore.cacheRow(ColumnFamilyStore.java:1181)
> at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1221)
> at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1168)
> at org.apache.cassandra.db.Table.getRow(Table.java:385)
> at org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:58)
> at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:641)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (CASSANDRA-3065) Major file corruption after
running nodetool cleanup
Posted by "Jonathan Ellis (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis reassigned CASSANDRA-3065:
-----------------------------------------
Assignee: Sylvain Lebresne
> Major file corruption after running nodetool cleanup
> ----------------------------------------------------
>
> Key: CASSANDRA-3065
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3065
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.3
> Reporter: Benjamin Schrauwen
> Assignee: Sylvain Lebresne
>
> After running nodetool cleanup on two of the nodes in my 4 node cluster, almost all SSTables on those those machine got corrupted. I am not able to read them anymore with sstable2json, and the cassandra daemon is repetitively throwing:
> ERROR [ReadStage:11] 2011-08-20 04:44:46,846 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[ReadStage:11,5,main]
> java.lang.RuntimeException: java.lang.IndexOutOfBoundsException
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IndexOutOfBoundsException
> at java.nio.Buffer.checkIndex(Buffer.java:514)
> at java.nio.DirectByteBuffer.get(DirectByteBuffer.java:209)
> at org.apache.cassandra.io.util.MappedFileDataInput.read(MappedFileDataInput.java:104)
> at java.io.InputStream.read(InputStream.java:154)
> at org.apache.cassandra.io.util.AbstractDataInput.readInt(AbstractDataInput.java:196)
> at org.apache.cassandra.io.sstable.IndexHelper.skipIndex(IndexHelper.java:61)
> at org.apache.cassandra.db.columniterator.SimpleSliceReader.<init>(SimpleSliceReader.java:58)
> at org.apache.cassandra.db.columniterator.SSTableSliceIterator.createReader(SSTableSliceIterator.java:91)
> at org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SSTableSliceIterator.java:67)
> at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:66)
> at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:80)
> at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1314)
> at org.apache.cassandra.db.ColumnFamilyStore.cacheRow(ColumnFamilyStore.java:1181)
> at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1221)
> at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1168)
> at org.apache.cassandra.db.Table.getRow(Table.java:385)
> at org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:58)
> at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:641)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3065) Major file corruption after
running nodetool cleanup
Posted by "Benjamin Schrauwen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13088191#comment-13088191 ]
Benjamin Schrauwen commented on CASSANDRA-3065:
-----------------------------------------------
Scrub is repetitively printing:
INFO [CompactionExecutor:6] 2011-08-20 14:40:53,302 CompactionManager.java (line 741) Retrying from row index; data is -8 bytes starting at 77698
WARN [CompactionExecutor:6] 2011-08-20 14:40:53,302 CompactionManager.java (line 765) Retry failed too. Skipping to next row (retry's stacktrace follows)
java.io.IOError: java.io.EOFException: bloom filter claims to be 2037737077 bytes, longer than entire row size -8
at org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:149)
at org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:90)
at org.apache.cassandra.db.compaction.CompactionManager.scrubOne(CompactionManager.java:746)
at org.apache.cassandra.db.compaction.CompactionManager.doScrub(CompactionManager.java:631)
at org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:65)
at org.apache.cassandra.db.compaction.CompactionManager$3.call(CompactionManager.java:251)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.io.EOFException: bloom filter claims to be 2037737077 bytes, longer than entire row size -8
at org.apache.cassandra.io.sstable.IndexHelper.defreezeBloomFilter(IndexHelper.java:111)
at org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:119)
... 10 more
WARN [CompactionExecutor:6] 2011-08-20 14:40:53,302 CompactionManager.java (line 735) Non-fatal error reading row (stacktrace follows)
java.io.IOError: java.io.IOException: Impossible row size 2642626510010996
at org.apache.cassandra.db.compaction.CompactionManager.scrubOne(CompactionManager.java:717)
at org.apache.cassandra.db.compaction.CompactionManager.doScrub(CompactionManager.java:631)
at org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:65)
at org.apache.cassandra.db.compaction.CompactionManager$3.call(CompactionManager.java:251)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.io.IOException: Impossible row size 2642626510010996
... 9 more
> Major file corruption after running nodetool cleanup
> ----------------------------------------------------
>
> Key: CASSANDRA-3065
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3065
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.3
> Reporter: Benjamin Schrauwen
>
> After running nodetool cleanup on two of the nodes in my 4 node cluster, almost all SSTables on those those machine got corrupted. I am not able to read them anymore with sstable2json, and the cassandra daemon is repetitively throwing:
> ERROR [ReadStage:11] 2011-08-20 04:44:46,846 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[ReadStage:11,5,main]
> java.lang.RuntimeException: java.lang.IndexOutOfBoundsException
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IndexOutOfBoundsException
> at java.nio.Buffer.checkIndex(Buffer.java:514)
> at java.nio.DirectByteBuffer.get(DirectByteBuffer.java:209)
> at org.apache.cassandra.io.util.MappedFileDataInput.read(MappedFileDataInput.java:104)
> at java.io.InputStream.read(InputStream.java:154)
> at org.apache.cassandra.io.util.AbstractDataInput.readInt(AbstractDataInput.java:196)
> at org.apache.cassandra.io.sstable.IndexHelper.skipIndex(IndexHelper.java:61)
> at org.apache.cassandra.db.columniterator.SimpleSliceReader.<init>(SimpleSliceReader.java:58)
> at org.apache.cassandra.db.columniterator.SSTableSliceIterator.createReader(SSTableSliceIterator.java:91)
> at org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SSTableSliceIterator.java:67)
> at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:66)
> at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:80)
> at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1314)
> at org.apache.cassandra.db.ColumnFamilyStore.cacheRow(ColumnFamilyStore.java:1181)
> at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1221)
> at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1168)
> at org.apache.cassandra.db.Table.getRow(Table.java:385)
> at org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:58)
> at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:641)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3065) Major file corruption after
running nodetool cleanup
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13088118#comment-13088118 ]
Jonathan Ellis commented on CASSANDRA-3065:
-------------------------------------------
What does sstable2json say?
What happens if you switch from mmap i/o to buffered?
What happens if you run scrub?
> Major file corruption after running nodetool cleanup
> ----------------------------------------------------
>
> Key: CASSANDRA-3065
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3065
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.3
> Reporter: Benjamin Schrauwen
>
> After running nodetool cleanup on two of the nodes in my 4 node cluster, almost all SSTables on those those machine got corrupted. I am not able to read them anymore with sstable2json, and the cassandra daemon is repetitively throwing:
> ERROR [ReadStage:11] 2011-08-20 04:44:46,846 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[ReadStage:11,5,main]
> java.lang.RuntimeException: java.lang.IndexOutOfBoundsException
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IndexOutOfBoundsException
> at java.nio.Buffer.checkIndex(Buffer.java:514)
> at java.nio.DirectByteBuffer.get(DirectByteBuffer.java:209)
> at org.apache.cassandra.io.util.MappedFileDataInput.read(MappedFileDataInput.java:104)
> at java.io.InputStream.read(InputStream.java:154)
> at org.apache.cassandra.io.util.AbstractDataInput.readInt(AbstractDataInput.java:196)
> at org.apache.cassandra.io.sstable.IndexHelper.skipIndex(IndexHelper.java:61)
> at org.apache.cassandra.db.columniterator.SimpleSliceReader.<init>(SimpleSliceReader.java:58)
> at org.apache.cassandra.db.columniterator.SSTableSliceIterator.createReader(SSTableSliceIterator.java:91)
> at org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SSTableSliceIterator.java:67)
> at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:66)
> at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:80)
> at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1314)
> at org.apache.cassandra.db.ColumnFamilyStore.cacheRow(ColumnFamilyStore.java:1181)
> at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1221)
> at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1168)
> at org.apache.cassandra.db.Table.getRow(Table.java:385)
> at org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:58)
> at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:641)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3065) Major file corruption after
running nodetool cleanup
Posted by "Benjamin Schrauwen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13088282#comment-13088282 ]
Benjamin Schrauwen commented on CASSANDRA-3065:
-----------------------------------------------
I already had to recover the nodes in the cluster, but ran scrub on a local cassandra using the corrupt file I saved. I get these exceptions when running scrub:
WARN 18:29:51,454 Row at 29174 is unreadable; skipping to next
DEBUG 18:29:51,455 Reading row at 29174
DEBUG 18:29:51,455 row 01 is 288230376420147200 bytes
WARN 18:29:51,455 Non-fatal error reading row (stacktrace follows)
java.io.IOError: java.io.IOException: Impossible row size 288230376420147200
at org.apache.cassandra.db.compaction.CompactionManager.scrubOne(CompactionManager.java:717)
at org.apache.cassandra.db.compaction.CompactionManager.doScrub(CompactionManager.java:631)
at org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:65)
at org.apache.cassandra.db.compaction.CompactionManager$3.call(CompactionManager.java:251)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)
Caused by: java.io.IOException: Impossible row size 288230376420147200
... 9 more
WARN 18:29:51,580 Non-fatal error reading row (stacktrace follows)
java.io.IOError: java.io.EOFException: bloom filter claims to be -779103867 bytes, longer than entire row size -3408150975012023934
at org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:149)
at org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:90)
at org.apache.cassandra.db.compaction.CompactionManager.scrubOne(CompactionManager.java:718)
at org.apache.cassandra.db.compaction.CompactionManager.doScrub(CompactionManager.java:631)
at org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:65)
at org.apache.cassandra.db.compaction.CompactionManager$3.call(CompactionManager.java:251)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)
Caused by: java.io.EOFException: bloom filter claims to be -779103867 bytes, longer than entire row size -3408150975012023934
at org.apache.cassandra.io.sstable.IndexHelper.defreezeBloomFilter(IndexHelper.java:111)
at org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:119)
... 10 more
> Major file corruption after running nodetool cleanup
> ----------------------------------------------------
>
> Key: CASSANDRA-3065
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3065
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.3
> Reporter: Benjamin Schrauwen
>
> After running nodetool cleanup on two of the nodes in my 4 node cluster, almost all SSTables on those those machine got corrupted. I am not able to read them anymore with sstable2json, and the cassandra daemon is repetitively throwing:
> ERROR [ReadStage:11] 2011-08-20 04:44:46,846 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[ReadStage:11,5,main]
> java.lang.RuntimeException: java.lang.IndexOutOfBoundsException
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IndexOutOfBoundsException
> at java.nio.Buffer.checkIndex(Buffer.java:514)
> at java.nio.DirectByteBuffer.get(DirectByteBuffer.java:209)
> at org.apache.cassandra.io.util.MappedFileDataInput.read(MappedFileDataInput.java:104)
> at java.io.InputStream.read(InputStream.java:154)
> at org.apache.cassandra.io.util.AbstractDataInput.readInt(AbstractDataInput.java:196)
> at org.apache.cassandra.io.sstable.IndexHelper.skipIndex(IndexHelper.java:61)
> at org.apache.cassandra.db.columniterator.SimpleSliceReader.<init>(SimpleSliceReader.java:58)
> at org.apache.cassandra.db.columniterator.SSTableSliceIterator.createReader(SSTableSliceIterator.java:91)
> at org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SSTableSliceIterator.java:67)
> at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:66)
> at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:80)
> at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1314)
> at org.apache.cassandra.db.ColumnFamilyStore.cacheRow(ColumnFamilyStore.java:1181)
> at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1221)
> at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1168)
> at org.apache.cassandra.db.Table.getRow(Table.java:385)
> at org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:58)
> at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:641)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CASSANDRA-3065) Major file corruption after
running nodetool cleanup
Posted by "Jonathan Ellis (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis resolved CASSANDRA-3065.
---------------------------------------
Resolution: Cannot Reproduce
Assignee: (was: Sylvain Lebresne)
Unfortunately a corrupt data file doesn't help us a whole lot, especially without the index (which gives us redundant row size information in case the data file's is hosed, which this is).
I'd recommend taking regular snapshots so that if you run into more problems you can see if it reproduces on a relatively recent dataset.
It's also worth trying out compression in 1.0, which as part of the block format adds checksumming so Cassandra can easily recognize corruption at that level.
> Major file corruption after running nodetool cleanup
> ----------------------------------------------------
>
> Key: CASSANDRA-3065
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3065
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.3
> Reporter: Benjamin Schrauwen
>
> After running nodetool cleanup on two of the nodes in my 4 node cluster, almost all SSTables on those those machine got corrupted. I am not able to read them anymore with sstable2json, and the cassandra daemon is repetitively throwing:
> ERROR [ReadStage:11] 2011-08-20 04:44:46,846 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[ReadStage:11,5,main]
> java.lang.RuntimeException: java.lang.IndexOutOfBoundsException
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IndexOutOfBoundsException
> at java.nio.Buffer.checkIndex(Buffer.java:514)
> at java.nio.DirectByteBuffer.get(DirectByteBuffer.java:209)
> at org.apache.cassandra.io.util.MappedFileDataInput.read(MappedFileDataInput.java:104)
> at java.io.InputStream.read(InputStream.java:154)
> at org.apache.cassandra.io.util.AbstractDataInput.readInt(AbstractDataInput.java:196)
> at org.apache.cassandra.io.sstable.IndexHelper.skipIndex(IndexHelper.java:61)
> at org.apache.cassandra.db.columniterator.SimpleSliceReader.<init>(SimpleSliceReader.java:58)
> at org.apache.cassandra.db.columniterator.SSTableSliceIterator.createReader(SSTableSliceIterator.java:91)
> at org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SSTableSliceIterator.java:67)
> at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:66)
> at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:80)
> at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1314)
> at org.apache.cassandra.db.ColumnFamilyStore.cacheRow(ColumnFamilyStore.java:1181)
> at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1221)
> at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1168)
> at org.apache.cassandra.db.Table.getRow(Table.java:385)
> at org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:58)
> at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:641)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3065) Major file corruption after
running nodetool cleanup
Posted by "Sylvain Lebresne (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13130770#comment-13130770 ]
Sylvain Lebresne commented on CASSANDRA-3065:
---------------------------------------------
Realizing this one has been sitting over there for some time now. If you still have that corrupted sstable around, I can offer to take a look. Otherwise, there isn't much to chew on on those stacktraces (I mean, other than there is something wrong with those sstables).
> Major file corruption after running nodetool cleanup
> ----------------------------------------------------
>
> Key: CASSANDRA-3065
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3065
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.3
> Reporter: Benjamin Schrauwen
> Assignee: Sylvain Lebresne
>
> After running nodetool cleanup on two of the nodes in my 4 node cluster, almost all SSTables on those those machine got corrupted. I am not able to read them anymore with sstable2json, and the cassandra daemon is repetitively throwing:
> ERROR [ReadStage:11] 2011-08-20 04:44:46,846 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[ReadStage:11,5,main]
> java.lang.RuntimeException: java.lang.IndexOutOfBoundsException
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IndexOutOfBoundsException
> at java.nio.Buffer.checkIndex(Buffer.java:514)
> at java.nio.DirectByteBuffer.get(DirectByteBuffer.java:209)
> at org.apache.cassandra.io.util.MappedFileDataInput.read(MappedFileDataInput.java:104)
> at java.io.InputStream.read(InputStream.java:154)
> at org.apache.cassandra.io.util.AbstractDataInput.readInt(AbstractDataInput.java:196)
> at org.apache.cassandra.io.sstable.IndexHelper.skipIndex(IndexHelper.java:61)
> at org.apache.cassandra.db.columniterator.SimpleSliceReader.<init>(SimpleSliceReader.java:58)
> at org.apache.cassandra.db.columniterator.SSTableSliceIterator.createReader(SSTableSliceIterator.java:91)
> at org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SSTableSliceIterator.java:67)
> at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:66)
> at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:80)
> at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1314)
> at org.apache.cassandra.db.ColumnFamilyStore.cacheRow(ColumnFamilyStore.java:1181)
> at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1221)
> at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1168)
> at org.apache.cassandra.db.Table.getRow(Table.java:385)
> at org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:58)
> at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:641)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3065) Major file corruption after
running nodetool cleanup
Posted by "Benjamin Schrauwen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13088190#comment-13088190 ]
Benjamin Schrauwen commented on CASSANDRA-3065:
-----------------------------------------------
How do you switch the mmap/buffered io in 0.8.3, that option seems to be gone from the config file?
> Major file corruption after running nodetool cleanup
> ----------------------------------------------------
>
> Key: CASSANDRA-3065
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3065
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.3
> Reporter: Benjamin Schrauwen
>
> After running nodetool cleanup on two of the nodes in my 4 node cluster, almost all SSTables on those those machine got corrupted. I am not able to read them anymore with sstable2json, and the cassandra daemon is repetitively throwing:
> ERROR [ReadStage:11] 2011-08-20 04:44:46,846 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[ReadStage:11,5,main]
> java.lang.RuntimeException: java.lang.IndexOutOfBoundsException
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IndexOutOfBoundsException
> at java.nio.Buffer.checkIndex(Buffer.java:514)
> at java.nio.DirectByteBuffer.get(DirectByteBuffer.java:209)
> at org.apache.cassandra.io.util.MappedFileDataInput.read(MappedFileDataInput.java:104)
> at java.io.InputStream.read(InputStream.java:154)
> at org.apache.cassandra.io.util.AbstractDataInput.readInt(AbstractDataInput.java:196)
> at org.apache.cassandra.io.sstable.IndexHelper.skipIndex(IndexHelper.java:61)
> at org.apache.cassandra.db.columniterator.SimpleSliceReader.<init>(SimpleSliceReader.java:58)
> at org.apache.cassandra.db.columniterator.SSTableSliceIterator.createReader(SSTableSliceIterator.java:91)
> at org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SSTableSliceIterator.java:67)
> at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:66)
> at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:80)
> at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1314)
> at org.apache.cassandra.db.ColumnFamilyStore.cacheRow(ColumnFamilyStore.java:1181)
> at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1221)
> at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1168)
> at org.apache.cassandra.db.Table.getRow(Table.java:385)
> at org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:58)
> at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:641)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3065) Major file corruption after
running nodetool cleanup
Posted by "Benjamin Schrauwen (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13131100#comment-13131100 ]
Benjamin Schrauwen commented on CASSANDRA-3065:
-----------------------------------------------
I lost the index file, but a corrupt datafile can be found here: http://dl.dropbox.com/u/3030404/corrupt_sstable-g-6296-Data.db.bz2
> Major file corruption after running nodetool cleanup
> ----------------------------------------------------
>
> Key: CASSANDRA-3065
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3065
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.3
> Reporter: Benjamin Schrauwen
> Assignee: Sylvain Lebresne
>
> After running nodetool cleanup on two of the nodes in my 4 node cluster, almost all SSTables on those those machine got corrupted. I am not able to read them anymore with sstable2json, and the cassandra daemon is repetitively throwing:
> ERROR [ReadStage:11] 2011-08-20 04:44:46,846 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[ReadStage:11,5,main]
> java.lang.RuntimeException: java.lang.IndexOutOfBoundsException
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IndexOutOfBoundsException
> at java.nio.Buffer.checkIndex(Buffer.java:514)
> at java.nio.DirectByteBuffer.get(DirectByteBuffer.java:209)
> at org.apache.cassandra.io.util.MappedFileDataInput.read(MappedFileDataInput.java:104)
> at java.io.InputStream.read(InputStream.java:154)
> at org.apache.cassandra.io.util.AbstractDataInput.readInt(AbstractDataInput.java:196)
> at org.apache.cassandra.io.sstable.IndexHelper.skipIndex(IndexHelper.java:61)
> at org.apache.cassandra.db.columniterator.SimpleSliceReader.<init>(SimpleSliceReader.java:58)
> at org.apache.cassandra.db.columniterator.SSTableSliceIterator.createReader(SSTableSliceIterator.java:91)
> at org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SSTableSliceIterator.java:67)
> at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:66)
> at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:80)
> at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1314)
> at org.apache.cassandra.db.ColumnFamilyStore.cacheRow(ColumnFamilyStore.java:1181)
> at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1221)
> at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1168)
> at org.apache.cassandra.db.Table.getRow(Table.java:385)
> at org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:58)
> at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:641)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3065) Major file corruption after
running nodetool cleanup
Posted by "Benjamin Schrauwen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13088189#comment-13088189 ]
Benjamin Schrauwen commented on CASSANDRA-3065:
-----------------------------------------------
output of sstable2json:
Exception in thread "main" java.io.IOError: java.io.EOFException: EOF after 120443748 bytes out of 1819241532
at org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:149)
at org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:90)
at org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:74)
at org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:179)
at org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:144)
at org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:136)
at org.apache.cassandra.tools.SSTableExport.export(SSTableExport.java:304)
at org.apache.cassandra.tools.SSTableExport.export(SSTableExport.java:335)
at org.apache.cassandra.tools.SSTableExport.export(SSTableExport.java:348)
at org.apache.cassandra.tools.SSTableExport.main(SSTableExport.java:406)
Caused by: java.io.EOFException: EOF after 120443748 bytes out of 1819241532
at org.apache.cassandra.io.util.FileUtils.skipBytesFully(FileUtils.java:229)
at org.apache.cassandra.io.sstable.IndexHelper.skipBloomFilter(IndexHelper.java:50)
at org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:140)
... 9 more
> Major file corruption after running nodetool cleanup
> ----------------------------------------------------
>
> Key: CASSANDRA-3065
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3065
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.3
> Reporter: Benjamin Schrauwen
>
> After running nodetool cleanup on two of the nodes in my 4 node cluster, almost all SSTables on those those machine got corrupted. I am not able to read them anymore with sstable2json, and the cassandra daemon is repetitively throwing:
> ERROR [ReadStage:11] 2011-08-20 04:44:46,846 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[ReadStage:11,5,main]
> java.lang.RuntimeException: java.lang.IndexOutOfBoundsException
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IndexOutOfBoundsException
> at java.nio.Buffer.checkIndex(Buffer.java:514)
> at java.nio.DirectByteBuffer.get(DirectByteBuffer.java:209)
> at org.apache.cassandra.io.util.MappedFileDataInput.read(MappedFileDataInput.java:104)
> at java.io.InputStream.read(InputStream.java:154)
> at org.apache.cassandra.io.util.AbstractDataInput.readInt(AbstractDataInput.java:196)
> at org.apache.cassandra.io.sstable.IndexHelper.skipIndex(IndexHelper.java:61)
> at org.apache.cassandra.db.columniterator.SimpleSliceReader.<init>(SimpleSliceReader.java:58)
> at org.apache.cassandra.db.columniterator.SSTableSliceIterator.createReader(SSTableSliceIterator.java:91)
> at org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SSTableSliceIterator.java:67)
> at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:66)
> at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:80)
> at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1314)
> at org.apache.cassandra.db.ColumnFamilyStore.cacheRow(ColumnFamilyStore.java:1181)
> at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1221)
> at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1168)
> at org.apache.cassandra.db.Table.getRow(Table.java:385)
> at org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:58)
> at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:641)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3065) Major file corruption after
running nodetool cleanup
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13088198#comment-13088198 ]
Jonathan Ellis commented on CASSANDRA-3065:
-------------------------------------------
Please re-run scrub with debug logging enabled.
You can set disk i/o mode the same way you always did, it's just not part of the example config anymore.
> Major file corruption after running nodetool cleanup
> ----------------------------------------------------
>
> Key: CASSANDRA-3065
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3065
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.3
> Reporter: Benjamin Schrauwen
>
> After running nodetool cleanup on two of the nodes in my 4 node cluster, almost all SSTables on those those machine got corrupted. I am not able to read them anymore with sstable2json, and the cassandra daemon is repetitively throwing:
> ERROR [ReadStage:11] 2011-08-20 04:44:46,846 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[ReadStage:11,5,main]
> java.lang.RuntimeException: java.lang.IndexOutOfBoundsException
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IndexOutOfBoundsException
> at java.nio.Buffer.checkIndex(Buffer.java:514)
> at java.nio.DirectByteBuffer.get(DirectByteBuffer.java:209)
> at org.apache.cassandra.io.util.MappedFileDataInput.read(MappedFileDataInput.java:104)
> at java.io.InputStream.read(InputStream.java:154)
> at org.apache.cassandra.io.util.AbstractDataInput.readInt(AbstractDataInput.java:196)
> at org.apache.cassandra.io.sstable.IndexHelper.skipIndex(IndexHelper.java:61)
> at org.apache.cassandra.db.columniterator.SimpleSliceReader.<init>(SimpleSliceReader.java:58)
> at org.apache.cassandra.db.columniterator.SSTableSliceIterator.createReader(SSTableSliceIterator.java:91)
> at org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SSTableSliceIterator.java:67)
> at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:66)
> at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:80)
> at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1314)
> at org.apache.cassandra.db.ColumnFamilyStore.cacheRow(ColumnFamilyStore.java:1181)
> at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1221)
> at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1168)
> at org.apache.cassandra.db.Table.getRow(Table.java:385)
> at org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:58)
> at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:641)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira