You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Jonathan Ellis (JIRA)" <ji...@apache.org> on 2011/04/19 16:18:06 UTC
[jira] [Commented] (CASSANDRA-2408) Faulty memory causes adjacent
nodes to have invalid data and fail compactation (java.io.IOException: Keys
must be written in ascending order.)
[ https://issues.apache.org/jira/browse/CASSANDRA-2408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13021602#comment-13021602 ]
Jonathan Ellis commented on CASSANDRA-2408:
-------------------------------------------
Did you run any repairs _before_ seeing this?
SSTableWriter checks order (of data written out of a memtable) at write time, so the only way to get data out-of-order should be to stream it from a corrupt node.
Repair would explain why it affects adjacent noces too.
> Faulty memory causes adjacent nodes to have invalid data and fail compactation (java.io.IOException: Keys must be written in ascending order.)
> ----------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-2408
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2408
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.4
> Reporter: Thibaut
> Fix For: 0.7.6
>
>
> Hi,
> We had to replace a node with faulty ram. Besides the cassandra cluster getting unresponsive, we also observerd a "keys must be written in ascending order" exception on all adjacent quorum nodes, causing their compactation to fail.
> We had another node with faulty ram a week earlier, and it was causing the same errors on it's neighbours.
> A faulty node shouldn't affect the key ordering of adjacent nodes?
> INFO [CompactionExecutor:1] 2011-03-31 09:57:29,529 CompactionManager.java (line 396) Compacting [SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1793-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1798-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1803-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1808-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1832-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1841-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1852-Data.db')]
> INFO [CompactionExecutor:1] 2011-03-31 09:58:26,437 SSTableWriter.java (line 108) Last written key : DecoratedKey(ba578bed5e75a06d1156dabccc68abd5_www.abc.com_hostinlinks_9223370735413988125_870cc25d-7e4f-437b-aa83-a311dfbf7f88, 62613537386265643565373561303664313135366461626363633638616264355f7777772e6b6f7475736f7a6c756b2e636f6d5f686f7374696e6c696e6b735f393232333337303733353431333938383132355f38373063633235642d376534662d343337622d616138332d613331316466626637663838)
> INFO [CompactionExecutor:1] 2011-03-31 09:58:26,515 SSTableWriter.java (line 109) Current key : DecoratedKey(ba578bed5e75a06d1156dabccc68abd5_www.abc.com_hostinlinks_9223370735413988136_17070303-ab24-4207-83bd-1604e2c159e4, 62613537386265643565373561303664313135366461626363633638616264355f7777772e6b677475736f7a6c756b2e636f6d5f686f7374696e6c696e6b735f393232333337303733353431333938383133365f31373037303330332d616232342d343230372d383362642d313630346532633135396534)
> INFO [CompactionExecutor:1] 2011-03-31 09:58:26,515 SSTableWriter.java (line 110) Writing into file /cassandra/data/table_lists/table_lists-tmp-f-1861-Data.db
> ERROR [CompactionExecutor:1] 2011-03-31 09:58:26,516 AbstractCassandraDaemon.java (line 112) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.io.IOException: Keys must be written in ascending order.
> at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:111)
> at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:128)
> at org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:452)
> at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:124)
> at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:94)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira