You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Thibaut (JIRA)" <ji...@apache.org> on 2011/03/31 10:53:05 UTC

[jira] [Created] (CASSANDRA-2408) Faulty memory causes adjacent nodes to have invalid data and fail compactation (java.io.IOException: Keys must be written in ascending order.)

Faulty memory causes adjacent nodes to have invalid data and fail compactation (java.io.IOException: Keys must be written in ascending order.)
----------------------------------------------------------------------------------------------------------------------------------------------

                 Key: CASSANDRA-2408
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2408
             Project: Cassandra
          Issue Type: Bug
          Components: Core
    Affects Versions: 0.7.4
            Reporter: Thibaut
             Fix For: 0.7.5


Hi,

We had to replace a node with faulty ram. Besides the cassandra cluster getting unresponsive, we also observerd a "keys must be written in ascending order" exception on all adjacent quorum nodes, causing the compactation to fail.
We had another node with faulty ram a week earlier, and it was causing the same errors on it's neighbours.

A faulty node shouldn't affect the key ordering of adjacent nodes?

 INFO [CompactionExecutor:1] 2011-03-31 09:57:29,529 CompactionManager.java (line 396) Compacting [SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1793-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1798-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1803-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1808-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1832-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1841-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1852-Data.db')]
 INFO [CompactionExecutor:1] 2011-03-31 09:58:26,437 SSTableWriter.java (line 108) Last written key : DecoratedKey(ba578bed5e75a06d1156dabccc68abd5_www.abc.com_hostinlinks_9223370735413988125_870cc25d-7e4f-437b-aa83-a311dfbf7f88, 62613537386265643565373561303664313135366461626363633638616264355f7777772e6b6f7475736f7a6c756b2e636f6d5f686f7374696e6c696e6b735f393232333337303733353431333938383132355f38373063633235642d376534662d343337622d616138332d613331316466626637663838)
 INFO [CompactionExecutor:1] 2011-03-31 09:58:26,515 SSTableWriter.java (line 109) Current key : DecoratedKey(ba578bed5e75a06d1156dabccc68abd5_www.abc.com_hostinlinks_9223370735413988136_17070303-ab24-4207-83bd-1604e2c159e4, 62613537386265643565373561303664313135366461626363633638616264355f7777772e6b677475736f7a6c756b2e636f6d5f686f7374696e6c696e6b735f393232333337303733353431333938383133365f31373037303330332d616232342d343230372d383362642d313630346532633135396534)
 INFO [CompactionExecutor:1] 2011-03-31 09:58:26,515 SSTableWriter.java (line 110) Writing into file /cassandra/data/table_lists/table_lists-tmp-f-1861-Data.db
ERROR [CompactionExecutor:1] 2011-03-31 09:58:26,516 AbstractCassandraDaemon.java (line 112) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
java.io.IOException: Keys must be written in ascending order.
      at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:111)
      at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:128)
      at org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:452)
      at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:124)
      at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:94)
      at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
      at java.util.concurrent.FutureTask.run(FutureTask.java:138)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      at java.lang.Thread.run(Thread.java:662)


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2408) Faulty memory causes adjacent nodes to have invalid data and fail compactation (java.io.IOException: Keys must be written in ascending order.)

Posted by "Thibaut (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-2408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13021612#comment-13021612 ] 

Thibaut commented on CASSANDRA-2408:
------------------------------------

I might have run a repair on the faulty node before, but I don't think I ran a repair on the adjacent nodes, as there was no logical need to do this. However, I can't completely exclude it.

We also replaced about 8 machines so far with faulty memory, and this problem only occured twice (on 6 different machines).

If I do a write ONE on the faulty node, the faulty node might also be buffering delayed writes for the other nodes? What if the a bit flip happens there?



> Faulty memory causes adjacent nodes to have invalid data and fail compactation (java.io.IOException: Keys must be written in ascending order.)
> ----------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-2408
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2408
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 0.7.4
>            Reporter: Thibaut
>             Fix For: 0.7.6
>
>
> Hi,
> We had to replace a node with faulty ram. Besides the cassandra cluster getting unresponsive, we also observerd a "keys must be written in ascending order" exception on all adjacent quorum nodes, causing their compactation to fail.
> We had another node with faulty ram a week earlier, and it was causing the same errors on it's neighbours.
> A faulty node shouldn't affect the key ordering of adjacent nodes?
>  INFO [CompactionExecutor:1] 2011-03-31 09:57:29,529 CompactionManager.java (line 396) Compacting [SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1793-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1798-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1803-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1808-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1832-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1841-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1852-Data.db')]
>  INFO [CompactionExecutor:1] 2011-03-31 09:58:26,437 SSTableWriter.java (line 108) Last written key : DecoratedKey(ba578bed5e75a06d1156dabccc68abd5_www.abc.com_hostinlinks_9223370735413988125_870cc25d-7e4f-437b-aa83-a311dfbf7f88, 62613537386265643565373561303664313135366461626363633638616264355f7777772e6b6f7475736f7a6c756b2e636f6d5f686f7374696e6c696e6b735f393232333337303733353431333938383132355f38373063633235642d376534662d343337622d616138332d613331316466626637663838)
>  INFO [CompactionExecutor:1] 2011-03-31 09:58:26,515 SSTableWriter.java (line 109) Current key : DecoratedKey(ba578bed5e75a06d1156dabccc68abd5_www.abc.com_hostinlinks_9223370735413988136_17070303-ab24-4207-83bd-1604e2c159e4, 62613537386265643565373561303664313135366461626363633638616264355f7777772e6b677475736f7a6c756b2e636f6d5f686f7374696e6c696e6b735f393232333337303733353431333938383133365f31373037303330332d616232342d343230372d383362642d313630346532633135396534)
>  INFO [CompactionExecutor:1] 2011-03-31 09:58:26,515 SSTableWriter.java (line 110) Writing into file /cassandra/data/table_lists/table_lists-tmp-f-1861-Data.db
> ERROR [CompactionExecutor:1] 2011-03-31 09:58:26,516 AbstractCassandraDaemon.java (line 112) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.io.IOException: Keys must be written in ascending order.
>       at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:111)
>       at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:128)
>       at org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:452)
>       at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:124)
>       at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:94)
>       at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>       at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>       at java.lang.Thread.run(Thread.java:662)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2408) Faulty memory causes adjacent nodes to have invalid data and fail compactation (java.io.IOException: Keys must be written in ascending order.)

Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-2408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13021602#comment-13021602 ] 

Jonathan Ellis commented on CASSANDRA-2408:
-------------------------------------------

Did you run any repairs _before_ seeing this?

SSTableWriter checks order (of data written out of a memtable) at write time, so the only way to get data out-of-order should be to stream it from a corrupt node.

Repair would explain why it affects adjacent noces too.

> Faulty memory causes adjacent nodes to have invalid data and fail compactation (java.io.IOException: Keys must be written in ascending order.)
> ----------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-2408
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2408
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 0.7.4
>            Reporter: Thibaut
>             Fix For: 0.7.6
>
>
> Hi,
> We had to replace a node with faulty ram. Besides the cassandra cluster getting unresponsive, we also observerd a "keys must be written in ascending order" exception on all adjacent quorum nodes, causing their compactation to fail.
> We had another node with faulty ram a week earlier, and it was causing the same errors on it's neighbours.
> A faulty node shouldn't affect the key ordering of adjacent nodes?
>  INFO [CompactionExecutor:1] 2011-03-31 09:57:29,529 CompactionManager.java (line 396) Compacting [SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1793-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1798-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1803-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1808-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1832-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1841-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1852-Data.db')]
>  INFO [CompactionExecutor:1] 2011-03-31 09:58:26,437 SSTableWriter.java (line 108) Last written key : DecoratedKey(ba578bed5e75a06d1156dabccc68abd5_www.abc.com_hostinlinks_9223370735413988125_870cc25d-7e4f-437b-aa83-a311dfbf7f88, 62613537386265643565373561303664313135366461626363633638616264355f7777772e6b6f7475736f7a6c756b2e636f6d5f686f7374696e6c696e6b735f393232333337303733353431333938383132355f38373063633235642d376534662d343337622d616138332d613331316466626637663838)
>  INFO [CompactionExecutor:1] 2011-03-31 09:58:26,515 SSTableWriter.java (line 109) Current key : DecoratedKey(ba578bed5e75a06d1156dabccc68abd5_www.abc.com_hostinlinks_9223370735413988136_17070303-ab24-4207-83bd-1604e2c159e4, 62613537386265643565373561303664313135366461626363633638616264355f7777772e6b677475736f7a6c756b2e636f6d5f686f7374696e6c696e6b735f393232333337303733353431333938383133365f31373037303330332d616232342d343230372d383362642d313630346532633135396534)
>  INFO [CompactionExecutor:1] 2011-03-31 09:58:26,515 SSTableWriter.java (line 110) Writing into file /cassandra/data/table_lists/table_lists-tmp-f-1861-Data.db
> ERROR [CompactionExecutor:1] 2011-03-31 09:58:26,516 AbstractCassandraDaemon.java (line 112) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.io.IOException: Keys must be written in ascending order.
>       at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:111)
>       at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:128)
>       at org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:452)
>       at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:124)
>       at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:94)
>       at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>       at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>       at java.lang.Thread.run(Thread.java:662)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2408) Faulty memory causes adjacent nodes to have invalid data and fail compactation (java.io.IOException: Keys must be written in ascending order.)

Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-2408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13021669#comment-13021669 ] 

Jonathan Ellis commented on CASSANDRA-2408:
-------------------------------------------

If the coordinator flips bits before sending the write to a data node, the data node will write out the flipped key, but in the correct order (for the flipped version).

> Faulty memory causes adjacent nodes to have invalid data and fail compactation (java.io.IOException: Keys must be written in ascending order.)
> ----------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-2408
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2408
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 0.7.4
>            Reporter: Thibaut
>             Fix For: 0.7.6
>
>
> Hi,
> We had to replace a node with faulty ram. Besides the cassandra cluster getting unresponsive, we also observerd a "keys must be written in ascending order" exception on all adjacent quorum nodes, causing their compactation to fail.
> We had another node with faulty ram a week earlier, and it was causing the same errors on it's neighbours.
> A faulty node shouldn't affect the key ordering of adjacent nodes?
>  INFO [CompactionExecutor:1] 2011-03-31 09:57:29,529 CompactionManager.java (line 396) Compacting [SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1793-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1798-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1803-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1808-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1832-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1841-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1852-Data.db')]
>  INFO [CompactionExecutor:1] 2011-03-31 09:58:26,437 SSTableWriter.java (line 108) Last written key : DecoratedKey(ba578bed5e75a06d1156dabccc68abd5_www.abc.com_hostinlinks_9223370735413988125_870cc25d-7e4f-437b-aa83-a311dfbf7f88, 62613537386265643565373561303664313135366461626363633638616264355f7777772e6b6f7475736f7a6c756b2e636f6d5f686f7374696e6c696e6b735f393232333337303733353431333938383132355f38373063633235642d376534662d343337622d616138332d613331316466626637663838)
>  INFO [CompactionExecutor:1] 2011-03-31 09:58:26,515 SSTableWriter.java (line 109) Current key : DecoratedKey(ba578bed5e75a06d1156dabccc68abd5_www.abc.com_hostinlinks_9223370735413988136_17070303-ab24-4207-83bd-1604e2c159e4, 62613537386265643565373561303664313135366461626363633638616264355f7777772e6b677475736f7a6c756b2e636f6d5f686f7374696e6c696e6b735f393232333337303733353431333938383133365f31373037303330332d616232342d343230372d383362642d313630346532633135396534)
>  INFO [CompactionExecutor:1] 2011-03-31 09:58:26,515 SSTableWriter.java (line 110) Writing into file /cassandra/data/table_lists/table_lists-tmp-f-1861-Data.db
> ERROR [CompactionExecutor:1] 2011-03-31 09:58:26,516 AbstractCassandraDaemon.java (line 112) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.io.IOException: Keys must be written in ascending order.
>       at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:111)
>       at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:128)
>       at org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:452)
>       at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:124)
>       at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:94)
>       at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>       at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>       at java.lang.Thread.run(Thread.java:662)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (CASSANDRA-2408) Faulty memory causes adjacent nodes to have invalid data and fail compactation (java.io.IOException: Keys must be written in ascending order.)

Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-2408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13021602#comment-13021602 ] 

Jonathan Ellis edited comment on CASSANDRA-2408 at 4/19/11 2:21 PM:
--------------------------------------------------------------------

Did you run any repairs _before_ seeing this?

SSTableWriter checks order (of data written out of a memtable) at write time, so the only way to get data out-of-order should be to stream it from a corrupt node.

Repair would explain why it affects adjacent nodes too.

      was (Author: jbellis):
    Did you run any repairs _before_ seeing this?

SSTableWriter checks order (of data written out of a memtable) at write time, so the only way to get data out-of-order should be to stream it from a corrupt node.

Repair would explain why it affects adjacent noces too.
  
> Faulty memory causes adjacent nodes to have invalid data and fail compactation (java.io.IOException: Keys must be written in ascending order.)
> ----------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-2408
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2408
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 0.7.4
>            Reporter: Thibaut
>             Fix For: 0.7.6
>
>
> Hi,
> We had to replace a node with faulty ram. Besides the cassandra cluster getting unresponsive, we also observerd a "keys must be written in ascending order" exception on all adjacent quorum nodes, causing their compactation to fail.
> We had another node with faulty ram a week earlier, and it was causing the same errors on it's neighbours.
> A faulty node shouldn't affect the key ordering of adjacent nodes?
>  INFO [CompactionExecutor:1] 2011-03-31 09:57:29,529 CompactionManager.java (line 396) Compacting [SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1793-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1798-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1803-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1808-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1832-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1841-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1852-Data.db')]
>  INFO [CompactionExecutor:1] 2011-03-31 09:58:26,437 SSTableWriter.java (line 108) Last written key : DecoratedKey(ba578bed5e75a06d1156dabccc68abd5_www.abc.com_hostinlinks_9223370735413988125_870cc25d-7e4f-437b-aa83-a311dfbf7f88, 62613537386265643565373561303664313135366461626363633638616264355f7777772e6b6f7475736f7a6c756b2e636f6d5f686f7374696e6c696e6b735f393232333337303733353431333938383132355f38373063633235642d376534662d343337622d616138332d613331316466626637663838)
>  INFO [CompactionExecutor:1] 2011-03-31 09:58:26,515 SSTableWriter.java (line 109) Current key : DecoratedKey(ba578bed5e75a06d1156dabccc68abd5_www.abc.com_hostinlinks_9223370735413988136_17070303-ab24-4207-83bd-1604e2c159e4, 62613537386265643565373561303664313135366461626363633638616264355f7777772e6b677475736f7a6c756b2e636f6d5f686f7374696e6c696e6b735f393232333337303733353431333938383133365f31373037303330332d616232342d343230372d383362642d313630346532633135396534)
>  INFO [CompactionExecutor:1] 2011-03-31 09:58:26,515 SSTableWriter.java (line 110) Writing into file /cassandra/data/table_lists/table_lists-tmp-f-1861-Data.db
> ERROR [CompactionExecutor:1] 2011-03-31 09:58:26,516 AbstractCassandraDaemon.java (line 112) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.io.IOException: Keys must be written in ascending order.
>       at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:111)
>       at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:128)
>       at org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:452)
>       at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:124)
>       at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:94)
>       at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>       at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>       at java.lang.Thread.run(Thread.java:662)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (CASSANDRA-2408) Faulty memory causes adjacent nodes to have invalid data and fail compactation (java.io.IOException: Keys must be written in ascending order.)

Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/CASSANDRA-2408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jonathan Ellis resolved CASSANDRA-2408.
---------------------------------------

       Resolution: Duplicate
    Fix Version/s:     (was: 0.7.6)

CASSANDRA-1717 will address not propagating corrupt data.

> Faulty memory causes adjacent nodes to have invalid data and fail compactation (java.io.IOException: Keys must be written in ascending order.)
> ----------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-2408
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2408
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 0.7.4
>            Reporter: Thibaut
>
> Hi,
> We had to replace a node with faulty ram. Besides the cassandra cluster getting unresponsive, we also observerd a "keys must be written in ascending order" exception on all adjacent quorum nodes, causing their compactation to fail.
> We had another node with faulty ram a week earlier, and it was causing the same errors on it's neighbours.
> A faulty node shouldn't affect the key ordering of adjacent nodes?
>  INFO [CompactionExecutor:1] 2011-03-31 09:57:29,529 CompactionManager.java (line 396) Compacting [SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1793-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1798-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1803-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1808-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1832-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1841-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1852-Data.db')]
>  INFO [CompactionExecutor:1] 2011-03-31 09:58:26,437 SSTableWriter.java (line 108) Last written key : DecoratedKey(ba578bed5e75a06d1156dabccc68abd5_www.abc.com_hostinlinks_9223370735413988125_870cc25d-7e4f-437b-aa83-a311dfbf7f88, 62613537386265643565373561303664313135366461626363633638616264355f7777772e6b6f7475736f7a6c756b2e636f6d5f686f7374696e6c696e6b735f393232333337303733353431333938383132355f38373063633235642d376534662d343337622d616138332d613331316466626637663838)
>  INFO [CompactionExecutor:1] 2011-03-31 09:58:26,515 SSTableWriter.java (line 109) Current key : DecoratedKey(ba578bed5e75a06d1156dabccc68abd5_www.abc.com_hostinlinks_9223370735413988136_17070303-ab24-4207-83bd-1604e2c159e4, 62613537386265643565373561303664313135366461626363633638616264355f7777772e6b677475736f7a6c756b2e636f6d5f686f7374696e6c696e6b735f393232333337303733353431333938383133365f31373037303330332d616232342d343230372d383362642d313630346532633135396534)
>  INFO [CompactionExecutor:1] 2011-03-31 09:58:26,515 SSTableWriter.java (line 110) Writing into file /cassandra/data/table_lists/table_lists-tmp-f-1861-Data.db
> ERROR [CompactionExecutor:1] 2011-03-31 09:58:26,516 AbstractCassandraDaemon.java (line 112) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.io.IOException: Keys must be written in ascending order.
>       at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:111)
>       at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:128)
>       at org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:452)
>       at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:124)
>       at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:94)
>       at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>       at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>       at java.lang.Thread.run(Thread.java:662)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (CASSANDRA-2408) Faulty memory causes adjacent nodes to have invalid data and fail compactation (java.io.IOException: Keys must be written in ascending order.)

Posted by "Thibaut (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/CASSANDRA-2408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Thibaut updated CASSANDRA-2408:
-------------------------------

    Description: 
Hi,

We had to replace a node with faulty ram. Besides the cassandra cluster getting unresponsive, we also observerd a "keys must be written in ascending order" exception on all adjacent quorum nodes, causing their compactation to fail.
We had another node with faulty ram a week earlier, and it was causing the same errors on it's neighbours.

A faulty node shouldn't affect the key ordering of adjacent nodes?

 INFO [CompactionExecutor:1] 2011-03-31 09:57:29,529 CompactionManager.java (line 396) Compacting [SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1793-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1798-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1803-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1808-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1832-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1841-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1852-Data.db')]
 INFO [CompactionExecutor:1] 2011-03-31 09:58:26,437 SSTableWriter.java (line 108) Last written key : DecoratedKey(ba578bed5e75a06d1156dabccc68abd5_www.abc.com_hostinlinks_9223370735413988125_870cc25d-7e4f-437b-aa83-a311dfbf7f88, 62613537386265643565373561303664313135366461626363633638616264355f7777772e6b6f7475736f7a6c756b2e636f6d5f686f7374696e6c696e6b735f393232333337303733353431333938383132355f38373063633235642d376534662d343337622d616138332d613331316466626637663838)
 INFO [CompactionExecutor:1] 2011-03-31 09:58:26,515 SSTableWriter.java (line 109) Current key : DecoratedKey(ba578bed5e75a06d1156dabccc68abd5_www.abc.com_hostinlinks_9223370735413988136_17070303-ab24-4207-83bd-1604e2c159e4, 62613537386265643565373561303664313135366461626363633638616264355f7777772e6b677475736f7a6c756b2e636f6d5f686f7374696e6c696e6b735f393232333337303733353431333938383133365f31373037303330332d616232342d343230372d383362642d313630346532633135396534)
 INFO [CompactionExecutor:1] 2011-03-31 09:58:26,515 SSTableWriter.java (line 110) Writing into file /cassandra/data/table_lists/table_lists-tmp-f-1861-Data.db
ERROR [CompactionExecutor:1] 2011-03-31 09:58:26,516 AbstractCassandraDaemon.java (line 112) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
java.io.IOException: Keys must be written in ascending order.
      at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:111)
      at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:128)
      at org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:452)
      at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:124)
      at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:94)
      at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
      at java.util.concurrent.FutureTask.run(FutureTask.java:138)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      at java.lang.Thread.run(Thread.java:662)


  was:
Hi,

We had to replace a node with faulty ram. Besides the cassandra cluster getting unresponsive, we also observerd a "keys must be written in ascending order" exception on all adjacent quorum nodes, causing the compactation to fail.
We had another node with faulty ram a week earlier, and it was causing the same errors on it's neighbours.

A faulty node shouldn't affect the key ordering of adjacent nodes?

 INFO [CompactionExecutor:1] 2011-03-31 09:57:29,529 CompactionManager.java (line 396) Compacting [SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1793-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1798-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1803-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1808-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1832-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1841-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1852-Data.db')]
 INFO [CompactionExecutor:1] 2011-03-31 09:58:26,437 SSTableWriter.java (line 108) Last written key : DecoratedKey(ba578bed5e75a06d1156dabccc68abd5_www.abc.com_hostinlinks_9223370735413988125_870cc25d-7e4f-437b-aa83-a311dfbf7f88, 62613537386265643565373561303664313135366461626363633638616264355f7777772e6b6f7475736f7a6c756b2e636f6d5f686f7374696e6c696e6b735f393232333337303733353431333938383132355f38373063633235642d376534662d343337622d616138332d613331316466626637663838)
 INFO [CompactionExecutor:1] 2011-03-31 09:58:26,515 SSTableWriter.java (line 109) Current key : DecoratedKey(ba578bed5e75a06d1156dabccc68abd5_www.abc.com_hostinlinks_9223370735413988136_17070303-ab24-4207-83bd-1604e2c159e4, 62613537386265643565373561303664313135366461626363633638616264355f7777772e6b677475736f7a6c756b2e636f6d5f686f7374696e6c696e6b735f393232333337303733353431333938383133365f31373037303330332d616232342d343230372d383362642d313630346532633135396534)
 INFO [CompactionExecutor:1] 2011-03-31 09:58:26,515 SSTableWriter.java (line 110) Writing into file /cassandra/data/table_lists/table_lists-tmp-f-1861-Data.db
ERROR [CompactionExecutor:1] 2011-03-31 09:58:26,516 AbstractCassandraDaemon.java (line 112) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
java.io.IOException: Keys must be written in ascending order.
      at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:111)
      at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:128)
      at org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:452)
      at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:124)
      at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:94)
      at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
      at java.util.concurrent.FutureTask.run(FutureTask.java:138)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      at java.lang.Thread.run(Thread.java:662)



> Faulty memory causes adjacent nodes to have invalid data and fail compactation (java.io.IOException: Keys must be written in ascending order.)
> ----------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-2408
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2408
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 0.7.4
>            Reporter: Thibaut
>             Fix For: 0.7.5
>
>
> Hi,
> We had to replace a node with faulty ram. Besides the cassandra cluster getting unresponsive, we also observerd a "keys must be written in ascending order" exception on all adjacent quorum nodes, causing their compactation to fail.
> We had another node with faulty ram a week earlier, and it was causing the same errors on it's neighbours.
> A faulty node shouldn't affect the key ordering of adjacent nodes?
>  INFO [CompactionExecutor:1] 2011-03-31 09:57:29,529 CompactionManager.java (line 396) Compacting [SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1793-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1798-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1803-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1808-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1832-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1841-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1852-Data.db')]
>  INFO [CompactionExecutor:1] 2011-03-31 09:58:26,437 SSTableWriter.java (line 108) Last written key : DecoratedKey(ba578bed5e75a06d1156dabccc68abd5_www.abc.com_hostinlinks_9223370735413988125_870cc25d-7e4f-437b-aa83-a311dfbf7f88, 62613537386265643565373561303664313135366461626363633638616264355f7777772e6b6f7475736f7a6c756b2e636f6d5f686f7374696e6c696e6b735f393232333337303733353431333938383132355f38373063633235642d376534662d343337622d616138332d613331316466626637663838)
>  INFO [CompactionExecutor:1] 2011-03-31 09:58:26,515 SSTableWriter.java (line 109) Current key : DecoratedKey(ba578bed5e75a06d1156dabccc68abd5_www.abc.com_hostinlinks_9223370735413988136_17070303-ab24-4207-83bd-1604e2c159e4, 62613537386265643565373561303664313135366461626363633638616264355f7777772e6b677475736f7a6c756b2e636f6d5f686f7374696e6c696e6b735f393232333337303733353431333938383133365f31373037303330332d616232342d343230372d383362642d313630346532633135396534)
>  INFO [CompactionExecutor:1] 2011-03-31 09:58:26,515 SSTableWriter.java (line 110) Writing into file /cassandra/data/table_lists/table_lists-tmp-f-1861-Data.db
> ERROR [CompactionExecutor:1] 2011-03-31 09:58:26,516 AbstractCassandraDaemon.java (line 112) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.io.IOException: Keys must be written in ascending order.
>       at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:111)
>       at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:128)
>       at org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:452)
>       at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:124)
>       at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:94)
>       at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>       at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>       at java.lang.Thread.run(Thread.java:662)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira