exception during add node due to test beforeAppend on SSTableWriter

["Desimpel, Ignace" <Ig...@nuance.com>]

4 node, byte ordered, LCS, 3 Compaction Executors, replication factor 1
Code is 2.0.4 version but with patch for CASSANDRA-6638<https://issues.apache.org/jira/browse/CASSANDRA-6638> However, no cleanup is run so patch should not play a roll

4 node cluster is started and insert/queries are done up to about only 10 GB of data on each node.
Then decommission one node, and delete local files.
Then add node again.
Exception : see below.

Any idea?

Regards,
Ignace Desimpel


  *   2014-01-31 17:12:02.600 ==>> Bootstrap is streaming data from other nodes... Please wait ...
  *   2014-01-31 17:12:02.600 ==>> Bootstrap stream state : rx= 29.00 tx= 100.00 Please wait ...
  *   2014-01-31 17:12:18.908 Enqueuing flush of Memtable-compactions_in_progress@350895652(0/0 serialized/live bytes, 1 ops)
  *   2014-01-31 17:12:18.908 Writing Memtable-compactions_in_progress@350895652(0/0 serialized/live bytes, 1 ops)
  *   2014-01-31 17:12:19.009 Completed flushing ../../../../data/cdi.cassandra.cdi/dbdatafile/system/compactions_in_progress/system-compactions_in_progress-jb-74-Data.db (42 bytes) for commitlog position ReplayPosition(segmentId=1391184546183, position=561494)
  *   2014-01-31 17:12:19.018 Exception in thread Thread[CompactionExecutor:1,1,main]
  *   java.lang.RuntimeException: Last written key DecoratedKey(8afc923701038000000000178575, 8afc923701038000000000178575) >= current key DecoratedKey(6e0bb95501038000000003dfdd1d, 6e0bb95501038000000003dfdd1d) writing into /media/datadrive1/cdi.cassandra.cdi/dbdatafile/Ks100K/ForwardLongFunction/Ks100K-ForwardLongFunction-tmp-jb-159-Data.db
  *   at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:142) ~[apache-cassandra-2.0.4-SNAPSHOT.jar:2.0.4-SNAPSHOT]
  *   at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:165) ~[apache-cassandra-2.0.4-SNAPSHOT.jar:2.0.4-SNAPSHOT]
  *   at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:160) ~[apache-cassandra-2.0.4-SNAPSHOT.jar:2.0.4-SNAPSHOT]
  *   at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) ~[apache-cassandra-2.0.4-SNAPSHOT.jar:2.0.4-SNAPSHOT]
  *   at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.0.4-SNAPSHOT.jar:2.0.4-SNAPSHOT]
  *   at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60) ~[apache-cassandra-2.0.4-SNAPSHOT.jar:2.0.4-SNAPSHOT]
  *   at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59) ~[apache-cassandra-2.0.4-SNAPSHOT.jar:2.0.4-SNAPSHOT]
  *   at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:197) ~[apache-cassandra-2.0.4-SNAPSHOT.jar:2.0.4-SNAPSHOT]
  *   at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.7.0_40]
  *   at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_40]
  *   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_40]
  *   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_40]
  *   at java.lang.Thread.run(Thread.java:724) [na:1.7.0_40]