You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Dmitrii Saprykin (JIRA)" <ji...@apache.org> on 2015/08/22 13:58:46 UTC
[jira] [Commented] (CASSANDRA-9126) java.lang.RuntimeException:
Last written key DecoratedKey >= current key DecoratedKey
[ https://issues.apache.org/jira/browse/CASSANDRA-9126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14708008#comment-14708008 ]
Dmitrii Saprykin commented on CASSANDRA-9126:
---------------------------------------------
I also experience the same issue. I did 'nodetool scrub' for affected columnfamily without any errors. And issue persists after scrub
We use:
Cassandra 2.0.15, LCS, HSHA.
Keyspace is 8 nodes, RF=3, ~300GB per node (~1TB of unique data without RF), ~10.000.000 keys
Before scrub we had:
{noformat}
ERROR [CompactionExecutor:499] 2015-08-21 20:26:24,438 CassandraDaemon.java (line 258) Exception in thread Thread[CompactionExecutor:499,1,main]
java.lang.RuntimeException: Last written key DecoratedKey(1928386083417163406, 0313ad7e) >= current key DecoratedKey(1911131194341551311, 010b2fb4) writing into /data/cassandra/storage/data/bigks/chunks/bigks-chunks-tmp-jb-26880-Data.db
at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:143)
at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:166)
at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:170)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:198)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
ERROR [CompactionExecutor:498] 2015-08-21 20:29:14,655 CassandraDaemon.java (line 258) Exception in thread Thread[CompactionExecutor:498,1,main]
java.lang.RuntimeException: Last written key DecoratedKey(1928386083417163406, 0313ad7e) >= current key DecoratedKey(1911131194341551311, 010b2fb4) writing into /data/cassandra/storage/data/bigks/chunks/bigks-chunks-tmp-jb-26954-Data.db
at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:143)
at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:166)
at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:170)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:198)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{noformat}
After scrub I see following errors during compaction:
{noformat}
ERROR [CompactionExecutor:815] 2015-08-22 03:16:17,153 CassandraDaemon.java (line 258) Exception in thread Thread[CompactionExecutor:815,1,main]
java.lang.RuntimeException: Last written key DecoratedKey(1922197470632862656, 02c3a415) >= current key DecoratedKey(1916775706087412870, 013a445b) writing into /data/cassandra/storage/data/bigks/chunks/bigks-chunks-tmp-jb-28982-Data.db
at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:143)
at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:166)
at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:170)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:198)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
ERROR [CompactionExecutor:819] 2015-08-22 03:18:10,152 CassandraDaemon.java (line 258) Exception in thread Thread[CompactionExecutor:819,1,main]
java.lang.RuntimeException: Last written key DecoratedKey(1922197470632862656, 02c3a415) >= current key DecoratedKey(1916775706087412870, 013a445b) writing into /data/cassandra/storage/data/bigks/chunks/bigks-chunks-tmp-jb-28999-Data.db
at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:143)
at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:166)
at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:170)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:198)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
ERROR [ValidationExecutor:592] 2015-08-22 03:20:33,374 Validator.java (line 255) Failed creating a merkle tree for [repair #a76b0d50-48a6-11e5-849b-3bfefac09ede on bigks/chunks, (1916658571049540871,1922266576662767350]], /130.14.18.93 (see log for details)
ERROR [ValidationExecutor:592] 2015-08-22 03:20:33,374 CassandraDaemon.java (line 258) Exception in thread Thread[ValidationExecutor:592,1,main]
java.lang.AssertionError: row DecoratedKey(1916775706087412870, 013a445b) received out of order wrt DecoratedKey(1922197470632862656, 02c3a415)
at org.apache.cassandra.repair.Validator.add(Validator.java:137)
at org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:806)
at org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:63)
at org.apache.cassandra.db.compaction.CompactionManager$8.call(CompactionManager.java:404)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
ERROR [ValidationExecutor:592] 2015-08-22 03:20:37,287 Validator.java (line 255) Failed creating a merkle tree for [repair #aaab6b40-48a6-11e5-849b-3bfefac09ede on bigks/chunks, (1924415850871700726,1932964198982195249]], /130.14.18.93 (see log for details)
ERROR [ValidationExecutor:592] 2015-08-22 03:20:37,287 CassandraDaemon.java (line 258) Exception in thread Thread[ValidationExecutor:592,1,main]
java.lang.AssertionError: row DecoratedKey(1924416666703877076, 00e57790) received out of order wrt DecoratedKey(1928782589289822441, 030cb3d9)
at org.apache.cassandra.repair.Validator.add(Validator.java:137)
at org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:806)
at org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:63)
at org.apache.cassandra.db.compaction.CompactionManager$8.call(CompactionManager.java:404)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
ERROR [CompactionExecutor:1005] 2015-08-22 06:07:15,676 CassandraDaemon.java (line 258) Exception in thread Thread[CompactionExecutor:1005,1,main]
java.lang.RuntimeException: Last written key DecoratedKey(1922197470632862656, 02c3a415) >= current key DecoratedKey(1916775706087412870, 013a445b) writing into /data/cassandra/storage/data/bigks/chunks/bigks-chunks-tmp-jb-29048-Data.db
at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:143)
at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:166)
at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:170)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:198)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
ERROR [CompactionExecutor:1009] 2015-08-22 06:11:24,670 CassandraDaemon.java (line 258) Exception in thread Thread[CompactionExecutor:1009,1,main]
java.lang.RuntimeException: Last written key DecoratedKey(1922197470632862656, 02c3a415) >= current key DecoratedKey(1916775706087412870, 013a445b) writing into /data/cassandra/storage/data/bigks/chunks/bigks-chunks-tmp-jb-29071-Data.db
at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:143)
at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:166)
at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:170)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:198)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{noformat}
> java.lang.RuntimeException: Last written key DecoratedKey >= current key DecoratedKey
> -------------------------------------------------------------------------------------
>
> Key: CASSANDRA-9126
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9126
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: srinivasu gottipati
> Priority: Critical
> Fix For: 2.0.x
>
>
> Cassandra V: 2.0.14,
> Getting the following exceptions while trying to compact (I see this issue was raised in earlier versions and marked as closed. However it still appears in 2.0.14). In our case, compaction is not getting succeeded and keep failing with this error.:
> {code}java.lang.RuntimeException: Last written key DecoratedKey(3462767860784856708, 354038323137333038305f3330325f31355f474d4543454f) >= current key DecoratedKey(3462334604624154281, 354036333036353334315f3336315f31355f474d4543454f) writing into {code}
> .......
> Stacktrace:{code}
> at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:143)
> at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:166)
> at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:167)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
> at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
> at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:198)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745){code}
> Any help is greatly appreciated
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)