You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "jean carlo rivera ura (JIRA)" <ji...@apache.org> on 2016/06/01 08:19:00 UTC
[jira] [Comment Edited] (CASSANDRA-9126)
java.lang.RuntimeException: Last written key DecoratedKey >= current key
DecoratedKey
[ https://issues.apache.org/jira/browse/CASSANDRA-9126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15304173#comment-15304173 ]
jean carlo rivera ura edited comment on CASSANDRA-9126 at 6/1/16 8:18 AM:
--------------------------------------------------------------------------
[~mambocab] We are using cassandra 2.1.12
In production we got a first error doing a repair (-pr -par)
{code:none}
ERROR [ValidationExecutor:3283] 2016-05-26 09:37:37,911 Validator.java:245 - Failed creating a merkle tree for [repair #b16c1f30-2314-11e6-97fe-dd058ca99653 on keyspace1/table1, (8699512854132214411,8702471448538509513]], /192.168.1.144 (see log for details)
ERROR [ValidationExecutor:3283] 2016-05-26 09:37:37,913 CassandraDaemon.java:227 - Exception in thread Thread[ValidationExecutor:3283,1,main]
java.lang.AssertionError: row DecoratedKey(8699513492008207074, 434c503031303030303030303036363632323233) received out of order wrt DecoratedKey(8702470199759211565, 4b454e4f42494a52432d43553130333031353232313032)
at org.apache.cassandra.repair.Validator.add(Validator.java:126) ~[apache-cassandra-2.1.12.jar:2.1.12]
at org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:1038) ~[apache-cassandra-2.1.12.jar:2.1.12]
at org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:89) ~[apache-cassandra-2.1.12.jar:2.1.12]
at org.apache.cassandra.db.compaction.CompactionManager$9.call(CompactionManager.java:649) ~[apache-cassandra-2.1.12.jar:2.1.12]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_60]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_60]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_60]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
{code}
We decided to do another a repair over this token range 8699512854132214411,8702471448538509513 using the next command.
{code:none}
root@node021:~$ nodetool repair keyspace1 -st 8699512854132214411 -et 8702471448538509513
[2016-05-27 14:19:20,768] Starting repair command #37, repairing 1 ranges for keyspace keyspace1 (parallelism=SEQUENTIAL, full=true)
[2016-05-27 14:19:38,489] Repair session 3dd2f250-2405-11e6-b33f-ab1665ddb9c0 for range (8699512854132214411,8702471448538509513] failed with error org.apache.cassandra.exceptions.RepairException: [repair #3dd2f250-2405-11e6-b33f-ab1665ddb9c0 on keyspace1/table2, (8699512854132214411,8702471448538509513]] Validation failed in /192.168.2.137
[2016-05-27 14:19:38,489] Repair command #37 finished
error: nodetool failed, check server logs
-- StackTrace --
java.lang.RuntimeException: nodetool failed, check server logs
at org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:292)
at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:204)
{code}
Unlike the first time, this error comes from the node 192.168.2.137. Checking the log, the node failed creating a merkle tree for the table table2. This table is LCS
{code:none}
ERROR [ValidationExecutor:3427] 2016-05-27 14:19:38,466 Validator.java:245 - Failed creating a merkle tree for [repair #3dd2f250-2405-11e6-b33f-ab1665ddb9c0 on keyspace1/table2, (8699512854132214411,8702471448538509513]], /192.168.1.154 (see log for details)
ERROR [ValidationExecutor:3427] 2016-05-27 14:19:38,467 CassandraDaemon.java:227 - Exception in thread Thread[ValidationExecutor:3427,1,main]
java.lang.AssertionError: row DecoratedKey(8699513197702636918, 49442d5350502d3130302d54513944486f7333704e68492b387a6e786870653347614d766a6832597744446a5543446d4b397a393545) received out of order wrt DecoratedKey(8702471248515353880, 49442d5350502d3130302d704a3063676b4446714b5075644b68654273672f686c6a4759767264724f4b6e664d3176765a7a70416d6b)
at org.apache.cassandra.repair.Validator.add(Validator.java:126) ~[apache-cassandra-2.1.12.jar:2.1.12]
at org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:1038) ~[apache-cassandra-2.1.12.jar:2.1.12]
at org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:89) ~[apache-cassandra-2.1.12.jar:2.1.12]
at org.apache.cassandra.db.compaction.CompactionManager$9.call(CompactionManager.java:649) ~[apache-cassandra-2.1.12.jar:2.1.12]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_60]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_60]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_60]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
{code}
We tried to run again the same repair on the same node with the same tokenrange and it finished without error
{code:none}
root@node021:~$ nodetool repair keyspace1 -st 8699512854132214411 -et 8702471448538509513
[2016-05-27 14:27:00,419] Starting repair command #38, repairing 1 ranges for keyspace keyspace1 (parallelism=SEQUENTIAL, full=true)
[2016-05-27 14:36:18,239] Repair session 4fcda620-2406-11e6-b33f-ab1665ddb9c0 for range (8699512854132214411,8702471448538509513] finished
[2016-05-27 14:36:18,239] Repair command #38 finished
{code}
It seems this error comes up by hazard. We notice that our first repair -pr -par has finished repairing all the tokenranges, but that one we got the error.
was (Author: carlo_4002):
[~mambocab] We are using cassandra 2.1.12
In production we got a first error doing a repair (-pr -par)
{code:none}
ERROR [ValidationExecutor:3283] 2016-05-26 09:37:37,911 Validator.java:245 - Failed creating a merkle tree for [repair #b16c1f30-2314-11e6-97fe-dd058ca99653 on pns_fr_prod/pig, (8699512854132214411,8702471448538509513]], /10.98.255.144 (see log for details)
ERROR [ValidationExecutor:3283] 2016-05-26 09:37:37,913 CassandraDaemon.java:227 - Exception in thread Thread[ValidationExecutor:3283,1,main]
node021.cassandra.prod.pns.s1.p.fti.net: /var/opt/hosting/log/cassandra/system.log-java.lang.AssertionError: row DecoratedKey(8699513492008207074, 434c503031303030303030303036363632323233) received out of order wrt DecoratedKey(8702470199759211565, 4b454e4f42494a52432d435531303330313532323130
{code}
We decided to do another a repair over this token range 8699512854132214411,8702471448538509513 using the next command.
{code:none}
root@node021[SPH][PROD][PnS3]:~$ nodetool repair pns_fr_prod -st 8699512854132214411 -et 8702471448538509513
[2016-05-27 14:19:20,768] Starting repair command #37, repairing 1 ranges for keyspace pns_fr_prod (parallelism=SEQUENTIAL, full=true)
[2016-05-27 14:19:38,489] Repair session 3dd2f250-2405-11e6-b33f-ab1665ddb9c0 for range (8699512854132214411,8702471448538509513] failed with error org.apache.cassandra.exceptions.RepairException: [repair #3dd2f250-2405-11e6-b33f-ab1665ddb9c0 on pns_fr_prod/syndic, (8699512854132214411,8702471448538509513]] Validation failed in /10.234.72.137
[2016-05-27 14:19:38,489] Repair command #37 finished
error: nodetool failed, check server logs
-- StackTrace --
java.lang.RuntimeException: nodetool failed, check server logs
at org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:292)
at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:204)
{code}
Unlike the first time, this error comes from the node 10.234.72.137. Checking the log, the node failed creating a merkle tree for the table syndic. This table is LCS
{code:none}
ERROR [ValidationExecutor:3427] 2016-05-27 14:19:38,466 Validator.java:245 - Failed creating a merkle tree for [repair #3dd2f250-2405-11e6-b33f-ab1665ddb9c0 on pns_fr_prod/syndic, (8699512854132214411,8702471448538509513]], /10.98.255.154 (see log for details)
ERROR [ValidationExecutor:3427] 2016-05-27 14:19:38,467 CassandraDaemon.java:227 - Exception in thread Thread[ValidationExecutor:3427,1,main]
java.lang.AssertionError: row DecoratedKey(8699513197702636918, 49442d5350502d3130302d54513944486f7333704e68492b387a6e786870653347614d766a6832597744446a5543446d4b397a393545) received out of order wrt DecoratedKey(8702471248515353880, 49442d5350502d3130302d704a3063676b4446714b5075644b68654273672f686c6a4759767264724f4b6e664d3176765a7a70416d6b)
at org.apache.cassandra.repair.Validator.add(Validator.java:126) ~[apache-cassandra-2.1.12.jar:2.1.12]
at org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:1038) ~[apache-cassandra-2.1.12.jar:2.1.12]
at org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:89) ~[apache-cassandra-2.1.12.jar:2.1.12]
at org.apache.cassandra.db.compaction.CompactionManager$9.call(CompactionManager.java:649) ~[apache-cassandra-2.1.12.jar:2.1.12]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_60]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_60]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_60]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
{code}
We tried to run again the same repair on the same node with the same tokenrange and it finished without error
{code:none}
root@node021[SPH][PROD][PnS3]:~$ nodetool repair pns_fr_prod -st 8699512854132214411 -et 8702471448538509513
[2016-05-27 14:27:00,419] Starting repair command #38, repairing 1 ranges for keyspace pns_fr_prod (parallelism=SEQUENTIAL, full=true)
[2016-05-27 14:36:18,239] Repair session 4fcda620-2406-11e6-b33f-ab1665ddb9c0 for range (8699512854132214411,8702471448538509513] finished
[2016-05-27 14:36:18,239] Repair command #38 finished
{code}
It seems this error comes up by hazard. We notice that our first repair -pr -par has finished repairing all the tokenranges, but that one we got the error.
> java.lang.RuntimeException: Last written key DecoratedKey >= current key DecoratedKey
> -------------------------------------------------------------------------------------
>
> Key: CASSANDRA-9126
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9126
> Project: Cassandra
> Issue Type: Bug
> Reporter: srinivasu gottipati
> Priority: Critical
> Attachments: cassandra-system.log
>
>
> Cassandra V: 2.0.14,
> Getting the following exceptions while trying to compact (I see this issue was raised in earlier versions and marked as closed. However it still appears in 2.0.14). In our case, compaction is not getting succeeded and keep failing with this error.:
> {code}java.lang.RuntimeException: Last written key DecoratedKey(3462767860784856708, 354038323137333038305f3330325f31355f474d4543454f) >= current key DecoratedKey(3462334604624154281, 354036333036353334315f3336315f31355f474d4543454f) writing into {code}
> .......
> Stacktrace:{code}
> at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:143)
> at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:166)
> at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:167)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
> at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
> at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:198)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745){code}
> Any help is greatly appreciated
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)