You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Marcus Eriksson (JIRA)" <ji...@apache.org> on 2015/01/21 15:59:35 UTC
[jira] [Resolved] (CASSANDRA-8120) Failed creating a merkle tree
during repair.
[ https://issues.apache.org/jira/browse/CASSANDRA-8120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Marcus Eriksson resolved CASSANDRA-8120.
----------------------------------------
Resolution: Duplicate
I think this is a duplicate of CASSANDRA-8211 - if we have overlap, this is the error we would see
> Failed creating a merkle tree during repair.
> --------------------------------------------
>
> Key: CASSANDRA-8120
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8120
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Ubuntu 12.04.5 LTS. 3.2.0-69-virtual.
> Reporter: jason milliron
>
> Seeing this while running nodetool repair on several nodes of our new (not an upgrade) 2.1.0 cluster. Multiple keyspaces. Doesn't hang the repair job. Storage is SSD ephemeral (EC2). No storage errors in syslog.
> {noformat}
> ERROR [ValidationExecutor:39] 2014-10-13 22:20:51,140 Validator.java:242 - Failed creating a merkle tree for [repair #1523dfa0-5327-11e4-925c-65a001aa6d80 on censored/censored, (-4468426774357114003,-4433737571269222163]], /10.37.666.175 (see log for details)
> INFO [AntiEntropySessions:37] 2014-10-13 22:20:51,141 RepairSession.java:260 - [repair #308e2750-5327-11e4-925c-65a001aa6d80] new session: will sync ip-10-37-666-175.ec2.internal/10.37.666.175, /10.167.666.137, /10.164.666.81 on range (6870883176570587821,6871062738154464368] for censored.[censored]
> INFO [AntiEntropyStage:1] 2014-10-13 22:20:51,142 ColumnFamilyStore.java:856 - Enqueuing flush of censored: 3227760 (0%) on-heap, 0 (0%) off-heap
> INFO [MemtableFlushWriter:2295] 2014-10-13 22:20:51,145 Memtable.java:326 - Writing Memtable-censored@2052537453(380369 serialized bytes, 8235 ops, 0%/0% of on/off-heap limit)
> ERROR [AntiEntropySessions:34] 2014-10-13 22:20:51,148 RepairSession.java:303 - [repair #1523dfa0-5327-11e4-925c-65a001aa6d80] session completed with the following error
> org.apache.cassandra.exceptions.RepairException: [repair #1523dfa0-5327-11e4-925c-65a001aa6d80 on censored/censored, (-4468426774357114003,-4433737571269222163]] Validation failed in /10.37.666.175
> at org.apache.cassandra.repair.RepairSession.validationComplete(RepairSession.java:166) ~[apache-cassandra-2.1.0.jar:2.1.0]
> at org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:384) ~[apache-cassandra-2.1.0.jar:2.1.0]
> at org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:123) ~[apache-cassandra-2.1.0.jar:2.1.0]
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62) ~[apache-cassandra-2.1.0.jar:2.1.0]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_67]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_67]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67]
> ERROR [ValidationExecutor:39] 2014-10-13 22:20:51,148 CassandraDaemon.java:166 - Exception in thread Thread[ValidationExecutor:39,1,main]
> java.lang.AssertionError: row DecoratedKey(-4468391015221140553, 00040000000100000800000000005abc2d00) received out of order wrt DecoratedKey(-4436255891196926644, 0004000000010000080000000000617f4400)
> at org.apache.cassandra.repair.Validator.add(Validator.java:136) ~[apache-cassandra-2.1.0.jar:2.1.0]
> at org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:916) ~[apache-cassandra-2.1.0.jar:2.1.0]
> at org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:100) ~[apache-cassandra-2.1.0.jar:2.1.0]
> at org.apache.cassandra.db.compaction.CompactionManager$9.call(CompactionManager.java:560) ~[apache-cassandra-2.1.0.jar:2.1.0]
> at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_67]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_67]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_67]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67]
> ERROR [AntiEntropySessions:34] 2014-10-13 22:20:51,149 CassandraDaemon.java:166 - Exception in thread Thread[AntiEntropySessions:34,5,RMI Runtime]
> java.lang.RuntimeException: org.apache.cassandra.exceptions.RepairException: [repair #1523dfa0-5327-11e4-925c-65a001aa6d80 on censored/censored, (-4468426774357114003,-4433737571269222163]] Validation failed in /10.37.666.175
> at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.jar:na]
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32) ~[apache-cassandra-2.1.0.jar:2.1.0]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.7.0_67]
> at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_67]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_67]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_67]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67]
> Caused by: org.apache.cassandra.exceptions.RepairException: [repair #1523dfa0-5327-11e4-925c-65a001aa6d80 on censored/censored, (-4468426774357114003,-4433737571269222163]] Validation failed in /10.37.666.175
> at org.apache.cassandra.repair.RepairSession.validationComplete(RepairSession.java:166) ~[apache-cassandra-2.1.0.jar:2.1.0]
> at org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:384) ~[apache-cassandra-2.1.0.jar:2.1.0]
> at org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:123) ~[apache-cassandra-2.1.0.jar:2.1.0]
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62) ~[apache-cassandra-2.1.0.jar:2.1.0]
> ... 3 common frames omitted
> INFO [MemtableFlushWriter:2295] 2014-10-13 22:20:51,180 Memtable.java:360 - Completed flushing /ephem/data/censored/censored-f69e4e9049ac11e4bc96bfea6c7ad14a/censored-censored-ka-3185-Data.db (126124 bytes) for commitlog position ReplayPosition(segmentId=1412194655943, position=28904247)
> INFO [RepairJobTask:3] 2014-10-13 22:20:51,259 RepairJob.java:145 - [repair #308e2750-5327-11e4-925c-65a001aa6d80] requesting merkle trees for censored (to [/10.167.666.137, /10.164.666.81, ip-10-37-666-175.ec2.internal/10.37.666.175])
> INFO [AntiEntropyStage:1] 2014-10-13 22:20:51,351 RepairSession.java:171 - [repair #308e2750-5327-11e4-925c-65a001aa6d80] Received merkle tree for censored from /10.167.666.137
> INFO [AntiEntropyStage:1] 2014-10-13 22:20:51,446 RepairSession.java:171 - [repair #308e2750-5327-11e4-925c-65a001aa6d80] Received merkle tree for censored from /10.164.666.81
> INFO [ValidationExecutor:39] 2014-10-13 22:20:51,452 SSTableReader.java:383 - Opening /ephem/data/censored/censored-f69e4e9049ac11e4bc96bfea6c7ad14a/snapshots/308e2750-5327-11e4-925c-65a001aa6d80/censored-censored-ka-3173 (87123 bytes)
> INFO [ValidationExecutor:39] 2014-10-13 22:20:51,453 SSTableReader.java:383 - Opening /ephem/data/censored/censored-f69e4e9049ac11e4bc96bfea6c7ad14a/snapshots/308e2750-5327-11e4-925c-65a001aa6d80/censored-censored-ka-3181 (7010 bytes)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)