You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Ruggero Marchei (JIRA)" <ji...@apache.org> on 2015/09/19 13:19:04 UTC
[jira] [Created] (CASSANDRA-10375) nodetool repair fails
Ruggero Marchei created CASSANDRA-10375:
-------------------------------------------
Summary: nodetool repair fails
Key: CASSANDRA-10375
URL: https://issues.apache.org/jira/browse/CASSANDRA-10375
Project: Cassandra
Issue Type: Bug
Components: Core, Tools
Environment: multi-dc Cassandra 2.2.1 cluster with 6 nodes, 3 per DC
Reporter: Ruggero Marchei
When I'm running a *nodetool repair* it often gets stalled after few seconds:
{code}[2015-09-19 11:12:13,807] Repair session 479ca1c0-5ebf-11e5-9619-3f4813058061 for range (40511972970986385,59154612555757611] failed with error [repair #479ca1c0-5ebf-11e5-9619-3f4813058061 on static_assets/assets, (40511972970986385,59154612555757611]] Validation failed in /10.8.34.113 (progress: 0%)
[2015-09-19 11:12:13,812] Repair session 479cc8d1-5ebf-11e5-9619-3f4813058061 for range (6553929828848556033,6576029219234973671] failed with error [repair #479cc8d1-5ebf-11e5-9619-3f4813058061 on static_assets/assets, (6553929828848556033,6576029219234973671]] Validation failed in /10.8.34.113 (progress: 0%)
{code}
At the same time I have this exception on another node:
{code}
ERROR [ValidationExecutor:66] 2015-09-19 11:12:13,825 CompactionManager.java:1070 - Cannot start multiple repair sessions over the same sstables
ERROR [ValidationExecutor:66] 2015-09-19 11:12:13,826 Validator.java:246 - Failed creating a merkle tree for [repair #479c2c90-5ebf-11e5-9619-3f4813058061 on static_assets/assets, (-2926621365236563900,-2916361392298929067]], /10.8.34.113 (see log for details)
ERROR [ValidationExecutor:66] 2015-09-19 11:12:13,826 CassandraDaemon.java:183 - Exception in thread Thread[ValidationExecutor:66,1,main]
java.lang.RuntimeException: Cannot start multiple repair sessions over the same sstables
at org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:1071) ~[apache-cassandra-2.2.1.jar:2.2.1]
at org.apache.cassandra.db.compaction.CompactionManager.access$700(CompactionManager.java:94) ~[apache-cassandra-2.2.1.jar:2.2.1]
at org.apache.cassandra.db.compaction.CompactionManager$10.call(CompactionManager.java:669) ~[apache-cassandra-2.2.1.jar:2.2.1]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_45]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_45]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_45]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
ERROR [ValidationExecutor:66] 2015-09-19 11:12:13,900 CompactionManager.java:1070 - Cannot start multiple repair sessions over the same sstables
ERROR [ValidationExecutor:66] 2015-09-19 11:12:13,900 Validator.java:246 - Failed creating a merkle tree for [repair #479c53a1-5ebf-11e5-9619-3f4813058061 on static_assets/assets, (8236929501578674892,8238760988019827700]], /10.8.34.113 (see log for details)
ERROR [ValidationExecutor:68] 2015-09-19 11:12:13,900 CompactionManager.java:1070 - Cannot start multiple repair sessions over the same sstables
ERROR [ValidationExecutor:67] 2015-09-19 11:12:13,900 CompactionManager.java:1070 - Cannot start multiple repair sessions over the same sstables
ERROR [ValidationExecutor:66] 2015-09-19 11:12:13,901 CassandraDaemon.java:183 - Exception in thread Thread[ValidationExecutor:66,1,main]
java.lang.RuntimeException: Cannot start multiple repair sessions over the same sstables
at org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:1071) ~[apache-cassandra-2.2.1.jar:2.2.1]
at org.apache.cassandra.db.compaction.CompactionManager.access$700(CompactionManager.java:94) ~[apache-cassandra-2.2.1.jar:2.2.1]
at org.apache.cassandra.db.compaction.CompactionManager$10.call(CompactionManager.java:669) ~[apache-cassandra-2.2.1.jar:2.2.1]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_45]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_45]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_45]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
ERROR [ValidationExecutor:68] 2015-09-19 11:12:13,901 Validator.java:246 - Failed creating a merkle tree for [repair #479cc8d1-5ebf-11e5-9619-3f4813058061 on static_assets/assets, (6553929828848556033,6576029219234973671]], /10.8.34.113 (see log for details)
ERROR [ValidationExecutor:67] 2015-09-19 11:12:13,901 Validator.java:246 - Failed creating a merkle tree for [repair #479ca1c0-5ebf-11e5-9619-3f4813058061 on static_assets/assets, (40511972970986385,59154612555757611]], /10.8.34.113 (see log for details)
ERROR [ValidationExecutor:68] 2015-09-19 11:12:13,901 CassandraDaemon.java:183 - Exception in thread Thread[ValidationExecutor:68,1,main]
java.lang.RuntimeException: Cannot start multiple repair sessions over the same sstables
at org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:1071) ~[apache-cassandra-2.2.1.jar:2.2.1]
at org.apache.cassandra.db.compaction.CompactionManager.access$700(CompactionManager.java:94) ~[apache-cassandra-2.2.1.jar:2.2.1]
at org.apache.cassandra.db.compaction.CompactionManager$10.call(CompactionManager.java:669) ~[apache-cassandra-2.2.1.jar:2.2.1]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_45]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_45]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_45]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
ERROR [ValidationExecutor:67] 2015-09-19 11:12:13,901 CassandraDaemon.java:183 - Exception in thread Thread[ValidationExecutor:67,1,main]
java.lang.RuntimeException: Cannot start multiple repair sessions over the same sstables
at org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:1071) ~[apache-cassandra-2.2.1.jar:2.2.1]
at org.apache.cassandra.db.compaction.CompactionManager.access$700(CompactionManager.java:94) ~[apache-cassandra-2.2.1.jar:2.2.1]
at org.apache.cassandra.db.compaction.CompactionManager$10.call(CompactionManager.java:669) ~[apache-cassandra-2.2.1.jar:2.2.1]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_45]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_45]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_45]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
{code}
And on the remaining nodes I have something similar to this:
{code}
ERROR [Reference-Reaper:1] 2015-09-19 11:12:14,147 Ref.java:187 - LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@320d7d92) to class org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier@780334113:/var/lib/cassandra/data/static_assets/assets-ceb62f1056df11e5865db530abf562fa/la-2447-big was not released before the reference was garbage collected
ERROR [Reference-Reaper:1] 2015-09-19 11:12:14,147 Ref.java:187 - LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@972fce9) to class org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier@389940435:/var/lib/cassandra/data/static_assets/assets-ceb62f1056df11e5865db530abf562fa/la-6-big was not released before the reference was garbage collected
ERROR [Reference-Reaper:1] 2015-09-19 11:12:14,147 Ref.java:187 - LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@5a2a0617) to class org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier@156533865:/var/lib/cassandra/data/static_assets/assets-ceb62f1056df11e5865db530abf562fa/la-4-big was not released before the reference was garbage collected
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)