You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Igor Zubchenok (JIRA)" <ji...@apache.org> on 2018/02/08 22:49:00 UTC
[jira] [Updated] (CASSANDRA-14224) Failed to create merkel tree
[ https://issues.apache.org/jira/browse/CASSANDRA-14224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Igor Zubchenok updated CASSANDRA-14224:
---------------------------------------
Description:
While repairing a node with nodetool -full -pr -dcpar, another node throws an assertion exception and failed to create merkel tree:
{code:java}
DEBUG [ValidationExecutor:25] 2018-02-08 21:31:42,581 SSTableReader.java:503 - Opening /var/lib/cassandra/data/prod/invoices-a8454000c10711e4896155ab730ce02d/snapshots/689d7a80-0d17-11e8-950c-37c5a1498597/mc-35-big (1.876KiB)
DEBUG [ValidationExecutor:39] 2018-02-08 21:31:42,581 SSTableReader.java:503 - Opening /var/lib/cassandra/data/prod/invoices-a8454000c10711e4896155ab730ce02d/snapshots/689d7a80-0d17-11e8-950c-37c5a1498597/mc-35-big (1.876KiB)
DEBUG [ValidationExecutor:39] 2018-02-08 21:31:42,586 SSTableReader.java:503 - Opening /var/lib/cassandra/data/prod/invoices-a8454000c10711e4896155ab730ce02d/snapshots/689d7a80-0d17-11e8-950c-37c5a1498597/mc-36-big (0.688KiB)
ERROR [ValidationExecutor:25] 2018-02-08 21:31:42,586 Validator.java:268 - Failed creating a merkle tree for [repair #69803961-0d17-11e8-950c-37c5a1498597 on prod/invoices, [(-4165282778092010776,-4162024061639141985]]], /88.99.35.178 (see log for details)
ERROR [ValidationExecutor:25] 2018-02-08 21:31:42,587 CassandraDaemon.java:228 - Exception in thread Thread[ValidationExecutor:25,1,main]
java.lang.AssertionError: null
at org.apache.cassandra.io.sstable.format.SSTableReader$GlobalTidy.get(SSTableReader.java:2313) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier.setup(SSTableReader.java:2138) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.io.sstable.format.SSTableReader.setup(SSTableReader.java:2081) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:519) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.ColumnFamilyStore.getSnapshotSSTableReader(ColumnFamilyStore.java:1901) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:1335) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.compaction.CompactionManager.access$700(CompactionManager.java:86) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.compaction.CompactionManager$13.call(CompactionManager.java:942) ~[apache-cassandra-3.11.1.jar:3.11.1]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_151]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_151]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_151]
at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81) [apache-cassandra-3.11.1.jar:3.11.1]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_151]
DEBUG [ValidationExecutor:39] 2018-02-08 21:31:42,588 SSTableReader.java:503 - Opening /var/lib/cassandra/data/prod/invoices-a8454000c10711e4896155ab730ce02d/snapshots/689d7a80-0d17-11e8-950c-37c5a1498597/mc-33-big (9.891MiB)
ERROR [NonPeriodicTasks:1] 2018-02-08 21:31:42,589 CassandraDaemon.java:228 - Exception in thread Thread[NonPeriodicTasks:1,5,main]
java.lang.NullPointerException: null
at org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier$1.run(SSTableReader.java:2192) ~[apache-cassandra-3.11.1.jar:3.11.1]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_151]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_151]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[na:1.8.0_151]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[na:1.8.0_151]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_151]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_151]
at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81) [apache-cassandra-3.11.1.jar:3.11.1]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_151]{code}
was:
While repairing a node with nodetool -full -pr -dcpar, another node throws an assertion exception and failed to create merkel tree:
{code:java}
DEBUG [ValidationExecutor:25] 2018-02-08 21:31:42,581 SSTableReader.java:503 - Opening /var/lib/cassandra/data/prod/invoices-a8454000c10711e4896155ab730ce02d/snapshots/689d7a80-0d17-11e8-950c-37c5a1498597/mc-35-big (1.876KiB) DEBUG [ValidationExecutor:39] 2018-02-08 21:31:42,581 SSTableReader.java:503 - Opening /var/lib/cassandra/data/prod/invoices-a8454000c10711e4896155ab730ce02d/snapshots/689d7a80-0d17-11e8-950c-37c5a1498597/mc-35-big (1.876KiB) DEBUG [ValidationExecutor:39] 2018-02-08 21:31:42,586 SSTableReader.java:503 - Opening /var/lib/cassandra/data/prod/invoices-a8454000c10711e4896155ab730ce02d/snapshots/689d7a80-0d17-11e8-950c-37c5a1498597/mc-36-big (0.688KiB) ERROR [ValidationExecutor:25] 2018-02-08 21:31:42,586 Validator.java:268 - Failed creating a merkle tree for [repair #69803961-0d17-11e8-950c-37c5a1498597 on prod/invoices, [(-4165282778092010776,-4162024061639141985]]], /88.99.35.178 (see log for details) ERROR [ValidationExecutor:25] 2018-02-08 21:31:42,587 CassandraDaemon.java:228 - Exception in thread Thread[ValidationExecutor:25,1,main] java.lang.AssertionError: null at org.apache.cassandra.io.sstable.format.SSTableReader$GlobalTidy.get(SSTableReader.java:2313) ~[apache-cassandra-3.11.1.jar:3.11.1] at org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier.setup(SSTableReader.java:2138) ~[apache-cassandra-3.11.1.jar:3.11.1] at org.apache.cassandra.io.sstable.format.SSTableReader.setup(SSTableReader.java:2081) ~[apache-cassandra-3.11.1.jar:3.11.1] at org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:519) ~[apache-cassandra-3.11.1.jar:3.11.1] at org.apache.cassandra.db.ColumnFamilyStore.getSnapshotSSTableReader(ColumnFamilyStore.java:1901) ~[apache-cassandra-3.11.1.jar:3.11.1] at org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:1335) ~[apache-cassandra-3.11.1.jar:3.11.1] at org.apache.cassandra.db.compaction.CompactionManager.access$700(CompactionManager.java:86) ~[apache-cassandra-3.11.1.jar:3.11.1] at org.apache.cassandra.db.compaction.CompactionManager$13.call(CompactionManager.java:942) ~[apache-cassandra-3.11.1.jar:3.11.1] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_151] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_151] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_151] at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81) [apache-cassandra-3.11.1.jar:3.11.1] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_151] DEBUG [ValidationExecutor:39] 2018-02-08 21:31:42,588 SSTableReader.java:503 - Opening /var/lib/cassandra/data/prod/invoices-a8454000c10711e4896155ab730ce02d/snapshots/689d7a80-0d17-11e8-950c-37c5a1498597/mc-33-big (9.891MiB) ERROR [NonPeriodicTasks:1] 2018-02-08 21:31:42,589 CassandraDaemon.java:228 - Exception in thread Thread[NonPeriodicTasks:1,5,main] java.lang.NullPointerException: null at org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier$1.run(SSTableReader.java:2192) ~[apache-cassandra-3.11.1.jar:3.11.1] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_151] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_151] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[na:1.8.0_151] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[na:1.8.0_151] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_151] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_151] at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81) [apache-cassandra-3.11.1.jar:3.11.1] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_151]
{code}
> Failed to create merkel tree
> ----------------------------
>
> Key: CASSANDRA-14224
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14224
> Project: Cassandra
> Issue Type: Bug
> Components: Repair
> Reporter: Igor Zubchenok
> Priority: Major
>
> While repairing a node with nodetool -full -pr -dcpar, another node throws an assertion exception and failed to create merkel tree:
>
> {code:java}
> DEBUG [ValidationExecutor:25] 2018-02-08 21:31:42,581 SSTableReader.java:503 - Opening /var/lib/cassandra/data/prod/invoices-a8454000c10711e4896155ab730ce02d/snapshots/689d7a80-0d17-11e8-950c-37c5a1498597/mc-35-big (1.876KiB)
> DEBUG [ValidationExecutor:39] 2018-02-08 21:31:42,581 SSTableReader.java:503 - Opening /var/lib/cassandra/data/prod/invoices-a8454000c10711e4896155ab730ce02d/snapshots/689d7a80-0d17-11e8-950c-37c5a1498597/mc-35-big (1.876KiB)
> DEBUG [ValidationExecutor:39] 2018-02-08 21:31:42,586 SSTableReader.java:503 - Opening /var/lib/cassandra/data/prod/invoices-a8454000c10711e4896155ab730ce02d/snapshots/689d7a80-0d17-11e8-950c-37c5a1498597/mc-36-big (0.688KiB)
> ERROR [ValidationExecutor:25] 2018-02-08 21:31:42,586 Validator.java:268 - Failed creating a merkle tree for [repair #69803961-0d17-11e8-950c-37c5a1498597 on prod/invoices, [(-4165282778092010776,-4162024061639141985]]], /88.99.35.178 (see log for details)
> ERROR [ValidationExecutor:25] 2018-02-08 21:31:42,587 CassandraDaemon.java:228 - Exception in thread Thread[ValidationExecutor:25,1,main]
> java.lang.AssertionError: null
> at org.apache.cassandra.io.sstable.format.SSTableReader$GlobalTidy.get(SSTableReader.java:2313) ~[apache-cassandra-3.11.1.jar:3.11.1]
> at org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier.setup(SSTableReader.java:2138) ~[apache-cassandra-3.11.1.jar:3.11.1]
> at org.apache.cassandra.io.sstable.format.SSTableReader.setup(SSTableReader.java:2081) ~[apache-cassandra-3.11.1.jar:3.11.1]
> at org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:519) ~[apache-cassandra-3.11.1.jar:3.11.1]
> at org.apache.cassandra.db.ColumnFamilyStore.getSnapshotSSTableReader(ColumnFamilyStore.java:1901) ~[apache-cassandra-3.11.1.jar:3.11.1]
> at org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:1335) ~[apache-cassandra-3.11.1.jar:3.11.1]
> at org.apache.cassandra.db.compaction.CompactionManager.access$700(CompactionManager.java:86) ~[apache-cassandra-3.11.1.jar:3.11.1]
> at org.apache.cassandra.db.compaction.CompactionManager$13.call(CompactionManager.java:942) ~[apache-cassandra-3.11.1.jar:3.11.1]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_151]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_151]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_151]
> at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81) [apache-cassandra-3.11.1.jar:3.11.1]
> at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_151]
> DEBUG [ValidationExecutor:39] 2018-02-08 21:31:42,588 SSTableReader.java:503 - Opening /var/lib/cassandra/data/prod/invoices-a8454000c10711e4896155ab730ce02d/snapshots/689d7a80-0d17-11e8-950c-37c5a1498597/mc-33-big (9.891MiB)
> ERROR [NonPeriodicTasks:1] 2018-02-08 21:31:42,589 CassandraDaemon.java:228 - Exception in thread Thread[NonPeriodicTasks:1,5,main]
> java.lang.NullPointerException: null
> at org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier$1.run(SSTableReader.java:2192) ~[apache-cassandra-3.11.1.jar:3.11.1]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_151]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_151]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[na:1.8.0_151]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[na:1.8.0_151]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_151]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_151]
> at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81) [apache-cassandra-3.11.1.jar:3.11.1]
> at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_151]{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org