You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Johan Bjork (JIRA)" <ji...@apache.org> on 2014/09/05 22:11:28 UTC
[jira] [Updated] (CASSANDRA-7892) Hashmap IllegalStateException in
anticompaction
[ https://issues.apache.org/jira/browse/CASSANDRA-7892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Johan Bjork updated CASSANDRA-7892:
-----------------------------------
Description:
{noformat}
ERROR [AntiEntropyStage:4] 2014-09-05 19:55:06,439 CassandraDaemon.java:166 - Exception in thread Thread[AntiEntropyStage:4,5,main]
java.lang.RuntimeException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.IllegalStateException
at org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:117) ~[apache-cassandra-2.1.0-rc5.jar:2.1.0-rc5]
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62) ~[apache-cassandra-2.1.0-rc5.jar:2.1.0-rc5]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_60]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.7.0_60]
at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_60]
Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.IllegalStateException
at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:411) ~[apache-cassandra-2.1.0-rc5.jar:2.1.0-rc5]
at org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:400) ~[apache-cassandra-2.1.0-rc5.jar:2.1.0-rc5]
at org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:113) ~[apache-cassandra-2.1.0-rc5.jar:2.1.0-rc5]
... 4 common frames omitted
Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalStateException
at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[na:1.7.0_60]
at java.util.concurrent.FutureTask.get(FutureTask.java:188) ~[na:1.7.0_60]
at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:407) ~[apache-cassandra-2.1.0-rc5.jar:2.1.0-rc5]
... 6 common frames omitted
Caused by: java.lang.IllegalStateException: null
at java.util.HashMap$HashIterator.remove(HashMap.java:938) ~[na:1.7.0_60]
at org.apache.cassandra.db.compaction.CompactionManager.performAnticompaction(CompactionManager.java:425) ~[apache-cassandra-2.1.0-rc5.jar:2.1.0-rc5]
at org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:381) ~[apache-cassandra-2.1.0-rc5.jar:2.1.0-rc5]
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.1.0-rc5.jar:2.1.0-rc5]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.7.0_60]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_60]
... 3 common frames omitted
{noformat}
And on a different host:
{noformat}
ERROR [CompactionExecutor:43] 2014-09-05 19:55:06,376 CassandraDaemon.java:166 - Exception in thread Thread[CompactionExecutor:4
3,1,main]
java.lang.IllegalStateException: null
at java.util.HashMap$HashIterator.remove(HashMap.java:938) ~[na:1.7.0_60]
at org.apache.cassandra.db.compaction.CompactionManager.performAnticompaction(CompactionManager.java:425) ~[apache-cassa
ndra-2.1.0-rc5.jar:2.1.0-rc5]
at org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:381) ~[apache-cassandra-2.1
.0-rc5.jar:2.1.0-rc5]
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.1.0-rc5.jar:2.1.0-rc5]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.7.0_60]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_60]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_60]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_60]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_60]
{noformat}
The exceptions happened in our test environment after I added a bunch of new-nodes at the same time that was in a different (previously non-existing) datacenter. The datacenter is determined using GossipingPropertyFileSnitch and started repairs on a bunch of nodes at the same time using
{noformat}
nodetool -h <node> repair -inc -par
{noformat}
The nodes in the old datacenter had previously been repaired using the -inc option.
was:
{noformat}
ERROR [AntiEntropyStage:4] 2014-09-05 19:55:06,439 CassandraDaemon.java:166 - Exception in thread Thread[AntiEntropyStage:4,5,main]
java.lang.RuntimeException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.IllegalStateException
at org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:117) ~[apache-cassandra-2.1.0-rc5.jar:2.1.0-rc5]
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62) ~[apache-cassandra-2.1.0-rc5.jar:2.1.0-rc5]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_60]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.7.0_60]
at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_60]
Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.IllegalStateException
at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:411) ~[apache-cassandra-2.1.0-rc5.jar:2.1.0-rc5]
at org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:400) ~[apache-cassandra-2.1.0-rc5.jar:2.1.0-rc5]
at org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:113) ~[apache-cassandra-2.1.0-rc5.jar:2.1.0-rc5]
... 4 common frames omitted
Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalStateException
at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[na:1.7.0_60]
at java.util.concurrent.FutureTask.get(FutureTask.java:188) ~[na:1.7.0_60]
at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:407) ~[apache-cassandra-2.1.0-rc5.jar:2.1.0-rc5]
... 6 common frames omitted
Caused by: java.lang.IllegalStateException: null
at java.util.HashMap$HashIterator.remove(HashMap.java:938) ~[na:1.7.0_60]
at org.apache.cassandra.db.compaction.CompactionManager.performAnticompaction(CompactionManager.java:425) ~[apache-cassandra-2.1.0-rc5.jar:2.1.0-rc5]
at org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:381) ~[apache-cassandra-2.1.0-rc5.jar:2.1.0-rc5]
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.1.0-rc5.jar:2.1.0-rc5]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.7.0_60]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_60]
... 3 common frames omitted
{noformat}
And on a different host:
{noformat}
ERROR [CompactionExecutor:43] 2014-09-05 19:55:06,376 CassandraDaemon.java:166 - Exception in thread Thread[CompactionExecutor:4
3,1,main]
java.lang.IllegalStateException: null
at java.util.HashMap$HashIterator.remove(HashMap.java:938) ~[na:1.7.0_60]
at org.apache.cassandra.db.compaction.CompactionManager.performAnticompaction(CompactionManager.java:425) ~[apache-cassa
ndra-2.1.0-rc5.jar:2.1.0-rc5]
at org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:381) ~[apache-cassandra-2.1
.0-rc5.jar:2.1.0-rc5]
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.1.0-rc5.jar:2.1.0-rc5]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.7.0_60]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_60]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_60]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_60]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_60]
{noformat}
The exceptions happened in our test environment after I added a bunch of new-nodes at the same time that was in a different (previously non-existing) datacenter. The datacenter is determined using GossipingPropertyFileSnitch and started repairs on a bunch of nodes at the same time using
{noformat}
nodetool -h bfstor4.aoa repair -inc -par
{noformat}
The nodes in the old datacenter had previously been repaired using the -inc option.
> Hashmap IllegalStateException in anticompaction
> -----------------------------------------------
>
> Key: CASSANDRA-7892
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7892
> Project: Cassandra
> Issue Type: Bug
> Environment: Cassandra 2.1 rc 5
> Reporter: Johan Bjork
>
> {noformat}
> ERROR [AntiEntropyStage:4] 2014-09-05 19:55:06,439 CassandraDaemon.java:166 - Exception in thread Thread[AntiEntropyStage:4,5,main]
> java.lang.RuntimeException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.IllegalStateException
> at org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:117) ~[apache-cassandra-2.1.0-rc5.jar:2.1.0-rc5]
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62) ~[apache-cassandra-2.1.0-rc5.jar:2.1.0-rc5]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_60]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.7.0_60]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_60]
> Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.IllegalStateException
> at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:411) ~[apache-cassandra-2.1.0-rc5.jar:2.1.0-rc5]
> at org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:400) ~[apache-cassandra-2.1.0-rc5.jar:2.1.0-rc5]
> at org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:113) ~[apache-cassandra-2.1.0-rc5.jar:2.1.0-rc5]
> ... 4 common frames omitted
> Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalStateException
> at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[na:1.7.0_60]
> at java.util.concurrent.FutureTask.get(FutureTask.java:188) ~[na:1.7.0_60]
> at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:407) ~[apache-cassandra-2.1.0-rc5.jar:2.1.0-rc5]
> ... 6 common frames omitted
> Caused by: java.lang.IllegalStateException: null
> at java.util.HashMap$HashIterator.remove(HashMap.java:938) ~[na:1.7.0_60]
> at org.apache.cassandra.db.compaction.CompactionManager.performAnticompaction(CompactionManager.java:425) ~[apache-cassandra-2.1.0-rc5.jar:2.1.0-rc5]
> at org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:381) ~[apache-cassandra-2.1.0-rc5.jar:2.1.0-rc5]
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.1.0-rc5.jar:2.1.0-rc5]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.7.0_60]
> at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_60]
> ... 3 common frames omitted
> {noformat}
> And on a different host:
> {noformat}
> ERROR [CompactionExecutor:43] 2014-09-05 19:55:06,376 CassandraDaemon.java:166 - Exception in thread Thread[CompactionExecutor:4
> 3,1,main]
> java.lang.IllegalStateException: null
> at java.util.HashMap$HashIterator.remove(HashMap.java:938) ~[na:1.7.0_60]
> at org.apache.cassandra.db.compaction.CompactionManager.performAnticompaction(CompactionManager.java:425) ~[apache-cassa
> ndra-2.1.0-rc5.jar:2.1.0-rc5]
> at org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:381) ~[apache-cassandra-2.1
> .0-rc5.jar:2.1.0-rc5]
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.1.0-rc5.jar:2.1.0-rc5]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.7.0_60]
> at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_60]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_60]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_60]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_60]
> {noformat}
> The exceptions happened in our test environment after I added a bunch of new-nodes at the same time that was in a different (previously non-existing) datacenter. The datacenter is determined using GossipingPropertyFileSnitch and started repairs on a bunch of nodes at the same time using
> {noformat}
> nodetool -h <node> repair -inc -par
> {noformat}
> The nodes in the old datacenter had previously been repaired using the -inc option.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)