You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "David Capwell (Jira)" <ji...@apache.org> on 2020/02/20 17:04:00 UTC
[jira] [Updated] (CASSANDRA-15592) IllegalStateException in gossip
after removing node
[ https://issues.apache.org/jira/browse/CASSANDRA-15592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
David Capwell updated CASSANDRA-15592:
--------------------------------------
Bug Category: Parent values: Correctness(12982)Level 1 values: API / Semantic Implementation(12988)
Complexity: Normal
Discovered By: User Report
Severity: Normal
Status: Open (was: Triage Needed)
> IllegalStateException in gossip after removing node
> ---------------------------------------------------
>
> Key: CASSANDRA-15592
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15592
> Project: Cassandra
> Issue Type: Bug
> Components: Cluster/Gossip
> Reporter: Marcus Olsson
> Assignee: Marcus Olsson
> Priority: Normal
>
> In one of our test environments we encountered the following exception:
> {noformat}
> 2020-02-02T10:50:13.276+0100 [GossipTasks:1] ERROR o.a.c.u.NoSpamLogger$NoSpamLogStatement:97 log java.lang.IllegalStateException: Attempting gossip state mutation from illegal thread: GossipTasks:1
> at org.apache.cassandra.gms.Gossiper.checkProperThreadForStateMutation(Gossiper.java:178)
> at org.apache.cassandra.gms.Gossiper.evictFromMembership(Gossiper.java:465)
> at org.apache.cassandra.gms.Gossiper.doStatusCheck(Gossiper.java:895)
> at org.apache.cassandra.gms.Gossiper.access$700(Gossiper.java:78)
> at org.apache.cassandra.gms.Gossiper$GossipTask.run(Gossiper.java:240)
> at org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:118)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:84)
> at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.lang.Thread.run(Thread.java:748)
> java.lang.IllegalStateException: Attempting gossip state mutation from illegal thread: GossipTasks:1
> at org.apache.cassandra.gms.Gossiper.checkProperThreadForStateMutation(Gossiper.java:178) [apache-cassandra-3.11.5.jar:3.11.5]
> at org.apache.cassandra.gms.Gossiper.evictFromMembership(Gossiper.java:465) [apache-cassandra-3.11.5.jar:3.11.5]
> at org.apache.cassandra.gms.Gossiper.doStatusCheck(Gossiper.java:895) [apache-cassandra-3.11.5.jar:3.11.5]
> at org.apache.cassandra.gms.Gossiper.access$700(Gossiper.java:78) [apache-cassandra-3.11.5.jar:3.11.5]
> at org.apache.cassandra.gms.Gossiper$GossipTask.run(Gossiper.java:240) [apache-cassandra-3.11.5.jar:3.11.5]
> at org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:118) [apache-cassandra-3.11.5.jar:3.11.5]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_231]
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_231]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_231]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_231]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_231]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_231]
> at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:84) [apache-cassandra-3.11.5.jar:3.11.5]
> at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-all-4.1.42.Final.jar:4.1.42.Final]
> at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_231]
> {noformat}
> Since CASSANDRA-15059 we check that all state changes are performed in the GossipStage but it seems like it was still performed in the "current" thread [here|https://github.com/apache/cassandra/blob/cassandra-3.11/src/java/org/apache/cassandra/gms/Gossiper.java#L895]. It should be as simple as adding a
> {code:java}
> runInGossipStageBlocking(() ->)
> {code}
> for it.
> I'll upload patches for 3.0, 3.11 and 4.0.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org