You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Caleb Rackliffe (Jira)" <ji...@apache.org> on 2021/03/26 15:34:00 UTC
[jira] [Comment Edited] (CASSANDRA-16495) Scheduled (Delayed)
Schema Pull Tasks May Run After MIGRATION Stage Shutdown During
Decommission
[ https://issues.apache.org/jira/browse/CASSANDRA-16495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17309478#comment-17309478 ]
Caleb Rackliffe edited comment on CASSANDRA-16495 at 3/26/21, 3:33 PM:
-----------------------------------------------------------------------
As long as we agree on the analysis, better stability of {{HintedHandoffAddRemoveTest}} over time here will be good enough proof for me. I'll review shortly. In terms of 3.0 and 3.11, I'd favor fixing it there as well, assuming it merges cleanly (and given how small the change is).
was (Author: maedhroz):
As long as we agree on the analysis, better stability of {{HintedHandoffAddRemoveTest}} over time here will be good enough proof for me. I'll review shortly. In terms of 3.0 and 3.11, I'd favor fixing it there as well, assuming it merges cleanly (and given how small it is).
> Scheduled (Delayed) Schema Pull Tasks May Run After MIGRATION Stage Shutdown During Decommission
> ------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-16495
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16495
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Startup and Shutdown
> Reporter: Caleb Rackliffe
> Assignee: Ekaterina Dimitrova
> Priority: Normal
> Fix For: 3.0.x, 3.11.x, 4.0-rc
>
>
> A new test added in CASSANDRA-16181 stumbled across this, although it doesn’t happen consistently. When [failure occurs|https://app.circleci.com/pipelines/github/maedhroz/cassandra/235/workflows/eb8133ce-9373-4136-b404-ceca167353f6/jobs/1355/tests], it appears to be because a delayed schema pull happens after decommission shuts down the MIGRATION stage’s thread pool.
> {noformat}
> ERROR [node1_isolatedExecutor:1] node1 2021-02-15 19:35:36,284 CassandraDaemon.java:579 - Exception in thread Thread[node1_NonPeriodicTasks:1,5,node1] java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut down at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:72)
> at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:825)
> at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1355)
> at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.execute(DebuggableThreadPoolExecutor.java:176)
> at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118) at org.apache.cassandra.concurrent.Stage.submit(Stage.java:129)
> at org.apache.cassandra.schema.MigrationCoordinator.lambda$scheduleSchemaPull$2(MigrationCoordinator.java:362)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.base/java.lang.Thread.run(Thread.java:834)
> {noformat}
> A fix might be as simple as shutting down ScheduledExecutors.nonPeriodicTasks in StorageService#decommission(). See the original discussion [here|https://issues.apache.org/jira/browse/CASSANDRA-16181?focusedCommentId=17293329&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17293329].
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org