You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Michael Semb Wever (Jira)" <ji...@apache.org> on 2026/07/09 09:08:00 UTC
[jira] [Comment Edited] (CASSANDRA-20610) nodetool unreachable shows all other instances down
[ https://issues.apache.org/jira/browse/CASSANDRA-20610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18094935#comment-18094935 ]
Michael Semb Wever edited comment on CASSANDRA-20610 at 7/9/26 9:07 AM:
------------------------------------------------------------------------
[~chrisjmiller] , the 20251014 thread dump (looks to me) matches [~chovatia.jaydeep@gmail.com]'s observations: 31/35 MutationStage threads parked in {{{}SubAllocator.allocate(MemtableAllocator.java:195){}}}, both MemtableFlushWriter threads at {{writeBarrier.await()}} ({{{}Flush.run:1193{}}}), MemtablePostFlush stuck.
But [~chovatia.jaydeep@gmail.com]… IIUC, shouldn't {{writeBarrier.markBlocking()}} (called immediately before {{await()}} in {{{}Flush.run{}}}) release your {_}Thread-1{_}? (i.e. what [~benedict] writes here)
If so, then the pre-barrier {{Group}} parked at line 195 should be woken via {{isBlockingSignal}} and permitted to exceed the memory limit, and a post-barrier {{Group}} isn't waited on by the barrier. So maybe we would still require one of (a) a lost wakeup in the markBlocking path, (b) a leaked {{Group}} with no thread behind it, or (c) a pre-barrier op parked somewhere {{markBlocking()}} that can't be reached ??
Possibly relevant to (c): the only write-path thread in the 20251014 dump outside the allocator is {{{}read-hotness-tracker:1{}}}, holding an open write {{Group}} inside {{beginWrite}} → {{CommitLog.add}} → {{PeriodicCommitLogService.maybeWaitForSync}} → {{awaitSyncAt}} . (We've witnessed elsewhere something similar with {{BatchCommitLogService}} too.)
[~chrisjmiller] , [~chovatia.jaydeep@gmail.com] , could it help with a heap dump, to walk the {{Keyspace.writeOrder}} {{Group}} chain to see which {{Group}} the barrier is waiting on ?
was (Author: michaelsembwever):
[~chrisjmiller] , the 20251014 dump (looks to me) matches [~chovatia.jaydeep@gmail.com]'s observations: 31/35 MutationStage threads parked in {{{}SubAllocator.allocate(MemtableAllocator.java:195){}}}, both MemtableFlushWriter threads at {{writeBarrier.await()}} ({{{}Flush.run:1193{}}}), MemtablePostFlush stuck.
But [~chovatia.jaydeep@gmail.com]… IIUC, shouldn't {{writeBarrier.markBlocking()}} (called immediately before {{await()}} in {{{}Flush.run{}}}) release your {_}Thread-1{_}? (i.e. what [~benedict] writes here)
If so, then the pre-barrier {{Group}} parked at line 195 should be woken via {{isBlockingSignal}} and permitted to exceed the memory limit, and a post-barrier {{Group}} isn't waited on by the barrier. So maybe we would still require one of (a) a lost wakeup in the markBlocking path, (b) a leaked {{Group}} with no thread behind it, or (c) a pre-barrier op parked somewhere {{markBlocking()}} that can't be reached ??
Possibly relevant to (c): the only write-path thread in this dump outside the allocator is {{{}read-hotness-tracker:1{}}}, holding an open write {{Group}} inside {{beginWrite}} → {{CommitLog.add}} → {{PeriodicCommitLogService.maybeWaitForSync}} → {{awaitSyncAt}} . (We've witnessed elsewhere something similar with {{BatchCommitLogService}} too.)
[~chrisjmiller] , [~chovatia.jaydeep@gmail.com] , would it help with a heap dump, to walk the {{Keyspace.writeOrder}} {{Group}} chain to see which {{Group}} the barrier is waiting on ?
> nodetool unreachable shows all other instances down
> ---------------------------------------------------
>
> Key: CASSANDRA-20610
> URL: https://issues.apache.org/jira/browse/CASSANDRA-20610
> Project: Apache Cassandra
> Issue Type: Bug
> Reporter: Chris Miller
> Priority: Normal
> Attachments: CASSANDRA-20610_cassandra.yaml, cassandra_thread_dump_20250530.txt, cassandra_thread_dump_20251014.txt
>
>
> see first comment
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org