[jira] [Updated] (CASSANDRA-21249) nodetool assassinate blocks the GOSSIP stage, causing the executing node to be marked down and the liveness check to be ineffective

["Stefan Miklosovic (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/CASSANDRA-21249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Stefan Miklosovic updated CASSANDRA-21249:
------------------------------------------
    Test and Documentation Plan: ci
                         Status: Patch Available  (was: In Progress)

> nodetool assassinate blocks the GOSSIP stage, causing the executing node to be marked down and the liveness check to be ineffective
> -----------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-21249
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-21249
>             Project: Apache Cassandra
>          Issue Type: Bug
>          Components: Cluster/Gossip
>            Reporter: Runtian Liu
>            Assignee: Runtian Liu
>            Priority: Normal
>             Fix For: 4.0.x, 4.1.x, 5.0.x
>
>
> h2. Summary
> When {{nodetool assassinate}} is executed against a remote endpoint, two problems occur:
> The node running the command is marked as DOWN by other nodes in the cluster. Peers lose the ability to gossip with the executing node for ~34 seconds, causing the phi accrual failure detector to convict it.
> The liveness check is ineffective. The command is supposed to refuse to assassinate a live node by checking whether the target's heartbeat changes during a 30-second observation window. In practice, the check always passes trivially, meaning a fully healthy node can be assassinated without any warning.
> h2. Root Cause
> The {{assassinateEndpoint}} method in {{Gossiper.java}} runs its entire body — including a 30-second sleep ({{{}RING_DELAY{}}}) and a subsequent 4-second sleep — inside {{{}runInGossipStageBlocking(){}}}. This submits the work to {{{}Stage.GOSSIP{}}}, which is a single-threaded executor.
> While the ~34 seconds of sleep execute on the GOSSIP stage thread:
> All inbound gossip message processing is blocked. {{{}GossipDigestSyn{}}}, {{{}GossipDigestAck{}}}, and {{GossipDigestAck2}} handlers are all dispatched to {{{}Stage.GOSSIP{}}}. They queue up and cannot execute.
> The failure detector on peer nodes receives no heartbeat updates from the executing node, because heartbeat reporting ({{{}notifyFailureDetector{}}}) only happens during ACK/ACK2 processing — which is blocked. After ~10–20 seconds of silence, peers convict the executing node as dead.
> The liveness check cannot detect a live target. The check compares the target's heartbeat before and after the 30-second sleep. However, the only code path that updates a remote endpoint's heartbeat on the local node is {{applyNewStates}} → {{{}setHeartBeatState{}}}, which runs on {{Stage.GOSSIP}} during gossip message processing — the very processing that is blocked. The heartbeat cannot change during the sleep, so the comparison always passes.
> h2. Regression Origin
> This was introduced by CASSANDRA-15059 , "Fix assorted gossip races and add related runtime checks", 2019-03-21).
> The commit's goal was to ensure that gossip state mutations only happen on the GOSSIP stage thread, adding {{runInGossipStageBlocking()}} wrappers throughout {{{}Gossiper{}}}. This was correct for methods like {{{}convict{}}}, {{{}markAsShutdown{}}}, {{{}removeEndpoint{}}}, and {{{}evictFromMembership{}}}. However, wrapping the entire {{assassinateEndpoint}} moved the long-duration sleeps onto the GOSSIP stage as an unintended side effect.
> Before CASSANDRA-15059: {{assassinateEndpoint}} ran entirely on the JMX thread. The GOSSIP stage remained free to process messages during the 30-second sleep, heartbeat updates from the target were visible, and the liveness check correctly detected live nodes.
> After CASSANDRA-15059: The entire method body runs on the GOSSIP stage. The liveness check is broken and the executing node gets convicted.
> h2. Affected Versions
> All versions carrying CASSANDRA-15059 that use gossip-based assassination: 3.0.28+, 3.11.x, 4.0.x, 4.1.x, 5.0.x. Trunk (5.1+) is not affected because gossip-based assassination was replaced by the TCM system.
> h2. Proposed Fix
> Split {{assassinateEndpoint}} into phases so that only the state mutations run on the GOSSIP stage, while reads and sleeps run on the caller (JMX) thread:
> Phase 1 (JMX thread): Snapshot the target's heartbeat generation and version. This is safe off the GOSSIP stage because {{endpointStateMap}} is a {{ConcurrentHashMap}} and {{HeartBeatState}} fields are {{{}volatile{}}}.
> Phase 2 (JMX thread): Sleep for {{RING_DELAY}} — the GOSSIP stage remains free to process messages, so if the target is alive, its heartbeat updates will be received.
> Phase 3 (GOSSIP stage): Re-read the target's heartbeat. If generation or version changed, throw {{{}RuntimeException("Endpoint still alive"){}}}. Otherwise, perform the assassination (set status to LEFT, call {{{}handleMajorStateChange{}}}).
> Phase 4 (JMX thread): Sleep for {{intervalInMillis * 4}} to allow gossip propagation.
> This restores the pre-CASSANDRA-15059 behavior: the GOSSIP stage is only held for the brief duration of state mutations in Phase 3, the liveness check works because heartbeat updates are processed during the sleep, and the executing node is not convicted by peers.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org