You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Brandon Williams (Jira)" <ji...@apache.org> on 2024/01/06 02:35:00 UTC
[jira] [Comment Edited] (CASSANDRA-19187) nodetool assassinate may cause thread serialization for that node
[ https://issues.apache.org/jira/browse/CASSANDRA-19187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17803731#comment-17803731 ]
Brandon Williams edited comment on CASSANDRA-19187 at 1/6/24 2:34 AM:
----------------------------------------------------------------------
||Branch||CI||
|[4.0|https://github.com/driftx/cassandra/tree/CASSANDRA-19187-test2]|[repeated|https://app.circleci.com/pipelines/github/driftx/cassandra/1450/workflows/c84aa22d-5bad-4074-a0d7-5811412ca379/jobs/70537]|
|[4.1|https://github.com/driftx/cassandra/tree/CASSANDRA-19187-4.1]|[repeated|https://app.circleci.com/pipelines/github/driftx/cassandra/1452/workflows/ca3174ea-f163-4aef-bc48-d3da4081b8a8/jobs/70748]|
|[5.0|https://github.com/driftx/cassandra/tree/CASSANDRA-19187-5.0]|[repeated|https://app.circleci.com/pipelines/github/driftx/cassandra/1451/workflows/0629e608-44b2-44c6-aace-0c63e198bb7c/jobs/70749]|
Everything passed, +1 from me again.
was (Author: brandon.williams):
||Branch||CI||
|[4.0|https://github.com/driftx/cassandra/tree/CASSANDRA-19187-test2]|[repeated|https://app.circleci.com/pipelines/github/driftx/cassandra/1450/workflows/c84aa22d-5bad-4074-a0d7-5811412ca379/jobs/70537]|
|[4.1|https://github.com/driftx/cassandra/tree/CASSANDRA-19187-4.1]|[repeated|https://app.circleci.com/pipelines/github/driftx/cassandra/1452/workflows/ca3174ea-f163-4aef-bc48-d3da4081b8a8]|
|[5.0|https://github.com/driftx/cassandra/tree/CASSANDRA-19187-5.0]|[repeated|https://app.circleci.com/pipelines/github/driftx/cassandra/1451/workflows/0629e608-44b2-44c6-aace-0c63e198bb7c]|
Everything passed, +1 from me again.
> nodetool assassinate may cause thread serialization for that node
> -----------------------------------------------------------------
>
> Key: CASSANDRA-19187
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19187
> Project: Cassandra
> Issue Type: Bug
> Components: Cluster/Membership
> Reporter: Runtian Liu
> Assignee: Runtian Liu
> Priority: Normal
> Fix For: 4.0.x, 4.1.x, 5.0.x
>
>
> When assassinate an ip address that is not in the gossip map, a "corrupted" entry will be inserted into the gossip map. [(1)|https://github.com/apache/cassandra/blob/cassandra-4.0/src/java/org/apache/cassandra/gms/Gossiper.java#L810]
> For example, if we do "nodetool assassinate 10.1.1.1"
> we will get an entry like below by running "nodetool gossipinfo":
>
> {code:java}
> /10.1.1.1
> generation:1702006511
> heartbeat:9999
> STATUS:209516:LEFT,-8393921141401589197,1702265651923
> STATUS_WITH_PORT:209515:LEFT,-8393921141401589197,1702265651923
> TOKENS: not present {code}
>
> This entry in endpointStateMap will cause issue for [isUpgradingFromVersionLowerThan|https://github.com/apache/cassandra/blob/cassandra-4.0/src/java/org/apache/cassandra/gms/Gossiper.java#L2284] function. Because the [upgradeFromVersionSupplier|https://github.com/apache/cassandra/blob/cassandra-4.0/src/java/org/apache/cassandra/gms/Gossiper.java#L191] supplier will always set the [allHostsHaveKnownVersion|https://github.com/apache/cassandra/blob/cassandra-4.0/src/java/org/apache/cassandra/gms/Gossiper.java#L216] flag to false so no memoized value will be returned. The "get" function will always require a lock from this [line|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/utils/ExpiringMemoizingSupplier.java#L66].
> If application is using "fetchAll", the native-transport-requests thread will hit this [line|https://github.com/apache/cassandra/blob/cassandra-4.0/src/java/org/apache/cassandra/db/filter/ColumnFilter.java#L574]. This means all the native-transport-requests thread is serialized, also, the lock is shared by GossipStage threads. It means if a node in a cluster with the corrupted gossip map is restart, the node will run into this problem.
> To fix the issue,
> # Why we want to add a dummy entry for nodetool assassinate if the endpoint is not in the map anymore. Should we do nothing or throw exception if the node is not in the gossip map anymore?
> # Before checking if a version is null, we should make sure the node is not a dead node. A decommissioned node, a left node should not be considered part of the cluster anymore when calculating "upgradeInProgressPossible"
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org