You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Cameron Zemek (Jira)" <ji...@apache.org> on 2024/08/04 23:11:00 UTC

[jira] [Commented] (CASSANDRA-18543) Waiting for gossip to settle does not wait for live endpoints

    [ https://issues.apache.org/jira/browse/CASSANDRA-18543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17870881#comment-17870881 ] 

Cameron Zemek commented on CASSANDRA-18543:
-------------------------------------------

[~Aburadeh] can you refer to https://issues.apache.org/jira/browse/CASSANDRA-19580 to see if that is what you are running into.

> Waiting for gossip to settle does not wait for live endpoints
> -------------------------------------------------------------
>
>                 Key: CASSANDRA-18543
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-18543
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Cluster/Gossip
>            Reporter: Cameron Zemek
>            Assignee: Stefan Miklosovic
>            Priority: Normal
>             Fix For: 3.11.16, 4.0.11, 4.1.3, 5.0-alpha1, 5.0
>
>         Attachments: gossip.patch, gossip4.patch
>
>          Time Spent: 1h
>  Remaining Estimate: 0h
>
> When a node starts it will get endpoint states (via shadow round) but have all nodes marked as down. The problem is the wait to settle only checks the size of endpoint states is stable before starting Native transport. Once native transport starts it will receive queries and fail consistency levels such as LOCAL_QUORUM since it still thinks nodes are down.
> This is problem for a number of large clusters for our customers. The cluster has quorum but due to this issue a node restart is causing a bunch of query errors.
> My initial solution to this was to only check live endpoints size in addition to size of endpoint states. This worked but I noticed in testing this fix that there also a lot of duplication of checking the same node (via Echo messages) for liveness. So the patch also removes this duplication of checking node is UP in markAlive.
> The final problem I found while testing is sometimes could still not see a change in live endpoints due to only 1 second polling, so the patch allows for overridding the settle parameters. I could not reliability reproduce this but think its worth providing a way to override these hardcoded values.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org