You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Aleksei Zotov (Jira)" <ji...@apache.org> on 2021/09/20 15:51:00 UTC
[jira] [Commented] (CASSANDRA-14930) decommission may cause timeout
because messaging backlog is cleared
[ https://issues.apache.org/jira/browse/CASSANDRA-14930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17417702#comment-17417702 ]
Aleksei Zotov commented on CASSANDRA-14930:
-------------------------------------------
[~brandon.williams]
I feel confused. I see the proposed changes already merged as a part of CASSANDRA-14616 ticket. Here are the existing commits from the repo:
3.0: [https://github.com/apache/cassandra/commit/bbf7dac87cdc41bf8e138a99f630e7a827ad0d98]
3.11: [https://github.com/apache/cassandra/commit/4dd7faa75210f635af36c0852e9b0d2e8bdbb95c]
Also the changes referred in the description seem to be simply taken from CASSANDRA-14616 ticket:
{quote}Fix cassandra-stress write hang with default options (*CASSANDRA-14616*)
{quote}
Moreover, they do not seem to be related to the description. I feel the wrong code was accidentally pushed.
[~jasonstack] could you please double check the branches mentioned in the description and confirm they have the code you want to merge as a part of this ticket.
> decommission may cause timeout because messaging backlog is cleared
> --------------------------------------------------------------------
>
> Key: CASSANDRA-14930
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14930
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Coordination, Legacy/Core
> Reporter: Zhao Yang
> Assignee: Zhao Yang
> Priority: Normal
> Fix For: 3.0.x, 3.11.x
>
>
> On a 3-node cluster with RF=2, decommissioning a node may cause quorum write timeout because messaging backlog to decommissioned node is cleared via {{Gossiper#removeEndpoint() -> OutboundTcpConnection#closeSocket()}}.
> (Timeout is less likely to happen with RF=3, because we can afford one less response)
> {code:java}
> What happened:
> 1. [WriteStage] before the leaving node is removed from tokenmetadata, the write endpoints are generated ( leaving endpoint is included )
> 2. [GossipStage] the leaving node is removed from tokenmetadata, no more future write handler will include leaving endpoints
> 3. [WriteStage] write handlers sends messages to messaging-service backlog
> 4. [GossipStage] messaging-service backlog is cleared, messages are not sent and connection closed
> 5. [WriteStage] write time out
> {code}
> |patch|
> |[3.0|https://github.com/jasonstack/cassandra/commits/decommission_timeout_3.0]|
> |[3.11|https://github.com/jasonstack/cassandra/commits/decommission_timeout_3.11]|
> We can avoid it by delaying to destroy messaging connection so that messages are sent and responded. This patch also avoids reopening already closed connection on {{MessagingService#convict()}}.
> New messaging framework rewrite in {{Trunk}} avoids the issues by not clearing messaging backlog.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org