You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Simon Zhou (JIRA)" <ji...@apache.org> on 2018/03/01 22:23:00 UTC
[jira] [Commented] (CASSANDRA-14252) Use zero as default score in
DynamicEndpointSnitch
[ https://issues.apache.org/jira/browse/CASSANDRA-14252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16382756#comment-16382756 ]
Simon Zhou commented on CASSANDRA-14252:
----------------------------------------
Talked with [~dikanggu] offline. Previously I thought that timeout wouldn't be counted as part of latency score. Actually it is, so setting replica score as 0 by default is less of a problem but only exposes a small vulnerability window:
Say if you have multiple replicas in a remote data center and you don't have score for one of them, thus it will be assigned score 0. This might cause traffic burst on this replica, for a short period of time and most of time it won't even be noticed.
This can be mitigated by assigning a larger score (such as maximum score of all the replicas) to the replica with null score. I'd defer this decision to [~dikanggu]. Otherwise the patch looks good to me.
> Use zero as default score in DynamicEndpointSnitch
> --------------------------------------------------
>
> Key: CASSANDRA-14252
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14252
> Project: Cassandra
> Issue Type: Bug
> Components: Coordination
> Reporter: Dikang Gu
> Assignee: Dikang Gu
> Priority: Major
> Fix For: 4.0, 3.0.17, 3.11.3
>
>
> The problem I want to solve is that I found in our deployment, one slow but alive data node can slow down the whole cluster, even caused timeout of our requests.
> We are using DynamicEndpointSnitch, with badness_threshold 0.1. I expect the DynamicEndpointSnitch switch to sortByProximityWithScore, if local data node latency is too high.
> I added some debug log, and figured out that in a lot of cases, the score from remote data node was not populated, so the fallback to sortByProximityWithScore never happened. That's why a single slow data node, can cause huge problems to the whole cluster.
> In this jira, I'd like to use zero as default score, so that we will get a chance to try remote data node, if local one is slow.
> I tested it in our test cluster, it improved the client latency in single slow data node case significantly.
> I flag this as a Bug, because it caused problems to our use cases multiple times.
> ==== logs ===
> _2018-02-21_23:08:57.54145 WARN 23:08:57 [RPC-Thread:978]: sortByProximityWithBadness: after sorting by proximity, addresses order change to [ip1, ip2], with scores [1.0]_
> _2018-02-21_23:08:57.54319 WARN 23:08:57 [RPC-Thread:967]: sortByProximityWithBadness: after sorting by proximity, addresses order change to [ip1, ip2], with scores [0.0]_
> _2018-02-21_23:08:57.55111 WARN 23:08:57 [RPC-Thread:453]: sortByProximityWithBadness: after sorting by proximity, addresses order change to [ip1, ip2], with scores [1.0]_
> _2018-02-21_23:08:57.55687 WARN 23:08:57 [RPC-Thread:753]: sortByProximityWithBadness: after sorting by proximity, addresses order change to [ip1, ip2], with scores [1.0]_
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org