You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Joseph Lynch (JIRA)" <ji...@apache.org> on 2018/07/20 17:59:00 UTC
[jira] [Commented] (CASSANDRA-14459) DynamicEndpointSnitch should
never prefer latent nodes
[ https://issues.apache.org/jira/browse/CASSANDRA-14459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16551064#comment-16551064 ]
Joseph Lynch commented on CASSANDRA-14459:
------------------------------------------
Ok based on the feedback I've moved the latency probes into a scheduled executor that runs every second and sends a single message to one node needing latency probes. If no probes are needed then they are not sent. I've gone a step further and posted a follow up commit which replaces all the uses of histograms with a simpler (and 10x more performant) EMA implementation. The second patch is very optional but I know that some folks have mentioned they don't like all the garbage that the Histograms create so I fixed it.
||trunk||trunk+EMA||
|[patch|https://github.com/apache/cassandra/compare/trunk...jolynch:CASSANDRA-14459]|[patch|https://github.com/apache/cassandra/compare/trunk...jolynch:CASSANDRA-14459-EMA]|
|[unit tests|https://github.com/apache/cassandra/compare/trunk...jolynch:CASSANDRA-14459#diff-f1baf62fd510947bf60e9a9def69810b] [!https://circleci.com/gh/jolynch/cassandra/tree/CASSANDRA-14459.png?circle-token= 1102a59698d04899ec971dd36e925928f7b521f5!|https://circleci.com/gh/jolynch/cassandra/tree/CASSANDRA-14459]|[unit tests|https://github.com/apache/cassandra/compare/trunk...jolynch:CASSANDRA-14459-EMA#diff-f1baf62fd510947bf60e9a9def69810b] [microbenchmark|https://github.com/apache/cassandra/compare/trunk...jolynch:CASSANDRA-14459-EMA#diff-f48455469e3cd6a2e225e3cb2b389d93] [!https://circleci.com/gh/jolynch/cassandra/tree/CASSANDRA-14459.png?circle-token= 1102a59698d04899ec971dd36e925928f7b521f5!|https://circleci.com/gh/jolynch/cassandra/tree/CASSANDRA-14459-EMA]|
> DynamicEndpointSnitch should never prefer latent nodes
> ------------------------------------------------------
>
> Key: CASSANDRA-14459
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14459
> Project: Cassandra
> Issue Type: Improvement
> Components: Coordination
> Reporter: Joseph Lynch
> Assignee: Joseph Lynch
> Priority: Minor
> Fix For: 4.x
>
>
> The DynamicEndpointSnitch has two unfortunate behaviors that allow it to provide latent hosts as replicas:
> # Loses all latency information when Cassandra restarts
> # Clears latency information entirely every ten minutes (by default), allowing global queries to be routed to _other datacenters_ (and local queries cross racks/azs)
> This means that the first few queries after restart/reset could be quite slow compared to average latencies. I propose we solve this by resetting to the minimum observed latency instead of completely clearing the samples and extending the {{isLatencyForSnitch}} idea to a three state variable instead of two, in particular {{YES}}, {{NO}}, {{MAYBE}}. This extension allows {{EchoMessages}} and {{PingMessages}} to send {{MAYBE}} indicating that the DS should use those measurements if it only has one or fewer samples for a host. This fixes both problems because on process restart we send out {{PingMessages}} / {{EchoMessages}} as part of startup, and we would reset to effectively the RTT of the hosts (also at that point normal gossip {{EchoMessages}} have an opportunity to add an additional latency measurement).
> This strategy also nicely deals with the "a host got slow but now it's fine" problem that the DS resets were (afaik) designed to stop because the {{EchoMessage}} ping latency will count only after the reset for that host. Ping latency is a more reasonable lower bound on host latency (as opposed to status quo of zero).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org