[jira] [Created] (CASSANDRA-20743) Inflation for speculative retry 99% threshold if one replica is slow

["Dmitry Konstantinov (Jira)" <ji...@apache.org>]

Dmitry Konstantinov created CASSANDRA-20743:
-----------------------------------------------

             Summary: Inflation for speculative retry 99% threshold if one replica is slow
                 Key: CASSANDRA-20743
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-20743
             Project: Apache Cassandra
          Issue Type: Bug
          Components: Consistency/Coordination
            Reporter: Dmitry Konstantinov
            Assignee: Dmitry Konstantinov


I have executed a set of LOCAL_QUORUM read tests with 3 node Cassandra cluster (4.1.x) when for one of the nodes a slow disk IO read is emulated using a configured delay added to SSTable disk-level reads with a configured probability. The purpose of these tests is to ensure that Cassandra does not degrade a lot from latency point of view if a single replica is not healthy.

During such tests I observe an interesting behaviour: drift/inflation for speculative retry threshold value.
We have a coordinator node, which is a replica as well. Let's assume we have an injected read delay = 100ms with 2% probability within this node and 2 other nodes are healthy. Usual read is executed from the local node + one of the remote nodes.
Because of the introduced delay for 2% of requests we cross speculative retry threshold value and run a speculative retry to the second remote replica.
The speculative retry threshold value is calculated as a +coordinator latency+ 99% by default; in these 2% cases the coordinator latency is actually equal to time to wait till speculative retry + time to execute the request to a remote replica, so we contribute this value back to our coordinator latency metric and actually create a degradation feedback loop: while the 2% delay for the local disk reads is in place the speculative retry threshold value will grow in steps = time to execute the request to a remote replica, degrading more and more.

A possible WA is to use MIN(99p,Xms) speculative retry option introduced in CASSANDRA-14293 but it is env specific, may depends on workload, so it can be not so easy to define the right value for X..

I have found the same issue reported for ScyllaDB - https://github.com/scylladb/scylladb/pull/8783 , to address it they started to use replica read response times instead of a full coordinator read time for speculative retry threshold value evaluation.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org