RE: Why Would a Single Node Drive Read Latency 5x Worse & Shutting itDown Improves it

[Fd Habash <fm...@gmail.com>]

And the fact that we I restart C* on this seed node, cluster read latency plummets back to near 1 second. Looks like this ‘unhealthiness’ survives a C* restart. There is a persistent issue on this node.

This may explain why read on this node is slow. But why do the other 8 nodes slow down with it. 

----------------
Thank you

From: Jeff Jirsa
Sent: Friday, March 9, 2018 10:11 AM
To: user@cassandra.apache.org
Subject: Re: Why Would a Single Node Drive Read Latency 5x Worse & Shutting itDown Improves it

Usually some combination of:

- CASSANDRA-10726
- Using percentiles for speculative retry
- having your client continue to use the unhealthy host as coordinator

-- 
Jeff Jirsa


On Mar 9, 2018, at 6:46 AM, Fd Habash <fm...@gmail.com> wrote:
 
Tying to come up with an explanation for this real-life C* behavior. Under a constant load test of 2k reads/sec, the very top graph is the read latency for a single node which is a seed one. 
 
You may not see it here, but there are 8 other nodes that follow it. Since we have 4 nodes in each of the 3 AZ’s, I know these other 8 nodes (4 per AZ) hold the secondary replicas for the seed node.
 
At the time of taking this metric snapshot, there were 0 compactions, 0 anti-compactions, 0 repairs. Cluster did nothing else but serve read requests.
 
Shortly after the 0900, I stopped C* on the top seed node and read latency dropped from 1 sec to < 150 ms. 
 
I know this probably not enough diagnostics to pin-point a cause. What explanations are these for this behavior? 
 
Misconfigured client connections? 
 
The seed node EC2 instance tested fine for general health. 
 
During the load test, this node had a ‘repair -pr’ job running on it which finished successfully, but read latency did not improve afterwards. 
 

 
----------------
Thank you
 
<ED27648BD8884412BEDF33F4AD5B209D.png>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org