RE: [EXTERNAL] Re: A Question About Hints

["Durity, Sean R" <SE...@homedepot.com>]

Versions 2.0 and 2.1 were generally very stable, so I can understand a reticence to move when there are so many other things competing for time and attention.

Sean Durity




From: shalom sagges <sh...@gmail.com>
Sent: Monday, March 04, 2019 4:21 PM
To: user@cassandra.apache.org
Subject: [EXTERNAL] Re: A Question About Hints

Everyone really should move off of the 2.x versions just like you are doing.
Tell me about it... But since there are a lot of groups involved, these things take time unfortunately.

Thanks for your assistance Kenneth

On Mon, Mar 4, 2019 at 11:04 PM Kenneth Brotman <ke...@yahoo.com.invalid>> wrote:
Since you are in the process of upgrading, I’d do nothing on the settings right now.  But if you wanted to do something on the settings in the meantime, based on my read of the information available, I’d maybe double the default settings. The upgrade will help a lot of things as you know.

Everyone really should move off of the 2.x versions just like you are doing.

From: shalom sagges [mailto:shalomsagges@gmail.com<ma...@gmail.com>]
Sent: Monday, March 04, 2019 12:34 PM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: Re: A Question About Hints

See my comments inline.

Do the 8 nodes clusters have the problem too?
Yes

To the same extent?
It depends on the throughput, but basically the smaller clusters get low throughput, so the problem is naturally smaller.

Is it any cluster across multi-DC’s?
Yes

Do all the clusters use nodes with similar specs?
All nodes have similar specs within a cluster but different specs on different clusters.

The version of Cassandra you are on can make a difference.  What version are you on?
Currently I'm on various versions, 2.0.14, 2.1.15 and 3.0.12. In the process of upgrading to 3.11.4

Did you see Edward Capriolo’s presentation at 26:19 into the YouTube video at: https://www.youtube.com/watch?v=uN4FtAjYmLU<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.youtube.com_watch-3Fv-3DuN4FtAjYmLU&d=DwMFaQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=rADb6b8CTim6Qjxi48jQvnRdL_GqKB5hM33yBEKS0Y4&s=oeJjPZZkhxRhyg5r3eApRMk-_tk6qTUrmtF_9eDY5fU&e=> where he briefly mentions you can get into trouble if you go to fast or two slow?
I guess you can say it about almost any parameter you change :)

BTW, I thought the comments at the end of the article you mentioned were really good.
The entire article is very good, but I wonder if it's still valid since it was created around 4 years ago.

Thanks!




On Mon, Mar 4, 2019 at 9:37 PM Kenneth Brotman <ke...@yahoocom.invalid>> wrote:
Makes sense  If you have time and don’t mind, could you answer the following:
Do the 8 nodes clusters have the problem too?
To the same extent?
Is it just the clusters with the large node count?
Is it any cluster across multi-DC’s?
Do all the clusters use nodes with similar specs?

The version of Cassandra you are on can make a difference.  What version are you on?

Did you see Edward Capriolo’s presentation at 26:19 into the YouTube video at: https://www.youtube.com/watch?v=uN4FtAjYmLU<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.youtube.com_watch-3Fv-3DuN4FtAjYmLU&d=DwMFaQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=rADb6b8CTim6Qjxi48jQvnRdL_GqKB5hM33yBEKS0Y4&s=oeJjPZZkhxRhyg5r3eApRMk-_tk6qTUrmtF_9eDY5fU&e=> where he briefly mentions you can get into trouble if you go to fast or two slow?
BTW, I thought the comments at the end of the article you mentioned were really good.



From: shalom sagges [mailto:shalomsagges@gmail.com<ma...@gmail.com>]
Sent: Monday, March 04, 2019 11:04 AM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: Re: A Question About Hints

It varies...
Some clusters have 48 nodes, others 24 nodes and some 8 nodes.
Both settings are on default.

I’d try making a single conservative change to one or the other, measure and reassess.  Then do same to other setting.
That's the plan, but I thought I might first get some valuable information from someone in the community that has already experienced in this type of change.

Thanks!

On Mon, Mar 4, 2019 at 8:27 PM Kenneth Brotman <ke...@yahoo.com.invalid>> wrote:
It sounds like your use case might be appropriate for tuning those two settings some.

How many nodes are in the cluster?
Are both settings definitely on the default values currently?

I’d try making a single conservative change to one or the other, measure and reassess.  Then do same to other setting.

Then of course share your results with us.

From: shalom sagges [mailto:shalomsagges@gmail.com<ma...@gmail.com>]
Sent: Monday, March 04, 2019 9:54 AM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: Re: A Question About Hints

Hi Kenneth,

The concern is that in some cases, hints accumulate on nodes, and it takes a while until they are delivered (multi DCs).
I see that whenever there are  a lot of hints in play,like after a rolling restart, the cluster works harder. That's why I want to decrease the hints delivery time.
I didn't want to change the configuration blindly and thought the community might have some experience on this subject.

I went over the cassandra.yaml file but didn't find any information on optimizing these attributes, just that the max_throttle is divided between nodes in the cluster and that I should increase the max_hints_delivery_threads because I have multi-dc deployments.

# Maximum throttle in KBs per second, per delivery thread.  This will be
# reduced proportionally to the number of nodes in the cluster.  (If there
# are two nodes in the cluster, each delivery thread will use the maximum
# rate; if there are three, each will throttle to half of the maximum,
# since we expect two nodes to be delivering hints simultaneously)
hinted_handoff_throttle_in_kb: 1024

# Number of threads with which to deliver hints;
# Consider increasing this number when you have multi-dc deployments, since
# cross-dc handoff tends to be slower
max_hints_delivery_threads: 2


Thanks for your help!


On Mon, Mar 4, 2019 at 6:44 PM Kenneth Brotman <ke...@yahoo.com.invalid>> wrote:
What is the concern?  Why are you looking there?  The casssandra.yml file has some notes about it.  Did you read them?

From: shalom sagges [mailto:shalomsagges@gmail.com<ma...@gmail.com>]
Sent: Monday, March 04, 2019 7:22 AM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: A Question About Hints

Hi All,

Does anyone know what is the most optimal hints configuration (multiple DCs) in terms of
max_hints_delivery_threads and hinted_handoff_throttle_in_kb?
If it's different for various use cases, is there a rule of thumb I can work with?

I found this post but it's quite old:
http://www.uberobert.com/bandwidth-cassandra-hinted-handoff/<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.uberobert.com_bandwidth-2Dcassandra-2Dhinted-2Dhandoff_&d=DwMFaQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=rADb6b8CTim6Qjxi48jQvnRdL_GqKB5hM33yBEKS0Y4&s=8BghQxt9y6HaOXfo3zrv7SXcsccnbXu377WLRHXjNlw&e=>

Thanks!

________________________________

The information in this Internet Email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this Email by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. When addressed to our clients any opinions or advice contained in this Email are subject to the terms and conditions expressed in any applicable governing The Home Depot terms of business or client engagement letter. The Home Depot disclaims all responsibility and liability for the accuracy and content of this attachment and for any damages or losses arising from any inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other items of a destructive nature, which may be contained in this attachment and shall not be liable for direct, indirect, consequential or special damages in connection with this e-mail message or its attachment.