Hints files are not getting truncated

[Meg Mara <mm...@digitalriver.com>]

Hello,

I am facing an issue with Hinted Handoff files in Cassandra v3.0.10. A DC1 node is storing large number of hints for DC2 nodes (we are facing connection timeout issues). The problem is that the hint files which are created on DC1 are not getting deleted after the 3 hour window. Hints are now being stored as flat files in the Cassandra home directory and I can see that old hints are being deleted but at a very slow pace. It still contains hints from May.
max_hint_window_in_ms: 10800000
max_hints_delivery_threads: 2

Why do you suppose this is happening? Any suggestions or recommendations would be much appreciated.

Thanks for your time.
Meg Mara

RE: Hints files are not getting truncated

[Meg Mara <mm...@digitalriver.com>]

Anuj,

Thank you for your explanation. I had misread the documentation and was under the impression all these days that Hints_window refers to ‘time until deletion’. Also, thanks for the information about TCP settings, I will have to recheck the existing configuration. I will also test the performance improvement of increasing the threads for the hint delivery.

Thank you,
Meg Mara

From: Anuj Wadehra [mailto:anujw_2003@yahoo.co.in.INVALID]
Sent: Tuesday, June 27, 2017 11:56 AM
To: user@cassandra.apache.org
Subject: Re: Hints files are not getting truncated

Hi Meg,

max_hint_window_in_ms =3 hrs means that if a node is down/unresponsive for more than 3 hrs, hints would not be stored for it any further until it becomes responsive again. It should not mean that already stored hints would be truncated after 3 hours.

Regarding connection timeouts between DCs, please check your firewall settings and tcp settings on node. Firewall between the DC must not kill an idle connection which is still considered to be usable by Cassandra. Please see http://docs.datastax.com/en/archived/cassandra/2.0/cassandra/troubleshooting/trblshootIdleFirewall.html .

In multi dc setup, documentation recommends to increase number of threads. You can try increasing it and check whether it improves the situation.

Thanks
Anuj


On Tue, Jun 27, 2017 at 9:47 PM, Meg Mara
<mm...@digitalriver.com>> wrote:

Hello,



I am facing an issue with Hinted Handoff files in Cassandra v3.0.10. A DC1 node is storing large number of hints for DC2 nodes (we are facing connection timeout issues). The problem is that the hint files which are created on DC1 are not getting deleted after the 3 hour window. Hints are now being stored as flat files in the Cassandra home directory and I can see that old hints are being deleted but at a very slow pace. It still contains hints from May.

max_hint_window_in_ms: 10800000

max_hints_delivery_threads: 2



Why do you suppose this is happening? Any suggestions or recommendations would be much appreciated.



Thanks for your time.

Meg Mara

Re: Hints files are not getting truncated

[Anuj Wadehra <an...@yahoo.co.in.INVALID>]

Hi Meg,
max_hint_window_in_ms =3 hrs means that if a node is down/unresponsive for more than 3 hrs, hints would not be stored for it any further until it becomes responsive again. It should not mean that already stored hints would be truncated after 3 hours.
Regarding connection timeouts between DCs, please check your firewall settings and tcp settings on node. Firewall between the DC must not kill an idle connection which is still considered to be usable by Cassandra. Please see http://docs.datastax.com/en/archived/cassandra/2.0/cassandra/troubleshooting/trblshootIdleFirewall.html .
In multi dc setup, documentation recommends to increase number of threads. You can try increasing it and check whether it improves the situation.
ThanksAnuj


 
 
  On Tue, Jun 27, 2017 at 9:47 PM, Meg Mara<mm...@digitalriver.com> wrote:    <!--#yiv5811390882 _filtered #yiv5811390882 {font-family:"Cambria Math";panose-1:2 4 5 3 5 4 6 3 2 4;} _filtered #yiv5811390882 {font-family:Calibri;panose-1:2 15 5 2 2 2 4 3 2 4;}#yiv5811390882 #yiv5811390882 p.yiv5811390882MsoNormal, #yiv5811390882 li.yiv5811390882MsoNormal, #yiv5811390882 div.yiv5811390882MsoNormal {margin:0in;margin-bottom:.0001pt;font-size:11.0pt;font-family:"Calibri", sans-serif;}#yiv5811390882 a:link, #yiv5811390882 span.yiv5811390882MsoHyperlink {color:#0563C1;text-decoration:underline;}#yiv5811390882 a:visited, #yiv5811390882 span.yiv5811390882MsoHyperlinkFollowed {color:#954F72;text-decoration:underline;}#yiv5811390882 span.yiv5811390882EmailStyle17 {font-family:"Calibri", sans-serif;color:windowtext;}#yiv5811390882 .yiv5811390882MsoChpDefault {font-family:"Calibri", sans-serif;} _filtered #yiv5811390882 {margin:1.0in 1.0in 1.0in 1.0in;}#yiv5811390882 div.yiv5811390882WordSection1 {}-->
Hello,
 
  
 
I am facing an issue with Hinted Handoff files in Cassandra v3.0.10. A DC1 node is storing large number of hints for DC2 nodes (we are facing connection timeout issues). The problem is that the hint files which are created on DC1 are not getting deleted after the 3 hour window. Hints are now being stored as flat files in the Cassandra home directory and I can see that old hints are being deleted but at a very slow pace. It still contains hints from May.
 
max_hint_window_in_ms: 10800000
 
max_hints_delivery_threads: 2
 
  
 
Why do you suppose this is happening? Any suggestions or recommendations would be much appreciated.
 
  
 
Thanks for your time.
 
Meg Mara
 
  
   

RE: Hints files are not getting truncated

[Meg Mara <mm...@digitalriver.com>]

Hannu,

Thanks for your response. Yes our gc grace period is at default setting, so you are right, I do not need hints which are a month old. Especially since I have scheduled repairs running on the cluster almost every day and cross DC repair running every week. The network connection is slow but we have never faced any major issues apart from some dropped mutation messages from time to time.

I will consider your advice about turning off hints.

Thank you,
Meg Mara

From: Hannu Kröger [mailto:hkroger@gmail.com]
Sent: Tuesday, June 27, 2017 11:37 AM
To: Meg Mara <mm...@digitalriver.com>; user@cassandra.apache.org
Subject: Re: Hints files are not getting truncated

Hi,

First of all, I don’t know why they get delivered so slowly.

However, if your gc grace seconds is the default 10 days then those hints from May are not needed and could/should be truncated. If the hint delivery is causing problems, then one option is that you could just disable it and rely on periodic repair doing its job and getting the data synchronized.

Now my question to you: Do you have repairs running periodically across the cluster and actually succeeding if the connection is flaky?

Hannu


On 27 June 2017 at 19:17:26, Meg Mara (mmara@digitalriver.com<ma...@digitalriver.com>) wrote:
Hello,

I am facing an issue with Hinted Handoff files in Cassandra v3.0.10. A DC1 node is storing large number of hints for DC2 nodes (we are facing connection timeout issues). The problem is that the hint files which are created on DC1 are not getting deleted after the 3 hour window. Hints are now being stored as flat files in the Cassandra home directory and I can see that old hints are being deleted but at a very slow pace. It still contains hints from May.
max_hint_window_in_ms: 10800000
max_hints_delivery_threads: 2

Why do you suppose this is happening? Any suggestions or recommendations would be much appreciated.

Thanks for your time.
Meg Mara

Re: Hints files are not getting truncated

[Hannu Kröger <hk...@gmail.com>]

Hi,

First of all, I don’t know why they get delivered so slowly.

However, if your gc grace seconds is the default 10 days then those hints
from May are not needed and could/should be truncated. If the hint delivery
is causing problems, then one option is that you could just disable it and
rely on periodic repair doing its job and getting the data synchronized.

Now my question to you: Do you have repairs running periodically across the
cluster and actually succeeding if the connection is flaky?

Hannu

On 27 June 2017 at 19:17:26, Meg Mara (mmara@digitalriver.com) wrote:

Hello,



I am facing an issue with Hinted Handoff files in Cassandra v3.0.10. A DC1
node is storing large number of hints for DC2 nodes (we are facing
connection timeout issues). The problem is that the hint files which are
created on DC1 are not getting deleted after the 3 hour window. Hints are
now being stored as flat files in the Cassandra home directory and I can
see that old hints are being deleted but at a very slow pace. It still
contains hints from May.

max_hint_window_in_ms: 10800000

max_hints_delivery_threads: 2



Why do you suppose this is happening? Any suggestions or recommendations
would be much appreciated.



Thanks for your time.

Meg Mara