[jira] [Commented] (CASSANDRA-11090) Hinted Handoff loop

["Cameron Zemek (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-11090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15429960#comment-15429960 ] 

Cameron Zemek commented on CASSANDRA-11090:
-------------------------------------------

In both 3.5 and 3.7 we seeing hinted handoff loop

Aug 04 07:20:33 ip-10-222-104-29.ec2.internal cassandra[31062]: INFO  o.a.c.hints.HintsDispatchExecutor Finished hinted handoff of file 4566c9ad-69df-45fd-84d2-6386072d3efa-1470295200416-1.hints to endpoint 4566c9ad-69df-45fd-84d2-6386072d3efa
Aug 04 07:21:03 ip-10-222-104-29.ec2.internal cassandra[31062]: INFO  o.apache.cassandra.hints.HintsStore Deleted hint file 4566c9ad-69df-45fd-84d2-6386072d3efa-1470295230416-1.hints
Aug 04 07:21:03 ip-10-222-104-29.ec2.internal cassandra[31062]: INFO  o.a.c.hints.HintsDispatchExecutor Finished hinted handoff of file 4566c9ad-69df-45fd-84d2-6386072d3efa-1470295230416-1.hints to endpoint 4566c9ad-69df-45fd-84d2-6386072d3efa
Aug 04 07:21:33 ip-10-222-104-29.ec2.internal cassandra[31062]: INFO  o.apache.cassandra.hints.HintsStore Deleted hint file 4566c9ad-69df-45fd-84d2-6386072d3efa-1470295260416-1.hints

and it just keeps repeating this. Have to rolling restart the cluster to resolve. Waiting for it to happen again to collect more information on the cause for this loop.

> Hinted Handoff loop
> -------------------
>
>                 Key: CASSANDRA-11090
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-11090
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Blake Eggleston
>            Assignee: Aleksey Yeschenko
>            Priority: Minor
>             Fix For: 3.0.4, 3.4
>
>
> After the hints executor finishes sending hints, another hints file is created for the same host, which is then processed on the next go around. This continues indefinitely. The new hint files are empty, so there shouldn't be any network traffic. However, there's still unnecessary hint activity, and the hint crc file isn't deleted in the 3.0 branch after hints are dispatched (but is in trunk), so any hint activity will trigger ~8640 files being created per day until the node is restarted. Restarting the node fixes the problem, after the existing hint files are processed. 
> This can be duplicated on cassandra-3.0 to trunk with this script: https://gist.github.com/bdeggleston/13fbb9e70c0c0bd277c7



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)