[jira] [Commented] (CASSANDRA-5902) Dealing with hints after a topology change

["Aleksey Yeschenko (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-5902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15058454#comment-15058454 ] 

Aleksey Yeschenko commented on CASSANDRA-5902:
----------------------------------------------

This is fine, conceptually.

That said, if our working assumption is that this situation should be rare (and I agree with the assumption here), then we might go a slightly simpler route:
- for hints files for which the node is no longer part of the ring, instead of regular delivery, write the new hints locally instead, just to their new destinations - and have them replay later via the regular hints replay path
- on the receiving end, if the hint no longer belongs, write a hint on that node for all the replicas instead, and have them be delivered later via the regular replay path

We do pay a bit more price in file IO - in redundant writes, but writing a hint for multiple replicas is relatively cheap (serialised just once, using the same space in the buffer), and this should be rare. 

As a bonus, {{Hint::apply()}} remains local only, without calls to {{StorageProxy}}, plus we could avoid further extending {{AWRH}}.

Please let me know if I'm just being silly and none of this makes any sense - I haven't head enough tea today.

> Dealing with hints after a topology change
> ------------------------------------------
>
>                 Key: CASSANDRA-5902
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-5902
>             Project: Cassandra
>          Issue Type: Sub-task
>          Components: Coordination
>            Reporter: Jonathan Ellis
>            Assignee: Branimir Lambov
>            Priority: Minor
>             Fix For: 3.0.x, 3.x
>
>
> Hints are stored and delivered by destination node id.  This allows them to survive IP changes in the target, while making "scan all the hints for a given destination" an efficient operation.  However, we do not detect and handle new node assuming responsibility for the hinted row via bootstrap before it can be delivered.
> I think we have to take a performance hit in this case -- we need to deliver such a hint to *all* replicas, since we don't know which is the "new" one.  This happens infrequently enough, however -- requiring first the target node to be down to create the hint, then the hint owner to be down long enough for the target to both recover and stream to a new node -- that this should be okay.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)