[jira] [Commented] (CASSANDRA-2496) Gossip should handle 'dead' states

["Brandon Williams (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-2496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042254#comment-13042254 ] 

Brandon Williams commented on CASSANDRA-2496:
---------------------------------------------

Some explanation of what changed and why it was necessary:

Consider nodes A through D. D is partitioned, and C is dead and needs to be removed. A removetoken will be issued to A for this.

In the current way we do things, A will modify it's own state by appending information to its status indicating that it will be removing C. B will see this, re-replicate as needed, then report to A that is is done. The problem however, is that since A modified its own state, A is also free to wipe that state out, either by restarting, or simple remove another token, because there's only space for one. If A reboots and then D's partition heals, D will never know C was removed. Worse, it will still have state for C that neither A nor B do, and so it will repopulate the ring with C again.

This patch changes this by instead having A sleep for RING_DELAY to make sure the generation for C is stable, and then it modifies C's state to indicate it is being removed, just as if C itself had done this. It also appends some extra state to indicate that A will be the removal coordinator. The others nodes see this, re-replicate and report back to A, which then modifies C's state once more to indicate it is completely removed. At this point, it doesn't matter if A dies completely and D's partition heals, since the state is stored in C's gossip information. If A reboots, it will be able to get the correct state information from B, or any other node.

If A fails while the other nodes are re-replicating, a new removetoken can be started elsewhere, or in the case of other replicas being down preventing removetoken from completing, a removetoken force will remove the node and then repair can be run to restore the replica count.

> Gossip should handle 'dead' states
> ----------------------------------
>
>                 Key: CASSANDRA-2496
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2496
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>            Reporter: Brandon Williams
>            Assignee: Brandon Williams
>         Attachments: 0001-Rework-token-removal-process.txt, 0002-add-2115-back.txt
>
>
> For background, see CASSANDRA-2371

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira