[jira] [Commented] (KUDU-2183) Leaders changing to an unstable/stuck configuration

["Todd Lipcon (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KUDU-2183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16196460#comment-16196460 ] 

Todd Lipcon commented on KUDU-2183:
-----------------------------------

As noted above, KUDU-2169 would help resolve this since 'D' would be able to vote at the end. Additionally, "3-4-3" re-replication might also help.

In the meantime, though, it seems like we should do something to avoid the configuration change at step 7. A simple heuristic would be to not evict a node unless the remaining nodes in the config have been able to form a majority within the last N heartbeat intervals. i.e if we have (A,B,C) and both B and C are non-responsive, we should not evict either one of them.

> Leaders changing to an unstable/stuck configuration
> ---------------------------------------------------
>
>                 Key: KUDU-2183
>                 URL: https://issues.apache.org/jira/browse/KUDU-2183
>             Project: Kudu
>          Issue Type: Bug
>          Components: consensus
>    Affects Versions: 1.4.0, 1.5.0, 1.6.0
>            Reporter: Todd Lipcon
>
> We've long had this issue but finally figured out the sequence of events that result in a "stuck" raft configuration after some nodes crash and recover:
> Initial state: A(LEADER), B, C
> 1) B goes down
> 2) A evicts B, commits config change: new config (A,C)
> 3) master sends a config change to A to add a new replica D
> 4) A commits config change to (A,C,D)
> 5) D is busy or actually not running and therefore doesn't accept the tablet copy request
> 6) C goes down
> (at this point the tablet is not operational because C is down and D has no replica yet)
> 7) A creates a pending config change to evict C (new config A, D)
> (it can't commit the config change because D still has no replica)
> 8) Tablet server "A" reboots so it is no longer the leader.
> At this point even if we bring every server back up, we end up with:
> A: has a pending config change from step 7 to (A, D), but can't be elected because D has no replica (thus won't vote until we implement KUDU-2169)
> B: comes up and gets its replica deleted because it was evicted
> C: comes up with config (A, C, D) but can't be elected because D has no replica and A has a higher OpId
> D: has no replica



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)