[jira] [Commented] (ZOOKEEPER-261) Reinitialized servers should not participate in leader election

["Brian Nixon (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15727016#comment-15727016 ] 

Brian Nixon commented on ZOOKEEPER-261:
---------------------------------------

Ben and I discussed this offline. When starting up without any local data, the safest thing to do is view this lack with extreme suspicion and not participate in voting until you can pull down the data tree from the rest of the ensemble. Such a server is not qualified to confirm which servers are up to date and could inadvertently elect a server that is missing some data. The one exception is the creation of a fresh ensemble, when there is no data to repopulate the local data tree. It's not clear that an ensemble can detect that it is in this state on its own since in the worst case, every server will be subject to the same data losing fault (in which case you should recover from backups instead of coming online as an empty data base). This extra information needs to come from the admin.

With the changes from ZOOKEEPER-2325, a server with no local data tree starts with a zxid of 0. I'll submit a pull request that changes that initial zxid to -1 unless a special 'initialize' file is present in the data directory and removes voting privileges from members reporting -1. The idea is that creating the 'initialize' file alongside 'myid' will be a standard part of ensemble creation - the extra information from the admin. The 'initialize' file will be automatically cleaned up by the server and subsequent restarts can view missing data directories as a sign they are legitimately missing context (e.g. adding to an existing ensemble).


> Reinitialized servers should not participate in leader election
> ---------------------------------------------------------------
>
>                 Key: ZOOKEEPER-261
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-261
>             Project: ZooKeeper
>          Issue Type: Improvement
>          Components: leaderElection, quorum
>            Reporter: Benjamin Reed
>
> A server that has lost its data should not participate in leader election until it has resynced with a leader. Our leader election algorithm and NEW_LEADER commit assumes that the followers voting on a leader have not lost any of their data. We should have a flag in the data directory saying whether or not the data is preserved so that the the flag will be cleared if the data is ever cleared.
> Here is the problematic scenario: you have have ensemble of machines A, B, and C. C is down. the last transaction seen by C is z. a transaction, z+1, is committed on A and B. Now there is a power outage. B's data gets reinitialized. when power comes back up, B and C comes up, but A does not. C will be elected leader and transaction z+1 is lost. (note, this can happen even if all three machines are up and C just responds quickly. in that case C would tell A to truncate z+1 from its log.) in theory we haven't violated our 2f+1 guarantee, since A is failed and B still hasn't recovered from failure, but it would be nice if when we don't have quorum that system stops working rather than works incorrectly if we lose quorum.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)