[jira] [Created] (ZOOKEEPER-4685) Unnecessary system unavailability due to Leader shutdown when follower sent ACK of PROPOSAL before sending ACK of NEWLEADER in log recovery

["Sirius (Jira)" <ji...@apache.org>]

Sirius created ZOOKEEPER-4685:
---------------------------------

             Summary: Unnecessary system unavailability due to Leader shutdown when follower sent ACK of PROPOSAL before sending ACK of NEWLEADER in log recovery
                 Key: ZOOKEEPER-4685
                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-4685
             Project: ZooKeeper
          Issue Type: Bug
          Components: quorum, server
    Affects Versions: 3.8.1, 3.7.1, 3.8.0, 3.7.0, 3.6.3
            Reporter: Sirius


When a follower is processing the NEWLEADER message in SYNC phase, it will call {{logRequest(..)}} to submit the txn persistence task to the SyncRequestProcessor thread. The SyncRequestProcessor thread may persist txns and reply ACK of that txn before replying ACK-LD (i.e. ACK of NEWLEADER) to the leader. This may cause the consequence that the leader cannot collect enough number of ACK-LDs successfully, followed by the leader's shutdown and a new round of election. This introduces unnecessary recovery procedures, consumes extra time before servers get into the BROADCAST phase and reduces the service's availability a lot. 

The following trace can be generated in the latest version nowadays.

 
h2. Trace

Start the ensemble with three nodes: S{+}0{+}, +S1+ & {+}S2{+}.
 - +S2+ is elected leader.
 - +S2+ logs a new txn <1, 1> and makes a broadcast.
 - +S0+ restarts & +S1+ crashes before receiving the proposal of <1, 1>.
 - +S2+ is elected leader again. 
 - +S2+ syncs with +S0+ using DIFF, and sends the proposal of <1, 1> during SYNC.
 - After +S0+ receives NEWLEADER,  {+}S0{+}'s sync thread may persist the txn <1, 1> and reply corresponding ACK to the leader +S2+ before {+}S0{+}'s QuorumPeer thread replies ACK-LD to the leader +S2+ .(This is possible because txn logging is processed asynchronously by Sync thread! )
 - The corresponding learnerHandler on +S2+ cannot recognize the ACK of some proposal before ACK-LD, and is going to be blocked at _waitForStartup()_ until the leader turn its state to {_}state.RUNNING{_}.
 - However, the quorumPeer thread of the leader +S2+ cannot receive enough number of ACK-LD, and then throws _InterruptedException_ during {_}waitForNewLeaderAck(..){_}.
 - After that, the leader will shutdown and a new round of election is raised, which consumes extra time for establishing the quorum and reduces availability a lot.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)