You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@zookeeper.apache.org by GitBox <gi...@apache.org> on 2021/05/03 06:20:41 UTC

[GitHub] [zookeeper] BinyuHuang-nju opened a new pull request #1690: Provide formal specification and verification using TLA+ for Zab

BinyuHuang-nju opened a new pull request #1690:
URL: https://github.com/apache/zookeeper/pull/1690


   We've made a formal specification for Zab in paper "High-performance broadcast for primary-backup systems" and a certain scale of model checking to verify the correctness of Zab(currently still running). There may exist differences between our specification and Zab in paper or in implementation. Please let us know flaws of specification if you have any question.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] BinyuHuang-nju edited a comment on pull request #1690: ZOOKEEPER-3615: Provide formal specification and verification using TLA+ for Zab

Posted by GitBox <gi...@apache.org>.
BinyuHuang-nju edited a comment on pull request #1690:
URL: https://github.com/apache/zookeeper/pull/1690#issuecomment-850846721


   @Vanlightly @hanm Hi! I have read some code about zab. But I still cannot understand some details. I want to ask you about them.
   
   1. At line 630 in function 'syncWithLeader' In [Learner.java](https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/quorum/Learner.java), I see follower in phase SYNCHRONIZATION may receive PROPOSAL and COMMIT. But in paper, leader will add a new server into its 'forwardingFollowers'(Q in paper) only when receiving ack of NEWLEADER from it, so in paper this new server should not receive PROPOSAL or COMMIT because in phase SYNCHRONIZATION it has not sent ACK-NEWLEADER.
   So here, my question is in ZAB 1.0, when does leader add a newly joined server into its 'forwardingFollowers'?(I am sorry that I didn't find this in code)
   
   2. My understanding of question above is: in phase 2, leader syncs with each follower and ''sends'' NEWLEADER only to this corresponding follower that has completed sync, not ''broadcasts'' NEWLEADER. So when leader is in phase 3, there may exist some follower in phase 2, and has not completed sync with leader since it hasn't received NEWLEADER, but it is allowed to receive PROPOSAL and COMMIT.
   I am not sure whether my understanding is right. If so, I think my spec about phase 2 is that, for each connected follower: first, leader syncs with  the follower by an abstract action  RECOVERY_SYNC; second, in the action, follower will have the latest history of leader, and leader sends NEWLEADER to it; then, leader and follower communicate using ACK-NEWLEADER and UPTODATE and follower turns to phase 3. Leader turns to phase 3 upon receiving ACK-NEWLEADER from a quorum. 
   
   3. At line 766 in function 'syncWithLeader' in [Learner.java](https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/quorum/Learner.java), I see when receiving NEWLEADER, the variable 'packetsNotCommitted' is set as empty, and when receiving UPTODATE, there is no action handling transactions left to commit them.
   Does that mean follower thinks all transactions committed upon receiving NEWLEADER not UPTODATE?(Actually in papaer and my spec of zab pre-1.0, follower thinks them committed upon receiving UPTODATE)
   
   4. In phase 1 in manual tha we mentioned above, there's one word: "Note, if a follower is connecting, but the leader is already established (in phase 3) the follower follows the phases, but the leader ignores any ACKs." 
   It seems like that leader in phase 3(BROADCAST) will ignre ACKEPOCH in phase1 and ACK of NEWLEADER in phase 2, then how dose this newly connecting follower receive UPTODATE to go to phase BROADCAST?
   
   Since these details are related to how to let follower sync with leader in my spec of ZAB 1.0, I want to have a further understanding of them. Thank you!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] hanm commented on pull request #1690: Provide formal specification and verification using TLA+ for Zab

Posted by GitBox <gi...@apache.org>.
hanm commented on pull request #1690:
URL: https://github.com/apache/zookeeper/pull/1690#issuecomment-831434710


   Thanks for the pull request @BinyuHuang-nju , glad someone finally picked this up. I'll provide some feedback this week.
   
   Quick notes:
   
   >> how can I pass Travis CI building
   
   You would need put Apache License on the TLA specification files: Zab.tla and ZabWithQTest.tla. Hopefully this will make the build happy. Just give you an example of what the license looks like: https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/ZooKeeper.java#L1
   
   Also, can you please update the title of the pull request with a JIRA number so this pull request can get merged after it's reviewed? I think ZOOKEEPER-3615 is a good candidate, and I assume @maoling don't mind that you steal this JIRA from them.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] Vanlightly commented on pull request #1690: ZOOKEEPER-3615: Provide formal specification and verification using TLA+ for Zab

Posted by GitBox <gi...@apache.org>.
Vanlightly commented on pull request #1690:
URL: https://github.com/apache/zookeeper/pull/1690#issuecomment-909366826


   I have started looking at the new version. One thing I have done is add a TypeOK invariant checking that the types of all variables are correct in all states. This helps me understand the spec better as it acts as a reference for what the different variables hold but can also be useful for catching type errors. I have it running with simulation for a few hours so far with no violations.
   
   Here's the TypeOK:
   
   ```
   ServersIncNullPoint == Server \union {NullPoint} 
   
   Zxid ==
       Seq(Nat)
       
   HistoryItem ==
        [epoch: Nat,
         counter: Nat,
         value: Value]    
       
   Proposal ==
       [msource: Server, mtype: {"RECOVERYSYNC"}, mepoch: Nat, mproposals: Seq(HistoryItem)] \union
       [msource: Server, mtype: {PROPOSAL}, mepoch: Nat, mproposal: HistoryItem]   
   
   Message ==
       [mtype: {FOLLOWERINFO}, mepoch: Nat] \union
       [mtype: {NEWLEADER}, mepoch: Nat, mlastZxid: Zxid] \union
       [mtype: {ACKLD}, mepoch: Nat] \union
       [mtype: {LEADERINFO}, mepoch: Nat] \union
       [mtype: {ACKEPOCH}, mepoch: Nat, mlastEpoch: Nat, mlastZxid: Zxid] \union
       [mtype: {UPTODATE}, mepoch: Nat, mcommit: Nat] \union
       [mtype: {PROPOSAL}, mepoch: Nat, mproposal: HistoryItem] \union
       [mtype: {ACK}, mepoch: Nat, mzxid: Zxid] \union
       [mtype: {COMMIT}, mepoch: Nat, mzxid: Zxid]
       
   ElectionState == {LOOKING, FOLLOWING, LEADING}
   
   Vote ==
       [proposedLeader: ServersIncNullPoint,
        proposedZxid: Zxid,
        proposedEpoch: Nat]
        
   ElectionVote ==
       [vote: Vote, round: Nat, state: ElectionState, version: Nat]
   
   ElectionMsg ==
       [mtype: {NOTIFICATION}, msource: Server, mstate: ElectionState, mround: Nat, mvote: Vote] \union
       [mtype: {NONE}]
               
   TypeOK ==
       /\ zabState \in [Server -> {ELECTION, DISCOVERY, SYNCHRONIZATION, BROADCAST}]
       /\ acceptedEpoch \in [Server -> Nat]
       /\ history \in [Server -> Seq(HistoryItem)] 
       /\ commitIndex \in [Server -> Nat]
       /\ learners \in [Server -> SUBSET ServersIncNullPoint]
       /\ cepochRecv \in [Server -> SUBSET ServersIncNullPoint]
       /\ ackeRecv \in [Server -> SUBSET ServersIncNullPoint]
       /\ ackldRecv \in [Server -> SUBSET ServersIncNullPoint]
       /\ ackIndex \in [Server -> [Server -> Nat]]
       /\ currentCounter \in [Server -> Nat]
       /\ sendCounter \in [Server -> Nat]
       /\ committedIndex \in [Server -> Nat]
       /\ committedCounter \in [Server -> [Server -> Nat]]
       /\ forwarding \in [Server -> SUBSET ServersIncNullPoint]
       /\ initialHistory \in [Server -> Seq(HistoryItem)] 
       /\ tempMaxEpoch \in [Server -> Nat]
       /\ cepochSent \in [Server -> BOOLEAN]
       /\ leaderAddr \in [Server -> ServersIncNullPoint]
       /\ synced \in [Server -> BOOLEAN]
       /\ proposalMsgsLog \in SUBSET Proposal
       /\ epochLeader \in [1..MAXEPOCH -> SUBSET ServersIncNullPoint]
       /\ inherentViolated \in BOOLEAN
       /\ forwarding \in [Server -> SUBSET Server]
       /\ msgs \in [Server -> [Server -> Seq(Message)]]
       
       \* Fast Leader Election
       /\ electionMsgs \in [Server -> [Server -> Seq(ElectionMsg)]]
       /\ recvQueue \in [Server -> Seq(ElectionMsg)]
       /\ leadingVoteSet \in [Server -> SUBSET Server]
       /\ receiveVotes \in [Server -> [Server -> ElectionVote]]
       /\ currentVote \in [Server -> Vote]
       /\ outOfElection \in [Server -> [Server -> ElectionVote]]
       /\ lastZxid \in [Server -> Zxid]
       /\ state \in [Server -> ElectionState]
       /\ waitNotmsg \in [Server -> BOOLEAN]
       /\ currentEpoch \in [Server -> Nat]
       /\ logicalClock \in [Server -> Nat]
   ```
   
   I still need to dig into the logic further so I understand it and check it against ZK code.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@zookeeper.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] BinyuHuang-nju edited a comment on pull request #1690: ZOOKEEPER-3615: Provide formal specification and verification using TLA+ for Zab

Posted by GitBox <gi...@apache.org>.
BinyuHuang-nju edited a comment on pull request #1690:
URL: https://github.com/apache/zookeeper/pull/1690#issuecomment-850846721


   @Vanlightly @hanm Hi! I have read some code about zab. But I still cannot understand some details. I want to ask you about them.
   
   1. At line 630 in function 'syncWithLeader' In [Learner.java](https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/quorum/Learner.java), I see follower in phase SYNCHRONIZATION may receive PROPOSAL and COMMIT. But in paper, leader will add a new server into its 'forwardingFollowers'(Q in paper) only when receiving ack of NEWLEADER from it, so in paper this new server should not receive PROPOSAL or COMMIT because in phase SYNCHRONIZATION it has not sent ACK-NEWLEADER.
   So here, my question is in ZAB 1.0, when does leader add a newly joined server into its 'forwardingFollowers'?(I am sorry that I didn't find this in code)
   
   2. At line 766 in function 'syncWithLeader' in [Learner.java](https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/quorum/Learner.java), I see when receiving NEWLEADER, the variable 'packetsNotCommitted' is set as empty, and when receiving UPTODATE, there is no action handling transactions left to commit them.
   Does that mean follower thinks all transactions committed upon receiving NEWLEADER not UPTODATE?(Actually in papaer and my spec of zab pre-1.0, follower thinks them committed upon receiving UPTODATE)
   
   Since these details are related to how to let follower sync with leader in my spec of ZAB 1.0, I want to have a further understanding of them. Thank you!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] BinyuHuang-nju commented on pull request #1690: ZOOKEEPER-3615: Provide formal specification and verification using TLA+ for Zab

Posted by GitBox <gi...@apache.org>.
BinyuHuang-nju commented on pull request #1690:
URL: https://github.com/apache/zookeeper/pull/1690#issuecomment-912909935


   @Vanlightly It's very nice for you to give me the suggestions. I will take it into spec, and update spec later! Thank you! If you find any question, please just tell me anytime.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@zookeeper.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] BinyuHuang-nju commented on a change in pull request #1690: ZOOKEEPER-3615: Provide formal specification and verification using TLA+ for Zab

Posted by GitBox <gi...@apache.org>.
BinyuHuang-nju commented on a change in pull request #1690:
URL: https://github.com/apache/zookeeper/pull/1690#discussion_r630750303



##########
File path: zookeeper-contrib/zookeeper-contrib-zabtla/README.md
##########
@@ -0,0 +1,71 @@
+# Zab-tla
+
+## Overview
+This project is devoted to providing formal specification and verification using TLA+ for the Zookeeper Atomic Broadcast(Zab) consensus protocol proposed in *Junqueira F P, Reed B C, Serafini M. Zab: High-performance broadcast for primary-backup systems[C]//2011 IEEE/IFIP 41st International Conference on Dependable Systems & Networks (DSN). IEEE, 2011: 245-256*.  
+
+We have made a formal [specification](Zab.tla) for Zab using TLA+ toolbox, and we have done a certain scale of model checking to verify the correctness of Zab.
+
+Due to the simplification of Zab algorithm description in the paper, some details in specification were modified and added. If you have any question, please let us know.
+
+You can find this document in chinese in [doc-in-chinsese](doc-in-chinese/README.md).
+
+## Requirements
+TLA+ toolbox version 1.7.0
+
+## Run
+Create specification and run models in the usual way.  

Review comment:
       Thank you! I can write a more detailed manual  in README about how to configure TLC parameters. Since the .toolbox space is too large, I can upload some more important files in .toolbox folder(such as .settings, .projejct, Model/MC.tla, Model/MC.cfg).
   Currrently I have something to do, and I will implement your suggestions like modifying ZabWithQTest.tla and README, reading ZAB 1.0 as soon as possible in several days!




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] Vanlightly commented on pull request #1690: ZOOKEEPER-3615: Provide formal specification and verification using TLA+ for Zab

Posted by GitBox <gi...@apache.org>.
Vanlightly commented on pull request #1690:
URL: https://github.com/apache/zookeeper/pull/1690#issuecomment-909366826


   I have started looking at the new version. One thing I have done is add a TypeOK invariant checking that the types of all variables are correct in all states. This helps me understand the spec better as it acts as a reference for what the different variables hold but can also be useful for catching type errors. I have it running with simulation for a few hours so far with no violations.
   
   Here's the TypeOK:
   
   ```
   ServersIncNullPoint == Server \union {NullPoint} 
   
   Zxid ==
       Seq(Nat)
       
   HistoryItem ==
        [epoch: Nat,
         counter: Nat,
         value: Value]    
       
   Proposal ==
       [msource: Server, mtype: {"RECOVERYSYNC"}, mepoch: Nat, mproposals: Seq(HistoryItem)] \union
       [msource: Server, mtype: {PROPOSAL}, mepoch: Nat, mproposal: HistoryItem]   
   
   Message ==
       [mtype: {FOLLOWERINFO}, mepoch: Nat] \union
       [mtype: {NEWLEADER}, mepoch: Nat, mlastZxid: Zxid] \union
       [mtype: {ACKLD}, mepoch: Nat] \union
       [mtype: {LEADERINFO}, mepoch: Nat] \union
       [mtype: {ACKEPOCH}, mepoch: Nat, mlastEpoch: Nat, mlastZxid: Zxid] \union
       [mtype: {UPTODATE}, mepoch: Nat, mcommit: Nat] \union
       [mtype: {PROPOSAL}, mepoch: Nat, mproposal: HistoryItem] \union
       [mtype: {ACK}, mepoch: Nat, mzxid: Zxid] \union
       [mtype: {COMMIT}, mepoch: Nat, mzxid: Zxid]
       
   ElectionState == {LOOKING, FOLLOWING, LEADING}
   
   Vote ==
       [proposedLeader: ServersIncNullPoint,
        proposedZxid: Zxid,
        proposedEpoch: Nat]
        
   ElectionVote ==
       [vote: Vote, round: Nat, state: ElectionState, version: Nat]
   
   ElectionMsg ==
       [mtype: {NOTIFICATION}, msource: Server, mstate: ElectionState, mround: Nat, mvote: Vote] \union
       [mtype: {NONE}]
               
   TypeOK ==
       /\ zabState \in [Server -> {ELECTION, DISCOVERY, SYNCHRONIZATION, BROADCAST}]
       /\ acceptedEpoch \in [Server -> Nat]
       /\ history \in [Server -> Seq(HistoryItem)] 
       /\ commitIndex \in [Server -> Nat]
       /\ learners \in [Server -> SUBSET ServersIncNullPoint]
       /\ cepochRecv \in [Server -> SUBSET ServersIncNullPoint]
       /\ ackeRecv \in [Server -> SUBSET ServersIncNullPoint]
       /\ ackldRecv \in [Server -> SUBSET ServersIncNullPoint]
       /\ ackIndex \in [Server -> [Server -> Nat]]
       /\ currentCounter \in [Server -> Nat]
       /\ sendCounter \in [Server -> Nat]
       /\ committedIndex \in [Server -> Nat]
       /\ committedCounter \in [Server -> [Server -> Nat]]
       /\ forwarding \in [Server -> SUBSET ServersIncNullPoint]
       /\ initialHistory \in [Server -> Seq(HistoryItem)] 
       /\ tempMaxEpoch \in [Server -> Nat]
       /\ cepochSent \in [Server -> BOOLEAN]
       /\ leaderAddr \in [Server -> ServersIncNullPoint]
       /\ synced \in [Server -> BOOLEAN]
       /\ proposalMsgsLog \in SUBSET Proposal
       /\ epochLeader \in [1..MAXEPOCH -> SUBSET ServersIncNullPoint]
       /\ inherentViolated \in BOOLEAN
       /\ forwarding \in [Server -> SUBSET Server]
       /\ msgs \in [Server -> [Server -> Seq(Message)]]
       
       \* Fast Leader Election
       /\ electionMsgs \in [Server -> [Server -> Seq(ElectionMsg)]]
       /\ recvQueue \in [Server -> Seq(ElectionMsg)]
       /\ leadingVoteSet \in [Server -> SUBSET Server]
       /\ receiveVotes \in [Server -> [Server -> ElectionVote]]
       /\ currentVote \in [Server -> Vote]
       /\ outOfElection \in [Server -> [Server -> ElectionVote]]
       /\ lastZxid \in [Server -> Zxid]
       /\ state \in [Server -> ElectionState]
       /\ waitNotmsg \in [Server -> BOOLEAN]
       /\ currentEpoch \in [Server -> Nat]
       /\ logicalClock \in [Server -> Nat]
   ```
   
   I still need to dig into the logic further so I understand it and check it against ZK code.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@zookeeper.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] Vanlightly commented on a change in pull request #1690: ZOOKEEPER-3615: Provide formal specification and verification using TLA+ for Zab

Posted by GitBox <gi...@apache.org>.
Vanlightly commented on a change in pull request #1690:
URL: https://github.com/apache/zookeeper/pull/1690#discussion_r705452337



##########
File path: zookeeper-specifications/zab-1.0/ZabWithFLE.tla
##########
@@ -0,0 +1,1020 @@
+(*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *)
+
+----------------------------- MODULE ZabWithFLE -----------------------------
+(* This is the formal specification for the Zab consensus algorithm,
+   which means Zookeeper Atomic Broadcast.*)
+
+(* Reference:
+   FLE: FastLeaderElection.java, Vote.java, QuorumPeer.java in https://github.com/apache/zookeeper.
+   ZAB: QuorumPeer.java, Learner.java, Follower.java, LearnerHandler.java, Leader.java 
+        in https://github.com/apache/zookeeper.
+        https://cwiki.apache.org/confluence/display/ZOOKEEPER/Zab1.0.
+ *)
+EXTENDS FastLeaderElection
+
+-----------------------------------------------------------------------------
+(* Defined in FastLeaderElection.tla:
+\* The set of server identifiers
+CONSTANT Server
+
+\* Server states
+CONSTANTS LOOKING, FOLLOWING, LEADING
+
+\* Message types
+CONSTANTS NOTIFICATION
+
+\* Timeout signal
+CONSTANT NONE
+*)
+\* The set of requests that can go into history
+CONSTANT Value
+
+\* Zab states
+CONSTANTS ELECTION, DISCOVERY, SYNCHRONIZATION, BROADCAST
+
+\* Message types
+CONSTANTS FOLLOWERINFO, LEADERINFO, ACKEPOCH, NEWLEADER, ACKLD, UPTODATE, PROPOSAL, ACK, COMMIT
+(* Additional message types used for recovery in synchronization(TRUNC/DIFF/SNAP) are not needed
+   since we abstract this part.(see action RECOVERYSYNC) *)
+   
+-----------------------------------------------------------------------------
+(* Defined in FastLeaderElection.tla: Quorums, NullPoint *)
+\* Return the maximum value from the set S
+Maximum(S) == IF S = {} THEN -1
+                        ELSE CHOOSE n \in S: \A m \in S: n >= m
+
+\* Return the minimum value from the set S
+Minimum(S) == IF S = {} THEN -1
+                        ELSE CHOOSE n \in S: \A m \in S: n <= m
+
+MAXEPOCH == 10
+
+-----------------------------------------------------------------------------
+(* Defined in FastLeaderElection.tla:
+   serverVars: <<state, currentEpoch, lastZxid>>,
+   electionVars: <<currentVote, logicalClock, receiveVotes, outOfElection, recvQueue, waitNotmsg>>,
+   leaderVars: <<leadingVoteSet>>,
+   electionMsgs,
+   idTable *)
+\* The current phase of server(ELECTION,DISCOVERY,SYNCHRONIZATION,BROADCAST)
+VARIABLE zabState
+
+\* The epoch number of the last NEWEPOCH(LEADERINFO) packet accepted
+\* namely f.p in paper, and currentEpoch in Zab.tla.
+VARIABLE acceptedEpoch
+
+\* The history of servers as the sequence of transactions.
+VARIABLE history
+
+\* commitIndex[i]: The maximum index of transactions that have been saved in a quorum of servers
+\*                 in the perspective of server i.(increases monotonically before restarting)
+VARIABLE commitIndex
+
+(* These transactions whose index \le commitIndex[i] can be applied to state machine immediately.
+   So if we have a variable applyIndex, we can suppose that applyIndex[i] = commitIndex[i] when verifying properties.
+   But in phase SYNC, follower will apply all queued proposals to state machine when receiving NEWLEADER.
+   But follower only serves traffic after receiving UPTODATE, so sequential consistency is not violated.
+   
+   So when we verify properties, we still suppose applyIndex[i] = commitIndex[i], because this is an engineering detail.*)
+
+\* learners[i]: The set of servers which leader i think are connected wich i.
+VARIABLE learners
+
+\* The messages representing requests and responses sent from one server to another.
+\* msgs[i][j] means the input buffer of server j from server i.
+VARIABLE msgs
+
+\* The set of followers who has successfully sent CEPOCH(FOLLOWERINFO) to leader.(equals to connectingFollowers in code)
+VARIABLE cepochRecv
+
+\* The set of followers who has successfully sent ACK-E to leader.(equals to electingFollowers in code)
+VARIABLE ackeRecv
+
+\* The set of followers who has successfully sent ACK-LD to leader in leader.(equals to newLeaderProposal in code)
+VARIABLE ackldRecv
+
+\* The set of servers which leader i broadcasts PROPOSAL and COMMIT to.(equals to forwardingFollowers in code)
+VARIABLE forwarding
+
+\* ackIndex[i][j]: The latest index that leader i has received from follower j via ACK.
+VARIABLE ackIndex
+
+\* currentCounter[i]: The count of transactions that clients request leader i.
+VARIABLE currentCounter
+
+\* sendCounter[i]: The count of transactions that leader i has broadcast in PROPOSAL.
+VARIABLE sendCounter
+
+\* committedIndex[i]: The maximum index of trasactions that leader i has broadcast in COMMIT.
+VARIABLE committedIndex
+
+\* committedCounter[i][j]: The latest counter of transaction that leader i has confirmed that follower j has committed.
+VARIABLE committedCounter
+
+\* initialHistory[i]: The initial history if leader i in epoch acceptedEpoch[i].
+VARIABLE initialHistory
+
+\* the maximum epoch in CEPOCH the prospective leader received from followers.
+VARIABLE tempMaxEpoch
+
+\* cepochSent[i] = TRUE means follower i has sent CEPOCH(FOLLOWERINFO) to leader.
+VARIABLE cepochSent
+
+\* leaderAddr[i]: The leader id of follower i. We use leaderAddr to express whether follower i has connected or lost connection.
+VARIABLE leaderAddr
+
+\* synced[i] = TRUE: follower i has completed sync with leader.
+VARIABLE synced
+
+\* The set of leaders in every epoch, only used in verifying properties.
+VARIABLE epochLeader
+
+\* The set of all broadcast messages, only used in verifying properties.
+VARIABLE proposalMsgsLog
+
+\* A variable used to check whether there are conditions contrary to the facts.
+VARIABLE inherentViolated
+
+serverVarsZ == <<state, currentEpoch, lastZxid, zabState, acceptedEpoch, history, commitIndex>>         \* 7 variables
+
+electionVarsZ == electionVars  \* 6 variables
+
+leaderVarsZ == <<leadingVoteSet, learners, cepochRecv, ackeRecv, ackldRecv, forwarding, 
+                 ackIndex, currentCounter, sendCounter, committedIndex, committedCounter>>              \* 11 variables
+
+tempVarsZ == <<initialHistory, tempMaxEpoch>>     \* 2 variables
+  
+followerVarsZ == <<cepochSent, leaderAddr, synced>>                 \* 3 variables
+
+verifyVarsZ == <<proposalMsgsLog, epochLeader, inherentViolated>>   \* 3 variables
+
+msgVarsZ == <<msgs, electionMsgs>>                \* 2 variables
+
+vars == <<serverVarsZ, electionVarsZ, leaderVarsZ, tempVarsZ, followerVarsZ, verifyVarsZ, msgVarsZ, idTable>>  \* 35 variables
+-----------------------------------------------------------------------------
+\* Add a message to msgs - add a message m to msgs[i][j].
+Send(i, j, m) == msgs' = [msgs EXCEPT ![i][j] = Append(msgs[i][j], m)]
+
+\* Remove a message from msgs - discard head of msgs[i][j].
+Discard(i, j) == msgs' = IF msgs[i][j] /= << >> THEN [msgs EXCEPT ![i][j] = Tail(msgs[i][j])]
+                                                ELSE msgs
+
+\* Leader broadcasts a message(PROPOSAL/COMMIT) to all other servers in forwardingFollowers.
+Broadcast(i, m) == msgs' = [msgs EXCEPT ![i] = [v \in Server |-> IF /\ v \in forwarding[i]
+                                                                    /\ v /= i
+                                                                    /\ \/ /\ m.mtype = PROPOSAL
+                                                                          /\ ackIndex[i][v] < Len(initialHistory[i]) + m.mproposal.counter
+                                                                       \/ /\ m.mtype = COMMIT
+                                                                          /\ committedCounter[i][v] < m.mzxid[2]
+                                                                  THEN Append(msgs[i][v], m)
+                                                                  ELSE msgs[i][v]]]
+
+BroadcastLEADERINFO(i, m) == msgs' = [msgs EXCEPT ![i] = [v \in Server |-> IF /\ v \in cepochRecv[i]
+                                                                              /\ v \in learners[i]
+                                                                              /\ v /= i THEN Append(msgs[i][v], m)
+                                                                                        ELSE msgs[i][v]]]
+
+BroadcastUPTODATE(i, m) == msgs' = [msgs EXCEPT ![i] = [v \in Server |-> IF /\ v \in ackldRecv[i]
+                                                                            /\ v \in learners[i]
+                                                                            /\ v /= i THEN Append(msgs[i][v], m)
+                                                                                      ELSE msgs[i][v]]]
+
+\* Combination of Send and Discard - discard head of msgs[j][i] and add m into msgs[i][j].
+Reply(i, j, m) == msgs' = [msgs EXCEPT ![j][i] = Tail(msgs[j][i]),
+                                       ![i][j] = Append(msgs[i][j], m)]
+
+\* shuffle the input buffer from server j(i) in server i(j).
+Clean(i, j) == msgs' = [msgs EXCEPT ![j][i] = << >>, ![i][j] = << >>]
+
+-----------------------------------------------------------------------------
+PZxidEqual(p, z) == p.epoch = z[1] /\ p.counter = z[2]
+
+TransactionEqual(t1, t2) == /\ t1.epoch   = t2.epoch
+                            /\ t1.counter = t2.counter
+                            
+TransactionPrecede(t1, t2) == \/ t1.epoch < t2.epoch
+                              \/ /\ t1.epoch   = t2.epoch
+                                 /\ t1.counter < t2.counter
+                                 
+-----------------------------------------------------------------------------
+\* Define initial values for all variables 
+InitServerVarsZ == /\ InitServerVars
+                   /\ zabState      = [s \in Server |-> ELECTION]
+                   /\ acceptedEpoch = [s \in Server |-> 0]
+                   /\ history       = [s \in Server |-> << >>]
+                   /\ commitIndex   = [s \in Server |-> 0]
+
+InitLeaderVarsZ == /\ InitLeaderVars
+                   /\ learners         = [s \in Server |-> {}]
+                   /\ cepochRecv       = [s \in Server |-> {}]
+                   /\ ackeRecv         = [s \in Server |-> {}]
+                   /\ ackldRecv        = [s \in Server |-> {}]
+                   /\ ackIndex         = [s \in Server |-> [v \in Server |-> 0]]
+                   /\ currentCounter   = [s \in Server |-> 0]
+                   /\ sendCounter      = [s \in Server |-> 0]
+                   /\ committedIndex   = [s \in Server |-> 0]
+                   /\ committedCounter = [s \in Server |-> [v \in Server |-> 0]]
+                   /\ forwarding       = [s \in Server |-> {}]
+
+InitElectionVarsZ == InitElectionVars
+
+InitTempVarsZ == /\ initialHistory = [s \in Server |-> << >>]
+                 /\ tempMaxEpoch   = [s \in Server |-> 0]
+
+InitFollowerVarsZ == /\ cepochSent = [s \in Server |-> FALSE]
+                     /\ leaderAddr = [s \in Server |-> NullPoint]
+                     /\ synced     = [s \in Server |-> FALSE]
+
+InitVerifyVarsZ == /\ proposalMsgsLog  = {}
+                   /\ epochLeader      = [i \in 1..MAXEPOCH |-> {}]
+                   /\ inherentViolated = FALSE
+                   
+InitMsgVarsZ == /\ msgs         = [s \in Server |-> [v \in Server |-> << >>]]
+                /\ electionMsgs = [s \in Server |-> [v \in Server |-> << >>]]
+
+InitZ == /\ InitServerVarsZ
+         /\ InitLeaderVarsZ
+         /\ InitElectionVarsZ
+         /\ InitTempVarsZ
+         /\ InitFollowerVarsZ
+         /\ InitVerifyVarsZ
+         /\ InitMsgVarsZ
+         /\ idTable = InitializeIdTable(Server)
+         
+-----------------------------------------------------------------------------
+ZabTurnToLeading(i) ==
+        /\ zabState'       = [zabState   EXCEPT ![i] = DISCOVERY]
+        /\ learners'       = [learners   EXCEPT ![i] = {i}]
+        /\ cepochRecv'     = [cepochRecv EXCEPT ![i] = {i}]
+        /\ ackeRecv'       = [ackeRecv   EXCEPT ![i] = {i}]
+        /\ ackldRecv'      = [ackldRecv  EXCEPT ![i] = {i}]
+        /\ forwarding'     = [forwarding EXCEPT ![i] = {}]
+        /\ ackIndex'       = [ackIndex   EXCEPT ![i] = [v \in Server |-> IF v = i THEN Len(history[i])
+                                                                                  ELSE 0]]
+        /\ currentCounter'   = [currentCounter   EXCEPT ![i] = 0]
+        /\ sendCounter'      = [sendCounter      EXCEPT ![i] = 0]
+        /\ commitIndex'      = [commitIndex      EXCEPT ![i] = 0]
+        /\ committedIndex'   = [committedIndex   EXCEPT ![i] = 0]
+        /\ committedCounter' = [committedCounter EXCEPT ![i] = [v \in Server |-> IF v = i THEN Len(history[i])
+                                                                                          ELSE 0]]
+        /\ initialHistory' = [initialHistory EXCEPT ![i] = history[i]]
+        /\ tempMaxEpoch'   = [tempMaxEpoch   EXCEPT ![i] = acceptedEpoch[i]]
+
+ZabTurnToFollowing(i) ==
+        /\ zabState'    = [zabState   EXCEPT ![i] = DISCOVERY]
+        /\ cepochSent'  = [cepochSent EXCEPT ![i] = FALSE]
+        /\ synced'      = [synced     EXCEPT ![i] = FALSE]
+        /\ commitIndex' = [commitIndex      EXCEPT ![i] = 0]
+        
+          
+(* Fast Leader Election *)
+FLEReceiveNotmsg(i, j) ==
+        /\ ReceiveNotmsg(i, j)
+        /\ UNCHANGED <<zabState, acceptedEpoch, history, commitIndex,learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter,
+                       sendCounter, committedIndex, committedCounter, tempVarsZ, followerVarsZ, verifyVarsZ, msgs>>
+
+FLENotmsgTimeout(i) ==
+        /\ NotmsgTimeout(i)
+        /\ UNCHANGED <<zabState, acceptedEpoch, history, commitIndex,learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter,
+                       sendCounter, committedIndex, committedCounter, tempVarsZ, followerVarsZ, verifyVarsZ, msgs>>
+
+FLEHandleNotmsg(i) ==
+        /\ HandleNotmsg(i)
+        /\ LET newState == state'[i]
+           IN
+           \/ /\ newState = LEADING
+              /\ ZabTurnToLeading(i)
+              /\ UNCHANGED <<cepochSent, synced>>
+           \/ /\ newState = FOLLOWING
+              /\ ZabTurnToFollowing(i)
+              /\ UNCHANGED <<learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter, sendCounter, committedIndex, committedCounter, tempVarsZ>>
+           \/ /\ newState = LOOKING
+              /\ UNCHANGED <<zabState, learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter, sendCounter, commitIndex,
+                             committedIndex, committedCounter, tempVarsZ, cepochSent, synced>>
+        /\ UNCHANGED <<acceptedEpoch, history, leaderAddr, verifyVarsZ, msgs>>
+
+\* On the premise that ReceiveVotes.HasQuorums = TRUE, corresponding to logic in line 1050-1055 in LFE.java.
+FLEWaitNewNotmsg(i) ==
+        /\ WaitNewNotmsg(i)
+        /\ UNCHANGED <<zabState, acceptedEpoch, history, commitIndex,learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter,
+                       sendCounter, committedIndex, committedCounter, tempVarsZ, followerVarsZ, verifyVarsZ, msgs>>
+
+\* On the premise that ReceiveVotes.HasQuorums = TRUE, corresponding to logic in line 1061-1066 in LFE.java.
+FLEWaitNewNotmsgEnd(i) ==
+        /\ WaitNewNotmsgEnd(i)
+        /\ LET newState == state'[i]
+           IN
+           \/ /\ newState = LEADING
+              /\ ZabTurnToLeading(i)
+              /\ UNCHANGED <<cepochSent, synced>>
+           \/ /\ newState = FOLLOWING
+              /\ ZabTurnToFollowing(i)
+              /\ UNCHANGED <<learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter, sendCounter, committedIndex, committedCounter, tempVarsZ>>
+           \/ /\ newState = LOOKING
+              /\ PrintT("New state is LOOKING in FLEWaitNewNotmsgEnd, which should not happen.")
+              /\ UNCHANGED <<zabState, learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter, sendCounter, commitIndex,
+                             committedIndex, committedCounter, tempVarsZ, cepochSent, synced>>
+        /\ UNCHANGED <<acceptedEpoch, history, leaderAddr, verifyVarsZ, msgs>>
+              
+-----------------------------------------------------------------------------
+(* A sub-action describing how a server transitions from LEADING/FOLLOWING to LOOKING.
+   Initially I call it 'ZabTimeoutZ', but it will be called not only when timeout, 
+   but also when finding a low epoch from leader.*)
+FollowerShutdown(i) ==
+        /\ ZabTimeout(i)
+        /\ zabState'   = [zabState   EXCEPT ![i] = ELECTION]
+        /\ leaderAddr' = [leaderAddr EXCEPT ![i] = NullPoint]
+
+LeaderShutdown(i) ==
+        /\ ZabTimeout(i)
+        /\ zabState'   = [zabState   EXCEPT ![i] = ELECTION]
+        /\ leaderAddr' = [s \in Server |-> IF s \in learners[i] THEN NullPoint ELSE leaderAddr[s]]
+        /\ learners'   = [learners   EXCEPT ![i] = {}]
+        /\ forwarding' = [forwarding EXCEPT ![i] = {}]
+        /\ msgs'       = [s \in Server |-> [v \in Server |-> IF v \in learners[i] \/ s \in learners[i] THEN << >> ELSE msgs[s][v]]]
+
+FollowerTimout(i) ==
+        /\ state[i]      = FOLLOWING
+        /\ leaderAddr[i] = NullPoint
+        /\ FollowerShutdown(i)
+        /\ msgs' = [s \in Server |-> [v \in Server |-> IF v = i THEN << >> ELSE msgs[s][v]]]
+        /\ UNCHANGED <<acceptedEpoch, history, commitIndex, learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, 
+                       currentCounter, sendCounter, committedIndex, committedCounter, tempVarsZ, cepochSent, synced, verifyVarsZ>>
+
+LeaderTimeout(i) ==
+        /\ state[i] = LEADING
+        /\ learners[i] \notin Quorums
+        /\ LeaderShutdown(i)
+        /\ UNCHANGED <<acceptedEpoch, history, commitIndex, cepochRecv, ackeRecv, ackldRecv, ackIndex, currentCounter, sendCounter, committedIndex, committedCounter,
+                       tempVarsZ, cepochSent, synced, verifyVarsZ>>
+    
+-----------------------------------------------------------------------------
+(* Establish connection between leader i and follower j. 
+   It means i creates a learnerHandler for communicating with j, and j finds i's address.*)
+EstablishConnection(i, j) ==
+        /\ state[i] = LEADING   /\ state[j] = FOLLOWING
+        /\ j \notin learners[i] /\ leaderAddr[j] = NullPoint
+        /\ currentVote[j].proposedLeader = i
+        /\ learners'   = [learners   EXCEPT ![i] = learners[i] \union {j}] \* Leader:   'addLearnerHandler(peer)'
+        /\ leaderAddr' = [leaderAddr EXCEPT ![j] = i]                      \* Follower: 'connectToLeader(addr, hostname)'
+        /\ UNCHANGED <<serverVarsZ, electionVarsZ, leadingVoteSet, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, 
+                       currentCounter, sendCounter, committedIndex, committedCounter, tempVarsZ, cepochSent, synced, verifyVarsZ, msgVarsZ, idTable>>
+        
+(* The leader i finds timeout and TCP connection between i and j closes.*)       
+Timeout(i, j) ==
+        /\ state[i] = LEADING /\ state[j] = FOLLOWING
+        /\ j \in learners[i]  /\ leaderAddr[j] = i
+        (* The action of leader i.(corresponding to function 'removeLearnerHandler(peer)'.) *)
+        /\ learners'   = [learners   EXCEPT ![i] = learners[i] \ {j}] 
+        /\ forwarding' = [forwarding EXCEPT ![i] = IF j \in forwarding[i] THEN forwarding[i] \ {j} ELSE forwarding[i]]
+        /\ cepochRecv' = [cepochRecv EXCEPT ![i] = IF j \in cepochRecv[i] THEN cepochRecv[i] \ {j} ELSE cepochRecv[i]]
+        /\ ackeRecv'   = [ackeRecv   EXCEPT ![i] = IF j \in ackeRecv[i] THEN ackeRecv[i] \ {j} ELSE ackeRecv[i]]
+        /\ ackldRecv'  = [ackldRecv  EXCEPT ![i] = IF j \in ackldRecv[i] THEN ackldRecv[i] \ {j} ELSE ackldRecv[i]]
+        /\ ackIndex'   = [ackIndex   EXCEPT ![i][j] = 0]
+        /\ committedCounter' = [committedCounter EXCEPT ![i][j] = 0]
+        (* The action of follower j. *)
+        /\ FollowerShutdown(j)
+        (* Clean input buffer.*)
+        /\ Clean(i, j)
+        /\ UNCHANGED <<acceptedEpoch, history, commitIndex, currentCounter, sendCounter, committedIndex,
+                       tempVarsZ, cepochSent, synced, verifyVarsZ>>
+-----------------------------------------------------------------------------
+\* In phase f11, follower sends f.p to leader via FOLLOWERINFO(CEPOCH).
+FollowerSendFOLLOWERINFO(i) ==
+        /\ state[i]    = FOLLOWING
+        /\ zabState[i] = DISCOVERY
+        /\ leaderAddr[i] /= NullPoint
+        /\ \lnot cepochSent[i]
+        /\ Send(i, leaderAddr[i], [mtype  |-> FOLLOWERINFO,
+                                   mepoch |-> acceptedEpoch[i]])
+        /\ cepochSent' = [cepochSent EXCEPT ![i] = TRUE]
+        /\ UNCHANGED <<serverVarsZ, leaderVarsZ, electionVarsZ, tempVarsZ, leaderAddr, synced, verifyVarsZ, electionMsgs, idTable>>
+        
+(* In phase l11, leader waits for receiving FOLLOWERINFO from a quorum,
+   and then chooses a new epoch e' as its own epoch and broadcasts LEADERINFO. *)
+LeaderHandleFOLLOWERINFO(i, j) ==
+        /\ state[i] = LEADING
+        /\ msgs[j][i] /= << >>
+        /\ msgs[j][i][1].mtype = FOLLOWERINFO
+        /\ LET msg == msgs[j][i][1]
+           IN \/ /\ NullPoint \notin cepochRecv[i]   \* 1. has not broadcast LEADERINFO - modify tempMaxEpoch
+                 /\ LET newEpoch == Maximum({tempMaxEpoch[i], msg.mepoch})
+                    IN tempMaxEpoch' = [tempMaxEpoch EXCEPT ![i] = newEpoch]
+                 /\ Discard(j, i)
+              \/ /\ NullPoint \in cepochRecv[i]      \* 2. has broadcast LEADERINFO - no need to handle the msg, just send LEADERINFO to corresponding server
+                 /\ Reply(i, j, [mtype  |-> LEADERINFO,
+                                 mepoch |-> acceptedEpoch[i]])
+                 /\ UNCHANGED tempMaxEpoch
+        /\ cepochRecv' = [cepochRecv EXCEPT ![i] = IF j \in cepochRecv[i] THEN cepochRecv[i]
+                                                                          ELSE cepochRecv[i] \union {j}]
+        /\ UNCHANGED <<serverVarsZ, followerVarsZ, electionVarsZ, initialHistory, leadingVoteSet, learners, ackeRecv, ackldRecv, 
+                       forwarding, ackIndex, currentCounter, sendCounter, committedIndex, committedCounter, verifyVarsZ, electionMsgs, idTable>>
+
+LeaderDiscovery1(i) ==

Review comment:
       Action name is not descriptive, perhaps rename to something like LeaderBroadcastsLEADERINFO




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@zookeeper.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] Vanlightly commented on a change in pull request #1690: ZOOKEEPER-3615: Provide formal specification and verification using TLA+ for Zab

Posted by GitBox <gi...@apache.org>.
Vanlightly commented on a change in pull request #1690:
URL: https://github.com/apache/zookeeper/pull/1690#discussion_r705461715



##########
File path: zookeeper-specifications/zab-1.0/ZabWithFLE.tla
##########
@@ -0,0 +1,1020 @@
+(*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *)
+
+----------------------------- MODULE ZabWithFLE -----------------------------
+(* This is the formal specification for the Zab consensus algorithm,
+   which means Zookeeper Atomic Broadcast.*)
+
+(* Reference:
+   FLE: FastLeaderElection.java, Vote.java, QuorumPeer.java in https://github.com/apache/zookeeper.
+   ZAB: QuorumPeer.java, Learner.java, Follower.java, LearnerHandler.java, Leader.java 
+        in https://github.com/apache/zookeeper.
+        https://cwiki.apache.org/confluence/display/ZOOKEEPER/Zab1.0.
+ *)
+EXTENDS FastLeaderElection
+
+-----------------------------------------------------------------------------
+(* Defined in FastLeaderElection.tla:
+\* The set of server identifiers
+CONSTANT Server
+
+\* Server states
+CONSTANTS LOOKING, FOLLOWING, LEADING
+
+\* Message types
+CONSTANTS NOTIFICATION
+
+\* Timeout signal
+CONSTANT NONE
+*)
+\* The set of requests that can go into history
+CONSTANT Value
+
+\* Zab states
+CONSTANTS ELECTION, DISCOVERY, SYNCHRONIZATION, BROADCAST
+
+\* Message types
+CONSTANTS FOLLOWERINFO, LEADERINFO, ACKEPOCH, NEWLEADER, ACKLD, UPTODATE, PROPOSAL, ACK, COMMIT
+(* Additional message types used for recovery in synchronization(TRUNC/DIFF/SNAP) are not needed
+   since we abstract this part.(see action RECOVERYSYNC) *)
+   
+-----------------------------------------------------------------------------
+(* Defined in FastLeaderElection.tla: Quorums, NullPoint *)
+\* Return the maximum value from the set S
+Maximum(S) == IF S = {} THEN -1
+                        ELSE CHOOSE n \in S: \A m \in S: n >= m
+
+\* Return the minimum value from the set S
+Minimum(S) == IF S = {} THEN -1
+                        ELSE CHOOSE n \in S: \A m \in S: n <= m
+
+MAXEPOCH == 10
+
+-----------------------------------------------------------------------------
+(* Defined in FastLeaderElection.tla:
+   serverVars: <<state, currentEpoch, lastZxid>>,
+   electionVars: <<currentVote, logicalClock, receiveVotes, outOfElection, recvQueue, waitNotmsg>>,
+   leaderVars: <<leadingVoteSet>>,
+   electionMsgs,
+   idTable *)
+\* The current phase of server(ELECTION,DISCOVERY,SYNCHRONIZATION,BROADCAST)
+VARIABLE zabState
+
+\* The epoch number of the last NEWEPOCH(LEADERINFO) packet accepted
+\* namely f.p in paper, and currentEpoch in Zab.tla.
+VARIABLE acceptedEpoch
+
+\* The history of servers as the sequence of transactions.
+VARIABLE history
+
+\* commitIndex[i]: The maximum index of transactions that have been saved in a quorum of servers
+\*                 in the perspective of server i.(increases monotonically before restarting)
+VARIABLE commitIndex
+
+(* These transactions whose index \le commitIndex[i] can be applied to state machine immediately.
+   So if we have a variable applyIndex, we can suppose that applyIndex[i] = commitIndex[i] when verifying properties.
+   But in phase SYNC, follower will apply all queued proposals to state machine when receiving NEWLEADER.
+   But follower only serves traffic after receiving UPTODATE, so sequential consistency is not violated.
+   
+   So when we verify properties, we still suppose applyIndex[i] = commitIndex[i], because this is an engineering detail.*)
+
+\* learners[i]: The set of servers which leader i think are connected wich i.
+VARIABLE learners
+
+\* The messages representing requests and responses sent from one server to another.
+\* msgs[i][j] means the input buffer of server j from server i.
+VARIABLE msgs
+
+\* The set of followers who has successfully sent CEPOCH(FOLLOWERINFO) to leader.(equals to connectingFollowers in code)
+VARIABLE cepochRecv
+
+\* The set of followers who has successfully sent ACK-E to leader.(equals to electingFollowers in code)
+VARIABLE ackeRecv
+
+\* The set of followers who has successfully sent ACK-LD to leader in leader.(equals to newLeaderProposal in code)
+VARIABLE ackldRecv
+
+\* The set of servers which leader i broadcasts PROPOSAL and COMMIT to.(equals to forwardingFollowers in code)
+VARIABLE forwarding
+
+\* ackIndex[i][j]: The latest index that leader i has received from follower j via ACK.
+VARIABLE ackIndex
+
+\* currentCounter[i]: The count of transactions that clients request leader i.
+VARIABLE currentCounter
+
+\* sendCounter[i]: The count of transactions that leader i has broadcast in PROPOSAL.
+VARIABLE sendCounter
+
+\* committedIndex[i]: The maximum index of trasactions that leader i has broadcast in COMMIT.
+VARIABLE committedIndex
+
+\* committedCounter[i][j]: The latest counter of transaction that leader i has confirmed that follower j has committed.
+VARIABLE committedCounter
+
+\* initialHistory[i]: The initial history if leader i in epoch acceptedEpoch[i].
+VARIABLE initialHistory
+
+\* the maximum epoch in CEPOCH the prospective leader received from followers.
+VARIABLE tempMaxEpoch
+
+\* cepochSent[i] = TRUE means follower i has sent CEPOCH(FOLLOWERINFO) to leader.
+VARIABLE cepochSent
+
+\* leaderAddr[i]: The leader id of follower i. We use leaderAddr to express whether follower i has connected or lost connection.
+VARIABLE leaderAddr
+
+\* synced[i] = TRUE: follower i has completed sync with leader.
+VARIABLE synced
+
+\* The set of leaders in every epoch, only used in verifying properties.
+VARIABLE epochLeader
+
+\* The set of all broadcast messages, only used in verifying properties.
+VARIABLE proposalMsgsLog
+
+\* A variable used to check whether there are conditions contrary to the facts.
+VARIABLE inherentViolated
+
+serverVarsZ == <<state, currentEpoch, lastZxid, zabState, acceptedEpoch, history, commitIndex>>         \* 7 variables
+
+electionVarsZ == electionVars  \* 6 variables
+
+leaderVarsZ == <<leadingVoteSet, learners, cepochRecv, ackeRecv, ackldRecv, forwarding, 
+                 ackIndex, currentCounter, sendCounter, committedIndex, committedCounter>>              \* 11 variables
+
+tempVarsZ == <<initialHistory, tempMaxEpoch>>     \* 2 variables
+  
+followerVarsZ == <<cepochSent, leaderAddr, synced>>                 \* 3 variables
+
+verifyVarsZ == <<proposalMsgsLog, epochLeader, inherentViolated>>   \* 3 variables
+
+msgVarsZ == <<msgs, electionMsgs>>                \* 2 variables
+
+vars == <<serverVarsZ, electionVarsZ, leaderVarsZ, tempVarsZ, followerVarsZ, verifyVarsZ, msgVarsZ, idTable>>  \* 35 variables
+-----------------------------------------------------------------------------
+\* Add a message to msgs - add a message m to msgs[i][j].
+Send(i, j, m) == msgs' = [msgs EXCEPT ![i][j] = Append(msgs[i][j], m)]
+
+\* Remove a message from msgs - discard head of msgs[i][j].
+Discard(i, j) == msgs' = IF msgs[i][j] /= << >> THEN [msgs EXCEPT ![i][j] = Tail(msgs[i][j])]
+                                                ELSE msgs
+
+\* Leader broadcasts a message(PROPOSAL/COMMIT) to all other servers in forwardingFollowers.
+Broadcast(i, m) == msgs' = [msgs EXCEPT ![i] = [v \in Server |-> IF /\ v \in forwarding[i]
+                                                                    /\ v /= i
+                                                                    /\ \/ /\ m.mtype = PROPOSAL
+                                                                          /\ ackIndex[i][v] < Len(initialHistory[i]) + m.mproposal.counter
+                                                                       \/ /\ m.mtype = COMMIT
+                                                                          /\ committedCounter[i][v] < m.mzxid[2]
+                                                                  THEN Append(msgs[i][v], m)
+                                                                  ELSE msgs[i][v]]]
+
+BroadcastLEADERINFO(i, m) == msgs' = [msgs EXCEPT ![i] = [v \in Server |-> IF /\ v \in cepochRecv[i]
+                                                                              /\ v \in learners[i]
+                                                                              /\ v /= i THEN Append(msgs[i][v], m)
+                                                                                        ELSE msgs[i][v]]]
+
+BroadcastUPTODATE(i, m) == msgs' = [msgs EXCEPT ![i] = [v \in Server |-> IF /\ v \in ackldRecv[i]
+                                                                            /\ v \in learners[i]
+                                                                            /\ v /= i THEN Append(msgs[i][v], m)
+                                                                                      ELSE msgs[i][v]]]
+
+\* Combination of Send and Discard - discard head of msgs[j][i] and add m into msgs[i][j].
+Reply(i, j, m) == msgs' = [msgs EXCEPT ![j][i] = Tail(msgs[j][i]),
+                                       ![i][j] = Append(msgs[i][j], m)]
+
+\* shuffle the input buffer from server j(i) in server i(j).
+Clean(i, j) == msgs' = [msgs EXCEPT ![j][i] = << >>, ![i][j] = << >>]
+
+-----------------------------------------------------------------------------
+PZxidEqual(p, z) == p.epoch = z[1] /\ p.counter = z[2]
+
+TransactionEqual(t1, t2) == /\ t1.epoch   = t2.epoch
+                            /\ t1.counter = t2.counter
+                            
+TransactionPrecede(t1, t2) == \/ t1.epoch < t2.epoch
+                              \/ /\ t1.epoch   = t2.epoch
+                                 /\ t1.counter < t2.counter
+                                 
+-----------------------------------------------------------------------------
+\* Define initial values for all variables 
+InitServerVarsZ == /\ InitServerVars
+                   /\ zabState      = [s \in Server |-> ELECTION]
+                   /\ acceptedEpoch = [s \in Server |-> 0]
+                   /\ history       = [s \in Server |-> << >>]
+                   /\ commitIndex   = [s \in Server |-> 0]
+
+InitLeaderVarsZ == /\ InitLeaderVars
+                   /\ learners         = [s \in Server |-> {}]
+                   /\ cepochRecv       = [s \in Server |-> {}]
+                   /\ ackeRecv         = [s \in Server |-> {}]
+                   /\ ackldRecv        = [s \in Server |-> {}]
+                   /\ ackIndex         = [s \in Server |-> [v \in Server |-> 0]]
+                   /\ currentCounter   = [s \in Server |-> 0]
+                   /\ sendCounter      = [s \in Server |-> 0]
+                   /\ committedIndex   = [s \in Server |-> 0]
+                   /\ committedCounter = [s \in Server |-> [v \in Server |-> 0]]
+                   /\ forwarding       = [s \in Server |-> {}]
+
+InitElectionVarsZ == InitElectionVars
+
+InitTempVarsZ == /\ initialHistory = [s \in Server |-> << >>]
+                 /\ tempMaxEpoch   = [s \in Server |-> 0]
+
+InitFollowerVarsZ == /\ cepochSent = [s \in Server |-> FALSE]
+                     /\ leaderAddr = [s \in Server |-> NullPoint]
+                     /\ synced     = [s \in Server |-> FALSE]
+
+InitVerifyVarsZ == /\ proposalMsgsLog  = {}
+                   /\ epochLeader      = [i \in 1..MAXEPOCH |-> {}]
+                   /\ inherentViolated = FALSE
+                   
+InitMsgVarsZ == /\ msgs         = [s \in Server |-> [v \in Server |-> << >>]]
+                /\ electionMsgs = [s \in Server |-> [v \in Server |-> << >>]]
+
+InitZ == /\ InitServerVarsZ
+         /\ InitLeaderVarsZ
+         /\ InitElectionVarsZ
+         /\ InitTempVarsZ
+         /\ InitFollowerVarsZ
+         /\ InitVerifyVarsZ
+         /\ InitMsgVarsZ
+         /\ idTable = InitializeIdTable(Server)
+         
+-----------------------------------------------------------------------------
+ZabTurnToLeading(i) ==
+        /\ zabState'       = [zabState   EXCEPT ![i] = DISCOVERY]
+        /\ learners'       = [learners   EXCEPT ![i] = {i}]
+        /\ cepochRecv'     = [cepochRecv EXCEPT ![i] = {i}]
+        /\ ackeRecv'       = [ackeRecv   EXCEPT ![i] = {i}]
+        /\ ackldRecv'      = [ackldRecv  EXCEPT ![i] = {i}]
+        /\ forwarding'     = [forwarding EXCEPT ![i] = {}]
+        /\ ackIndex'       = [ackIndex   EXCEPT ![i] = [v \in Server |-> IF v = i THEN Len(history[i])
+                                                                                  ELSE 0]]
+        /\ currentCounter'   = [currentCounter   EXCEPT ![i] = 0]
+        /\ sendCounter'      = [sendCounter      EXCEPT ![i] = 0]
+        /\ commitIndex'      = [commitIndex      EXCEPT ![i] = 0]
+        /\ committedIndex'   = [committedIndex   EXCEPT ![i] = 0]
+        /\ committedCounter' = [committedCounter EXCEPT ![i] = [v \in Server |-> IF v = i THEN Len(history[i])
+                                                                                          ELSE 0]]
+        /\ initialHistory' = [initialHistory EXCEPT ![i] = history[i]]
+        /\ tempMaxEpoch'   = [tempMaxEpoch   EXCEPT ![i] = acceptedEpoch[i]]
+
+ZabTurnToFollowing(i) ==
+        /\ zabState'    = [zabState   EXCEPT ![i] = DISCOVERY]
+        /\ cepochSent'  = [cepochSent EXCEPT ![i] = FALSE]
+        /\ synced'      = [synced     EXCEPT ![i] = FALSE]
+        /\ commitIndex' = [commitIndex      EXCEPT ![i] = 0]
+        
+          
+(* Fast Leader Election *)
+FLEReceiveNotmsg(i, j) ==
+        /\ ReceiveNotmsg(i, j)
+        /\ UNCHANGED <<zabState, acceptedEpoch, history, commitIndex,learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter,
+                       sendCounter, committedIndex, committedCounter, tempVarsZ, followerVarsZ, verifyVarsZ, msgs>>
+
+FLENotmsgTimeout(i) ==
+        /\ NotmsgTimeout(i)
+        /\ UNCHANGED <<zabState, acceptedEpoch, history, commitIndex,learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter,
+                       sendCounter, committedIndex, committedCounter, tempVarsZ, followerVarsZ, verifyVarsZ, msgs>>
+
+FLEHandleNotmsg(i) ==
+        /\ HandleNotmsg(i)
+        /\ LET newState == state'[i]
+           IN
+           \/ /\ newState = LEADING
+              /\ ZabTurnToLeading(i)
+              /\ UNCHANGED <<cepochSent, synced>>
+           \/ /\ newState = FOLLOWING
+              /\ ZabTurnToFollowing(i)
+              /\ UNCHANGED <<learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter, sendCounter, committedIndex, committedCounter, tempVarsZ>>
+           \/ /\ newState = LOOKING
+              /\ UNCHANGED <<zabState, learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter, sendCounter, commitIndex,
+                             committedIndex, committedCounter, tempVarsZ, cepochSent, synced>>
+        /\ UNCHANGED <<acceptedEpoch, history, leaderAddr, verifyVarsZ, msgs>>
+
+\* On the premise that ReceiveVotes.HasQuorums = TRUE, corresponding to logic in line 1050-1055 in LFE.java.
+FLEWaitNewNotmsg(i) ==
+        /\ WaitNewNotmsg(i)
+        /\ UNCHANGED <<zabState, acceptedEpoch, history, commitIndex,learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter,
+                       sendCounter, committedIndex, committedCounter, tempVarsZ, followerVarsZ, verifyVarsZ, msgs>>
+
+\* On the premise that ReceiveVotes.HasQuorums = TRUE, corresponding to logic in line 1061-1066 in LFE.java.
+FLEWaitNewNotmsgEnd(i) ==
+        /\ WaitNewNotmsgEnd(i)
+        /\ LET newState == state'[i]
+           IN
+           \/ /\ newState = LEADING
+              /\ ZabTurnToLeading(i)
+              /\ UNCHANGED <<cepochSent, synced>>
+           \/ /\ newState = FOLLOWING
+              /\ ZabTurnToFollowing(i)
+              /\ UNCHANGED <<learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter, sendCounter, committedIndex, committedCounter, tempVarsZ>>
+           \/ /\ newState = LOOKING
+              /\ PrintT("New state is LOOKING in FLEWaitNewNotmsgEnd, which should not happen.")
+              /\ UNCHANGED <<zabState, learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter, sendCounter, commitIndex,
+                             committedIndex, committedCounter, tempVarsZ, cepochSent, synced>>
+        /\ UNCHANGED <<acceptedEpoch, history, leaderAddr, verifyVarsZ, msgs>>
+              
+-----------------------------------------------------------------------------
+(* A sub-action describing how a server transitions from LEADING/FOLLOWING to LOOKING.
+   Initially I call it 'ZabTimeoutZ', but it will be called not only when timeout, 
+   but also when finding a low epoch from leader.*)
+FollowerShutdown(i) ==
+        /\ ZabTimeout(i)
+        /\ zabState'   = [zabState   EXCEPT ![i] = ELECTION]
+        /\ leaderAddr' = [leaderAddr EXCEPT ![i] = NullPoint]
+
+LeaderShutdown(i) ==
+        /\ ZabTimeout(i)
+        /\ zabState'   = [zabState   EXCEPT ![i] = ELECTION]
+        /\ leaderAddr' = [s \in Server |-> IF s \in learners[i] THEN NullPoint ELSE leaderAddr[s]]
+        /\ learners'   = [learners   EXCEPT ![i] = {}]
+        /\ forwarding' = [forwarding EXCEPT ![i] = {}]
+        /\ msgs'       = [s \in Server |-> [v \in Server |-> IF v \in learners[i] \/ s \in learners[i] THEN << >> ELSE msgs[s][v]]]
+
+FollowerTimout(i) ==
+        /\ state[i]      = FOLLOWING
+        /\ leaderAddr[i] = NullPoint
+        /\ FollowerShutdown(i)
+        /\ msgs' = [s \in Server |-> [v \in Server |-> IF v = i THEN << >> ELSE msgs[s][v]]]
+        /\ UNCHANGED <<acceptedEpoch, history, commitIndex, learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, 
+                       currentCounter, sendCounter, committedIndex, committedCounter, tempVarsZ, cepochSent, synced, verifyVarsZ>>
+
+LeaderTimeout(i) ==
+        /\ state[i] = LEADING
+        /\ learners[i] \notin Quorums
+        /\ LeaderShutdown(i)
+        /\ UNCHANGED <<acceptedEpoch, history, commitIndex, cepochRecv, ackeRecv, ackldRecv, ackIndex, currentCounter, sendCounter, committedIndex, committedCounter,
+                       tempVarsZ, cepochSent, synced, verifyVarsZ>>
+    
+-----------------------------------------------------------------------------
+(* Establish connection between leader i and follower j. 
+   It means i creates a learnerHandler for communicating with j, and j finds i's address.*)
+EstablishConnection(i, j) ==
+        /\ state[i] = LEADING   /\ state[j] = FOLLOWING
+        /\ j \notin learners[i] /\ leaderAddr[j] = NullPoint
+        /\ currentVote[j].proposedLeader = i
+        /\ learners'   = [learners   EXCEPT ![i] = learners[i] \union {j}] \* Leader:   'addLearnerHandler(peer)'
+        /\ leaderAddr' = [leaderAddr EXCEPT ![j] = i]                      \* Follower: 'connectToLeader(addr, hostname)'
+        /\ UNCHANGED <<serverVarsZ, electionVarsZ, leadingVoteSet, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, 
+                       currentCounter, sendCounter, committedIndex, committedCounter, tempVarsZ, cepochSent, synced, verifyVarsZ, msgVarsZ, idTable>>
+        
+(* The leader i finds timeout and TCP connection between i and j closes.*)       
+Timeout(i, j) ==
+        /\ state[i] = LEADING /\ state[j] = FOLLOWING
+        /\ j \in learners[i]  /\ leaderAddr[j] = i
+        (* The action of leader i.(corresponding to function 'removeLearnerHandler(peer)'.) *)
+        /\ learners'   = [learners   EXCEPT ![i] = learners[i] \ {j}] 
+        /\ forwarding' = [forwarding EXCEPT ![i] = IF j \in forwarding[i] THEN forwarding[i] \ {j} ELSE forwarding[i]]
+        /\ cepochRecv' = [cepochRecv EXCEPT ![i] = IF j \in cepochRecv[i] THEN cepochRecv[i] \ {j} ELSE cepochRecv[i]]
+        /\ ackeRecv'   = [ackeRecv   EXCEPT ![i] = IF j \in ackeRecv[i] THEN ackeRecv[i] \ {j} ELSE ackeRecv[i]]
+        /\ ackldRecv'  = [ackldRecv  EXCEPT ![i] = IF j \in ackldRecv[i] THEN ackldRecv[i] \ {j} ELSE ackldRecv[i]]
+        /\ ackIndex'   = [ackIndex   EXCEPT ![i][j] = 0]
+        /\ committedCounter' = [committedCounter EXCEPT ![i][j] = 0]
+        (* The action of follower j. *)
+        /\ FollowerShutdown(j)
+        (* Clean input buffer.*)
+        /\ Clean(i, j)
+        /\ UNCHANGED <<acceptedEpoch, history, commitIndex, currentCounter, sendCounter, committedIndex,
+                       tempVarsZ, cepochSent, synced, verifyVarsZ>>
+-----------------------------------------------------------------------------
+\* In phase f11, follower sends f.p to leader via FOLLOWERINFO(CEPOCH).
+FollowerSendFOLLOWERINFO(i) ==
+        /\ state[i]    = FOLLOWING
+        /\ zabState[i] = DISCOVERY
+        /\ leaderAddr[i] /= NullPoint
+        /\ \lnot cepochSent[i]
+        /\ Send(i, leaderAddr[i], [mtype  |-> FOLLOWERINFO,
+                                   mepoch |-> acceptedEpoch[i]])
+        /\ cepochSent' = [cepochSent EXCEPT ![i] = TRUE]
+        /\ UNCHANGED <<serverVarsZ, leaderVarsZ, electionVarsZ, tempVarsZ, leaderAddr, synced, verifyVarsZ, electionMsgs, idTable>>
+        
+(* In phase l11, leader waits for receiving FOLLOWERINFO from a quorum,
+   and then chooses a new epoch e' as its own epoch and broadcasts LEADERINFO. *)
+LeaderHandleFOLLOWERINFO(i, j) ==
+        /\ state[i] = LEADING
+        /\ msgs[j][i] /= << >>
+        /\ msgs[j][i][1].mtype = FOLLOWERINFO
+        /\ LET msg == msgs[j][i][1]
+           IN \/ /\ NullPoint \notin cepochRecv[i]   \* 1. has not broadcast LEADERINFO - modify tempMaxEpoch
+                 /\ LET newEpoch == Maximum({tempMaxEpoch[i], msg.mepoch})
+                    IN tempMaxEpoch' = [tempMaxEpoch EXCEPT ![i] = newEpoch]
+                 /\ Discard(j, i)
+              \/ /\ NullPoint \in cepochRecv[i]      \* 2. has broadcast LEADERINFO - no need to handle the msg, just send LEADERINFO to corresponding server
+                 /\ Reply(i, j, [mtype  |-> LEADERINFO,
+                                 mepoch |-> acceptedEpoch[i]])
+                 /\ UNCHANGED tempMaxEpoch
+        /\ cepochRecv' = [cepochRecv EXCEPT ![i] = IF j \in cepochRecv[i] THEN cepochRecv[i]
+                                                                          ELSE cepochRecv[i] \union {j}]
+        /\ UNCHANGED <<serverVarsZ, followerVarsZ, electionVarsZ, initialHistory, leadingVoteSet, learners, ackeRecv, ackldRecv, 
+                       forwarding, ackIndex, currentCounter, sendCounter, committedIndex, committedCounter, verifyVarsZ, electionMsgs, idTable>>
+
+LeaderDiscovery1(i) ==
+        /\ state[i]    = LEADING
+        /\ zabState[i] = DISCOVERY
+        /\ cepochRecv[i] \in Quorums
+        /\ acceptedEpoch' = [acceptedEpoch EXCEPT ![i] = tempMaxEpoch[i] + 1]
+        /\ cepochRecv'    = [cepochRecv    EXCEPT ![i] = cepochRecv[i] \union {NullPoint}]
+        /\ BroadcastLEADERINFO(i, [mtype  |-> LEADERINFO,
+                                   mepoch |-> acceptedEpoch'[i]])
+        /\ UNCHANGED <<state, currentEpoch, lastZxid, zabState, history, commitIndex, electionVarsZ, leadingVoteSet, learners, ackeRecv, ackldRecv, 
+                       forwarding, ackIndex, currentCounter, sendCounter, committedIndex, committedCounter, 
+                       tempVarsZ, followerVarsZ, verifyVarsZ, electionMsgs, idTable>>
+
+(* In phase f12, follower receives NEWEPOCH. If e' > f.p, then follower sends ACK-E back,
+   and ACK-E contains f.a and lastZxid to let leader judge whether it is the latest.
+   After handling NEWEPOCH, follower's zabState turns to SYNCHRONIZATION. *)
+FollowerHandleLEADERINFO(i, j) ==
+        /\ state[i] = FOLLOWING
+        /\ msgs[j][i] /= << >>
+        /\ msgs[j][i][1].mtype = LEADERINFO
+        /\ LET msg     == msgs[j][i][1]
+               infoOk  == j = leaderAddr[i]
+               epochOk == /\ infoOk
+                          /\ msg.mepoch >= acceptedEpoch[i]
+               correct == /\ epochOk
+                          /\ zabState[i] = DISCOVERY
+           IN /\ infoOk
+              /\ \/ /\ epochOk                      \* 1. Normal case
+                    /\ \/ /\ correct
+                          /\ acceptedEpoch' = [acceptedEpoch EXCEPT ![i] = msg.mepoch]
+                          /\ Reply(i, j, [mtype      |-> ACKEPOCH,
+                                          mepoch     |-> msg.mepoch,
+                                          mlastEpoch |-> currentEpoch[i],
+                                          mlastZxid  |-> lastZxid[i]])
+                          /\ cepochSent' = [cepochSent EXCEPT ![i] = TRUE]
+                          /\ UNCHANGED inherentViolated
+                       \/ /\ ~correct
+                          /\ PrintT("Exception: Condition correct is false in FollowerHandleLEADERINFO(" \o ToString(i) \o ", " \o ToString(j) \o ").")
+                          /\ inherentViolated' = TRUE
+                          /\ Discard(j, i)
+                          /\ UNCHANGED <<acceptedEpoch, cepochSent>>
+                    /\ zabState' = [zabState EXCEPT ![i] = IF zabState[i] = DISCOVERY THEN SYNCHRONIZATION
+                                                                                      ELSE zabState[i]]
+                    /\ UNCHANGED <<varsL, leaderAddr>>
+                 \/ /\ ~epochOk                    \* 2. Abnormal case - go back to election
+                    /\ FollowerShutdown(i)
+                    /\ Clean(i, j)
+                    /\ UNCHANGED <<acceptedEpoch, cepochSent, inherentViolated>>
+        /\ UNCHANGED <<history, commitIndex, learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter, sendCounter, committedIndex,
+                       committedCounter, tempVarsZ, synced, proposalMsgsLog, epochLeader>>
+
+\* Abstraction of actions making follower synced with leader before leader sending NEWLEADER.
+subRECOVERYSYNC(i, j) ==
+        LET canSync == /\ state[i] = LEADING   /\ zabState[i] /= DISCOVERY      /\ j \in learners[i]  /\ j \in ackeRecv[i]
+                       /\ state[j] = FOLLOWING /\ zabState[j] = SYNCHRONIZATION /\ leaderAddr[j] = i  /\ synced[j] = FALSE
+        IN
+        \/ /\ canSync
+           /\ history'     = [history     EXCEPT ![j] = history[i]]
+           /\ lastZxid'    = [lastZxid    EXCEPT ![j] = lastZxid[i]]
+           /\ UpdateProposal(j, leaderAddr[j], lastZxid'[j], currentEpoch[j])
+           /\ commitIndex' = [commitIndex EXCEPT ![j] = commitIndex[i]]
+           /\ synced'      = [synced      EXCEPT ![j] = TRUE]
+           /\ forwarding'  = [forwarding  EXCEPT ![i] = forwarding[i] \union {j}]    \* j will receive PROPOSAL and COMMIT
+           /\ ackIndex'    = [ackIndex    EXCEPT ![i][j] = Len(history[i])]
+           /\ committedCounter' = [committedCounter EXCEPT ![i][j] = Maximum({commitIndex[i] - Len(initialHistory[i]), 0})]
+           /\ LET ms == [msource|->i, mtype|->"RECOVERYSYNC", mepoch|->acceptedEpoch[i], mproposals|->history[i]]
+              IN proposalMsgsLog' = IF ms \in proposalMsgsLog THEN proposalMsgsLog
+                                                              ELSE proposalMsgsLog \union {ms}
+           /\ Reply(i, j, [mtype     |-> NEWLEADER,
+                           mepoch    |-> acceptedEpoch[i],
+                           mlastZxid |-> lastZxid[i]])
+        \/ /\ ~canSync
+           /\ Discard(j, i)
+           /\ UNCHANGED <<history, lastZxid, currentVote, commitIndex, synced, forwarding, ackIndex, committedCounter, proposalMsgsLog>>
+        
+(* In phase l12, leader waits for receiving ACKEPOPCH from a quorum,
+   and check whether it has the latest history and epoch from them.
+   If so, leader's zabState turns to SYNCHRONIZATION. *)
+LeaderHandleACKEPOCH(i, j) ==
+        /\ state[i] = LEADING
+        /\ msgs[j][i] /= << >>
+        /\ msgs[j][i][1].mtype = ACKEPOCH
+        /\ LET msg == msgs[j][i][1]
+               infoOk == /\ j \in learners[i]            
+                         /\ acceptedEpoch[i] = msg.mepoch
+               logOk  == /\ infoOk                          \* logOk = TRUE means leader is more up-to-date than follower j
+                         /\ \/ currentEpoch[i] > msg.mlastEpoch
+                            \/ /\ currentEpoch[i] = msg.mlastEpoch
+                               /\ \/ lastZxid[i][1] > msg.mlastZxid[1]
+                                  \/ /\ lastZxid[i][1] = msg.mlastZxid[1]
+                                     /\ lastZxid[i][2] >= msg.mlastZxid[2]
+               replyOk == /\ infoOk
+                          /\ NullPoint \in ackeRecv[i]
+           IN /\ infoOk
+              /\ \/ /\ replyOk
+                    /\ subRECOVERYSYNC(i, j)
+                    /\ ackeRecv' = [ackeRecv EXCEPT ![i] = IF j \notin ackeRecv[i] THEN ackeRecv[i] \union {j}
+                                                                                   ELSE ackeRecv[i]]
+                    /\ UNCHANGED <<state, currentEpoch,logicalClock, receiveVotes, outOfElection, recvQueue, waitNotmsg, leadingVoteSet, electionMsgs, idTable,
+                                   zabState, leaderAddr, learners>>
+                 \/ /\ ~replyOk
+                    /\ \/ /\ logOk
+                          /\ ackeRecv' = [ackeRecv EXCEPT ![i] = IF j \notin ackeRecv[i] THEN ackeRecv[i] \union {j}
+                                                                                         ELSE ackeRecv[i]]
+                          /\ Discard(j, i)
+                          /\ UNCHANGED <<varsL, zabState, leaderAddr, learners, forwarding>>
+                       \/ /\ ~logOk            \* go back to election
+                          /\ LeaderShutdown(i)
+                          /\ UNCHANGED ackeRecv
+                    /\ UNCHANGED <<history, commitIndex, synced, forwarding, ackIndex, committedCounter, proposalMsgsLog>>
+        /\ UNCHANGED <<acceptedEpoch, cepochRecv, ackldRecv,currentCounter, sendCounter, committedIndex, tempVarsZ, cepochSent, epochLeader, inherentViolated>>
+                          
+LeaderDiscovery2(i) ==
+        /\ state[i] = LEADING
+        /\ zabState[i] = DISCOVERY
+        /\ ackeRecv[i] \in Quorums
+        /\ zabState'       = [zabState       EXCEPT ![i] = SYNCHRONIZATION]
+        /\ currentEpoch'   = [currentEpoch   EXCEPT ![i] = acceptedEpoch[i]]
+        /\ initialHistory' = [initialHistory EXCEPT ![i] = history[i]]
+        /\ ackeRecv'       = [ackeRecv       EXCEPT ![i] = ackeRecv[i] \union {NullPoint}]
+        /\ ackIndex'       = [ackIndex       EXCEPT ![i][i] = Len(history[i])]
+        /\ UpdateProposal(i, i, lastZxid[i], currentEpoch'[i])
+        /\ LET epoch == acceptedEpoch[i]
+           IN epochLeader' = [epochLeader EXCEPT ![epoch] = epochLeader[epoch] \union {i}]
+        /\ UNCHANGED <<state, lastZxid, acceptedEpoch, history, commitIndex, logicalClock, receiveVotes, outOfElection, recvQueue, waitNotmsg,
+                       leadingVoteSet, learners, cepochRecv, ackldRecv, forwarding, currentCounter, sendCounter, committedIndex, committedCounter,
+                       tempMaxEpoch, followerVarsZ, proposalMsgsLog, inherentViolated, msgVarsZ, idTable>>
+
+(* Note:  Set cepochRecv, ackeRecv, ackldRecv to {NullPoint} in corresponding three actions to
+          make sure that the prospective leader will not broadcast NEWEPOCH/NEWLEADER/COMMITLD twice.*)
+-----------------------------------------------------------------------------
+RECOVERYSYNC(i, j) ==
+        /\ state[i] = LEADING   /\ zabState[i] /= DISCOVERY      /\ j \in learners[i] /\ j \in ackeRecv[i]
+        /\ state[j] = FOLLOWING /\ zabState[j] = SYNCHRONIZATION /\ leaderAddr[j] = i /\ synced[j] = FALSE
+        \* /\ acceptedEpoch[i] = acceptedEpoch[j] \* This condition is unnecessary.
+        /\ history'     = [history     EXCEPT ![j] = history[i]]
+        /\ lastZxid'    = [lastZxid    EXCEPT ![j] = lastZxid[i]]
+        /\ UpdateProposal(j, leaderAddr[j], lastZxid'[j], currentEpoch[j])
+        /\ commitIndex' = [commitIndex EXCEPT ![j] = commitIndex[i]]
+        /\ synced'      = [synced      EXCEPT ![j] = TRUE]
+        /\ forwarding'  = [forwarding  EXCEPT ![i] = forwarding[i] \union {j}]
+        /\ ackIndex'    = [ackIndex    EXCEPT ![i][j] = Len(history[i])]
+        /\ committedCounter' = [committedCounter EXCEPT ![i][j] = Maximum({commitIndex[i] - Len(initialHistory[i]), 0})]
+        /\ LET ms == [msource|->i, mtype|->"RECOVERYSYNC", mepoch|->acceptedEpoch[i], mproposals|->history[i]]
+           IN proposalMsgsLog' = IF ms \in proposalMsgsLog THEN proposalMsgsLog
+                                                           ELSE proposalMsgsLog \union {ms}
+        /\ Send(i, j, [mtype     |-> NEWLEADER,
+                       mepoch    |-> acceptedEpoch[i],
+                       mlastZxid |-> lastZxid[i]])
+        /\ UNCHANGED <<state, zabState, acceptedEpoch, currentEpoch, logicalClock, receiveVotes, outOfElection, recvQueue, waitNotmsg, 
+                       leadingVoteSet, learners, cepochRecv, ackeRecv, ackldRecv, currentCounter, sendCounter, committedIndex, 
+                       tempVarsZ, cepochSent, leaderAddr, epochLeader, inherentViolated, electionMsgs, idTable>>
+
+(* In phase f21, follower receives NEWLEADER. The follower updates its epoch and history,
+   and sends back ACK-LD to leader. *)
+FollowerHandleNEWLEADER(i, j) ==
+        /\ state[i] = FOLLOWING
+        /\ msgs[j][i] /= << >>
+        /\ msgs[j][i][1].mtype = NEWLEADER
+        /\ LET msg     == msgs[j][i][1]
+               infoOk  == /\ leaderAddr[i] = j
+                          /\ acceptedEpoch[i] = msg.mepoch
+               correct == /\ infoOk
+                          /\ zabState[i] = SYNCHRONIZATION
+                          /\ synced[i]
+                          /\ ZxidEqual(lastZxid[i], msg.mlastZxid)
+           IN /\ infoOk
+              /\ currentEpoch' = [currentEpoch EXCEPT ![i] = msg.mepoch]
+              /\ UpdateProposal(i, j, lastZxid[i], currentEpoch'[i])
+              /\ \/ /\ correct
+                    /\ Reply(i, j, [mtype  |-> ACKLD,
+                                    mepoch |-> msg.mepoch])
+                    /\ UNCHANGED inherentViolated
+                 \/ /\ ~correct
+                    /\ PrintT("Exception: Condition correct is false in FollowerHandleNEWLEADER(" \o ToString(i) \o ", " \o ToString(j) \o ").")
+                    /\ inherentViolated' = TRUE
+                    /\ Discard(j, i)
+        /\ UNCHANGED <<state, lastZxid, zabState, acceptedEpoch, history, commitIndex, logicalClock, receiveVotes, outOfElection, recvQueue, waitNotmsg,
+                       leaderVarsZ, tempVarsZ, followerVarsZ, proposalMsgsLog, epochLeader, electionMsgs, idTable>>
+
+\* In phase l22, leader receives ACK-LD from a quorum of followers, and sends COMMIT-LD(UPTODATE) to followers.
+LeaderHandleACKLD(i, j) ==
+        /\ state[i] = LEADING
+        /\ msgs[j][i] /= << >>
+        /\ msgs[j][i][1].mtype = ACKLD
+        /\ LET msg     == msgs[j][i][1]
+               infoOk  == /\ acceptedEpoch[i] = msg.mepoch
+                          /\ j \in learners[i]
+               replyOk == /\ infoOk
+                          /\ NullPoint \in ackldRecv[i]
+           IN /\ infoOk
+              /\ \/ /\ replyOk
+                    /\ Reply(i, j, [mtype   |-> UPTODATE,
+                                    mepoch  |-> acceptedEpoch[i],
+                                    mcommit |-> commitIndex[i]])
+                    /\ committedCounter' = [committedCounter EXCEPT ![i][j] = Maximum({commitIndex[i] - Len(initialHistory[i]), committedCounter[i][j]})]
+                 \/ /\ ~replyOk
+                    /\ Discard(j, i)
+                    /\ UNCHANGED committedCounter
+              /\ ackldRecv' = [ackldRecv EXCEPT ![i] = IF j \notin ackldRecv[i] THEN ackldRecv[i] \union {j}
+                                                                                ELSE ackldRecv[i]]
+        /\ UNCHANGED <<serverVarsZ, electionVarsZ, leadingVoteSet, learners, cepochRecv, ackeRecv, forwarding, 
+                       ackIndex, currentCounter, sendCounter, committedIndex, tempVarsZ, followerVarsZ, verifyVarsZ, electionMsgs, idTable>>
+
+LeaderSync2(i) ==
+        /\ state[i] = LEADING
+        /\ zabState[i] = SYNCHRONIZATION
+        /\ ackldRecv[i] \in Quorums
+        /\ commitIndex'    = [commitIndex    EXCEPT ![i] = Len(history[i])]
+        /\ committedIndex' = [committedIndex EXCEPT ![i] = Len(history[i])]
+        /\ zabState'       = [zabState       EXCEPT ![i] = BROADCAST]
+        /\ currentCounter' = [currentCounter EXCEPT ![i] = 0]
+        /\ sendCounter'    = [sendCounter    EXCEPT ![i] = 0]
+        /\ ackldRecv'      = [ackldRecv      EXCEPT ![i] = ackldRecv[i] \union {NullPoint}]
+        /\ BroadcastUPTODATE(i, [mtype   |-> UPTODATE,
+                                 mepoch  |-> acceptedEpoch[i],
+                                 mcommit |-> Len(history[i])]) \* In actual UPTODATE doesn't carry this info
+        /\ UNCHANGED <<state, currentEpoch, lastZxid, acceptedEpoch, history, electionVarsZ, leadingVoteSet, learners, cepochRecv, ackeRecv, forwarding, ackIndex, 
+                       committedCounter, tempVarsZ, followerVarsZ, verifyVarsZ, electionMsgs, idTable>>
+
+FollowerHandleUPTODATE(i, j) ==
+        /\ state[i] = FOLLOWING
+        /\ msgs[j][i] /= << >>
+        /\ msgs[j][i][1].mtype = UPTODATE
+        /\ LET msg == msgs[j][i][1]
+               infoOk == /\ leaderAddr[i] = j
+                         /\ acceptedEpoch[i] = msg.mepoch
+               correct == /\ infoOk
+                          /\ zabState[i] = SYNCHRONIZATION
+                          /\ currentEpoch[i] = msg.mepoch
+           IN /\ infoOk
+              /\ \/ /\ correct
+                    /\ commitIndex' = [commitIndex EXCEPT ![i] = Maximum({commitIndex[i], msg.mcommit})] \* May have received COMMIT and have a bigger commit index.
+                    /\ zabState'    = [zabState    EXCEPT ![i] = BROADCAST]
+                    /\ UNCHANGED inherentViolated
+                 \/ /\ ~correct
+                    /\ PrintT("Exception: Condition correct is false in FollowerHandleUPTODATE(" \o ToString(i) \o ", " \o ToString(j) \o ").")
+                    /\ inherentViolated' = TRUE
+                    /\ UNCHANGED <<commitIndex, zabState>>
+        /\ Discard(j, i)
+        /\ UNCHANGED <<state, currentEpoch, lastZxid, acceptedEpoch, history, electionVarsZ, leaderVarsZ, tempVarsZ, followerVarsZ,
+                       proposalMsgsLog, epochLeader, electionMsgs, idTable>>
+
+-----------------------------------------------------------------------------
+(* In phase l31, leader receives client request and broadcasts PROPOSAL.
+   Note: In production, any server in traffic can receive requests and forward it to leader if necessary.
+   We choose to let leader be the sole one who can receive requests, to simplify spec and keep correctness at the same time. *)
+ClientRequest(i, v) ==
+        /\ state[i] = LEADING
+        /\ zabState[i] = BROADCAST
+        /\ currentCounter' = [currentCounter EXCEPT ![i] = currentCounter[i] + 1]
+        /\ LET newTransaction == [epoch   |-> acceptedEpoch[i],
+                                  counter |-> currentCounter'[i],
+                                  value   |-> v]
+           IN /\ history'  = [history  EXCEPT ![i] = Append(history[i], newTransaction)]
+              /\ lastZxid' = [lastZxid EXCEPT ![i] = <<acceptedEpoch[i], currentCounter'[i]>>]
+              /\ ackIndex' = [ackIndex EXCEPT ![i][i] = Len(history'[i])]
+              /\ UpdateProposal(i, i, lastZxid'[i], currentEpoch[i])
+        /\ UNCHANGED <<state, currentEpoch, zabState, acceptedEpoch, commitIndex, logicalClock, receiveVotes, outOfElection, recvQueue, waitNotmsg,
+                       leadingVoteSet, learners, cepochRecv, ackeRecv, ackldRecv, forwarding, sendCounter, committedIndex, committedCounter,
+                       tempVarsZ, followerVarsZ, verifyVarsZ, msgVarsZ, idTable>>
+
+LeaderBroadcast1(i) ==
+        /\ state[i] = LEADING
+        /\ zabState[i] = BROADCAST
+        /\ sendCounter[i] < currentCounter[i]
+        /\ LET toBeSentCounter == sendCounter[i] + 1
+               toBeSentIndex   == Len(initialHistory[i]) + toBeSentCounter
+               toBeSentEntry   == history[i][toBeSentIndex]
+           IN /\ Broadcast(i, [mtype     |-> PROPOSAL,
+                               mepoch    |-> acceptedEpoch[i],
+                               mproposal |-> toBeSentEntry])
+              /\ sendCounter' = [sendCounter EXCEPT ![i] = toBeSentCounter]
+              /\ LET m == [msource|-> i, mepoch|-> acceptedEpoch[i], mtype|-> PROPOSAL, mproposal|-> toBeSentEntry]
+                 IN proposalMsgsLog' = proposalMsgsLog \union {m}
+        /\ UNCHANGED <<serverVarsZ, electionVarsZ, leadingVoteSet, learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter, 
+                       committedIndex, committedCounter, tempVarsZ, followerVarsZ, epochLeader, inherentViolated, electionMsgs, idTable>>
+
+\* In phase f31, follower accepts proposal and append it to history.
+FollowerHandlePROPOSAL(i, j) ==
+        /\ state[i] = FOLLOWING
+        /\ msgs[j][i] /= << >>
+        /\ msgs[j][i][1].mtype = PROPOSAL
+        /\ LET msg    == msgs[j][i][1]
+               infoOk == /\ leaderAddr[i] = j
+                         /\ acceptedEpoch[i] = msg.mepoch
+               correct == /\ infoOk
+                          /\ zabState[i] /= DISCOVERY
+                          /\ synced[i]
+               logOk  == \/ /\ msg.mproposal.counter = 1     \* the first PROPOSAL in this epoch
+                            /\ \/ Len(history[i]) = 0
+                               \/ /\ Len(history[i]) > 0
+                                  /\ history[i][Len(history[i])].epoch < msg.mepoch
+                         \/ /\ msg.mproposal.counter > 1      \* not the first PROPOSAL in this epoch
+                            /\ Len(history[i]) > 0
+                            /\ history[i][Len(history[i])].epoch = msg.mepoch
+                            /\ history[i][Len(history[i])].counter = msg.mproposal.counter - 1
+           IN /\ infoOk
+              /\ \/ /\ correct
+                    /\ \/ /\ logOk
+                          /\ history'  = [history  EXCEPT ![i] = Append(history[i], msg.mproposal)]
+                          /\ lastZxid' = [lastZxid EXCEPT ![i] = <<msg.mepoch, msg.mproposal.counter>>]
+                          /\ UpdateProposal(i, j, lastZxid'[i], currentEpoch[i])
+                          /\ Reply(i, j, [mtype  |-> ACK,
+                                          mepoch |-> acceptedEpoch[i],
+                                          mzxid  |-> <<msg.mepoch, msg.mproposal.counter>>])
+                          /\ UNCHANGED inherentViolated
+                       \/ /\ ~logOk
+                          /\ PrintT("Exception: Condition logOk is false in FollowerHandlePROPOSAL(" \o ToString(i) \o ", " \o ToString(j) \o ").")
+                          /\ inherentViolated' = TRUE
+                          /\ Discard(j, i)
+                          /\ UNCHANGED <<history, lastZxid, currentVote>>
+                 \/ /\ ~correct
+                    /\ PrintT("Exception: Condition correct is false in FollowerHandlePROPOSAL(" \o ToString(i) \o ", " \o ToString(j) \o ").")
+                    /\ inherentViolated' = TRUE
+                    /\ Discard(j, i)
+                    /\ UNCHANGED <<history, lastZxid, currentVote>>
+        /\ UNCHANGED <<state, currentEpoch, zabState, acceptedEpoch, commitIndex, logicalClock, receiveVotes, outOfElection, recvQueue, waitNotmsg,
+                       leaderVarsZ, tempVarsZ, followerVarsZ, proposalMsgsLog, epochLeader, electionMsgs, idTable>>
+
+\* In phase l32, leader receives ack from a quorum of followers to a certain proposal, and commits the proposal.
+LeaderHandleACK(i, j) ==
+        /\ state[i] = LEADING
+        /\ msgs[j][i] /= << >>
+        /\ msgs[j][i][1].mtype = ACK
+        /\ LET msg == msgs[j][i][1]
+               infoOk == /\ j \in forwarding[i]
+                         /\ acceptedEpoch[i] = msg.mepoch
+               correct == /\ infoOk
+                          /\ zabState[i] = BROADCAST
+                          /\ sendCounter[i] >= msg.mzxid[2]
+               logOk == /\ infoOk
+                        /\ ackIndex[i][j] + 1 = Len(initialHistory[i]) + msg.mzxid[2]
+           IN /\ infoOk
+              /\ \/ /\ correct
+                    /\ \/ /\ logOk
+                          /\ ackIndex' = [ackIndex EXCEPT ![i][j] = ackIndex[i][j] + 1]
+                       \/ /\ ~logOk
+                          /\ PrintT("Note: redundant ACK.")
+                          /\ UNCHANGED ackIndex
+                    /\ UNCHANGED inherentViolated
+                 \/ /\ ~correct
+                    /\ PrintT("Exception: Condition correct is false in FollowerHandleACK(" \o ToString(i) \o ", " \o ToString(j) \o ").")
+                    /\ inherentViolated' = TRUE
+                    /\ UNCHANGED ackIndex
+        /\ Discard(j, i)
+        /\ UNCHANGED <<serverVarsZ, electionVarsZ, leadingVoteSet, learners, cepochRecv, ackeRecv, ackldRecv, forwarding, currentCounter, sendCounter, 
+                       committedIndex, committedCounter, tempVarsZ, followerVarsZ, proposalMsgsLog, epochLeader, electionMsgs, idTable>>
+
+LeaderAdvanceCommit(i) ==
+        /\ state[i] = LEADING
+        /\ zabState[i] = BROADCAST
+        /\ commitIndex[i] < Len(history[i])
+        /\ LET Agree(index)   == {i} \union {k \in (Server\{i}): ackIndex[i][k] >= index}
+               agreeIndexes   == {index \in (commitIndex[i] + 1)..Len(history[i]): Agree(index) \in Quorums}
+               newCommitIndex == IF agreeIndexes /= {} THEN Maximum(agreeIndexes)
+                                                       ELSE commitIndex[i]
+           IN commitIndex' = [commitIndex EXCEPT ![i] = newCommitIndex]
+        /\ UNCHANGED <<state, currentEpoch, lastZxid, zabState, acceptedEpoch, history, electionVarsZ, leaderVarsZ, tempVarsZ, followerVarsZ,
+                       verifyVarsZ, msgVarsZ, idTable>>
+
+LeaderBroadcast2(i) ==

Review comment:
       Action name is not descriptive, perhaps rename to something like to LeaderBroadcastCommit




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@zookeeper.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] BinyuHuang-nju edited a comment on pull request #1690: ZOOKEEPER-3615: Provide formal specification and verification using TLA+ for Zab

Posted by GitBox <gi...@apache.org>.
BinyuHuang-nju edited a comment on pull request #1690:
URL: https://github.com/apache/zookeeper/pull/1690#issuecomment-844163499


   @hanm Maybe it would be inappropriate for me to ask the following questions under this PR. Now I am learning ZAB 1.0 and prepare to write spec. Could you please let me ask about details in it?
   
   Currently I have several problems about ZAB 1.0.
   1. For some detailed industial implementation (like message with type PING, connection between leader and followers), do I need to implement these in spec?
   2. I see before sending NEWLEADER, leader will choose a best message from SNAP,TRUNC,DIFF to send according to the corresponding follower's state. Do I need to implement these, or abstract this part and just send NEWLEADER to sync with followers?
   3. I don't understand whether variable 'loracle' is more like a global variable that has the latest leader ID and every server can reach it, or 'loracle' is a local variable like 'votedFor' in Raft. (Actually I used the latter in Zab.tla, which corresponds to 'leaderOracle'. And I think it has no effect on correctness no matter which one is used.)
   4. The last problem what I want to ask is that about recovery, because I didn't see notes about this part in paper and the link. 
   I saw how the leader handles when a new server wants to join them, but I didn't see how a restarted server finds the latest leader. If 'loracle' is the former in question 3, I can understand this part. In Zab.tla, I use the method where the follower broadcasts messages to ask other servers what their local oracle is, and update its oracle when receiving the same oracle and epoch from a quorum(I used this method because I saw servers recover like this in View-stamped Replication).
   A follower must have received corresponding PROPOSAL when receiving COMMIT, if the follower is the initial server that joins Q. But I think this may not always be true when the follower is one that joins Q midway. So I want to know how follower handles to catch up state when receiving COMMIT corresponding to a transaction that not exists in its local history.(You could see this condition in the image at the bottom of README. When I wrote spec for Zab pre-1.0, I choose to let followers keep re-sending CEPOCH to obtain latest transactions until the conflict described above disappers.)
   
   I am very sorry if my problems bother you. Thank you for the various feedbacks and suggestions you have provided before!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] Vanlightly commented on pull request #1690: ZOOKEEPER-3615: Provide formal specification and verification using TLA+ for Zab

Posted by GitBox <gi...@apache.org>.
Vanlightly commented on pull request #1690:
URL: https://github.com/apache/zookeeper/pull/1690#issuecomment-844375695


   From my perspective:
   1. Implement the minimum required, in order to keep thing simpler and reduce the state space. If the only point of the PING message is to act as a keep alive when there is no data to replicate then don't include it. Do not implement timeouts explicitly, rather just allow for a node to go back to an election at any time.
   2. SNAP and DIFF exist as they optimize performance for the given scenario. For the TLA+ we don't worry about that. I would implement DIFF or SNAP but not both. SNAP would result in a smaller state space as it is only a single message. TRUNC as far as I know is when a follower is ahead of the new leader (with uncommitted entries) so would be required.
   
   Regarding 3 and 4, unfortunately the Zab 1.0 document is lacking details so someone with deep knowledge of the protocol will need explain this missing pieces. If I have time I might read the code to reverse engineer it back into detailed a protocol description.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] BinyuHuang-nju edited a comment on pull request #1690: ZOOKEEPER-3615: Provide formal specification and verification using TLA+ for Zab

Posted by GitBox <gi...@apache.org>.
BinyuHuang-nju edited a comment on pull request #1690:
URL: https://github.com/apache/zookeeper/pull/1690#issuecomment-850846721


   @Vanlightly @hanm Hi! I have read some code about zab. But I still cannot understand some details. I want to ask you about them.
   
   1. At line 630 in function 'syncWithLeader' In [Learner.java](https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/quorum/Learner.java), I see follower in phase SYNCHRONIZATION may receive PROPOSAL and COMMIT. But in paper, leader will add a new server into its 'forwardingFollowers'(Q in paper) only when receiving ack of NEWLEADER from it, so in paper this new server should not receive PROPOSAL or COMMIT because in phase SYNCHRONIZATION it has not sent ACK-NEWLEADER.
   So here, my question is in ZAB 1.0, when does leader add a newly joined server into its 'forwardingFollowers'?(I am sorry that I didn't find this in code)
   
   2. My understanding of question above is: in phase 2, leader syncs with each follower and ''sends'' NEWLEADER only to this corresponding follower that has completed sync, not ''broadcasts'' NEWLEADER. So when leader is in phase 3, there may exist some follower in phase 2, and has not completed sync with leader since it hasn't received NEWLEADER, but it is allowed to receive PROPOSAL and COMMIT.
   I am not sure whether my understanding is right. If so, I think my spec about phase 2 is that, for each connected follower: first, leader syncs with  the follower by an abstract action  RECOVERY_SYNC; second, in the action, follower will have the latest history of leader, and leader sends NEWLEADER to it; then, leader and follower communicate using ACK-NEWLEADER and UPTODATE and follower turns to phase 3. Leader turns to phase 3 upon receiving ACK-NEWLEADER from a quorum. 
   
   3. At line 766 in function 'syncWithLeader' in [Learner.java](https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/quorum/Learner.java), I see when receiving NEWLEADER, the variable 'packetsNotCommitted' is set as empty, and when receiving UPTODATE, there is no action handling transactions left to commit them.
   Does that mean follower thinks all transactions committed upon receiving NEWLEADER not UPTODATE? And it seems like that NEWLEADER, ACK-LEADER, UPTODATE carries little data except epoch, just used for acknowledgement between leader and followers.
   
   4. In phase 1 in manual tha we mentioned above, there's one word: "Note, if a follower is connecting, but the leader is already established (in phase 3) the follower follows the phases, but the leader ignores any ACKs." 
   It seems like that leader in phase 3(BROADCAST) will ignre ACKEPOCH in phase1 and ACK of NEWLEADER in phase 2, then how dose this newly connecting follower receive UPTODATE to go to phase BROADCAST?
   
   Since these details are related to how to let follower sync with leader in my spec of ZAB 1.0, I want to have a further understanding of them. Thank you!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] BinyuHuang-nju edited a comment on pull request #1690: ZOOKEEPER-3615: Provide formal specification and verification using TLA+ for Zab

Posted by GitBox <gi...@apache.org>.
BinyuHuang-nju edited a comment on pull request #1690:
URL: https://github.com/apache/zookeeper/pull/1690#issuecomment-850846721


   @Vanlightly @hanm Hi! I have read some code about zab. But I still cannot understand some details. I want to ask you about them.
   
   1. At line 630 in function 'syncWithLeader' In [Learner.java](https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/quorum/Learner.java), I see follower in phase SYNCHRONIZATION may receive PROPOSAL and COMMIT. But in paper, leader will add a new server into its 'forwardingFollowers'(Q in paper) only when receiving ack of NEWLEADER from it, so in paper this new server should not receive PROPOSAL or COMMIT because in phase SYNCHRONIZATION it has not sent ACK-NEWLEADER.
   So here, my question is in ZAB 1.0, when does leader add a newly joined server into its 'forwardingFollowers'?(I am sorry that I didn't find this in code)
   
   2. At line 766 in function 'syncWithLeader' in [Learner.java](https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/quorum/Learner.java), I see when receiving NEWLEADER, the variable 'packetsNotCommitted' is set as empty, and when receiving UPTODATE, there is no action handling transactions left to commit them.
   Does that mean follower thinks all transactions committed upon receiving NEWLEADER not UPTODATE?(Actually in papaer and my spec of zab pre-1.0, follower thinks them committed upon receiving UPTODATE)
   
   3. In phase 1 in manual that we we mentioned above, there's one word: "Note, if a follower is connecting, but the leader is already established (in phase 3) the follower follows the phases, but the leader ignores any ACKs." 
   It seems like that leader in phase 3(BROADCAST) will ignre ACKEPOCH in phase1 and ACK of NEWLEADER in phase 2, then how dose this newly connecting follower receive UPTODATE to go to phase BROADCAST?
   
   Since these details are related to how to let follower sync with leader in my spec of ZAB 1.0, I want to have a further understanding of them. Thank you!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] BinyuHuang-nju edited a comment on pull request #1690: ZOOKEEPER-3615: Provide formal specification and verification using TLA+ for Zab

Posted by GitBox <gi...@apache.org>.
BinyuHuang-nju edited a comment on pull request #1690:
URL: https://github.com/apache/zookeeper/pull/1690#issuecomment-832510959


   @Vanlightly Sorry, I didn't make it clear that I didn't do complete running. Actually, It is almost impossible to traverse all states due to state space explosion, and this is a disadvantage of "model checking". So it is very normal when you see the model cannot stop.
   
   We can assume situations when a server halts and restarts for multiple times, or re-elect a leader  for multiple times but not increase epoch because the last round of elections ended before the prospective leader broadcasts NEWEPOCH. So in principle, there are infinite large state spaces when running models.
   On the other hand, there are a lot of repetitive actions like restart, re-election happen continuously, which make state space expand rapidly.
   
   And the experience imparted to me by the senior’s work like Parallel Raft, TPaxos shows that it is quite normal that models cannot be run completely. So in actual, I run it for a long time to judge whether there's invariant violation or not. 
   
   Also, I have run models with excluding the restart situation(modify "currentEpoch[i] < 0" in Restart and RecoveryAfterRestart), and the result will produce a lot of actions we care more about. Until now, the results appears good. I think you could have a try if you have enough time.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] BinyuHuang-nju commented on pull request #1690: ZOOKEEPER-3615: Provide formal specification and verification using TLA+ for Zab

Posted by GitBox <gi...@apache.org>.
BinyuHuang-nju commented on pull request #1690:
URL: https://github.com/apache/zookeeper/pull/1690#issuecomment-974819434


   @Vanlightly That's right! In my opinion, invariant 'Leadership' makes sure there is at most one leader every epoch. So everytime when a proposal created, its zxid is unique and will never be modified. That's why I just compare zxid of two transaction in **TransactionEqual** and we can prove that is enough for verifying differences between transactions.
   I will update action when leader processes REQUEST, to let leader propose a unique value everytime, to avoid this misunderstanding. Thank you!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@zookeeper.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] BinyuHuang-nju commented on pull request #1690: Provide formal specification and verification using TLA+ for Zab

Posted by GitBox <gi...@apache.org>.
BinyuHuang-nju commented on pull request #1690:
URL: https://github.com/apache/zookeeper/pull/1690#issuecomment-831234732


   @Vanlightly Sorry. Ok, I have checked invariants including Leadership, PrefixConsistency, Integrity and Agreement, and these invariants appear good for several days. In principle, all invariants should keep unviolated when we can guarantee that Leadership and PrefixConsistency are true anytime. So I just focus on Leadership and PrefixConsistency.
   The invariants that hit violations are both related to variable proposalMsgsLog. So I think I made mistakes in writing spec related to proposalMsgsLog or in writing PrefixConsistency. 
   Please let me check this spec and reply to you later, and you can check whether invariants Leadership and PrefixConsistency will hit violations. Thank you for your feedback very much!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] BinyuHuang-nju commented on pull request #1690: ZOOKEEPER-3615: Provide formal specification and verification using TLA+ for Zab

Posted by GitBox <gi...@apache.org>.
BinyuHuang-nju commented on pull request #1690:
URL: https://github.com/apache/zookeeper/pull/1690#issuecomment-847826409


   @Vanlightly @hanm Thank you for your replies! Your interpretation of the problems will simplify my spec of zab 1.0.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] BinyuHuang-nju commented on pull request #1690: ZOOKEEPER-3615: Provide formal specification and verification using TLA+ for Zab

Posted by GitBox <gi...@apache.org>.
BinyuHuang-nju commented on pull request #1690:
URL: https://github.com/apache/zookeeper/pull/1690#issuecomment-855615675


   @hanm There's one point I don't get.
   > When follower receives NEWLEADER, it will apply all previous queued proposals from leader atomically to its state machine. 
   It seems like that when some learnerMaster sends NEWLEADER to the corresponding follower, the leader may have not completed sync with a quorum of followers. So some transaction saved by this follower may have not been saved by a quorum, which is not committed and can not be applied from my perspective. Here I don't understand why these queued proposals can be applied. Thank you!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] Vanlightly commented on pull request #1690: Provide formal specification and verification using TLA+ for Zab

Posted by GitBox <gi...@apache.org>.
Vanlightly commented on pull request #1690:
URL: https://github.com/apache/zookeeper/pull/1690#issuecomment-831134962


   Yes, I will review it and give feedback.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] BinyuHuang-nju commented on a change in pull request #1690: ZOOKEEPER-3615: Provide formal specification and verification using TLA+ for Zab

Posted by GitBox <gi...@apache.org>.
BinyuHuang-nju commented on a change in pull request #1690:
URL: https://github.com/apache/zookeeper/pull/1690#discussion_r630750303



##########
File path: zookeeper-contrib/zookeeper-contrib-zabtla/README.md
##########
@@ -0,0 +1,71 @@
+# Zab-tla
+
+## Overview
+This project is devoted to providing formal specification and verification using TLA+ for the Zookeeper Atomic Broadcast(Zab) consensus protocol proposed in *Junqueira F P, Reed B C, Serafini M. Zab: High-performance broadcast for primary-backup systems[C]//2011 IEEE/IFIP 41st International Conference on Dependable Systems & Networks (DSN). IEEE, 2011: 245-256*.  
+
+We have made a formal [specification](Zab.tla) for Zab using TLA+ toolbox, and we have done a certain scale of model checking to verify the correctness of Zab.
+
+Due to the simplification of Zab algorithm description in the paper, some details in specification were modified and added. If you have any question, please let us know.
+
+You can find this document in chinese in [doc-in-chinsese](doc-in-chinese/README.md).
+
+## Requirements
+TLA+ toolbox version 1.7.0
+
+## Run
+Create specification and run models in the usual way.  

Review comment:
       Thank you! I can write a more detailed manual  in README about how to configure TLC parameters. Since the .toolbox space is too large, I can upload some more important files in .toolbox folder(such as .settings, .projejct, Model/MC.tla, Model/MC.cfg).




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] hanm commented on a change in pull request #1690: ZOOKEEPER-3615: Provide formal specification and verification using TLA+ for Zab

Posted by GitBox <gi...@apache.org>.
hanm commented on a change in pull request #1690:
URL: https://github.com/apache/zookeeper/pull/1690#discussion_r631471863



##########
File path: zookeeper-contrib/zookeeper-contrib-zabtla/experiment/ZabWithQTest.tla
##########
@@ -0,0 +1,994 @@
+(*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *)
+-------------------------------- MODULE ZabWithQTest --------------------------------
+\* This is the test for formal specification for the Zab consensus algorithm,

Review comment:
       Thinking more about the folder structure of the  specification files, I am tempted to make a proposal that instead of putting them under zookeeper-contrib, we create a folder under root level called zookeeper-specifications, and put ZAB specifications there. I feel these specifications have more affinities to the zookeeper core project comparing to zookeeper-contrib. 
   
   The new root folder structures would look like:
   ├── bin
   ├── conf
   ├── dev
   ├── target
   ├── tools
   ├── zookeeper-assembly
   ├── zookeeper-client
   ├── zookeeper-compatibility-tests
   ├── zookeeper-contrib
   ├── zookeeper-docs
   ├── zookeeper-it
   ├── zookeeper-jute
   ├── zookeeper-metrics-providers
   ├── zookeeper-recipes
   ├── zookeeper-server
   └── zookeeper-specifications
   
   And the spec folder looks like
   └── zookeeper-specifications
       ├── leader-election
       ├── reconfig
       └── zab
   
   Here we put the zab spec, toolbox configs, tests, etc in zab folder.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] eolivelli commented on pull request #1690: Provide formal specification and verification using TLA+ for Zab

Posted by GitBox <gi...@apache.org>.
eolivelli commented on pull request #1690:
URL: https://github.com/apache/zookeeper/pull/1690#issuecomment-831054913


   @Vanlightly @fpj probably you are interested in this PR


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] Vanlightly commented on pull request #1690: ZOOKEEPER-3615: Provide formal specification and verification using TLA+ for Zab

Posted by GitBox <gi...@apache.org>.
Vanlightly commented on pull request #1690:
URL: https://github.com/apache/zookeeper/pull/1690#issuecomment-832489284


   @BinyuHuang-nju with 3 servers and 1 value it now hits all actions. It's been running for 20 hours and reached a diameter of 22 with over 17 trillion states,  2 trillion unique states and the queue size is over 1 trillion and growing. When I ran it with 2 servers it had reached a diameter of over 60 when I killed it. I see there are some constraints regarding epochs in the enabling conditions of the actions to keep it to only two epochs but even so we've got a very large state space. 
   
   In your results you describe much smaller state counts. Have you run this to completion or only as far as the state counts you have described? If you have run it to completion, what was your model?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] BinyuHuang-nju commented on pull request #1690: ZOOKEEPER-3615: Provide formal specification and verification using TLA+ for Zab

Posted by GitBox <gi...@apache.org>.
BinyuHuang-nju commented on pull request #1690:
URL: https://github.com/apache/zookeeper/pull/1690#issuecomment-916966706


   @Vanlightly It is very nice for you to have made so many suggestions. I will check each detail you mentioned one by one and modify them, and push new version here as soon as possible! Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@zookeeper.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] BinyuHuang-nju commented on a change in pull request #1690: ZOOKEEPER-3615: Provide formal specification and verification using TLA+ for Zab

Posted by GitBox <gi...@apache.org>.
BinyuHuang-nju commented on a change in pull request #1690:
URL: https://github.com/apache/zookeeper/pull/1690#discussion_r634132822



##########
File path: zookeeper-contrib/zookeeper-contrib-zabtla/README.md
##########
@@ -0,0 +1,71 @@
+# Zab-tla
+
+## Overview
+This project is devoted to providing formal specification and verification using TLA+ for the Zookeeper Atomic Broadcast(Zab) consensus protocol proposed in *Junqueira F P, Reed B C, Serafini M. Zab: High-performance broadcast for primary-backup systems[C]//2011 IEEE/IFIP 41st International Conference on Dependable Systems & Networks (DSN). IEEE, 2011: 245-256*.  
+
+We have made a formal [specification](Zab.tla) for Zab using TLA+ toolbox, and we have done a certain scale of model checking to verify the correctness of Zab.
+
+Due to the simplification of Zab algorithm description in the paper, some details in specification were modified and added. If you have any question, please let us know.
+
+You can find this document in chinese in [doc-in-chinsese](doc-in-chinese/README.md).
+
+## Requirements
+TLA+ toolbox version 1.7.0
+
+## Run
+Create specification and run models in the usual way.  

Review comment:
       Hi there! I have modified the project according to your requirements. Due to the insertion of Apache License in spec, I forgot to modify the line number in README (offset 17 lines backward). I will modify this minor issue in future commits. If you have any questions, please let me know. 
   Next, I will take time to learn about ZAB 1.0. And if necessary, I will optimize spec from the perspective of explosive state space of model checking, and this may be a relatively long work.
   By the way, since zookeeper-specifications is a new folder, Travis CI rejects this commit at the beginning of building. Could you please tell me how to pass it?(Actually I haven't touched Travis CI before)  Thank you!




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] BinyuHuang-nju commented on pull request #1690: ZOOKEEPER-3615: Provide formal specification and verification using TLA+ for Zab

Posted by GitBox <gi...@apache.org>.
BinyuHuang-nju commented on pull request #1690:
URL: https://github.com/apache/zookeeper/pull/1690#issuecomment-832752514


   @Vanlightly Hi! Since I previously thought that infinite state space is normal, I didn't pay much attention to how to constrain state space. Thanks for your reminder, I think I can modify spec ZabWithQTest.tla, to try to let models be run completely.
   
   I added global variables electionNum, totalRestartNum to restrain state space and try to run models to completion. I let
   electionNum < MaxElectionNum in Election, let totalRestartNum < MaxTotalRestartNum in Restart and RecoveryAfterRestart, and let Len(history[i]) < MaxTransactionNum in ClientRequest.
   When I set 2 servers, MaxElectionNum = 2, MaxTotalRestartNum = 3, MaxTransactionNum = 2, the model was run completely in 1 minute, and reached a diameter of 47 with 2518479 states. In this case, it is normal that actions involving election and restart and some low-probability actions such as BecomeFollower will not be traversed.
   
   So this time We may be able to run the model completely when the scale is not very large. I will try this way with bigger values like 3 servers, and tell you results tomorrow if it can be run completely. And I will push new spec here immediately.
   
   But, in principle state space is infinite in this spec using model checking. This way in spec makes my verification more like a testing, not model checking. I think it is a trade-off between verification of Zab correctness and model checking.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] hanm commented on pull request #1690: ZOOKEEPER-3615: Provide formal specification and verification using TLA+ for Zab

Posted by GitBox <gi...@apache.org>.
hanm commented on pull request #1690:
URL: https://github.com/apache/zookeeper/pull/1690#issuecomment-847186137


   > @hanm Maybe it would be inappropriate for me to ask the following questions under this PR. Now I am learning ZAB 1.0 and prepare to write spec. Could you please let me ask about details in it?
   > 
   > Currently I have several problems about ZAB 1.0.
   > 
   > 1. For some detailed industial implementation (like message with type PING, connection between leader and followers), do I need to implement these in spec?
   
   It depends on if the particular message type is critical to reveal important details of the protocol. Messages like NEWLEADER or UPTODATE is an integral part of the protocol, I don't think we can abstract it away. For Ping though, I think we can omit it for now.
   
   > 2. I see before sending NEWLEADER, leader will choose a best message from SNAP,TRUNC,DIFF to send according to the corresponding follower's state. Do I need to implement these, or abstract this part and just send NEWLEADER to sync with followers?
   
   I am inclined to abstract this part away using "RECOVERY_SYNC" operation. In fact, that's what the original pre-1.0 did where a leader always sync learners with full history. We can go more fine grained later. This will hopefully reduce the state space.
   
   > 3. I don't understand whether variable 'loracle' is more like a global variable that has the latest leader ID and every server can reach it, or 'loracle' is a local variable like 'votedFor' in Raft. (Actually I used the latter in Zab.tla, which corresponds to 'leaderOracle'. And I think it has no effect on correctness no matter which one is used.)
   
   ZAB 1.0 and its implementation does not use this "loracle". Instead, a server always starts a leader election and get the latest later ID through the leader election results. If we want an abstraction here, I would tend to think "loracle" is a global variable.
   
   > 4. The last problem what I want to ask is that about recovery, because I didn't see notes about this part in paper and the link.
   >    I saw how the leader handles when a new server wants to join them, but I didn't see how a restarted server finds the latest leader. If 'loracle' is the former in question 3, I can understand this part. In Zab.tla, I use the method where the follower broadcasts messages to ask other servers what their local oracle is, and update its oracle when receiving the same oracle and epoch from a quorum(I used this method because I saw servers recover like this in View-stamped Replication).
   
   As mentioned earlier, a restarted (or newly joined) server finds latest leader through leader election.
   
   >    A follower must have received corresponding PROPOSAL when receiving COMMIT, if the follower is the initial server that joins Q. But I think this may not always be true when the follower is one that joins Q midway. So I want to know how follower handles to catch up state when receiving COMMIT corresponding to a transaction that not exists in its local history.(You could see this condition in the image at the bottom of README. When I wrote spec for Zab pre-1.0, I choose to let followers keep re-sending CEPOCH to obtain latest transactions until the conflict described above disappers.)
   
   The "follow handles to catch up" part of the protocol is the recovery part (SNAP, DIFF, TRANC). Follower will not start broadcast unless the recovery phase is finished. The invariant here is that once recovery finished, follower should have the latest history of quorum / leader so any COMMIT it receives has a corresponding entry in its history. This is different comparing to Paxos or Raft where there was no dedicated recovery phase.
   
   > 
   > I am very sorry if my problems bother you. Thank you for the various feedbacks and suggestions you have provided before!
   
   No worries, I am not super active in community these days, but I  will answer questions to the issues that I am involved in (with undefined SLA).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] Vanlightly commented on a change in pull request #1690: ZOOKEEPER-3615: Provide formal specification and verification using TLA+ for Zab

Posted by GitBox <gi...@apache.org>.
Vanlightly commented on a change in pull request #1690:
URL: https://github.com/apache/zookeeper/pull/1690#discussion_r705462438



##########
File path: zookeeper-specifications/zab-1.0/ZabWithFLE.tla
##########
@@ -0,0 +1,1020 @@
+(*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *)
+
+----------------------------- MODULE ZabWithFLE -----------------------------
+(* This is the formal specification for the Zab consensus algorithm,
+   which means Zookeeper Atomic Broadcast.*)
+
+(* Reference:
+   FLE: FastLeaderElection.java, Vote.java, QuorumPeer.java in https://github.com/apache/zookeeper.
+   ZAB: QuorumPeer.java, Learner.java, Follower.java, LearnerHandler.java, Leader.java 
+        in https://github.com/apache/zookeeper.
+        https://cwiki.apache.org/confluence/display/ZOOKEEPER/Zab1.0.
+ *)
+EXTENDS FastLeaderElection
+
+-----------------------------------------------------------------------------
+(* Defined in FastLeaderElection.tla:
+\* The set of server identifiers
+CONSTANT Server
+
+\* Server states
+CONSTANTS LOOKING, FOLLOWING, LEADING
+
+\* Message types
+CONSTANTS NOTIFICATION
+
+\* Timeout signal
+CONSTANT NONE
+*)
+\* The set of requests that can go into history
+CONSTANT Value
+
+\* Zab states
+CONSTANTS ELECTION, DISCOVERY, SYNCHRONIZATION, BROADCAST
+
+\* Message types
+CONSTANTS FOLLOWERINFO, LEADERINFO, ACKEPOCH, NEWLEADER, ACKLD, UPTODATE, PROPOSAL, ACK, COMMIT
+(* Additional message types used for recovery in synchronization(TRUNC/DIFF/SNAP) are not needed
+   since we abstract this part.(see action RECOVERYSYNC) *)
+   
+-----------------------------------------------------------------------------
+(* Defined in FastLeaderElection.tla: Quorums, NullPoint *)
+\* Return the maximum value from the set S
+Maximum(S) == IF S = {} THEN -1
+                        ELSE CHOOSE n \in S: \A m \in S: n >= m
+
+\* Return the minimum value from the set S
+Minimum(S) == IF S = {} THEN -1
+                        ELSE CHOOSE n \in S: \A m \in S: n <= m
+
+MAXEPOCH == 10
+
+-----------------------------------------------------------------------------
+(* Defined in FastLeaderElection.tla:
+   serverVars: <<state, currentEpoch, lastZxid>>,
+   electionVars: <<currentVote, logicalClock, receiveVotes, outOfElection, recvQueue, waitNotmsg>>,
+   leaderVars: <<leadingVoteSet>>,
+   electionMsgs,
+   idTable *)
+\* The current phase of server(ELECTION,DISCOVERY,SYNCHRONIZATION,BROADCAST)
+VARIABLE zabState
+
+\* The epoch number of the last NEWEPOCH(LEADERINFO) packet accepted
+\* namely f.p in paper, and currentEpoch in Zab.tla.
+VARIABLE acceptedEpoch
+
+\* The history of servers as the sequence of transactions.
+VARIABLE history
+
+\* commitIndex[i]: The maximum index of transactions that have been saved in a quorum of servers
+\*                 in the perspective of server i.(increases monotonically before restarting)
+VARIABLE commitIndex
+
+(* These transactions whose index \le commitIndex[i] can be applied to state machine immediately.
+   So if we have a variable applyIndex, we can suppose that applyIndex[i] = commitIndex[i] when verifying properties.
+   But in phase SYNC, follower will apply all queued proposals to state machine when receiving NEWLEADER.
+   But follower only serves traffic after receiving UPTODATE, so sequential consistency is not violated.
+   
+   So when we verify properties, we still suppose applyIndex[i] = commitIndex[i], because this is an engineering detail.*)
+
+\* learners[i]: The set of servers which leader i think are connected wich i.
+VARIABLE learners
+
+\* The messages representing requests and responses sent from one server to another.
+\* msgs[i][j] means the input buffer of server j from server i.
+VARIABLE msgs
+
+\* The set of followers who has successfully sent CEPOCH(FOLLOWERINFO) to leader.(equals to connectingFollowers in code)
+VARIABLE cepochRecv
+
+\* The set of followers who has successfully sent ACK-E to leader.(equals to electingFollowers in code)
+VARIABLE ackeRecv
+
+\* The set of followers who has successfully sent ACK-LD to leader in leader.(equals to newLeaderProposal in code)
+VARIABLE ackldRecv
+
+\* The set of servers which leader i broadcasts PROPOSAL and COMMIT to.(equals to forwardingFollowers in code)
+VARIABLE forwarding
+
+\* ackIndex[i][j]: The latest index that leader i has received from follower j via ACK.
+VARIABLE ackIndex
+
+\* currentCounter[i]: The count of transactions that clients request leader i.
+VARIABLE currentCounter
+
+\* sendCounter[i]: The count of transactions that leader i has broadcast in PROPOSAL.
+VARIABLE sendCounter
+
+\* committedIndex[i]: The maximum index of trasactions that leader i has broadcast in COMMIT.
+VARIABLE committedIndex
+
+\* committedCounter[i][j]: The latest counter of transaction that leader i has confirmed that follower j has committed.
+VARIABLE committedCounter
+
+\* initialHistory[i]: The initial history if leader i in epoch acceptedEpoch[i].
+VARIABLE initialHistory
+
+\* the maximum epoch in CEPOCH the prospective leader received from followers.
+VARIABLE tempMaxEpoch
+
+\* cepochSent[i] = TRUE means follower i has sent CEPOCH(FOLLOWERINFO) to leader.
+VARIABLE cepochSent
+
+\* leaderAddr[i]: The leader id of follower i. We use leaderAddr to express whether follower i has connected or lost connection.
+VARIABLE leaderAddr
+
+\* synced[i] = TRUE: follower i has completed sync with leader.
+VARIABLE synced
+
+\* The set of leaders in every epoch, only used in verifying properties.
+VARIABLE epochLeader
+
+\* The set of all broadcast messages, only used in verifying properties.
+VARIABLE proposalMsgsLog
+
+\* A variable used to check whether there are conditions contrary to the facts.
+VARIABLE inherentViolated
+
+serverVarsZ == <<state, currentEpoch, lastZxid, zabState, acceptedEpoch, history, commitIndex>>         \* 7 variables
+
+electionVarsZ == electionVars  \* 6 variables
+
+leaderVarsZ == <<leadingVoteSet, learners, cepochRecv, ackeRecv, ackldRecv, forwarding, 
+                 ackIndex, currentCounter, sendCounter, committedIndex, committedCounter>>              \* 11 variables
+
+tempVarsZ == <<initialHistory, tempMaxEpoch>>     \* 2 variables
+  
+followerVarsZ == <<cepochSent, leaderAddr, synced>>                 \* 3 variables
+
+verifyVarsZ == <<proposalMsgsLog, epochLeader, inherentViolated>>   \* 3 variables
+
+msgVarsZ == <<msgs, electionMsgs>>                \* 2 variables
+
+vars == <<serverVarsZ, electionVarsZ, leaderVarsZ, tempVarsZ, followerVarsZ, verifyVarsZ, msgVarsZ, idTable>>  \* 35 variables
+-----------------------------------------------------------------------------
+\* Add a message to msgs - add a message m to msgs[i][j].
+Send(i, j, m) == msgs' = [msgs EXCEPT ![i][j] = Append(msgs[i][j], m)]
+
+\* Remove a message from msgs - discard head of msgs[i][j].
+Discard(i, j) == msgs' = IF msgs[i][j] /= << >> THEN [msgs EXCEPT ![i][j] = Tail(msgs[i][j])]
+                                                ELSE msgs
+
+\* Leader broadcasts a message(PROPOSAL/COMMIT) to all other servers in forwardingFollowers.
+Broadcast(i, m) == msgs' = [msgs EXCEPT ![i] = [v \in Server |-> IF /\ v \in forwarding[i]
+                                                                    /\ v /= i
+                                                                    /\ \/ /\ m.mtype = PROPOSAL
+                                                                          /\ ackIndex[i][v] < Len(initialHistory[i]) + m.mproposal.counter
+                                                                       \/ /\ m.mtype = COMMIT
+                                                                          /\ committedCounter[i][v] < m.mzxid[2]
+                                                                  THEN Append(msgs[i][v], m)
+                                                                  ELSE msgs[i][v]]]
+
+BroadcastLEADERINFO(i, m) == msgs' = [msgs EXCEPT ![i] = [v \in Server |-> IF /\ v \in cepochRecv[i]
+                                                                              /\ v \in learners[i]
+                                                                              /\ v /= i THEN Append(msgs[i][v], m)
+                                                                                        ELSE msgs[i][v]]]
+
+BroadcastUPTODATE(i, m) == msgs' = [msgs EXCEPT ![i] = [v \in Server |-> IF /\ v \in ackldRecv[i]
+                                                                            /\ v \in learners[i]
+                                                                            /\ v /= i THEN Append(msgs[i][v], m)
+                                                                                      ELSE msgs[i][v]]]
+
+\* Combination of Send and Discard - discard head of msgs[j][i] and add m into msgs[i][j].
+Reply(i, j, m) == msgs' = [msgs EXCEPT ![j][i] = Tail(msgs[j][i]),
+                                       ![i][j] = Append(msgs[i][j], m)]
+
+\* shuffle the input buffer from server j(i) in server i(j).
+Clean(i, j) == msgs' = [msgs EXCEPT ![j][i] = << >>, ![i][j] = << >>]
+
+-----------------------------------------------------------------------------
+PZxidEqual(p, z) == p.epoch = z[1] /\ p.counter = z[2]
+
+TransactionEqual(t1, t2) == /\ t1.epoch   = t2.epoch
+                            /\ t1.counter = t2.counter
+                            
+TransactionPrecede(t1, t2) == \/ t1.epoch < t2.epoch
+                              \/ /\ t1.epoch   = t2.epoch
+                                 /\ t1.counter < t2.counter
+                                 
+-----------------------------------------------------------------------------
+\* Define initial values for all variables 
+InitServerVarsZ == /\ InitServerVars
+                   /\ zabState      = [s \in Server |-> ELECTION]
+                   /\ acceptedEpoch = [s \in Server |-> 0]
+                   /\ history       = [s \in Server |-> << >>]
+                   /\ commitIndex   = [s \in Server |-> 0]
+
+InitLeaderVarsZ == /\ InitLeaderVars
+                   /\ learners         = [s \in Server |-> {}]
+                   /\ cepochRecv       = [s \in Server |-> {}]
+                   /\ ackeRecv         = [s \in Server |-> {}]
+                   /\ ackldRecv        = [s \in Server |-> {}]
+                   /\ ackIndex         = [s \in Server |-> [v \in Server |-> 0]]
+                   /\ currentCounter   = [s \in Server |-> 0]
+                   /\ sendCounter      = [s \in Server |-> 0]
+                   /\ committedIndex   = [s \in Server |-> 0]
+                   /\ committedCounter = [s \in Server |-> [v \in Server |-> 0]]
+                   /\ forwarding       = [s \in Server |-> {}]
+
+InitElectionVarsZ == InitElectionVars
+
+InitTempVarsZ == /\ initialHistory = [s \in Server |-> << >>]
+                 /\ tempMaxEpoch   = [s \in Server |-> 0]
+
+InitFollowerVarsZ == /\ cepochSent = [s \in Server |-> FALSE]
+                     /\ leaderAddr = [s \in Server |-> NullPoint]
+                     /\ synced     = [s \in Server |-> FALSE]
+
+InitVerifyVarsZ == /\ proposalMsgsLog  = {}
+                   /\ epochLeader      = [i \in 1..MAXEPOCH |-> {}]
+                   /\ inherentViolated = FALSE
+                   
+InitMsgVarsZ == /\ msgs         = [s \in Server |-> [v \in Server |-> << >>]]
+                /\ electionMsgs = [s \in Server |-> [v \in Server |-> << >>]]
+
+InitZ == /\ InitServerVarsZ
+         /\ InitLeaderVarsZ
+         /\ InitElectionVarsZ
+         /\ InitTempVarsZ
+         /\ InitFollowerVarsZ
+         /\ InitVerifyVarsZ
+         /\ InitMsgVarsZ
+         /\ idTable = InitializeIdTable(Server)
+         
+-----------------------------------------------------------------------------
+ZabTurnToLeading(i) ==
+        /\ zabState'       = [zabState   EXCEPT ![i] = DISCOVERY]
+        /\ learners'       = [learners   EXCEPT ![i] = {i}]
+        /\ cepochRecv'     = [cepochRecv EXCEPT ![i] = {i}]
+        /\ ackeRecv'       = [ackeRecv   EXCEPT ![i] = {i}]
+        /\ ackldRecv'      = [ackldRecv  EXCEPT ![i] = {i}]
+        /\ forwarding'     = [forwarding EXCEPT ![i] = {}]
+        /\ ackIndex'       = [ackIndex   EXCEPT ![i] = [v \in Server |-> IF v = i THEN Len(history[i])
+                                                                                  ELSE 0]]
+        /\ currentCounter'   = [currentCounter   EXCEPT ![i] = 0]
+        /\ sendCounter'      = [sendCounter      EXCEPT ![i] = 0]
+        /\ commitIndex'      = [commitIndex      EXCEPT ![i] = 0]
+        /\ committedIndex'   = [committedIndex   EXCEPT ![i] = 0]
+        /\ committedCounter' = [committedCounter EXCEPT ![i] = [v \in Server |-> IF v = i THEN Len(history[i])
+                                                                                          ELSE 0]]
+        /\ initialHistory' = [initialHistory EXCEPT ![i] = history[i]]
+        /\ tempMaxEpoch'   = [tempMaxEpoch   EXCEPT ![i] = acceptedEpoch[i]]
+
+ZabTurnToFollowing(i) ==
+        /\ zabState'    = [zabState   EXCEPT ![i] = DISCOVERY]
+        /\ cepochSent'  = [cepochSent EXCEPT ![i] = FALSE]
+        /\ synced'      = [synced     EXCEPT ![i] = FALSE]
+        /\ commitIndex' = [commitIndex      EXCEPT ![i] = 0]
+        
+          
+(* Fast Leader Election *)
+FLEReceiveNotmsg(i, j) ==
+        /\ ReceiveNotmsg(i, j)
+        /\ UNCHANGED <<zabState, acceptedEpoch, history, commitIndex,learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter,
+                       sendCounter, committedIndex, committedCounter, tempVarsZ, followerVarsZ, verifyVarsZ, msgs>>
+
+FLENotmsgTimeout(i) ==
+        /\ NotmsgTimeout(i)
+        /\ UNCHANGED <<zabState, acceptedEpoch, history, commitIndex,learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter,
+                       sendCounter, committedIndex, committedCounter, tempVarsZ, followerVarsZ, verifyVarsZ, msgs>>
+
+FLEHandleNotmsg(i) ==
+        /\ HandleNotmsg(i)
+        /\ LET newState == state'[i]
+           IN
+           \/ /\ newState = LEADING
+              /\ ZabTurnToLeading(i)
+              /\ UNCHANGED <<cepochSent, synced>>
+           \/ /\ newState = FOLLOWING
+              /\ ZabTurnToFollowing(i)
+              /\ UNCHANGED <<learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter, sendCounter, committedIndex, committedCounter, tempVarsZ>>
+           \/ /\ newState = LOOKING
+              /\ UNCHANGED <<zabState, learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter, sendCounter, commitIndex,
+                             committedIndex, committedCounter, tempVarsZ, cepochSent, synced>>
+        /\ UNCHANGED <<acceptedEpoch, history, leaderAddr, verifyVarsZ, msgs>>
+
+\* On the premise that ReceiveVotes.HasQuorums = TRUE, corresponding to logic in line 1050-1055 in LFE.java.
+FLEWaitNewNotmsg(i) ==
+        /\ WaitNewNotmsg(i)
+        /\ UNCHANGED <<zabState, acceptedEpoch, history, commitIndex,learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter,
+                       sendCounter, committedIndex, committedCounter, tempVarsZ, followerVarsZ, verifyVarsZ, msgs>>
+
+\* On the premise that ReceiveVotes.HasQuorums = TRUE, corresponding to logic in line 1061-1066 in LFE.java.
+FLEWaitNewNotmsgEnd(i) ==
+        /\ WaitNewNotmsgEnd(i)
+        /\ LET newState == state'[i]
+           IN
+           \/ /\ newState = LEADING
+              /\ ZabTurnToLeading(i)
+              /\ UNCHANGED <<cepochSent, synced>>
+           \/ /\ newState = FOLLOWING
+              /\ ZabTurnToFollowing(i)
+              /\ UNCHANGED <<learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter, sendCounter, committedIndex, committedCounter, tempVarsZ>>
+           \/ /\ newState = LOOKING
+              /\ PrintT("New state is LOOKING in FLEWaitNewNotmsgEnd, which should not happen.")
+              /\ UNCHANGED <<zabState, learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter, sendCounter, commitIndex,
+                             committedIndex, committedCounter, tempVarsZ, cepochSent, synced>>
+        /\ UNCHANGED <<acceptedEpoch, history, leaderAddr, verifyVarsZ, msgs>>
+              
+-----------------------------------------------------------------------------
+(* A sub-action describing how a server transitions from LEADING/FOLLOWING to LOOKING.
+   Initially I call it 'ZabTimeoutZ', but it will be called not only when timeout, 
+   but also when finding a low epoch from leader.*)
+FollowerShutdown(i) ==
+        /\ ZabTimeout(i)
+        /\ zabState'   = [zabState   EXCEPT ![i] = ELECTION]
+        /\ leaderAddr' = [leaderAddr EXCEPT ![i] = NullPoint]
+
+LeaderShutdown(i) ==
+        /\ ZabTimeout(i)
+        /\ zabState'   = [zabState   EXCEPT ![i] = ELECTION]
+        /\ leaderAddr' = [s \in Server |-> IF s \in learners[i] THEN NullPoint ELSE leaderAddr[s]]
+        /\ learners'   = [learners   EXCEPT ![i] = {}]
+        /\ forwarding' = [forwarding EXCEPT ![i] = {}]
+        /\ msgs'       = [s \in Server |-> [v \in Server |-> IF v \in learners[i] \/ s \in learners[i] THEN << >> ELSE msgs[s][v]]]
+
+FollowerTimout(i) ==
+        /\ state[i]      = FOLLOWING
+        /\ leaderAddr[i] = NullPoint
+        /\ FollowerShutdown(i)
+        /\ msgs' = [s \in Server |-> [v \in Server |-> IF v = i THEN << >> ELSE msgs[s][v]]]
+        /\ UNCHANGED <<acceptedEpoch, history, commitIndex, learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, 
+                       currentCounter, sendCounter, committedIndex, committedCounter, tempVarsZ, cepochSent, synced, verifyVarsZ>>
+
+LeaderTimeout(i) ==
+        /\ state[i] = LEADING
+        /\ learners[i] \notin Quorums
+        /\ LeaderShutdown(i)
+        /\ UNCHANGED <<acceptedEpoch, history, commitIndex, cepochRecv, ackeRecv, ackldRecv, ackIndex, currentCounter, sendCounter, committedIndex, committedCounter,
+                       tempVarsZ, cepochSent, synced, verifyVarsZ>>
+    
+-----------------------------------------------------------------------------
+(* Establish connection between leader i and follower j. 
+   It means i creates a learnerHandler for communicating with j, and j finds i's address.*)
+EstablishConnection(i, j) ==
+        /\ state[i] = LEADING   /\ state[j] = FOLLOWING
+        /\ j \notin learners[i] /\ leaderAddr[j] = NullPoint
+        /\ currentVote[j].proposedLeader = i
+        /\ learners'   = [learners   EXCEPT ![i] = learners[i] \union {j}] \* Leader:   'addLearnerHandler(peer)'
+        /\ leaderAddr' = [leaderAddr EXCEPT ![j] = i]                      \* Follower: 'connectToLeader(addr, hostname)'
+        /\ UNCHANGED <<serverVarsZ, electionVarsZ, leadingVoteSet, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, 
+                       currentCounter, sendCounter, committedIndex, committedCounter, tempVarsZ, cepochSent, synced, verifyVarsZ, msgVarsZ, idTable>>
+        
+(* The leader i finds timeout and TCP connection between i and j closes.*)       
+Timeout(i, j) ==
+        /\ state[i] = LEADING /\ state[j] = FOLLOWING
+        /\ j \in learners[i]  /\ leaderAddr[j] = i
+        (* The action of leader i.(corresponding to function 'removeLearnerHandler(peer)'.) *)
+        /\ learners'   = [learners   EXCEPT ![i] = learners[i] \ {j}] 
+        /\ forwarding' = [forwarding EXCEPT ![i] = IF j \in forwarding[i] THEN forwarding[i] \ {j} ELSE forwarding[i]]
+        /\ cepochRecv' = [cepochRecv EXCEPT ![i] = IF j \in cepochRecv[i] THEN cepochRecv[i] \ {j} ELSE cepochRecv[i]]
+        /\ ackeRecv'   = [ackeRecv   EXCEPT ![i] = IF j \in ackeRecv[i] THEN ackeRecv[i] \ {j} ELSE ackeRecv[i]]
+        /\ ackldRecv'  = [ackldRecv  EXCEPT ![i] = IF j \in ackldRecv[i] THEN ackldRecv[i] \ {j} ELSE ackldRecv[i]]
+        /\ ackIndex'   = [ackIndex   EXCEPT ![i][j] = 0]
+        /\ committedCounter' = [committedCounter EXCEPT ![i][j] = 0]
+        (* The action of follower j. *)
+        /\ FollowerShutdown(j)
+        (* Clean input buffer.*)
+        /\ Clean(i, j)
+        /\ UNCHANGED <<acceptedEpoch, history, commitIndex, currentCounter, sendCounter, committedIndex,
+                       tempVarsZ, cepochSent, synced, verifyVarsZ>>
+-----------------------------------------------------------------------------
+\* In phase f11, follower sends f.p to leader via FOLLOWERINFO(CEPOCH).
+FollowerSendFOLLOWERINFO(i) ==
+        /\ state[i]    = FOLLOWING
+        /\ zabState[i] = DISCOVERY
+        /\ leaderAddr[i] /= NullPoint
+        /\ \lnot cepochSent[i]
+        /\ Send(i, leaderAddr[i], [mtype  |-> FOLLOWERINFO,
+                                   mepoch |-> acceptedEpoch[i]])
+        /\ cepochSent' = [cepochSent EXCEPT ![i] = TRUE]
+        /\ UNCHANGED <<serverVarsZ, leaderVarsZ, electionVarsZ, tempVarsZ, leaderAddr, synced, verifyVarsZ, electionMsgs, idTable>>
+        
+(* In phase l11, leader waits for receiving FOLLOWERINFO from a quorum,
+   and then chooses a new epoch e' as its own epoch and broadcasts LEADERINFO. *)
+LeaderHandleFOLLOWERINFO(i, j) ==
+        /\ state[i] = LEADING
+        /\ msgs[j][i] /= << >>
+        /\ msgs[j][i][1].mtype = FOLLOWERINFO
+        /\ LET msg == msgs[j][i][1]
+           IN \/ /\ NullPoint \notin cepochRecv[i]   \* 1. has not broadcast LEADERINFO - modify tempMaxEpoch
+                 /\ LET newEpoch == Maximum({tempMaxEpoch[i], msg.mepoch})
+                    IN tempMaxEpoch' = [tempMaxEpoch EXCEPT ![i] = newEpoch]
+                 /\ Discard(j, i)
+              \/ /\ NullPoint \in cepochRecv[i]      \* 2. has broadcast LEADERINFO - no need to handle the msg, just send LEADERINFO to corresponding server
+                 /\ Reply(i, j, [mtype  |-> LEADERINFO,
+                                 mepoch |-> acceptedEpoch[i]])
+                 /\ UNCHANGED tempMaxEpoch
+        /\ cepochRecv' = [cepochRecv EXCEPT ![i] = IF j \in cepochRecv[i] THEN cepochRecv[i]
+                                                                          ELSE cepochRecv[i] \union {j}]
+        /\ UNCHANGED <<serverVarsZ, followerVarsZ, electionVarsZ, initialHistory, leadingVoteSet, learners, ackeRecv, ackldRecv, 
+                       forwarding, ackIndex, currentCounter, sendCounter, committedIndex, committedCounter, verifyVarsZ, electionMsgs, idTable>>
+
+LeaderDiscovery1(i) ==
+        /\ state[i]    = LEADING
+        /\ zabState[i] = DISCOVERY
+        /\ cepochRecv[i] \in Quorums
+        /\ acceptedEpoch' = [acceptedEpoch EXCEPT ![i] = tempMaxEpoch[i] + 1]
+        /\ cepochRecv'    = [cepochRecv    EXCEPT ![i] = cepochRecv[i] \union {NullPoint}]
+        /\ BroadcastLEADERINFO(i, [mtype  |-> LEADERINFO,
+                                   mepoch |-> acceptedEpoch'[i]])
+        /\ UNCHANGED <<state, currentEpoch, lastZxid, zabState, history, commitIndex, electionVarsZ, leadingVoteSet, learners, ackeRecv, ackldRecv, 
+                       forwarding, ackIndex, currentCounter, sendCounter, committedIndex, committedCounter, 
+                       tempVarsZ, followerVarsZ, verifyVarsZ, electionMsgs, idTable>>
+
+(* In phase f12, follower receives NEWEPOCH. If e' > f.p, then follower sends ACK-E back,
+   and ACK-E contains f.a and lastZxid to let leader judge whether it is the latest.
+   After handling NEWEPOCH, follower's zabState turns to SYNCHRONIZATION. *)
+FollowerHandleLEADERINFO(i, j) ==
+        /\ state[i] = FOLLOWING
+        /\ msgs[j][i] /= << >>
+        /\ msgs[j][i][1].mtype = LEADERINFO
+        /\ LET msg     == msgs[j][i][1]
+               infoOk  == j = leaderAddr[i]
+               epochOk == /\ infoOk
+                          /\ msg.mepoch >= acceptedEpoch[i]
+               correct == /\ epochOk
+                          /\ zabState[i] = DISCOVERY
+           IN /\ infoOk
+              /\ \/ /\ epochOk                      \* 1. Normal case
+                    /\ \/ /\ correct
+                          /\ acceptedEpoch' = [acceptedEpoch EXCEPT ![i] = msg.mepoch]
+                          /\ Reply(i, j, [mtype      |-> ACKEPOCH,
+                                          mepoch     |-> msg.mepoch,
+                                          mlastEpoch |-> currentEpoch[i],
+                                          mlastZxid  |-> lastZxid[i]])
+                          /\ cepochSent' = [cepochSent EXCEPT ![i] = TRUE]
+                          /\ UNCHANGED inherentViolated
+                       \/ /\ ~correct
+                          /\ PrintT("Exception: Condition correct is false in FollowerHandleLEADERINFO(" \o ToString(i) \o ", " \o ToString(j) \o ").")
+                          /\ inherentViolated' = TRUE
+                          /\ Discard(j, i)
+                          /\ UNCHANGED <<acceptedEpoch, cepochSent>>
+                    /\ zabState' = [zabState EXCEPT ![i] = IF zabState[i] = DISCOVERY THEN SYNCHRONIZATION
+                                                                                      ELSE zabState[i]]
+                    /\ UNCHANGED <<varsL, leaderAddr>>
+                 \/ /\ ~epochOk                    \* 2. Abnormal case - go back to election
+                    /\ FollowerShutdown(i)
+                    /\ Clean(i, j)
+                    /\ UNCHANGED <<acceptedEpoch, cepochSent, inherentViolated>>
+        /\ UNCHANGED <<history, commitIndex, learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter, sendCounter, committedIndex,
+                       committedCounter, tempVarsZ, synced, proposalMsgsLog, epochLeader>>
+
+\* Abstraction of actions making follower synced with leader before leader sending NEWLEADER.
+subRECOVERYSYNC(i, j) ==
+        LET canSync == /\ state[i] = LEADING   /\ zabState[i] /= DISCOVERY      /\ j \in learners[i]  /\ j \in ackeRecv[i]
+                       /\ state[j] = FOLLOWING /\ zabState[j] = SYNCHRONIZATION /\ leaderAddr[j] = i  /\ synced[j] = FALSE
+        IN
+        \/ /\ canSync
+           /\ history'     = [history     EXCEPT ![j] = history[i]]
+           /\ lastZxid'    = [lastZxid    EXCEPT ![j] = lastZxid[i]]
+           /\ UpdateProposal(j, leaderAddr[j], lastZxid'[j], currentEpoch[j])
+           /\ commitIndex' = [commitIndex EXCEPT ![j] = commitIndex[i]]
+           /\ synced'      = [synced      EXCEPT ![j] = TRUE]
+           /\ forwarding'  = [forwarding  EXCEPT ![i] = forwarding[i] \union {j}]    \* j will receive PROPOSAL and COMMIT
+           /\ ackIndex'    = [ackIndex    EXCEPT ![i][j] = Len(history[i])]
+           /\ committedCounter' = [committedCounter EXCEPT ![i][j] = Maximum({commitIndex[i] - Len(initialHistory[i]), 0})]
+           /\ LET ms == [msource|->i, mtype|->"RECOVERYSYNC", mepoch|->acceptedEpoch[i], mproposals|->history[i]]
+              IN proposalMsgsLog' = IF ms \in proposalMsgsLog THEN proposalMsgsLog
+                                                              ELSE proposalMsgsLog \union {ms}
+           /\ Reply(i, j, [mtype     |-> NEWLEADER,
+                           mepoch    |-> acceptedEpoch[i],
+                           mlastZxid |-> lastZxid[i]])
+        \/ /\ ~canSync
+           /\ Discard(j, i)
+           /\ UNCHANGED <<history, lastZxid, currentVote, commitIndex, synced, forwarding, ackIndex, committedCounter, proposalMsgsLog>>
+        
+(* In phase l12, leader waits for receiving ACKEPOPCH from a quorum,
+   and check whether it has the latest history and epoch from them.
+   If so, leader's zabState turns to SYNCHRONIZATION. *)
+LeaderHandleACKEPOCH(i, j) ==
+        /\ state[i] = LEADING
+        /\ msgs[j][i] /= << >>
+        /\ msgs[j][i][1].mtype = ACKEPOCH
+        /\ LET msg == msgs[j][i][1]
+               infoOk == /\ j \in learners[i]            
+                         /\ acceptedEpoch[i] = msg.mepoch
+               logOk  == /\ infoOk                          \* logOk = TRUE means leader is more up-to-date than follower j
+                         /\ \/ currentEpoch[i] > msg.mlastEpoch
+                            \/ /\ currentEpoch[i] = msg.mlastEpoch
+                               /\ \/ lastZxid[i][1] > msg.mlastZxid[1]
+                                  \/ /\ lastZxid[i][1] = msg.mlastZxid[1]
+                                     /\ lastZxid[i][2] >= msg.mlastZxid[2]
+               replyOk == /\ infoOk
+                          /\ NullPoint \in ackeRecv[i]
+           IN /\ infoOk
+              /\ \/ /\ replyOk
+                    /\ subRECOVERYSYNC(i, j)
+                    /\ ackeRecv' = [ackeRecv EXCEPT ![i] = IF j \notin ackeRecv[i] THEN ackeRecv[i] \union {j}
+                                                                                   ELSE ackeRecv[i]]
+                    /\ UNCHANGED <<state, currentEpoch,logicalClock, receiveVotes, outOfElection, recvQueue, waitNotmsg, leadingVoteSet, electionMsgs, idTable,
+                                   zabState, leaderAddr, learners>>
+                 \/ /\ ~replyOk
+                    /\ \/ /\ logOk
+                          /\ ackeRecv' = [ackeRecv EXCEPT ![i] = IF j \notin ackeRecv[i] THEN ackeRecv[i] \union {j}
+                                                                                         ELSE ackeRecv[i]]
+                          /\ Discard(j, i)
+                          /\ UNCHANGED <<varsL, zabState, leaderAddr, learners, forwarding>>
+                       \/ /\ ~logOk            \* go back to election
+                          /\ LeaderShutdown(i)
+                          /\ UNCHANGED ackeRecv
+                    /\ UNCHANGED <<history, commitIndex, synced, forwarding, ackIndex, committedCounter, proposalMsgsLog>>
+        /\ UNCHANGED <<acceptedEpoch, cepochRecv, ackldRecv,currentCounter, sendCounter, committedIndex, tempVarsZ, cepochSent, epochLeader, inherentViolated>>
+                          
+LeaderDiscovery2(i) ==
+        /\ state[i] = LEADING
+        /\ zabState[i] = DISCOVERY
+        /\ ackeRecv[i] \in Quorums
+        /\ zabState'       = [zabState       EXCEPT ![i] = SYNCHRONIZATION]
+        /\ currentEpoch'   = [currentEpoch   EXCEPT ![i] = acceptedEpoch[i]]
+        /\ initialHistory' = [initialHistory EXCEPT ![i] = history[i]]
+        /\ ackeRecv'       = [ackeRecv       EXCEPT ![i] = ackeRecv[i] \union {NullPoint}]
+        /\ ackIndex'       = [ackIndex       EXCEPT ![i][i] = Len(history[i])]
+        /\ UpdateProposal(i, i, lastZxid[i], currentEpoch'[i])
+        /\ LET epoch == acceptedEpoch[i]
+           IN epochLeader' = [epochLeader EXCEPT ![epoch] = epochLeader[epoch] \union {i}]
+        /\ UNCHANGED <<state, lastZxid, acceptedEpoch, history, commitIndex, logicalClock, receiveVotes, outOfElection, recvQueue, waitNotmsg,
+                       leadingVoteSet, learners, cepochRecv, ackldRecv, forwarding, currentCounter, sendCounter, committedIndex, committedCounter,
+                       tempMaxEpoch, followerVarsZ, proposalMsgsLog, inherentViolated, msgVarsZ, idTable>>
+
+(* Note:  Set cepochRecv, ackeRecv, ackldRecv to {NullPoint} in corresponding three actions to
+          make sure that the prospective leader will not broadcast NEWEPOCH/NEWLEADER/COMMITLD twice.*)
+-----------------------------------------------------------------------------
+RECOVERYSYNC(i, j) ==
+        /\ state[i] = LEADING   /\ zabState[i] /= DISCOVERY      /\ j \in learners[i] /\ j \in ackeRecv[i]
+        /\ state[j] = FOLLOWING /\ zabState[j] = SYNCHRONIZATION /\ leaderAddr[j] = i /\ synced[j] = FALSE
+        \* /\ acceptedEpoch[i] = acceptedEpoch[j] \* This condition is unnecessary.
+        /\ history'     = [history     EXCEPT ![j] = history[i]]
+        /\ lastZxid'    = [lastZxid    EXCEPT ![j] = lastZxid[i]]
+        /\ UpdateProposal(j, leaderAddr[j], lastZxid'[j], currentEpoch[j])
+        /\ commitIndex' = [commitIndex EXCEPT ![j] = commitIndex[i]]
+        /\ synced'      = [synced      EXCEPT ![j] = TRUE]
+        /\ forwarding'  = [forwarding  EXCEPT ![i] = forwarding[i] \union {j}]
+        /\ ackIndex'    = [ackIndex    EXCEPT ![i][j] = Len(history[i])]
+        /\ committedCounter' = [committedCounter EXCEPT ![i][j] = Maximum({commitIndex[i] - Len(initialHistory[i]), 0})]
+        /\ LET ms == [msource|->i, mtype|->"RECOVERYSYNC", mepoch|->acceptedEpoch[i], mproposals|->history[i]]
+           IN proposalMsgsLog' = IF ms \in proposalMsgsLog THEN proposalMsgsLog
+                                                           ELSE proposalMsgsLog \union {ms}
+        /\ Send(i, j, [mtype     |-> NEWLEADER,
+                       mepoch    |-> acceptedEpoch[i],
+                       mlastZxid |-> lastZxid[i]])
+        /\ UNCHANGED <<state, zabState, acceptedEpoch, currentEpoch, logicalClock, receiveVotes, outOfElection, recvQueue, waitNotmsg, 
+                       leadingVoteSet, learners, cepochRecv, ackeRecv, ackldRecv, currentCounter, sendCounter, committedIndex, 
+                       tempVarsZ, cepochSent, leaderAddr, epochLeader, inherentViolated, electionMsgs, idTable>>
+
+(* In phase f21, follower receives NEWLEADER. The follower updates its epoch and history,
+   and sends back ACK-LD to leader. *)
+FollowerHandleNEWLEADER(i, j) ==
+        /\ state[i] = FOLLOWING
+        /\ msgs[j][i] /= << >>
+        /\ msgs[j][i][1].mtype = NEWLEADER
+        /\ LET msg     == msgs[j][i][1]
+               infoOk  == /\ leaderAddr[i] = j
+                          /\ acceptedEpoch[i] = msg.mepoch
+               correct == /\ infoOk
+                          /\ zabState[i] = SYNCHRONIZATION
+                          /\ synced[i]
+                          /\ ZxidEqual(lastZxid[i], msg.mlastZxid)
+           IN /\ infoOk
+              /\ currentEpoch' = [currentEpoch EXCEPT ![i] = msg.mepoch]
+              /\ UpdateProposal(i, j, lastZxid[i], currentEpoch'[i])
+              /\ \/ /\ correct
+                    /\ Reply(i, j, [mtype  |-> ACKLD,
+                                    mepoch |-> msg.mepoch])
+                    /\ UNCHANGED inherentViolated
+                 \/ /\ ~correct
+                    /\ PrintT("Exception: Condition correct is false in FollowerHandleNEWLEADER(" \o ToString(i) \o ", " \o ToString(j) \o ").")
+                    /\ inherentViolated' = TRUE
+                    /\ Discard(j, i)
+        /\ UNCHANGED <<state, lastZxid, zabState, acceptedEpoch, history, commitIndex, logicalClock, receiveVotes, outOfElection, recvQueue, waitNotmsg,
+                       leaderVarsZ, tempVarsZ, followerVarsZ, proposalMsgsLog, epochLeader, electionMsgs, idTable>>
+
+\* In phase l22, leader receives ACK-LD from a quorum of followers, and sends COMMIT-LD(UPTODATE) to followers.
+LeaderHandleACKLD(i, j) ==
+        /\ state[i] = LEADING
+        /\ msgs[j][i] /= << >>
+        /\ msgs[j][i][1].mtype = ACKLD
+        /\ LET msg     == msgs[j][i][1]
+               infoOk  == /\ acceptedEpoch[i] = msg.mepoch
+                          /\ j \in learners[i]
+               replyOk == /\ infoOk
+                          /\ NullPoint \in ackldRecv[i]
+           IN /\ infoOk
+              /\ \/ /\ replyOk
+                    /\ Reply(i, j, [mtype   |-> UPTODATE,
+                                    mepoch  |-> acceptedEpoch[i],
+                                    mcommit |-> commitIndex[i]])
+                    /\ committedCounter' = [committedCounter EXCEPT ![i][j] = Maximum({commitIndex[i] - Len(initialHistory[i]), committedCounter[i][j]})]
+                 \/ /\ ~replyOk
+                    /\ Discard(j, i)
+                    /\ UNCHANGED committedCounter
+              /\ ackldRecv' = [ackldRecv EXCEPT ![i] = IF j \notin ackldRecv[i] THEN ackldRecv[i] \union {j}
+                                                                                ELSE ackldRecv[i]]
+        /\ UNCHANGED <<serverVarsZ, electionVarsZ, leadingVoteSet, learners, cepochRecv, ackeRecv, forwarding, 
+                       ackIndex, currentCounter, sendCounter, committedIndex, tempVarsZ, followerVarsZ, verifyVarsZ, electionMsgs, idTable>>
+
+LeaderSync2(i) ==
+        /\ state[i] = LEADING
+        /\ zabState[i] = SYNCHRONIZATION
+        /\ ackldRecv[i] \in Quorums
+        /\ commitIndex'    = [commitIndex    EXCEPT ![i] = Len(history[i])]
+        /\ committedIndex' = [committedIndex EXCEPT ![i] = Len(history[i])]
+        /\ zabState'       = [zabState       EXCEPT ![i] = BROADCAST]
+        /\ currentCounter' = [currentCounter EXCEPT ![i] = 0]
+        /\ sendCounter'    = [sendCounter    EXCEPT ![i] = 0]
+        /\ ackldRecv'      = [ackldRecv      EXCEPT ![i] = ackldRecv[i] \union {NullPoint}]
+        /\ BroadcastUPTODATE(i, [mtype   |-> UPTODATE,
+                                 mepoch  |-> acceptedEpoch[i],
+                                 mcommit |-> Len(history[i])]) \* In actual UPTODATE doesn't carry this info
+        /\ UNCHANGED <<state, currentEpoch, lastZxid, acceptedEpoch, history, electionVarsZ, leadingVoteSet, learners, cepochRecv, ackeRecv, forwarding, ackIndex, 
+                       committedCounter, tempVarsZ, followerVarsZ, verifyVarsZ, electionMsgs, idTable>>
+
+FollowerHandleUPTODATE(i, j) ==
+        /\ state[i] = FOLLOWING
+        /\ msgs[j][i] /= << >>
+        /\ msgs[j][i][1].mtype = UPTODATE
+        /\ LET msg == msgs[j][i][1]
+               infoOk == /\ leaderAddr[i] = j
+                         /\ acceptedEpoch[i] = msg.mepoch
+               correct == /\ infoOk
+                          /\ zabState[i] = SYNCHRONIZATION
+                          /\ currentEpoch[i] = msg.mepoch
+           IN /\ infoOk
+              /\ \/ /\ correct
+                    /\ commitIndex' = [commitIndex EXCEPT ![i] = Maximum({commitIndex[i], msg.mcommit})] \* May have received COMMIT and have a bigger commit index.
+                    /\ zabState'    = [zabState    EXCEPT ![i] = BROADCAST]
+                    /\ UNCHANGED inherentViolated
+                 \/ /\ ~correct
+                    /\ PrintT("Exception: Condition correct is false in FollowerHandleUPTODATE(" \o ToString(i) \o ", " \o ToString(j) \o ").")
+                    /\ inherentViolated' = TRUE
+                    /\ UNCHANGED <<commitIndex, zabState>>
+        /\ Discard(j, i)
+        /\ UNCHANGED <<state, currentEpoch, lastZxid, acceptedEpoch, history, electionVarsZ, leaderVarsZ, tempVarsZ, followerVarsZ,
+                       proposalMsgsLog, epochLeader, electionMsgs, idTable>>
+
+-----------------------------------------------------------------------------
+(* In phase l31, leader receives client request and broadcasts PROPOSAL.
+   Note: In production, any server in traffic can receive requests and forward it to leader if necessary.
+   We choose to let leader be the sole one who can receive requests, to simplify spec and keep correctness at the same time. *)
+ClientRequest(i, v) ==
+        /\ state[i] = LEADING
+        /\ zabState[i] = BROADCAST
+        /\ currentCounter' = [currentCounter EXCEPT ![i] = currentCounter[i] + 1]
+        /\ LET newTransaction == [epoch   |-> acceptedEpoch[i],
+                                  counter |-> currentCounter'[i],
+                                  value   |-> v]
+           IN /\ history'  = [history  EXCEPT ![i] = Append(history[i], newTransaction)]
+              /\ lastZxid' = [lastZxid EXCEPT ![i] = <<acceptedEpoch[i], currentCounter'[i]>>]
+              /\ ackIndex' = [ackIndex EXCEPT ![i][i] = Len(history'[i])]
+              /\ UpdateProposal(i, i, lastZxid'[i], currentEpoch[i])
+        /\ UNCHANGED <<state, currentEpoch, zabState, acceptedEpoch, commitIndex, logicalClock, receiveVotes, outOfElection, recvQueue, waitNotmsg,
+                       leadingVoteSet, learners, cepochRecv, ackeRecv, ackldRecv, forwarding, sendCounter, committedIndex, committedCounter,
+                       tempVarsZ, followerVarsZ, verifyVarsZ, msgVarsZ, idTable>>
+
+LeaderBroadcast1(i) ==
+        /\ state[i] = LEADING
+        /\ zabState[i] = BROADCAST
+        /\ sendCounter[i] < currentCounter[i]
+        /\ LET toBeSentCounter == sendCounter[i] + 1
+               toBeSentIndex   == Len(initialHistory[i]) + toBeSentCounter
+               toBeSentEntry   == history[i][toBeSentIndex]
+           IN /\ Broadcast(i, [mtype     |-> PROPOSAL,
+                               mepoch    |-> acceptedEpoch[i],
+                               mproposal |-> toBeSentEntry])
+              /\ sendCounter' = [sendCounter EXCEPT ![i] = toBeSentCounter]
+              /\ LET m == [msource|-> i, mepoch|-> acceptedEpoch[i], mtype|-> PROPOSAL, mproposal|-> toBeSentEntry]
+                 IN proposalMsgsLog' = proposalMsgsLog \union {m}
+        /\ UNCHANGED <<serverVarsZ, electionVarsZ, leadingVoteSet, learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter, 
+                       committedIndex, committedCounter, tempVarsZ, followerVarsZ, epochLeader, inherentViolated, electionMsgs, idTable>>
+
+\* In phase f31, follower accepts proposal and append it to history.
+FollowerHandlePROPOSAL(i, j) ==
+        /\ state[i] = FOLLOWING
+        /\ msgs[j][i] /= << >>
+        /\ msgs[j][i][1].mtype = PROPOSAL
+        /\ LET msg    == msgs[j][i][1]
+               infoOk == /\ leaderAddr[i] = j
+                         /\ acceptedEpoch[i] = msg.mepoch
+               correct == /\ infoOk
+                          /\ zabState[i] /= DISCOVERY
+                          /\ synced[i]
+               logOk  == \/ /\ msg.mproposal.counter = 1     \* the first PROPOSAL in this epoch
+                            /\ \/ Len(history[i]) = 0
+                               \/ /\ Len(history[i]) > 0
+                                  /\ history[i][Len(history[i])].epoch < msg.mepoch
+                         \/ /\ msg.mproposal.counter > 1      \* not the first PROPOSAL in this epoch
+                            /\ Len(history[i]) > 0
+                            /\ history[i][Len(history[i])].epoch = msg.mepoch
+                            /\ history[i][Len(history[i])].counter = msg.mproposal.counter - 1
+           IN /\ infoOk
+              /\ \/ /\ correct
+                    /\ \/ /\ logOk
+                          /\ history'  = [history  EXCEPT ![i] = Append(history[i], msg.mproposal)]
+                          /\ lastZxid' = [lastZxid EXCEPT ![i] = <<msg.mepoch, msg.mproposal.counter>>]
+                          /\ UpdateProposal(i, j, lastZxid'[i], currentEpoch[i])
+                          /\ Reply(i, j, [mtype  |-> ACK,
+                                          mepoch |-> acceptedEpoch[i],
+                                          mzxid  |-> <<msg.mepoch, msg.mproposal.counter>>])
+                          /\ UNCHANGED inherentViolated
+                       \/ /\ ~logOk
+                          /\ PrintT("Exception: Condition logOk is false in FollowerHandlePROPOSAL(" \o ToString(i) \o ", " \o ToString(j) \o ").")
+                          /\ inherentViolated' = TRUE
+                          /\ Discard(j, i)
+                          /\ UNCHANGED <<history, lastZxid, currentVote>>
+                 \/ /\ ~correct
+                    /\ PrintT("Exception: Condition correct is false in FollowerHandlePROPOSAL(" \o ToString(i) \o ", " \o ToString(j) \o ").")
+                    /\ inherentViolated' = TRUE
+                    /\ Discard(j, i)
+                    /\ UNCHANGED <<history, lastZxid, currentVote>>
+        /\ UNCHANGED <<state, currentEpoch, zabState, acceptedEpoch, commitIndex, logicalClock, receiveVotes, outOfElection, recvQueue, waitNotmsg,
+                       leaderVarsZ, tempVarsZ, followerVarsZ, proposalMsgsLog, epochLeader, electionMsgs, idTable>>
+
+\* In phase l32, leader receives ack from a quorum of followers to a certain proposal, and commits the proposal.
+LeaderHandleACK(i, j) ==
+        /\ state[i] = LEADING
+        /\ msgs[j][i] /= << >>
+        /\ msgs[j][i][1].mtype = ACK
+        /\ LET msg == msgs[j][i][1]
+               infoOk == /\ j \in forwarding[i]
+                         /\ acceptedEpoch[i] = msg.mepoch
+               correct == /\ infoOk
+                          /\ zabState[i] = BROADCAST
+                          /\ sendCounter[i] >= msg.mzxid[2]
+               logOk == /\ infoOk
+                        /\ ackIndex[i][j] + 1 = Len(initialHistory[i]) + msg.mzxid[2]
+           IN /\ infoOk
+              /\ \/ /\ correct
+                    /\ \/ /\ logOk
+                          /\ ackIndex' = [ackIndex EXCEPT ![i][j] = ackIndex[i][j] + 1]
+                       \/ /\ ~logOk
+                          /\ PrintT("Note: redundant ACK.")
+                          /\ UNCHANGED ackIndex
+                    /\ UNCHANGED inherentViolated
+                 \/ /\ ~correct
+                    /\ PrintT("Exception: Condition correct is false in FollowerHandleACK(" \o ToString(i) \o ", " \o ToString(j) \o ").")
+                    /\ inherentViolated' = TRUE
+                    /\ UNCHANGED ackIndex
+        /\ Discard(j, i)
+        /\ UNCHANGED <<serverVarsZ, electionVarsZ, leadingVoteSet, learners, cepochRecv, ackeRecv, ackldRecv, forwarding, currentCounter, sendCounter, 
+                       committedIndex, committedCounter, tempVarsZ, followerVarsZ, proposalMsgsLog, epochLeader, electionMsgs, idTable>>
+
+LeaderAdvanceCommit(i) ==

Review comment:
       Consider combining commit advancement with LeaderHandleACK to reduce state space.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@zookeeper.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] hanm commented on a change in pull request #1690: ZOOKEEPER-3615: Provide formal specification and verification using TLA+ for Zab

Posted by GitBox <gi...@apache.org>.
hanm commented on a change in pull request #1690:
URL: https://github.com/apache/zookeeper/pull/1690#discussion_r631473566



##########
File path: zookeeper-contrib/zookeeper-contrib-zabtla/README.md
##########
@@ -0,0 +1,71 @@
+# Zab-tla
+
+## Overview
+This project is devoted to providing formal specification and verification using TLA+ for the Zookeeper Atomic Broadcast(Zab) consensus protocol proposed in *Junqueira F P, Reed B C, Serafini M. Zab: High-performance broadcast for primary-backup systems[C]//2011 IEEE/IFIP 41st International Conference on Dependable Systems & Networks (DSN). IEEE, 2011: 245-256*.  
+
+We have made a formal [specification](Zab.tla) for Zab using TLA+ toolbox, and we have done a certain scale of model checking to verify the correctness of Zab.
+
+Due to the simplification of Zab algorithm description in the paper, some details in specification were modified and added. If you have any question, please let us know.
+
+You can find this document in chinese in [doc-in-chinsese](doc-in-chinese/README.md).
+
+## Requirements
+TLA+ toolbox version 1.7.0
+
+## Run
+Create specification and run models in the usual way.  

Review comment:
       Yeah, please don't feel any pressures and take your time. I'll also take some time to comment on details of the spec this week, I think even it's not ZAB 1.0 the relevant experience are valuable when we make the 1.0 spec.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] BinyuHuang-nju edited a comment on pull request #1690: ZOOKEEPER-3615: Provide formal specification and verification using TLA+ for Zab

Posted by GitBox <gi...@apache.org>.
BinyuHuang-nju edited a comment on pull request #1690:
URL: https://github.com/apache/zookeeper/pull/1690#issuecomment-844163499


   @hanm Maybe it would be inappropriate for me to ask the following questions under this PR. Now I am learning ZAB 1.0 and prepare to write spec. Could you please let me ask about details in it?
   
   Currently I have several problems about ZAB 1.0.
   1. For some detailed industial implementation (like message with type PING, connection between leader and followers), do I need to implement these in spec?
   2. I see before send NEWLEADER, leader will choose a best message from SNAP,TRUNC,DIFF to send according to the corresponding follower's state. Do I need to implement these, or abstract this part and just send NEWLEADER to sync with followers?
   3. I don't understand whether variable 'loracle' is more like a global variable that has the latest leader ID and every server can reach it, or 'loracle' is a local variable like 'votedFor' in Raft. (Actually I used the latter in Zab.tla, which corresponds to 'leaderOracle'. And I think it has no effect on correctness no matter which one is used.)
   4. The last problem what I want to ask is that about recovery, because I didn't see notes about this part in paper and the link. 
   I saw how the leader handles when a new server wants to join them, but I didn't see how a restarted server finds the latest leader. If 'loracle' is the former in question 3, I can understand this part. In Zab.tla, I use the method which the follower broadcasts messages to ack other servers what their local oracle is, and update its oracle when receiving the same oracle and epoch from a quorum(I used this method because I saw servers recover like this in View-stamped Replication).
   A follower must have received corresponding PROPOSAL when receiving COMMIT, if the follower is the initial server that joins Q. But I think this may not always be true when the follower is one that joins Q midway. So I want to know how follower handles to catch up state when receiving COMMIT corresponding to a transaction that not exists in its local history.(You could see this condition in the image at the bottom of README. When I wrote spec for Zab pre-1.0, I choose to let followers keep re-sending CEPOCH to obtain latest transactions until the conflict described above disappers.)
   
   I am very sorry if my problems bother you. Thank you for the various feedbacks and suggestions you have provided before!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] BinyuHuang-nju commented on a change in pull request #1690: ZOOKEEPER-3615: Provide formal specification and verification using TLA+ for Zab

Posted by GitBox <gi...@apache.org>.
BinyuHuang-nju commented on a change in pull request #1690:
URL: https://github.com/apache/zookeeper/pull/1690#discussion_r635014579



##########
File path: zookeeper-contrib/zookeeper-contrib-zabtla/README.md
##########
@@ -0,0 +1,71 @@
+# Zab-tla
+
+## Overview
+This project is devoted to providing formal specification and verification using TLA+ for the Zookeeper Atomic Broadcast(Zab) consensus protocol proposed in *Junqueira F P, Reed B C, Serafini M. Zab: High-performance broadcast for primary-backup systems[C]//2011 IEEE/IFIP 41st International Conference on Dependable Systems & Networks (DSN). IEEE, 2011: 245-256*.  
+
+We have made a formal [specification](Zab.tla) for Zab using TLA+ toolbox, and we have done a certain scale of model checking to verify the correctness of Zab.
+
+Due to the simplification of Zab algorithm description in the paper, some details in specification were modified and added. If you have any question, please let us know.
+
+You can find this document in chinese in [doc-in-chinsese](doc-in-chinese/README.md).
+
+## Requirements
+TLA+ toolbox version 1.7.0
+
+## Run
+Create specification and run models in the usual way.  

Review comment:
       Thank you! Then I know where the problem is. I just modified tla files and didn't modified other files at the same time.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] Vanlightly commented on pull request #1690: ZOOKEEPER-3615: Provide formal specification and verification using TLA+ for Zab

Posted by GitBox <gi...@apache.org>.
Vanlightly commented on pull request #1690:
URL: https://github.com/apache/zookeeper/pull/1690#issuecomment-836302010


   @BinyuHuang-nju good work getting it to reach a higher diameter, that improves our confidence in the spec. I will start digging into the functioning of the spec itself and provide feedback. I am interested in seeing if I can cause violations to occur by making subtle changes to the spec (making sure it detects problems that I introduce), I'm also interested in profiling it to see if any Java overrides could improve performance.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] BinyuHuang-nju commented on pull request #1690: ZOOKEEPER-3615: Provide formal specification and verification using TLA+ for Zab

Posted by GitBox <gi...@apache.org>.
BinyuHuang-nju commented on pull request #1690:
URL: https://github.com/apache/zookeeper/pull/1690#issuecomment-844163499


   @hanm Maybe it would be inappropriate for me to ask the following questions under this PR. Now I am learning ZAB 1.0 and prepare to write spec. Could you please let me ask about details in it?
   
   Currently I have several problems about ZAB 1.0.
   1. For some detailed industial implementation (like message with type PING, connection between leader and followers), do I need to implement these in spec?
   2. I see before send NEWLEADER, leader will choose a best message from SNAP,TRUNC,DIFF to send according to the corresponding follower's state. Do I need to implement these, or abstract this part and just send NEWLEADER to sync with followers?
   3. I don't understand whether variable 'loracle' is more like a global variable that has the latest leader ID and every server can reach it, or 'loracle' is a local variable like 'votedFor' in Raft. (Actually I used the latter in Zab.tla, which corresponds to 'leaderOracle'. And I think it has no effect on correctness no matter which one is used.)
   4. The last problem what I want to ask is that about recovery, because I didn't see notes about this part in paper and the link. 
   I saw how the leader handles when a new server wants to join them, but I didn't see how a restarted server finds the latest leader (If 'loracle' is the former in question 3, I can understand this part).
   A follower must have received corresponding PROPOSAL when receiving COMMIT, if the follower is the initial server that joins Q. But I think this may not always be true when the follower is one that joins Q midway. So I want to know how follower handles to catch up state when receiving COMMIT corresponding to a transaction that not exists in its local history.(You could see this condition in the image at the bottom of README. When I wrote spec for Zab pre-1.0, I choose to keep re-sending CEPOCH to obtain latest transactions until the conflict described above disappers.)
   
   I am very sorry if my problems bother you. Thank you for the various feedbacks and suggestions you have provided before!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] Vanlightly commented on pull request #1690: Provide formal specification and verification using TLA+ for Zab

Posted by GitBox <gi...@apache.org>.
Vanlightly commented on pull request #1690:
URL: https://github.com/apache/zookeeper/pull/1690#issuecomment-831155754


   @BinyuHuang-nju  I have run it a couple of times and hit some invariant violations, such as LocalPrimaryOrder and Integrity. I have not taken the time to understand the spec or these invariants yet but would you be able to specify which invariants you have enabled when you run it? I am using Servers {s1,s2} and Values {v1} as sets of model values, with the rest of the constants as model values.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] BinyuHuang-nju commented on a change in pull request #1690: ZOOKEEPER-3615: Provide formal specification and verification using TLA+ for Zab

Posted by GitBox <gi...@apache.org>.
BinyuHuang-nju commented on a change in pull request #1690:
URL: https://github.com/apache/zookeeper/pull/1690#discussion_r630746719



##########
File path: zookeeper-contrib/zookeeper-contrib-zabtla/README.md
##########
@@ -0,0 +1,71 @@
+# Zab-tla
+
+## Overview
+This project is devoted to providing formal specification and verification using TLA+ for the Zookeeper Atomic Broadcast(Zab) consensus protocol proposed in *Junqueira F P, Reed B C, Serafini M. Zab: High-performance broadcast for primary-backup systems[C]//2011 IEEE/IFIP 41st International Conference on Dependable Systems & Networks (DSN). IEEE, 2011: 245-256*.  

Review comment:
       Thank you for your suggestion! I didn't find a good manual describing Zab with new version before and this link will be very helpful to me. I will take time to learn about Zab 1.0!




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] BinyuHuang-nju edited a comment on pull request #1690: ZOOKEEPER-3615: Provide formal specification and verification using TLA+ for Zab

Posted by GitBox <gi...@apache.org>.
BinyuHuang-nju edited a comment on pull request #1690:
URL: https://github.com/apache/zookeeper/pull/1690#issuecomment-850846721


   @Vanlightly @hanm Hi! I have read some code about zab. But I still cannot understand some details. I want to ask you about them.
   
   1. At line 630 in function 'syncWithLeader' In [Learner.java](https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/quorum/Learner.java), I see follower in phase SYNCHRONIZATION may receive PROPOSAL and COMMIT. But in paper, leader will add a new server into its 'forwardingFollowers'(Q in paper) only when receiving ack of NEWLEADER from it, so in paper this new server should not receive PROPOSAL or COMMIT because in phase SYNCHRONIZATION it has not sent ACK-NEWLEADER.
   So here, my question is in ZAB 1.0, when does leader add a newly joined server into its 'forwardingFollowers'?(I am sorry that I didn't find this in code)
   
   2. My understanding of question above is: in phase 2, leader syncs with each follower and ''sends'' NEWLEADER only to this corresponding follower that has completed sync, not ''broadcasts'' NEWLEADER. So when leader is in phase 3, there may exist some follower in phase 2, and has not completed sync with leader since it hasn't received NEWLEADER, but it is allowed to receive PROPOSAL and COMMIT.
   I am not sure whether my understanding is right. 
   
   3. At line 766 in function 'syncWithLeader' in [Learner.java](https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/quorum/Learner.java), I see when receiving NEWLEADER, the variable 'packetsNotCommitted' is set as empty, and when receiving UPTODATE, there is no action handling transactions left to commit them.
   Does that mean follower thinks all transactions committed upon receiving NEWLEADER not UPTODATE?(Actually in papaer and my spec of zab pre-1.0, follower thinks them committed upon receiving UPTODATE)
   
   4. In phase 1 in manual tha we mentioned above, there's one word: "Note, if a follower is connecting, but the leader is already established (in phase 3) the follower follows the phases, but the leader ignores any ACKs." 
   It seems like that leader in phase 3(BROADCAST) will ignre ACKEPOCH in phase1 and ACK of NEWLEADER in phase 2, then how dose this newly connecting follower receive UPTODATE to go to phase BROADCAST?
   
   Since these details are related to how to let follower sync with leader in my spec of ZAB 1.0, I want to have a further understanding of them. Thank you!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] Vanlightly commented on pull request #1690: ZOOKEEPER-3615: Provide formal specification and verification using TLA+ for Zab

Posted by GitBox <gi...@apache.org>.
Vanlightly commented on pull request #1690:
URL: https://github.com/apache/zookeeper/pull/1690#issuecomment-974609840


   @BinyuHuang-nju I have checked it over and it's looking good. There's just one last point from my perspective.
   
   The Value constant is not really that useful as it stands right now. It is still possible for a client to write the same value more than once. More importantly the values are not checked in any invariants, so the values do not serve any purpose right now other than to increase the state space. 
   
   I would recommend adding a value equality check to `TransactionEqual` to ensure that not only the zxid matches but the value also. Then also prevent the same value from being written twice in `ClientRequestAndLeaderBroadcastProposal` and remove the check for transaction num.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@zookeeper.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] BinyuHuang-nju commented on pull request #1690: ZOOKEEPER-3615: Provide formal specification and verification using TLA+ for Zab

Posted by GitBox <gi...@apache.org>.
BinyuHuang-nju commented on pull request #1690:
URL: https://github.com/apache/zookeeper/pull/1690#issuecomment-974819434






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@zookeeper.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] Vanlightly commented on a change in pull request #1690: ZOOKEEPER-3615: Provide formal specification and verification using TLA+ for Zab

Posted by GitBox <gi...@apache.org>.
Vanlightly commented on a change in pull request #1690:
URL: https://github.com/apache/zookeeper/pull/1690#discussion_r705453486



##########
File path: zookeeper-specifications/zab-1.0/ZabWithFLE.tla
##########
@@ -0,0 +1,1020 @@
+(*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *)
+
+----------------------------- MODULE ZabWithFLE -----------------------------
+(* This is the formal specification for the Zab consensus algorithm,
+   which means Zookeeper Atomic Broadcast.*)
+
+(* Reference:
+   FLE: FastLeaderElection.java, Vote.java, QuorumPeer.java in https://github.com/apache/zookeeper.
+   ZAB: QuorumPeer.java, Learner.java, Follower.java, LearnerHandler.java, Leader.java 
+        in https://github.com/apache/zookeeper.
+        https://cwiki.apache.org/confluence/display/ZOOKEEPER/Zab1.0.
+ *)
+EXTENDS FastLeaderElection
+
+-----------------------------------------------------------------------------
+(* Defined in FastLeaderElection.tla:
+\* The set of server identifiers
+CONSTANT Server
+
+\* Server states
+CONSTANTS LOOKING, FOLLOWING, LEADING
+
+\* Message types
+CONSTANTS NOTIFICATION
+
+\* Timeout signal
+CONSTANT NONE
+*)
+\* The set of requests that can go into history
+CONSTANT Value
+
+\* Zab states
+CONSTANTS ELECTION, DISCOVERY, SYNCHRONIZATION, BROADCAST
+
+\* Message types
+CONSTANTS FOLLOWERINFO, LEADERINFO, ACKEPOCH, NEWLEADER, ACKLD, UPTODATE, PROPOSAL, ACK, COMMIT
+(* Additional message types used for recovery in synchronization(TRUNC/DIFF/SNAP) are not needed
+   since we abstract this part.(see action RECOVERYSYNC) *)
+   
+-----------------------------------------------------------------------------
+(* Defined in FastLeaderElection.tla: Quorums, NullPoint *)
+\* Return the maximum value from the set S
+Maximum(S) == IF S = {} THEN -1
+                        ELSE CHOOSE n \in S: \A m \in S: n >= m
+
+\* Return the minimum value from the set S
+Minimum(S) == IF S = {} THEN -1
+                        ELSE CHOOSE n \in S: \A m \in S: n <= m
+
+MAXEPOCH == 10
+
+-----------------------------------------------------------------------------
+(* Defined in FastLeaderElection.tla:
+   serverVars: <<state, currentEpoch, lastZxid>>,
+   electionVars: <<currentVote, logicalClock, receiveVotes, outOfElection, recvQueue, waitNotmsg>>,
+   leaderVars: <<leadingVoteSet>>,
+   electionMsgs,
+   idTable *)
+\* The current phase of server(ELECTION,DISCOVERY,SYNCHRONIZATION,BROADCAST)
+VARIABLE zabState
+
+\* The epoch number of the last NEWEPOCH(LEADERINFO) packet accepted
+\* namely f.p in paper, and currentEpoch in Zab.tla.
+VARIABLE acceptedEpoch
+
+\* The history of servers as the sequence of transactions.
+VARIABLE history
+
+\* commitIndex[i]: The maximum index of transactions that have been saved in a quorum of servers
+\*                 in the perspective of server i.(increases monotonically before restarting)
+VARIABLE commitIndex
+
+(* These transactions whose index \le commitIndex[i] can be applied to state machine immediately.
+   So if we have a variable applyIndex, we can suppose that applyIndex[i] = commitIndex[i] when verifying properties.
+   But in phase SYNC, follower will apply all queued proposals to state machine when receiving NEWLEADER.
+   But follower only serves traffic after receiving UPTODATE, so sequential consistency is not violated.
+   
+   So when we verify properties, we still suppose applyIndex[i] = commitIndex[i], because this is an engineering detail.*)
+
+\* learners[i]: The set of servers which leader i think are connected wich i.
+VARIABLE learners
+
+\* The messages representing requests and responses sent from one server to another.
+\* msgs[i][j] means the input buffer of server j from server i.
+VARIABLE msgs
+
+\* The set of followers who has successfully sent CEPOCH(FOLLOWERINFO) to leader.(equals to connectingFollowers in code)
+VARIABLE cepochRecv
+
+\* The set of followers who has successfully sent ACK-E to leader.(equals to electingFollowers in code)
+VARIABLE ackeRecv
+
+\* The set of followers who has successfully sent ACK-LD to leader in leader.(equals to newLeaderProposal in code)
+VARIABLE ackldRecv
+
+\* The set of servers which leader i broadcasts PROPOSAL and COMMIT to.(equals to forwardingFollowers in code)
+VARIABLE forwarding
+
+\* ackIndex[i][j]: The latest index that leader i has received from follower j via ACK.
+VARIABLE ackIndex
+
+\* currentCounter[i]: The count of transactions that clients request leader i.
+VARIABLE currentCounter
+
+\* sendCounter[i]: The count of transactions that leader i has broadcast in PROPOSAL.
+VARIABLE sendCounter
+
+\* committedIndex[i]: The maximum index of trasactions that leader i has broadcast in COMMIT.
+VARIABLE committedIndex
+
+\* committedCounter[i][j]: The latest counter of transaction that leader i has confirmed that follower j has committed.
+VARIABLE committedCounter
+
+\* initialHistory[i]: The initial history if leader i in epoch acceptedEpoch[i].
+VARIABLE initialHistory
+
+\* the maximum epoch in CEPOCH the prospective leader received from followers.
+VARIABLE tempMaxEpoch
+
+\* cepochSent[i] = TRUE means follower i has sent CEPOCH(FOLLOWERINFO) to leader.
+VARIABLE cepochSent
+
+\* leaderAddr[i]: The leader id of follower i. We use leaderAddr to express whether follower i has connected or lost connection.
+VARIABLE leaderAddr
+
+\* synced[i] = TRUE: follower i has completed sync with leader.
+VARIABLE synced
+
+\* The set of leaders in every epoch, only used in verifying properties.
+VARIABLE epochLeader
+
+\* The set of all broadcast messages, only used in verifying properties.
+VARIABLE proposalMsgsLog
+
+\* A variable used to check whether there are conditions contrary to the facts.
+VARIABLE inherentViolated
+
+serverVarsZ == <<state, currentEpoch, lastZxid, zabState, acceptedEpoch, history, commitIndex>>         \* 7 variables
+
+electionVarsZ == electionVars  \* 6 variables
+
+leaderVarsZ == <<leadingVoteSet, learners, cepochRecv, ackeRecv, ackldRecv, forwarding, 
+                 ackIndex, currentCounter, sendCounter, committedIndex, committedCounter>>              \* 11 variables
+
+tempVarsZ == <<initialHistory, tempMaxEpoch>>     \* 2 variables
+  
+followerVarsZ == <<cepochSent, leaderAddr, synced>>                 \* 3 variables
+
+verifyVarsZ == <<proposalMsgsLog, epochLeader, inherentViolated>>   \* 3 variables
+
+msgVarsZ == <<msgs, electionMsgs>>                \* 2 variables
+
+vars == <<serverVarsZ, electionVarsZ, leaderVarsZ, tempVarsZ, followerVarsZ, verifyVarsZ, msgVarsZ, idTable>>  \* 35 variables
+-----------------------------------------------------------------------------
+\* Add a message to msgs - add a message m to msgs[i][j].
+Send(i, j, m) == msgs' = [msgs EXCEPT ![i][j] = Append(msgs[i][j], m)]
+
+\* Remove a message from msgs - discard head of msgs[i][j].
+Discard(i, j) == msgs' = IF msgs[i][j] /= << >> THEN [msgs EXCEPT ![i][j] = Tail(msgs[i][j])]
+                                                ELSE msgs
+
+\* Leader broadcasts a message(PROPOSAL/COMMIT) to all other servers in forwardingFollowers.
+Broadcast(i, m) == msgs' = [msgs EXCEPT ![i] = [v \in Server |-> IF /\ v \in forwarding[i]
+                                                                    /\ v /= i
+                                                                    /\ \/ /\ m.mtype = PROPOSAL
+                                                                          /\ ackIndex[i][v] < Len(initialHistory[i]) + m.mproposal.counter
+                                                                       \/ /\ m.mtype = COMMIT
+                                                                          /\ committedCounter[i][v] < m.mzxid[2]
+                                                                  THEN Append(msgs[i][v], m)
+                                                                  ELSE msgs[i][v]]]
+
+BroadcastLEADERINFO(i, m) == msgs' = [msgs EXCEPT ![i] = [v \in Server |-> IF /\ v \in cepochRecv[i]
+                                                                              /\ v \in learners[i]
+                                                                              /\ v /= i THEN Append(msgs[i][v], m)
+                                                                                        ELSE msgs[i][v]]]
+
+BroadcastUPTODATE(i, m) == msgs' = [msgs EXCEPT ![i] = [v \in Server |-> IF /\ v \in ackldRecv[i]
+                                                                            /\ v \in learners[i]
+                                                                            /\ v /= i THEN Append(msgs[i][v], m)
+                                                                                      ELSE msgs[i][v]]]
+
+\* Combination of Send and Discard - discard head of msgs[j][i] and add m into msgs[i][j].
+Reply(i, j, m) == msgs' = [msgs EXCEPT ![j][i] = Tail(msgs[j][i]),
+                                       ![i][j] = Append(msgs[i][j], m)]
+
+\* shuffle the input buffer from server j(i) in server i(j).
+Clean(i, j) == msgs' = [msgs EXCEPT ![j][i] = << >>, ![i][j] = << >>]
+
+-----------------------------------------------------------------------------
+PZxidEqual(p, z) == p.epoch = z[1] /\ p.counter = z[2]
+
+TransactionEqual(t1, t2) == /\ t1.epoch   = t2.epoch
+                            /\ t1.counter = t2.counter
+                            
+TransactionPrecede(t1, t2) == \/ t1.epoch < t2.epoch
+                              \/ /\ t1.epoch   = t2.epoch
+                                 /\ t1.counter < t2.counter
+                                 
+-----------------------------------------------------------------------------
+\* Define initial values for all variables 
+InitServerVarsZ == /\ InitServerVars
+                   /\ zabState      = [s \in Server |-> ELECTION]
+                   /\ acceptedEpoch = [s \in Server |-> 0]
+                   /\ history       = [s \in Server |-> << >>]
+                   /\ commitIndex   = [s \in Server |-> 0]
+
+InitLeaderVarsZ == /\ InitLeaderVars
+                   /\ learners         = [s \in Server |-> {}]
+                   /\ cepochRecv       = [s \in Server |-> {}]
+                   /\ ackeRecv         = [s \in Server |-> {}]
+                   /\ ackldRecv        = [s \in Server |-> {}]
+                   /\ ackIndex         = [s \in Server |-> [v \in Server |-> 0]]
+                   /\ currentCounter   = [s \in Server |-> 0]
+                   /\ sendCounter      = [s \in Server |-> 0]
+                   /\ committedIndex   = [s \in Server |-> 0]
+                   /\ committedCounter = [s \in Server |-> [v \in Server |-> 0]]
+                   /\ forwarding       = [s \in Server |-> {}]
+
+InitElectionVarsZ == InitElectionVars
+
+InitTempVarsZ == /\ initialHistory = [s \in Server |-> << >>]
+                 /\ tempMaxEpoch   = [s \in Server |-> 0]
+
+InitFollowerVarsZ == /\ cepochSent = [s \in Server |-> FALSE]
+                     /\ leaderAddr = [s \in Server |-> NullPoint]
+                     /\ synced     = [s \in Server |-> FALSE]
+
+InitVerifyVarsZ == /\ proposalMsgsLog  = {}
+                   /\ epochLeader      = [i \in 1..MAXEPOCH |-> {}]
+                   /\ inherentViolated = FALSE
+                   
+InitMsgVarsZ == /\ msgs         = [s \in Server |-> [v \in Server |-> << >>]]
+                /\ electionMsgs = [s \in Server |-> [v \in Server |-> << >>]]
+
+InitZ == /\ InitServerVarsZ
+         /\ InitLeaderVarsZ
+         /\ InitElectionVarsZ
+         /\ InitTempVarsZ
+         /\ InitFollowerVarsZ
+         /\ InitVerifyVarsZ
+         /\ InitMsgVarsZ
+         /\ idTable = InitializeIdTable(Server)
+         
+-----------------------------------------------------------------------------
+ZabTurnToLeading(i) ==
+        /\ zabState'       = [zabState   EXCEPT ![i] = DISCOVERY]
+        /\ learners'       = [learners   EXCEPT ![i] = {i}]
+        /\ cepochRecv'     = [cepochRecv EXCEPT ![i] = {i}]
+        /\ ackeRecv'       = [ackeRecv   EXCEPT ![i] = {i}]
+        /\ ackldRecv'      = [ackldRecv  EXCEPT ![i] = {i}]
+        /\ forwarding'     = [forwarding EXCEPT ![i] = {}]
+        /\ ackIndex'       = [ackIndex   EXCEPT ![i] = [v \in Server |-> IF v = i THEN Len(history[i])
+                                                                                  ELSE 0]]
+        /\ currentCounter'   = [currentCounter   EXCEPT ![i] = 0]
+        /\ sendCounter'      = [sendCounter      EXCEPT ![i] = 0]
+        /\ commitIndex'      = [commitIndex      EXCEPT ![i] = 0]
+        /\ committedIndex'   = [committedIndex   EXCEPT ![i] = 0]
+        /\ committedCounter' = [committedCounter EXCEPT ![i] = [v \in Server |-> IF v = i THEN Len(history[i])
+                                                                                          ELSE 0]]
+        /\ initialHistory' = [initialHistory EXCEPT ![i] = history[i]]
+        /\ tempMaxEpoch'   = [tempMaxEpoch   EXCEPT ![i] = acceptedEpoch[i]]
+
+ZabTurnToFollowing(i) ==
+        /\ zabState'    = [zabState   EXCEPT ![i] = DISCOVERY]
+        /\ cepochSent'  = [cepochSent EXCEPT ![i] = FALSE]
+        /\ synced'      = [synced     EXCEPT ![i] = FALSE]
+        /\ commitIndex' = [commitIndex      EXCEPT ![i] = 0]
+        
+          
+(* Fast Leader Election *)
+FLEReceiveNotmsg(i, j) ==
+        /\ ReceiveNotmsg(i, j)
+        /\ UNCHANGED <<zabState, acceptedEpoch, history, commitIndex,learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter,
+                       sendCounter, committedIndex, committedCounter, tempVarsZ, followerVarsZ, verifyVarsZ, msgs>>
+
+FLENotmsgTimeout(i) ==
+        /\ NotmsgTimeout(i)
+        /\ UNCHANGED <<zabState, acceptedEpoch, history, commitIndex,learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter,
+                       sendCounter, committedIndex, committedCounter, tempVarsZ, followerVarsZ, verifyVarsZ, msgs>>
+
+FLEHandleNotmsg(i) ==
+        /\ HandleNotmsg(i)
+        /\ LET newState == state'[i]
+           IN
+           \/ /\ newState = LEADING
+              /\ ZabTurnToLeading(i)
+              /\ UNCHANGED <<cepochSent, synced>>
+           \/ /\ newState = FOLLOWING
+              /\ ZabTurnToFollowing(i)
+              /\ UNCHANGED <<learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter, sendCounter, committedIndex, committedCounter, tempVarsZ>>
+           \/ /\ newState = LOOKING
+              /\ UNCHANGED <<zabState, learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter, sendCounter, commitIndex,
+                             committedIndex, committedCounter, tempVarsZ, cepochSent, synced>>
+        /\ UNCHANGED <<acceptedEpoch, history, leaderAddr, verifyVarsZ, msgs>>
+
+\* On the premise that ReceiveVotes.HasQuorums = TRUE, corresponding to logic in line 1050-1055 in LFE.java.
+FLEWaitNewNotmsg(i) ==
+        /\ WaitNewNotmsg(i)
+        /\ UNCHANGED <<zabState, acceptedEpoch, history, commitIndex,learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter,
+                       sendCounter, committedIndex, committedCounter, tempVarsZ, followerVarsZ, verifyVarsZ, msgs>>
+
+\* On the premise that ReceiveVotes.HasQuorums = TRUE, corresponding to logic in line 1061-1066 in LFE.java.
+FLEWaitNewNotmsgEnd(i) ==
+        /\ WaitNewNotmsgEnd(i)
+        /\ LET newState == state'[i]
+           IN
+           \/ /\ newState = LEADING
+              /\ ZabTurnToLeading(i)
+              /\ UNCHANGED <<cepochSent, synced>>
+           \/ /\ newState = FOLLOWING
+              /\ ZabTurnToFollowing(i)
+              /\ UNCHANGED <<learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter, sendCounter, committedIndex, committedCounter, tempVarsZ>>
+           \/ /\ newState = LOOKING
+              /\ PrintT("New state is LOOKING in FLEWaitNewNotmsgEnd, which should not happen.")
+              /\ UNCHANGED <<zabState, learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter, sendCounter, commitIndex,
+                             committedIndex, committedCounter, tempVarsZ, cepochSent, synced>>
+        /\ UNCHANGED <<acceptedEpoch, history, leaderAddr, verifyVarsZ, msgs>>
+              
+-----------------------------------------------------------------------------
+(* A sub-action describing how a server transitions from LEADING/FOLLOWING to LOOKING.
+   Initially I call it 'ZabTimeoutZ', but it will be called not only when timeout, 
+   but also when finding a low epoch from leader.*)
+FollowerShutdown(i) ==
+        /\ ZabTimeout(i)
+        /\ zabState'   = [zabState   EXCEPT ![i] = ELECTION]
+        /\ leaderAddr' = [leaderAddr EXCEPT ![i] = NullPoint]
+
+LeaderShutdown(i) ==
+        /\ ZabTimeout(i)
+        /\ zabState'   = [zabState   EXCEPT ![i] = ELECTION]
+        /\ leaderAddr' = [s \in Server |-> IF s \in learners[i] THEN NullPoint ELSE leaderAddr[s]]
+        /\ learners'   = [learners   EXCEPT ![i] = {}]
+        /\ forwarding' = [forwarding EXCEPT ![i] = {}]
+        /\ msgs'       = [s \in Server |-> [v \in Server |-> IF v \in learners[i] \/ s \in learners[i] THEN << >> ELSE msgs[s][v]]]
+
+FollowerTimout(i) ==
+        /\ state[i]      = FOLLOWING
+        /\ leaderAddr[i] = NullPoint
+        /\ FollowerShutdown(i)
+        /\ msgs' = [s \in Server |-> [v \in Server |-> IF v = i THEN << >> ELSE msgs[s][v]]]
+        /\ UNCHANGED <<acceptedEpoch, history, commitIndex, learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, 
+                       currentCounter, sendCounter, committedIndex, committedCounter, tempVarsZ, cepochSent, synced, verifyVarsZ>>
+
+LeaderTimeout(i) ==
+        /\ state[i] = LEADING
+        /\ learners[i] \notin Quorums
+        /\ LeaderShutdown(i)
+        /\ UNCHANGED <<acceptedEpoch, history, commitIndex, cepochRecv, ackeRecv, ackldRecv, ackIndex, currentCounter, sendCounter, committedIndex, committedCounter,
+                       tempVarsZ, cepochSent, synced, verifyVarsZ>>
+    
+-----------------------------------------------------------------------------
+(* Establish connection between leader i and follower j. 
+   It means i creates a learnerHandler for communicating with j, and j finds i's address.*)
+EstablishConnection(i, j) ==
+        /\ state[i] = LEADING   /\ state[j] = FOLLOWING
+        /\ j \notin learners[i] /\ leaderAddr[j] = NullPoint
+        /\ currentVote[j].proposedLeader = i
+        /\ learners'   = [learners   EXCEPT ![i] = learners[i] \union {j}] \* Leader:   'addLearnerHandler(peer)'
+        /\ leaderAddr' = [leaderAddr EXCEPT ![j] = i]                      \* Follower: 'connectToLeader(addr, hostname)'
+        /\ UNCHANGED <<serverVarsZ, electionVarsZ, leadingVoteSet, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, 
+                       currentCounter, sendCounter, committedIndex, committedCounter, tempVarsZ, cepochSent, synced, verifyVarsZ, msgVarsZ, idTable>>
+        
+(* The leader i finds timeout and TCP connection between i and j closes.*)       
+Timeout(i, j) ==
+        /\ state[i] = LEADING /\ state[j] = FOLLOWING
+        /\ j \in learners[i]  /\ leaderAddr[j] = i
+        (* The action of leader i.(corresponding to function 'removeLearnerHandler(peer)'.) *)
+        /\ learners'   = [learners   EXCEPT ![i] = learners[i] \ {j}] 
+        /\ forwarding' = [forwarding EXCEPT ![i] = IF j \in forwarding[i] THEN forwarding[i] \ {j} ELSE forwarding[i]]
+        /\ cepochRecv' = [cepochRecv EXCEPT ![i] = IF j \in cepochRecv[i] THEN cepochRecv[i] \ {j} ELSE cepochRecv[i]]
+        /\ ackeRecv'   = [ackeRecv   EXCEPT ![i] = IF j \in ackeRecv[i] THEN ackeRecv[i] \ {j} ELSE ackeRecv[i]]
+        /\ ackldRecv'  = [ackldRecv  EXCEPT ![i] = IF j \in ackldRecv[i] THEN ackldRecv[i] \ {j} ELSE ackldRecv[i]]
+        /\ ackIndex'   = [ackIndex   EXCEPT ![i][j] = 0]
+        /\ committedCounter' = [committedCounter EXCEPT ![i][j] = 0]
+        (* The action of follower j. *)
+        /\ FollowerShutdown(j)
+        (* Clean input buffer.*)
+        /\ Clean(i, j)
+        /\ UNCHANGED <<acceptedEpoch, history, commitIndex, currentCounter, sendCounter, committedIndex,
+                       tempVarsZ, cepochSent, synced, verifyVarsZ>>
+-----------------------------------------------------------------------------
+\* In phase f11, follower sends f.p to leader via FOLLOWERINFO(CEPOCH).
+FollowerSendFOLLOWERINFO(i) ==
+        /\ state[i]    = FOLLOWING
+        /\ zabState[i] = DISCOVERY
+        /\ leaderAddr[i] /= NullPoint
+        /\ \lnot cepochSent[i]
+        /\ Send(i, leaderAddr[i], [mtype  |-> FOLLOWERINFO,
+                                   mepoch |-> acceptedEpoch[i]])
+        /\ cepochSent' = [cepochSent EXCEPT ![i] = TRUE]
+        /\ UNCHANGED <<serverVarsZ, leaderVarsZ, electionVarsZ, tempVarsZ, leaderAddr, synced, verifyVarsZ, electionMsgs, idTable>>
+        
+(* In phase l11, leader waits for receiving FOLLOWERINFO from a quorum,
+   and then chooses a new epoch e' as its own epoch and broadcasts LEADERINFO. *)
+LeaderHandleFOLLOWERINFO(i, j) ==
+        /\ state[i] = LEADING
+        /\ msgs[j][i] /= << >>
+        /\ msgs[j][i][1].mtype = FOLLOWERINFO
+        /\ LET msg == msgs[j][i][1]
+           IN \/ /\ NullPoint \notin cepochRecv[i]   \* 1. has not broadcast LEADERINFO - modify tempMaxEpoch
+                 /\ LET newEpoch == Maximum({tempMaxEpoch[i], msg.mepoch})
+                    IN tempMaxEpoch' = [tempMaxEpoch EXCEPT ![i] = newEpoch]
+                 /\ Discard(j, i)
+              \/ /\ NullPoint \in cepochRecv[i]      \* 2. has broadcast LEADERINFO - no need to handle the msg, just send LEADERINFO to corresponding server
+                 /\ Reply(i, j, [mtype  |-> LEADERINFO,
+                                 mepoch |-> acceptedEpoch[i]])
+                 /\ UNCHANGED tempMaxEpoch
+        /\ cepochRecv' = [cepochRecv EXCEPT ![i] = IF j \in cepochRecv[i] THEN cepochRecv[i]
+                                                                          ELSE cepochRecv[i] \union {j}]
+        /\ UNCHANGED <<serverVarsZ, followerVarsZ, electionVarsZ, initialHistory, leadingVoteSet, learners, ackeRecv, ackldRecv, 
+                       forwarding, ackIndex, currentCounter, sendCounter, committedIndex, committedCounter, verifyVarsZ, electionMsgs, idTable>>
+
+LeaderDiscovery1(i) ==
+        /\ state[i]    = LEADING
+        /\ zabState[i] = DISCOVERY
+        /\ cepochRecv[i] \in Quorums
+        /\ acceptedEpoch' = [acceptedEpoch EXCEPT ![i] = tempMaxEpoch[i] + 1]
+        /\ cepochRecv'    = [cepochRecv    EXCEPT ![i] = cepochRecv[i] \union {NullPoint}]
+        /\ BroadcastLEADERINFO(i, [mtype  |-> LEADERINFO,
+                                   mepoch |-> acceptedEpoch'[i]])
+        /\ UNCHANGED <<state, currentEpoch, lastZxid, zabState, history, commitIndex, electionVarsZ, leadingVoteSet, learners, ackeRecv, ackldRecv, 
+                       forwarding, ackIndex, currentCounter, sendCounter, committedIndex, committedCounter, 
+                       tempVarsZ, followerVarsZ, verifyVarsZ, electionMsgs, idTable>>
+
+(* In phase f12, follower receives NEWEPOCH. If e' > f.p, then follower sends ACK-E back,
+   and ACK-E contains f.a and lastZxid to let leader judge whether it is the latest.
+   After handling NEWEPOCH, follower's zabState turns to SYNCHRONIZATION. *)
+FollowerHandleLEADERINFO(i, j) ==
+        /\ state[i] = FOLLOWING
+        /\ msgs[j][i] /= << >>
+        /\ msgs[j][i][1].mtype = LEADERINFO
+        /\ LET msg     == msgs[j][i][1]
+               infoOk  == j = leaderAddr[i]
+               epochOk == /\ infoOk
+                          /\ msg.mepoch >= acceptedEpoch[i]
+               correct == /\ epochOk
+                          /\ zabState[i] = DISCOVERY
+           IN /\ infoOk
+              /\ \/ /\ epochOk                      \* 1. Normal case
+                    /\ \/ /\ correct
+                          /\ acceptedEpoch' = [acceptedEpoch EXCEPT ![i] = msg.mepoch]
+                          /\ Reply(i, j, [mtype      |-> ACKEPOCH,
+                                          mepoch     |-> msg.mepoch,
+                                          mlastEpoch |-> currentEpoch[i],
+                                          mlastZxid  |-> lastZxid[i]])
+                          /\ cepochSent' = [cepochSent EXCEPT ![i] = TRUE]
+                          /\ UNCHANGED inherentViolated
+                       \/ /\ ~correct
+                          /\ PrintT("Exception: Condition correct is false in FollowerHandleLEADERINFO(" \o ToString(i) \o ", " \o ToString(j) \o ").")
+                          /\ inherentViolated' = TRUE
+                          /\ Discard(j, i)
+                          /\ UNCHANGED <<acceptedEpoch, cepochSent>>
+                    /\ zabState' = [zabState EXCEPT ![i] = IF zabState[i] = DISCOVERY THEN SYNCHRONIZATION
+                                                                                      ELSE zabState[i]]
+                    /\ UNCHANGED <<varsL, leaderAddr>>
+                 \/ /\ ~epochOk                    \* 2. Abnormal case - go back to election
+                    /\ FollowerShutdown(i)
+                    /\ Clean(i, j)
+                    /\ UNCHANGED <<acceptedEpoch, cepochSent, inherentViolated>>
+        /\ UNCHANGED <<history, commitIndex, learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter, sendCounter, committedIndex,
+                       committedCounter, tempVarsZ, synced, proposalMsgsLog, epochLeader>>
+
+\* Abstraction of actions making follower synced with leader before leader sending NEWLEADER.
+subRECOVERYSYNC(i, j) ==
+        LET canSync == /\ state[i] = LEADING   /\ zabState[i] /= DISCOVERY      /\ j \in learners[i]  /\ j \in ackeRecv[i]
+                       /\ state[j] = FOLLOWING /\ zabState[j] = SYNCHRONIZATION /\ leaderAddr[j] = i  /\ synced[j] = FALSE
+        IN
+        \/ /\ canSync
+           /\ history'     = [history     EXCEPT ![j] = history[i]]
+           /\ lastZxid'    = [lastZxid    EXCEPT ![j] = lastZxid[i]]
+           /\ UpdateProposal(j, leaderAddr[j], lastZxid'[j], currentEpoch[j])
+           /\ commitIndex' = [commitIndex EXCEPT ![j] = commitIndex[i]]
+           /\ synced'      = [synced      EXCEPT ![j] = TRUE]
+           /\ forwarding'  = [forwarding  EXCEPT ![i] = forwarding[i] \union {j}]    \* j will receive PROPOSAL and COMMIT
+           /\ ackIndex'    = [ackIndex    EXCEPT ![i][j] = Len(history[i])]
+           /\ committedCounter' = [committedCounter EXCEPT ![i][j] = Maximum({commitIndex[i] - Len(initialHistory[i]), 0})]
+           /\ LET ms == [msource|->i, mtype|->"RECOVERYSYNC", mepoch|->acceptedEpoch[i], mproposals|->history[i]]
+              IN proposalMsgsLog' = IF ms \in proposalMsgsLog THEN proposalMsgsLog
+                                                              ELSE proposalMsgsLog \union {ms}
+           /\ Reply(i, j, [mtype     |-> NEWLEADER,
+                           mepoch    |-> acceptedEpoch[i],
+                           mlastZxid |-> lastZxid[i]])
+        \/ /\ ~canSync
+           /\ Discard(j, i)
+           /\ UNCHANGED <<history, lastZxid, currentVote, commitIndex, synced, forwarding, ackIndex, committedCounter, proposalMsgsLog>>
+        
+(* In phase l12, leader waits for receiving ACKEPOPCH from a quorum,
+   and check whether it has the latest history and epoch from them.
+   If so, leader's zabState turns to SYNCHRONIZATION. *)
+LeaderHandleACKEPOCH(i, j) ==
+        /\ state[i] = LEADING
+        /\ msgs[j][i] /= << >>
+        /\ msgs[j][i][1].mtype = ACKEPOCH
+        /\ LET msg == msgs[j][i][1]
+               infoOk == /\ j \in learners[i]            
+                         /\ acceptedEpoch[i] = msg.mepoch
+               logOk  == /\ infoOk                          \* logOk = TRUE means leader is more up-to-date than follower j
+                         /\ \/ currentEpoch[i] > msg.mlastEpoch
+                            \/ /\ currentEpoch[i] = msg.mlastEpoch
+                               /\ \/ lastZxid[i][1] > msg.mlastZxid[1]
+                                  \/ /\ lastZxid[i][1] = msg.mlastZxid[1]
+                                     /\ lastZxid[i][2] >= msg.mlastZxid[2]
+               replyOk == /\ infoOk
+                          /\ NullPoint \in ackeRecv[i]
+           IN /\ infoOk
+              /\ \/ /\ replyOk
+                    /\ subRECOVERYSYNC(i, j)
+                    /\ ackeRecv' = [ackeRecv EXCEPT ![i] = IF j \notin ackeRecv[i] THEN ackeRecv[i] \union {j}
+                                                                                   ELSE ackeRecv[i]]
+                    /\ UNCHANGED <<state, currentEpoch,logicalClock, receiveVotes, outOfElection, recvQueue, waitNotmsg, leadingVoteSet, electionMsgs, idTable,
+                                   zabState, leaderAddr, learners>>
+                 \/ /\ ~replyOk
+                    /\ \/ /\ logOk
+                          /\ ackeRecv' = [ackeRecv EXCEPT ![i] = IF j \notin ackeRecv[i] THEN ackeRecv[i] \union {j}
+                                                                                         ELSE ackeRecv[i]]
+                          /\ Discard(j, i)
+                          /\ UNCHANGED <<varsL, zabState, leaderAddr, learners, forwarding>>
+                       \/ /\ ~logOk            \* go back to election
+                          /\ LeaderShutdown(i)
+                          /\ UNCHANGED ackeRecv
+                    /\ UNCHANGED <<history, commitIndex, synced, forwarding, ackIndex, committedCounter, proposalMsgsLog>>
+        /\ UNCHANGED <<acceptedEpoch, cepochRecv, ackldRecv,currentCounter, sendCounter, committedIndex, tempVarsZ, cepochSent, epochLeader, inherentViolated>>
+                          
+LeaderDiscovery2(i) ==

Review comment:
       Action name is not descriptive, perhaps rename to something like LeaderTransitionsToSynchronization




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@zookeeper.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] Vanlightly commented on a change in pull request #1690: ZOOKEEPER-3615: Provide formal specification and verification using TLA+ for Zab

Posted by GitBox <gi...@apache.org>.
Vanlightly commented on a change in pull request #1690:
URL: https://github.com/apache/zookeeper/pull/1690#discussion_r699459102



##########
File path: zookeeper-specifications/zab-1.0/FastLeaderElection.tla
##########
@@ -0,0 +1,518 @@
+(*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *)
+------------------------- MODULE FastLeaderElection -------------------------
+\* This is the formal specification for Fast Leader Election in Zab protocol.
+(* Reference:
+   FastLeaderElection.java, Vote.java, QuorumPeer.java in https://github.com/apache/zookeeper.
+   Medeiros A. ZooKeeper��s atomic broadcast protocol: Theory and practice[J]. Aalto University School of Science, 2012.
+*)
+EXTENDS Integers, FiniteSets, Sequences, Naturals, TLC
+
+-----------------------------------------------------------------------------
+\* The set of server identifiers
+CONSTANT Server
+
+\* Server states
+CONSTANTS LOOKING, FOLLOWING, LEADING
+
+\* Message types
+CONSTANTS NOTIFICATION
+
+\* Timeout signal
+CONSTANT NONE
+-----------------------------------------------------------------------------
+Quorums == {Q \in SUBSET Server: Cardinality(Q)*2 > Cardinality(Server)}
+
+NullPoint == CHOOSE p: p \notin Server
+
+-----------------------------------------------------------------------------
+\* Server's state(LOOKING, FOLLOWING, LEADING).
+VARIABLE state
+
+\* The epoch number of the last NEWLEADER packet accepted, used for comparing.
+VARIABLE currentEpoch
+
+\* The zxid of the last transaction in history.
+VARIABLE lastZxid
+
+\* currentVote[i]: The server who i thinks is the current leader(id,zxid,peerEpoch,...).
+VARIABLE currentVote
+
+\* Election instance.(logicalClock in code)
+VARIABLE logicalClock
+
+\* The votes from the current leader election are stored in ReceiveVotes.
+VARIABLE receiveVotes
+
+(* The votes from previous leader elections, as well as the votes from the current leader election are
+   stored in outofelection. Note that notifications in a LOOKING state are not stored in outofelection.
+   Only FOLLOWING or LEADING notifications are stored in outofelection.  *)
+VARIABLE outOfElection
+
+\* recvQueue[i]: The queue of received notifications or timeout signals in server i.
+VARIABLE recvQueue
+
+\* A veriable to wait for new notifications, corresponding to line 1050 in FastLeaderElection.java.
+VARIABLE waitNotmsg
+
+\* leadingVoteSet[i]: The set of voters that follow i.
+VARIABLE leadingVoteSet
+
+(* The messages about election sent from one server to another.
+   electionMsgs[i][j] means the input buffer of server j from server i. *)
+VARIABLE electionMsgs
+
+\* Set used for mapping Server to Integers, to compare ids from different servers.
+VARIABLE idTable
+
+serverVars == <<state, currentEpoch, lastZxid>>
+
+electionVars == <<currentVote, logicalClock, receiveVotes, outOfElection, recvQueue, waitNotmsg>>
+
+leaderVars == <<leadingVoteSet>>
+
+varsL == <<serverVars, electionVars, leaderVars, electionMsgs, idTable>>
+-----------------------------------------------------------------------------
+\* Processing of electionMsgs
+BroadcastNotmsg(i, m) == electionMsgs' = [electionMsgs EXCEPT ![i] = [v \in Server |-> IF v /= i
+                                                                                       THEN Append(electionMsgs[i][v], m)
+                                                                                       ELSE electionMsgs[i][v]]]
+
+DiscardNotmsg(i, j) == electionMsgs' = [electionMsgs EXCEPT ![i][j] = IF electionMsgs[i][j] /= << >>
+                                                                      THEN Tail(electionMsgs[i][j])
+                                                                      ELSE << >>]
+
+ReplyNotmsg(i, j, m) == electionMsgs' = [electionMsgs EXCEPT ![i][j] = Append(electionMsgs[i][j], m),
+                                                             ![j][i] = Tail(electionMsgs[j][i])]
+                                               
+-----------------------------------------------------------------------------
+\* Processing of recvQueue
+RECURSIVE RemoveNone(_)
+RemoveNone(seq) == CASE seq =  << >> -> << >>
+                   []   seq /= << >> -> IF Head(seq).mtype = NONE THEN RemoveNone(Tail(seq))
+                                                                  ELSE <<Head(seq)>> \o RemoveNone(Tail(seq)) 
+
+\* Processing of idTable and order comparing
+RECURSIVE InitializeIdTable(_)
+InitializeIdTable(Remaining) == IF Remaining = {} THEN {}
+                                ELSE LET chosen == CHOOSE i \in Remaining: TRUE
+                                         re     == Remaining \ {chosen}
+                                     IN {<<chosen, Cardinality(Remaining)>>} \union InitializeIdTable(re)
+                                     
+\* FALSE: id1 < id2; TRUE: id1 > id2
+IdCompare(id1,id2) == LET item1 == CHOOSE item \in idTable: item[1] = id1
+                          item2 == CHOOSE item \in idTable: item[1] = id2
+                      IN item1[2] > item2[2]
+
+\* FALSE: zxid1 <= zxid2; TRUE: zxid1 > zxid2
+ZxidCompare(zxid1, zxid2) == \/ zxid1[1] > zxid2[1]
+                             \/ /\ zxid1[1] = zxid2[1]
+                                /\ zxid1[2] > zxid2[2]
+
+ZxidEqual(zxid1, zxid2) == zxid1[1] = zxid2[1] /\ zxid1[2] = zxid2[2]
+
+\* FALSE: vote1 <= vote2; TRUE: vote1 > vote2
+TotalOrderPredicate(vote1, vote2) == \/ vote1.proposedEpoch > vote2.proposedEpoch
+                                     \/ /\ vote1.proposedEpoch = vote2.proposedEpoch
+                                        /\ \/ ZxidCompare(vote1.proposedZxid, vote2.proposedZxid)
+                                           \/ /\ ZxidEqual(vote1.proposedZxid, vote2.proposedZxid)
+                                              /\ IdCompare(vote1.proposedLeader, vote2.proposedLeader)
+
+VoteEqual(vote1, round1, vote2, round2) == /\ vote1.proposedLeader = vote2.proposedLeader
+                                           /\ ZxidEqual(vote1.proposedZxid, vote2.proposedZxid)
+                                           /\ vote1.proposedEpoch  = vote2.proposedEpoch
+                                           /\ round1 = round2
+
+-----------------------------------------------------------------------------
+\* Processing of currentVote
+InitialVote == [proposedLeader |-> NullPoint,
+                proposedZxid   |-> <<0, 0>>,
+                proposedEpoch  |-> 0]
+
+SelfVote(i) == [proposedLeader |-> i,
+                proposedZxid   |-> lastZxid[i],
+                proposedEpoch  |-> currentEpoch[i]]
+
+UpdateProposal(i, nid, nzxid, nepoch) == currentVote' = [currentVote EXCEPT ![i].proposedLeader = nid, \* no need to record state in LOOKING
+                                                                            ![i].proposedZxid   = nzxid,
+                                                                            ![i].proposedEpoch  = nepoch]  
+                                                                            
+-----------------------------------------------------------------------------
+\* Processing of receiveVotes and outOfElection
+RvClear(i) == receiveVotes'  = [receiveVotes  EXCEPT ![i] = [v \in Server |-> [vote    |-> InitialVote,
+                                                                               round   |-> 0,
+                                                                               state   |-> LOOKING,
+                                                                               version |-> 0]]]
+
+RvPut(i, id, mvote, mround, mstate) == receiveVotes' = CASE receiveVotes[i][id].round < mround -> [receiveVotes EXCEPT ![i][id].vote    = mvote,
+                                                                                                                       ![i][id].round   = mround,
+                                                                                                                       ![i][id].state   = mstate,
+                                                                                                                       ![i][id].version = 1]
+                                                       []   receiveVotes[i][id].round = mround -> [receiveVotes EXCEPT ![i][id].vote    = mvote,
+                                                                                                                       ![i][id].state   = mstate,
+                                                                                                                       ![i][id].version = @ + 1]
+                                                       []   receiveVotes[i][id].round > mround -> receiveVotes
+
+Put(i, id, rcvset, mvote, mround, mstate) == CASE rcvset[id].round < mround -> [rcvset EXCEPT ![id].vote    = mvote,
+                                                                                              ![id].round   = mround,
+                                                                                              ![id].state   = mstate,
+                                                                                              ![id].version = 1]
+                                             []   rcvset[id].round = mround -> [rcvset EXCEPT ![id].vote    = mvote,
+                                                                                              ![id].state   = mstate,
+                                                                                              ![id].version = @ + 1]
+                                             []   rcvset[id].round > mround -> rcvset
+
+RvClearAndPut(i, id, vote, round) == receiveVotes' = LET oneVote == [vote    |-> vote, 
+                                                                     round   |-> round, 
+                                                                     state   |-> LOOKING,
+                                                                     version |-> 1]
+                                                     IN [receiveVotes EXCEPT ![i] = [v \in Server |-> IF v = id THEN oneVote
+                                                                                                                ELSE [vote    |-> InitialVote,
+                                                                                                                      round   |-> 0,
+                                                                                                                      state   |-> LOOKING,
+                                                                                                                      version |-> 0]]]                     
+
+VoteSet(i, msource, rcvset, thisvote, thisround) == {msource} \union {s \in (Server \ {msource}): VoteEqual(rcvset[s].vote, 

Review comment:
       The argument i is not used in VoteSet body.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@zookeeper.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] Vanlightly commented on a change in pull request #1690: ZOOKEEPER-3615: Provide formal specification and verification using TLA+ for Zab

Posted by GitBox <gi...@apache.org>.
Vanlightly commented on a change in pull request #1690:
URL: https://github.com/apache/zookeeper/pull/1690#discussion_r705458236



##########
File path: zookeeper-specifications/zab-1.0/ZabWithFLE.tla
##########
@@ -0,0 +1,1020 @@
+(*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *)
+
+----------------------------- MODULE ZabWithFLE -----------------------------
+(* This is the formal specification for the Zab consensus algorithm,
+   which means Zookeeper Atomic Broadcast.*)
+
+(* Reference:
+   FLE: FastLeaderElection.java, Vote.java, QuorumPeer.java in https://github.com/apache/zookeeper.
+   ZAB: QuorumPeer.java, Learner.java, Follower.java, LearnerHandler.java, Leader.java 
+        in https://github.com/apache/zookeeper.
+        https://cwiki.apache.org/confluence/display/ZOOKEEPER/Zab1.0.
+ *)
+EXTENDS FastLeaderElection
+
+-----------------------------------------------------------------------------
+(* Defined in FastLeaderElection.tla:
+\* The set of server identifiers
+CONSTANT Server
+
+\* Server states
+CONSTANTS LOOKING, FOLLOWING, LEADING
+
+\* Message types
+CONSTANTS NOTIFICATION
+
+\* Timeout signal
+CONSTANT NONE
+*)
+\* The set of requests that can go into history
+CONSTANT Value
+
+\* Zab states
+CONSTANTS ELECTION, DISCOVERY, SYNCHRONIZATION, BROADCAST
+
+\* Message types
+CONSTANTS FOLLOWERINFO, LEADERINFO, ACKEPOCH, NEWLEADER, ACKLD, UPTODATE, PROPOSAL, ACK, COMMIT
+(* Additional message types used for recovery in synchronization(TRUNC/DIFF/SNAP) are not needed
+   since we abstract this part.(see action RECOVERYSYNC) *)
+   
+-----------------------------------------------------------------------------
+(* Defined in FastLeaderElection.tla: Quorums, NullPoint *)
+\* Return the maximum value from the set S
+Maximum(S) == IF S = {} THEN -1
+                        ELSE CHOOSE n \in S: \A m \in S: n >= m
+
+\* Return the minimum value from the set S
+Minimum(S) == IF S = {} THEN -1
+                        ELSE CHOOSE n \in S: \A m \in S: n <= m
+
+MAXEPOCH == 10
+
+-----------------------------------------------------------------------------
+(* Defined in FastLeaderElection.tla:
+   serverVars: <<state, currentEpoch, lastZxid>>,
+   electionVars: <<currentVote, logicalClock, receiveVotes, outOfElection, recvQueue, waitNotmsg>>,
+   leaderVars: <<leadingVoteSet>>,
+   electionMsgs,
+   idTable *)
+\* The current phase of server(ELECTION,DISCOVERY,SYNCHRONIZATION,BROADCAST)
+VARIABLE zabState
+
+\* The epoch number of the last NEWEPOCH(LEADERINFO) packet accepted
+\* namely f.p in paper, and currentEpoch in Zab.tla.
+VARIABLE acceptedEpoch
+
+\* The history of servers as the sequence of transactions.
+VARIABLE history
+
+\* commitIndex[i]: The maximum index of transactions that have been saved in a quorum of servers
+\*                 in the perspective of server i.(increases monotonically before restarting)
+VARIABLE commitIndex
+
+(* These transactions whose index \le commitIndex[i] can be applied to state machine immediately.
+   So if we have a variable applyIndex, we can suppose that applyIndex[i] = commitIndex[i] when verifying properties.
+   But in phase SYNC, follower will apply all queued proposals to state machine when receiving NEWLEADER.
+   But follower only serves traffic after receiving UPTODATE, so sequential consistency is not violated.
+   
+   So when we verify properties, we still suppose applyIndex[i] = commitIndex[i], because this is an engineering detail.*)
+
+\* learners[i]: The set of servers which leader i think are connected wich i.
+VARIABLE learners
+
+\* The messages representing requests and responses sent from one server to another.
+\* msgs[i][j] means the input buffer of server j from server i.
+VARIABLE msgs
+
+\* The set of followers who has successfully sent CEPOCH(FOLLOWERINFO) to leader.(equals to connectingFollowers in code)
+VARIABLE cepochRecv
+
+\* The set of followers who has successfully sent ACK-E to leader.(equals to electingFollowers in code)
+VARIABLE ackeRecv
+
+\* The set of followers who has successfully sent ACK-LD to leader in leader.(equals to newLeaderProposal in code)
+VARIABLE ackldRecv
+
+\* The set of servers which leader i broadcasts PROPOSAL and COMMIT to.(equals to forwardingFollowers in code)
+VARIABLE forwarding
+
+\* ackIndex[i][j]: The latest index that leader i has received from follower j via ACK.
+VARIABLE ackIndex
+
+\* currentCounter[i]: The count of transactions that clients request leader i.
+VARIABLE currentCounter
+
+\* sendCounter[i]: The count of transactions that leader i has broadcast in PROPOSAL.
+VARIABLE sendCounter
+
+\* committedIndex[i]: The maximum index of trasactions that leader i has broadcast in COMMIT.
+VARIABLE committedIndex
+
+\* committedCounter[i][j]: The latest counter of transaction that leader i has confirmed that follower j has committed.
+VARIABLE committedCounter
+
+\* initialHistory[i]: The initial history if leader i in epoch acceptedEpoch[i].
+VARIABLE initialHistory
+
+\* the maximum epoch in CEPOCH the prospective leader received from followers.
+VARIABLE tempMaxEpoch
+
+\* cepochSent[i] = TRUE means follower i has sent CEPOCH(FOLLOWERINFO) to leader.
+VARIABLE cepochSent
+
+\* leaderAddr[i]: The leader id of follower i. We use leaderAddr to express whether follower i has connected or lost connection.
+VARIABLE leaderAddr
+
+\* synced[i] = TRUE: follower i has completed sync with leader.
+VARIABLE synced
+
+\* The set of leaders in every epoch, only used in verifying properties.
+VARIABLE epochLeader
+
+\* The set of all broadcast messages, only used in verifying properties.
+VARIABLE proposalMsgsLog
+
+\* A variable used to check whether there are conditions contrary to the facts.
+VARIABLE inherentViolated
+
+serverVarsZ == <<state, currentEpoch, lastZxid, zabState, acceptedEpoch, history, commitIndex>>         \* 7 variables
+
+electionVarsZ == electionVars  \* 6 variables
+
+leaderVarsZ == <<leadingVoteSet, learners, cepochRecv, ackeRecv, ackldRecv, forwarding, 
+                 ackIndex, currentCounter, sendCounter, committedIndex, committedCounter>>              \* 11 variables
+
+tempVarsZ == <<initialHistory, tempMaxEpoch>>     \* 2 variables
+  
+followerVarsZ == <<cepochSent, leaderAddr, synced>>                 \* 3 variables
+
+verifyVarsZ == <<proposalMsgsLog, epochLeader, inherentViolated>>   \* 3 variables
+
+msgVarsZ == <<msgs, electionMsgs>>                \* 2 variables
+
+vars == <<serverVarsZ, electionVarsZ, leaderVarsZ, tempVarsZ, followerVarsZ, verifyVarsZ, msgVarsZ, idTable>>  \* 35 variables
+-----------------------------------------------------------------------------
+\* Add a message to msgs - add a message m to msgs[i][j].
+Send(i, j, m) == msgs' = [msgs EXCEPT ![i][j] = Append(msgs[i][j], m)]
+
+\* Remove a message from msgs - discard head of msgs[i][j].
+Discard(i, j) == msgs' = IF msgs[i][j] /= << >> THEN [msgs EXCEPT ![i][j] = Tail(msgs[i][j])]
+                                                ELSE msgs
+
+\* Leader broadcasts a message(PROPOSAL/COMMIT) to all other servers in forwardingFollowers.
+Broadcast(i, m) == msgs' = [msgs EXCEPT ![i] = [v \in Server |-> IF /\ v \in forwarding[i]
+                                                                    /\ v /= i
+                                                                    /\ \/ /\ m.mtype = PROPOSAL
+                                                                          /\ ackIndex[i][v] < Len(initialHistory[i]) + m.mproposal.counter
+                                                                       \/ /\ m.mtype = COMMIT
+                                                                          /\ committedCounter[i][v] < m.mzxid[2]
+                                                                  THEN Append(msgs[i][v], m)
+                                                                  ELSE msgs[i][v]]]
+
+BroadcastLEADERINFO(i, m) == msgs' = [msgs EXCEPT ![i] = [v \in Server |-> IF /\ v \in cepochRecv[i]
+                                                                              /\ v \in learners[i]
+                                                                              /\ v /= i THEN Append(msgs[i][v], m)
+                                                                                        ELSE msgs[i][v]]]
+
+BroadcastUPTODATE(i, m) == msgs' = [msgs EXCEPT ![i] = [v \in Server |-> IF /\ v \in ackldRecv[i]
+                                                                            /\ v \in learners[i]
+                                                                            /\ v /= i THEN Append(msgs[i][v], m)
+                                                                                      ELSE msgs[i][v]]]
+
+\* Combination of Send and Discard - discard head of msgs[j][i] and add m into msgs[i][j].
+Reply(i, j, m) == msgs' = [msgs EXCEPT ![j][i] = Tail(msgs[j][i]),
+                                       ![i][j] = Append(msgs[i][j], m)]
+
+\* shuffle the input buffer from server j(i) in server i(j).
+Clean(i, j) == msgs' = [msgs EXCEPT ![j][i] = << >>, ![i][j] = << >>]
+
+-----------------------------------------------------------------------------
+PZxidEqual(p, z) == p.epoch = z[1] /\ p.counter = z[2]
+
+TransactionEqual(t1, t2) == /\ t1.epoch   = t2.epoch
+                            /\ t1.counter = t2.counter
+                            
+TransactionPrecede(t1, t2) == \/ t1.epoch < t2.epoch
+                              \/ /\ t1.epoch   = t2.epoch
+                                 /\ t1.counter < t2.counter
+                                 
+-----------------------------------------------------------------------------
+\* Define initial values for all variables 
+InitServerVarsZ == /\ InitServerVars
+                   /\ zabState      = [s \in Server |-> ELECTION]
+                   /\ acceptedEpoch = [s \in Server |-> 0]
+                   /\ history       = [s \in Server |-> << >>]
+                   /\ commitIndex   = [s \in Server |-> 0]
+
+InitLeaderVarsZ == /\ InitLeaderVars
+                   /\ learners         = [s \in Server |-> {}]
+                   /\ cepochRecv       = [s \in Server |-> {}]
+                   /\ ackeRecv         = [s \in Server |-> {}]
+                   /\ ackldRecv        = [s \in Server |-> {}]
+                   /\ ackIndex         = [s \in Server |-> [v \in Server |-> 0]]
+                   /\ currentCounter   = [s \in Server |-> 0]
+                   /\ sendCounter      = [s \in Server |-> 0]
+                   /\ committedIndex   = [s \in Server |-> 0]
+                   /\ committedCounter = [s \in Server |-> [v \in Server |-> 0]]
+                   /\ forwarding       = [s \in Server |-> {}]
+
+InitElectionVarsZ == InitElectionVars
+
+InitTempVarsZ == /\ initialHistory = [s \in Server |-> << >>]
+                 /\ tempMaxEpoch   = [s \in Server |-> 0]
+
+InitFollowerVarsZ == /\ cepochSent = [s \in Server |-> FALSE]
+                     /\ leaderAddr = [s \in Server |-> NullPoint]
+                     /\ synced     = [s \in Server |-> FALSE]
+
+InitVerifyVarsZ == /\ proposalMsgsLog  = {}
+                   /\ epochLeader      = [i \in 1..MAXEPOCH |-> {}]
+                   /\ inherentViolated = FALSE
+                   
+InitMsgVarsZ == /\ msgs         = [s \in Server |-> [v \in Server |-> << >>]]
+                /\ electionMsgs = [s \in Server |-> [v \in Server |-> << >>]]
+
+InitZ == /\ InitServerVarsZ
+         /\ InitLeaderVarsZ
+         /\ InitElectionVarsZ
+         /\ InitTempVarsZ
+         /\ InitFollowerVarsZ
+         /\ InitVerifyVarsZ
+         /\ InitMsgVarsZ
+         /\ idTable = InitializeIdTable(Server)
+         
+-----------------------------------------------------------------------------
+ZabTurnToLeading(i) ==
+        /\ zabState'       = [zabState   EXCEPT ![i] = DISCOVERY]
+        /\ learners'       = [learners   EXCEPT ![i] = {i}]
+        /\ cepochRecv'     = [cepochRecv EXCEPT ![i] = {i}]
+        /\ ackeRecv'       = [ackeRecv   EXCEPT ![i] = {i}]
+        /\ ackldRecv'      = [ackldRecv  EXCEPT ![i] = {i}]
+        /\ forwarding'     = [forwarding EXCEPT ![i] = {}]
+        /\ ackIndex'       = [ackIndex   EXCEPT ![i] = [v \in Server |-> IF v = i THEN Len(history[i])
+                                                                                  ELSE 0]]
+        /\ currentCounter'   = [currentCounter   EXCEPT ![i] = 0]
+        /\ sendCounter'      = [sendCounter      EXCEPT ![i] = 0]
+        /\ commitIndex'      = [commitIndex      EXCEPT ![i] = 0]
+        /\ committedIndex'   = [committedIndex   EXCEPT ![i] = 0]
+        /\ committedCounter' = [committedCounter EXCEPT ![i] = [v \in Server |-> IF v = i THEN Len(history[i])
+                                                                                          ELSE 0]]
+        /\ initialHistory' = [initialHistory EXCEPT ![i] = history[i]]
+        /\ tempMaxEpoch'   = [tempMaxEpoch   EXCEPT ![i] = acceptedEpoch[i]]
+
+ZabTurnToFollowing(i) ==
+        /\ zabState'    = [zabState   EXCEPT ![i] = DISCOVERY]
+        /\ cepochSent'  = [cepochSent EXCEPT ![i] = FALSE]
+        /\ synced'      = [synced     EXCEPT ![i] = FALSE]
+        /\ commitIndex' = [commitIndex      EXCEPT ![i] = 0]
+        
+          
+(* Fast Leader Election *)
+FLEReceiveNotmsg(i, j) ==
+        /\ ReceiveNotmsg(i, j)
+        /\ UNCHANGED <<zabState, acceptedEpoch, history, commitIndex,learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter,
+                       sendCounter, committedIndex, committedCounter, tempVarsZ, followerVarsZ, verifyVarsZ, msgs>>
+
+FLENotmsgTimeout(i) ==
+        /\ NotmsgTimeout(i)
+        /\ UNCHANGED <<zabState, acceptedEpoch, history, commitIndex,learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter,
+                       sendCounter, committedIndex, committedCounter, tempVarsZ, followerVarsZ, verifyVarsZ, msgs>>
+
+FLEHandleNotmsg(i) ==
+        /\ HandleNotmsg(i)
+        /\ LET newState == state'[i]
+           IN
+           \/ /\ newState = LEADING
+              /\ ZabTurnToLeading(i)
+              /\ UNCHANGED <<cepochSent, synced>>
+           \/ /\ newState = FOLLOWING
+              /\ ZabTurnToFollowing(i)
+              /\ UNCHANGED <<learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter, sendCounter, committedIndex, committedCounter, tempVarsZ>>
+           \/ /\ newState = LOOKING
+              /\ UNCHANGED <<zabState, learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter, sendCounter, commitIndex,
+                             committedIndex, committedCounter, tempVarsZ, cepochSent, synced>>
+        /\ UNCHANGED <<acceptedEpoch, history, leaderAddr, verifyVarsZ, msgs>>
+
+\* On the premise that ReceiveVotes.HasQuorums = TRUE, corresponding to logic in line 1050-1055 in LFE.java.
+FLEWaitNewNotmsg(i) ==
+        /\ WaitNewNotmsg(i)
+        /\ UNCHANGED <<zabState, acceptedEpoch, history, commitIndex,learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter,
+                       sendCounter, committedIndex, committedCounter, tempVarsZ, followerVarsZ, verifyVarsZ, msgs>>
+
+\* On the premise that ReceiveVotes.HasQuorums = TRUE, corresponding to logic in line 1061-1066 in LFE.java.
+FLEWaitNewNotmsgEnd(i) ==
+        /\ WaitNewNotmsgEnd(i)
+        /\ LET newState == state'[i]
+           IN
+           \/ /\ newState = LEADING
+              /\ ZabTurnToLeading(i)
+              /\ UNCHANGED <<cepochSent, synced>>
+           \/ /\ newState = FOLLOWING
+              /\ ZabTurnToFollowing(i)
+              /\ UNCHANGED <<learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter, sendCounter, committedIndex, committedCounter, tempVarsZ>>
+           \/ /\ newState = LOOKING
+              /\ PrintT("New state is LOOKING in FLEWaitNewNotmsgEnd, which should not happen.")
+              /\ UNCHANGED <<zabState, learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter, sendCounter, commitIndex,
+                             committedIndex, committedCounter, tempVarsZ, cepochSent, synced>>
+        /\ UNCHANGED <<acceptedEpoch, history, leaderAddr, verifyVarsZ, msgs>>
+              
+-----------------------------------------------------------------------------
+(* A sub-action describing how a server transitions from LEADING/FOLLOWING to LOOKING.
+   Initially I call it 'ZabTimeoutZ', but it will be called not only when timeout, 
+   but also when finding a low epoch from leader.*)
+FollowerShutdown(i) ==
+        /\ ZabTimeout(i)
+        /\ zabState'   = [zabState   EXCEPT ![i] = ELECTION]
+        /\ leaderAddr' = [leaderAddr EXCEPT ![i] = NullPoint]
+
+LeaderShutdown(i) ==
+        /\ ZabTimeout(i)
+        /\ zabState'   = [zabState   EXCEPT ![i] = ELECTION]
+        /\ leaderAddr' = [s \in Server |-> IF s \in learners[i] THEN NullPoint ELSE leaderAddr[s]]
+        /\ learners'   = [learners   EXCEPT ![i] = {}]
+        /\ forwarding' = [forwarding EXCEPT ![i] = {}]
+        /\ msgs'       = [s \in Server |-> [v \in Server |-> IF v \in learners[i] \/ s \in learners[i] THEN << >> ELSE msgs[s][v]]]
+
+FollowerTimout(i) ==
+        /\ state[i]      = FOLLOWING
+        /\ leaderAddr[i] = NullPoint
+        /\ FollowerShutdown(i)
+        /\ msgs' = [s \in Server |-> [v \in Server |-> IF v = i THEN << >> ELSE msgs[s][v]]]
+        /\ UNCHANGED <<acceptedEpoch, history, commitIndex, learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, 
+                       currentCounter, sendCounter, committedIndex, committedCounter, tempVarsZ, cepochSent, synced, verifyVarsZ>>
+
+LeaderTimeout(i) ==
+        /\ state[i] = LEADING
+        /\ learners[i] \notin Quorums
+        /\ LeaderShutdown(i)
+        /\ UNCHANGED <<acceptedEpoch, history, commitIndex, cepochRecv, ackeRecv, ackldRecv, ackIndex, currentCounter, sendCounter, committedIndex, committedCounter,
+                       tempVarsZ, cepochSent, synced, verifyVarsZ>>
+    
+-----------------------------------------------------------------------------
+(* Establish connection between leader i and follower j. 
+   It means i creates a learnerHandler for communicating with j, and j finds i's address.*)
+EstablishConnection(i, j) ==
+        /\ state[i] = LEADING   /\ state[j] = FOLLOWING
+        /\ j \notin learners[i] /\ leaderAddr[j] = NullPoint
+        /\ currentVote[j].proposedLeader = i
+        /\ learners'   = [learners   EXCEPT ![i] = learners[i] \union {j}] \* Leader:   'addLearnerHandler(peer)'
+        /\ leaderAddr' = [leaderAddr EXCEPT ![j] = i]                      \* Follower: 'connectToLeader(addr, hostname)'
+        /\ UNCHANGED <<serverVarsZ, electionVarsZ, leadingVoteSet, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, 
+                       currentCounter, sendCounter, committedIndex, committedCounter, tempVarsZ, cepochSent, synced, verifyVarsZ, msgVarsZ, idTable>>
+        
+(* The leader i finds timeout and TCP connection between i and j closes.*)       
+Timeout(i, j) ==
+        /\ state[i] = LEADING /\ state[j] = FOLLOWING
+        /\ j \in learners[i]  /\ leaderAddr[j] = i
+        (* The action of leader i.(corresponding to function 'removeLearnerHandler(peer)'.) *)
+        /\ learners'   = [learners   EXCEPT ![i] = learners[i] \ {j}] 
+        /\ forwarding' = [forwarding EXCEPT ![i] = IF j \in forwarding[i] THEN forwarding[i] \ {j} ELSE forwarding[i]]
+        /\ cepochRecv' = [cepochRecv EXCEPT ![i] = IF j \in cepochRecv[i] THEN cepochRecv[i] \ {j} ELSE cepochRecv[i]]
+        /\ ackeRecv'   = [ackeRecv   EXCEPT ![i] = IF j \in ackeRecv[i] THEN ackeRecv[i] \ {j} ELSE ackeRecv[i]]
+        /\ ackldRecv'  = [ackldRecv  EXCEPT ![i] = IF j \in ackldRecv[i] THEN ackldRecv[i] \ {j} ELSE ackldRecv[i]]
+        /\ ackIndex'   = [ackIndex   EXCEPT ![i][j] = 0]
+        /\ committedCounter' = [committedCounter EXCEPT ![i][j] = 0]
+        (* The action of follower j. *)
+        /\ FollowerShutdown(j)
+        (* Clean input buffer.*)
+        /\ Clean(i, j)
+        /\ UNCHANGED <<acceptedEpoch, history, commitIndex, currentCounter, sendCounter, committedIndex,
+                       tempVarsZ, cepochSent, synced, verifyVarsZ>>
+-----------------------------------------------------------------------------
+\* In phase f11, follower sends f.p to leader via FOLLOWERINFO(CEPOCH).
+FollowerSendFOLLOWERINFO(i) ==
+        /\ state[i]    = FOLLOWING
+        /\ zabState[i] = DISCOVERY
+        /\ leaderAddr[i] /= NullPoint
+        /\ \lnot cepochSent[i]
+        /\ Send(i, leaderAddr[i], [mtype  |-> FOLLOWERINFO,
+                                   mepoch |-> acceptedEpoch[i]])
+        /\ cepochSent' = [cepochSent EXCEPT ![i] = TRUE]
+        /\ UNCHANGED <<serverVarsZ, leaderVarsZ, electionVarsZ, tempVarsZ, leaderAddr, synced, verifyVarsZ, electionMsgs, idTable>>
+        
+(* In phase l11, leader waits for receiving FOLLOWERINFO from a quorum,
+   and then chooses a new epoch e' as its own epoch and broadcasts LEADERINFO. *)
+LeaderHandleFOLLOWERINFO(i, j) ==
+        /\ state[i] = LEADING
+        /\ msgs[j][i] /= << >>
+        /\ msgs[j][i][1].mtype = FOLLOWERINFO
+        /\ LET msg == msgs[j][i][1]
+           IN \/ /\ NullPoint \notin cepochRecv[i]   \* 1. has not broadcast LEADERINFO - modify tempMaxEpoch
+                 /\ LET newEpoch == Maximum({tempMaxEpoch[i], msg.mepoch})
+                    IN tempMaxEpoch' = [tempMaxEpoch EXCEPT ![i] = newEpoch]
+                 /\ Discard(j, i)
+              \/ /\ NullPoint \in cepochRecv[i]      \* 2. has broadcast LEADERINFO - no need to handle the msg, just send LEADERINFO to corresponding server
+                 /\ Reply(i, j, [mtype  |-> LEADERINFO,
+                                 mepoch |-> acceptedEpoch[i]])
+                 /\ UNCHANGED tempMaxEpoch
+        /\ cepochRecv' = [cepochRecv EXCEPT ![i] = IF j \in cepochRecv[i] THEN cepochRecv[i]
+                                                                          ELSE cepochRecv[i] \union {j}]
+        /\ UNCHANGED <<serverVarsZ, followerVarsZ, electionVarsZ, initialHistory, leadingVoteSet, learners, ackeRecv, ackldRecv, 
+                       forwarding, ackIndex, currentCounter, sendCounter, committedIndex, committedCounter, verifyVarsZ, electionMsgs, idTable>>
+
+LeaderDiscovery1(i) ==
+        /\ state[i]    = LEADING
+        /\ zabState[i] = DISCOVERY
+        /\ cepochRecv[i] \in Quorums
+        /\ acceptedEpoch' = [acceptedEpoch EXCEPT ![i] = tempMaxEpoch[i] + 1]
+        /\ cepochRecv'    = [cepochRecv    EXCEPT ![i] = cepochRecv[i] \union {NullPoint}]
+        /\ BroadcastLEADERINFO(i, [mtype  |-> LEADERINFO,
+                                   mepoch |-> acceptedEpoch'[i]])
+        /\ UNCHANGED <<state, currentEpoch, lastZxid, zabState, history, commitIndex, electionVarsZ, leadingVoteSet, learners, ackeRecv, ackldRecv, 
+                       forwarding, ackIndex, currentCounter, sendCounter, committedIndex, committedCounter, 
+                       tempVarsZ, followerVarsZ, verifyVarsZ, electionMsgs, idTable>>
+
+(* In phase f12, follower receives NEWEPOCH. If e' > f.p, then follower sends ACK-E back,
+   and ACK-E contains f.a and lastZxid to let leader judge whether it is the latest.
+   After handling NEWEPOCH, follower's zabState turns to SYNCHRONIZATION. *)
+FollowerHandleLEADERINFO(i, j) ==
+        /\ state[i] = FOLLOWING
+        /\ msgs[j][i] /= << >>
+        /\ msgs[j][i][1].mtype = LEADERINFO
+        /\ LET msg     == msgs[j][i][1]
+               infoOk  == j = leaderAddr[i]
+               epochOk == /\ infoOk
+                          /\ msg.mepoch >= acceptedEpoch[i]
+               correct == /\ epochOk
+                          /\ zabState[i] = DISCOVERY
+           IN /\ infoOk
+              /\ \/ /\ epochOk                      \* 1. Normal case
+                    /\ \/ /\ correct
+                          /\ acceptedEpoch' = [acceptedEpoch EXCEPT ![i] = msg.mepoch]
+                          /\ Reply(i, j, [mtype      |-> ACKEPOCH,
+                                          mepoch     |-> msg.mepoch,
+                                          mlastEpoch |-> currentEpoch[i],
+                                          mlastZxid  |-> lastZxid[i]])
+                          /\ cepochSent' = [cepochSent EXCEPT ![i] = TRUE]
+                          /\ UNCHANGED inherentViolated
+                       \/ /\ ~correct
+                          /\ PrintT("Exception: Condition correct is false in FollowerHandleLEADERINFO(" \o ToString(i) \o ", " \o ToString(j) \o ").")
+                          /\ inherentViolated' = TRUE
+                          /\ Discard(j, i)
+                          /\ UNCHANGED <<acceptedEpoch, cepochSent>>
+                    /\ zabState' = [zabState EXCEPT ![i] = IF zabState[i] = DISCOVERY THEN SYNCHRONIZATION
+                                                                                      ELSE zabState[i]]
+                    /\ UNCHANGED <<varsL, leaderAddr>>
+                 \/ /\ ~epochOk                    \* 2. Abnormal case - go back to election
+                    /\ FollowerShutdown(i)
+                    /\ Clean(i, j)
+                    /\ UNCHANGED <<acceptedEpoch, cepochSent, inherentViolated>>
+        /\ UNCHANGED <<history, commitIndex, learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter, sendCounter, committedIndex,
+                       committedCounter, tempVarsZ, synced, proposalMsgsLog, epochLeader>>
+
+\* Abstraction of actions making follower synced with leader before leader sending NEWLEADER.
+subRECOVERYSYNC(i, j) ==
+        LET canSync == /\ state[i] = LEADING   /\ zabState[i] /= DISCOVERY      /\ j \in learners[i]  /\ j \in ackeRecv[i]
+                       /\ state[j] = FOLLOWING /\ zabState[j] = SYNCHRONIZATION /\ leaderAddr[j] = i  /\ synced[j] = FALSE
+        IN
+        \/ /\ canSync
+           /\ history'     = [history     EXCEPT ![j] = history[i]]
+           /\ lastZxid'    = [lastZxid    EXCEPT ![j] = lastZxid[i]]
+           /\ UpdateProposal(j, leaderAddr[j], lastZxid'[j], currentEpoch[j])
+           /\ commitIndex' = [commitIndex EXCEPT ![j] = commitIndex[i]]
+           /\ synced'      = [synced      EXCEPT ![j] = TRUE]
+           /\ forwarding'  = [forwarding  EXCEPT ![i] = forwarding[i] \union {j}]    \* j will receive PROPOSAL and COMMIT
+           /\ ackIndex'    = [ackIndex    EXCEPT ![i][j] = Len(history[i])]
+           /\ committedCounter' = [committedCounter EXCEPT ![i][j] = Maximum({commitIndex[i] - Len(initialHistory[i]), 0})]
+           /\ LET ms == [msource|->i, mtype|->"RECOVERYSYNC", mepoch|->acceptedEpoch[i], mproposals|->history[i]]
+              IN proposalMsgsLog' = IF ms \in proposalMsgsLog THEN proposalMsgsLog
+                                                              ELSE proposalMsgsLog \union {ms}
+           /\ Reply(i, j, [mtype     |-> NEWLEADER,
+                           mepoch    |-> acceptedEpoch[i],
+                           mlastZxid |-> lastZxid[i]])
+        \/ /\ ~canSync
+           /\ Discard(j, i)
+           /\ UNCHANGED <<history, lastZxid, currentVote, commitIndex, synced, forwarding, ackIndex, committedCounter, proposalMsgsLog>>
+        
+(* In phase l12, leader waits for receiving ACKEPOPCH from a quorum,
+   and check whether it has the latest history and epoch from them.
+   If so, leader's zabState turns to SYNCHRONIZATION. *)
+LeaderHandleACKEPOCH(i, j) ==
+        /\ state[i] = LEADING
+        /\ msgs[j][i] /= << >>
+        /\ msgs[j][i][1].mtype = ACKEPOCH
+        /\ LET msg == msgs[j][i][1]
+               infoOk == /\ j \in learners[i]            
+                         /\ acceptedEpoch[i] = msg.mepoch
+               logOk  == /\ infoOk                          \* logOk = TRUE means leader is more up-to-date than follower j
+                         /\ \/ currentEpoch[i] > msg.mlastEpoch
+                            \/ /\ currentEpoch[i] = msg.mlastEpoch
+                               /\ \/ lastZxid[i][1] > msg.mlastZxid[1]
+                                  \/ /\ lastZxid[i][1] = msg.mlastZxid[1]
+                                     /\ lastZxid[i][2] >= msg.mlastZxid[2]
+               replyOk == /\ infoOk
+                          /\ NullPoint \in ackeRecv[i]
+           IN /\ infoOk
+              /\ \/ /\ replyOk
+                    /\ subRECOVERYSYNC(i, j)
+                    /\ ackeRecv' = [ackeRecv EXCEPT ![i] = IF j \notin ackeRecv[i] THEN ackeRecv[i] \union {j}
+                                                                                   ELSE ackeRecv[i]]
+                    /\ UNCHANGED <<state, currentEpoch,logicalClock, receiveVotes, outOfElection, recvQueue, waitNotmsg, leadingVoteSet, electionMsgs, idTable,
+                                   zabState, leaderAddr, learners>>
+                 \/ /\ ~replyOk
+                    /\ \/ /\ logOk
+                          /\ ackeRecv' = [ackeRecv EXCEPT ![i] = IF j \notin ackeRecv[i] THEN ackeRecv[i] \union {j}
+                                                                                         ELSE ackeRecv[i]]
+                          /\ Discard(j, i)
+                          /\ UNCHANGED <<varsL, zabState, leaderAddr, learners, forwarding>>
+                       \/ /\ ~logOk            \* go back to election
+                          /\ LeaderShutdown(i)
+                          /\ UNCHANGED ackeRecv
+                    /\ UNCHANGED <<history, commitIndex, synced, forwarding, ackIndex, committedCounter, proposalMsgsLog>>
+        /\ UNCHANGED <<acceptedEpoch, cepochRecv, ackldRecv,currentCounter, sendCounter, committedIndex, tempVarsZ, cepochSent, epochLeader, inherentViolated>>
+                          
+LeaderDiscovery2(i) ==
+        /\ state[i] = LEADING
+        /\ zabState[i] = DISCOVERY
+        /\ ackeRecv[i] \in Quorums
+        /\ zabState'       = [zabState       EXCEPT ![i] = SYNCHRONIZATION]
+        /\ currentEpoch'   = [currentEpoch   EXCEPT ![i] = acceptedEpoch[i]]
+        /\ initialHistory' = [initialHistory EXCEPT ![i] = history[i]]
+        /\ ackeRecv'       = [ackeRecv       EXCEPT ![i] = ackeRecv[i] \union {NullPoint}]
+        /\ ackIndex'       = [ackIndex       EXCEPT ![i][i] = Len(history[i])]
+        /\ UpdateProposal(i, i, lastZxid[i], currentEpoch'[i])
+        /\ LET epoch == acceptedEpoch[i]
+           IN epochLeader' = [epochLeader EXCEPT ![epoch] = epochLeader[epoch] \union {i}]
+        /\ UNCHANGED <<state, lastZxid, acceptedEpoch, history, commitIndex, logicalClock, receiveVotes, outOfElection, recvQueue, waitNotmsg,
+                       leadingVoteSet, learners, cepochRecv, ackldRecv, forwarding, currentCounter, sendCounter, committedIndex, committedCounter,
+                       tempMaxEpoch, followerVarsZ, proposalMsgsLog, inherentViolated, msgVarsZ, idTable>>
+
+(* Note:  Set cepochRecv, ackeRecv, ackldRecv to {NullPoint} in corresponding three actions to
+          make sure that the prospective leader will not broadcast NEWEPOCH/NEWLEADER/COMMITLD twice.*)
+-----------------------------------------------------------------------------
+RECOVERYSYNC(i, j) ==
+        /\ state[i] = LEADING   /\ zabState[i] /= DISCOVERY      /\ j \in learners[i] /\ j \in ackeRecv[i]
+        /\ state[j] = FOLLOWING /\ zabState[j] = SYNCHRONIZATION /\ leaderAddr[j] = i /\ synced[j] = FALSE
+        \* /\ acceptedEpoch[i] = acceptedEpoch[j] \* This condition is unnecessary.
+        /\ history'     = [history     EXCEPT ![j] = history[i]]
+        /\ lastZxid'    = [lastZxid    EXCEPT ![j] = lastZxid[i]]
+        /\ UpdateProposal(j, leaderAddr[j], lastZxid'[j], currentEpoch[j])
+        /\ commitIndex' = [commitIndex EXCEPT ![j] = commitIndex[i]]
+        /\ synced'      = [synced      EXCEPT ![j] = TRUE]
+        /\ forwarding'  = [forwarding  EXCEPT ![i] = forwarding[i] \union {j}]
+        /\ ackIndex'    = [ackIndex    EXCEPT ![i][j] = Len(history[i])]
+        /\ committedCounter' = [committedCounter EXCEPT ![i][j] = Maximum({commitIndex[i] - Len(initialHistory[i]), 0})]
+        /\ LET ms == [msource|->i, mtype|->"RECOVERYSYNC", mepoch|->acceptedEpoch[i], mproposals|->history[i]]
+           IN proposalMsgsLog' = IF ms \in proposalMsgsLog THEN proposalMsgsLog
+                                                           ELSE proposalMsgsLog \union {ms}
+        /\ Send(i, j, [mtype     |-> NEWLEADER,
+                       mepoch    |-> acceptedEpoch[i],
+                       mlastZxid |-> lastZxid[i]])
+        /\ UNCHANGED <<state, zabState, acceptedEpoch, currentEpoch, logicalClock, receiveVotes, outOfElection, recvQueue, waitNotmsg, 
+                       leadingVoteSet, learners, cepochRecv, ackeRecv, ackldRecv, currentCounter, sendCounter, committedIndex, 
+                       tempVarsZ, cepochSent, leaderAddr, epochLeader, inherentViolated, electionMsgs, idTable>>
+
+(* In phase f21, follower receives NEWLEADER. The follower updates its epoch and history,
+   and sends back ACK-LD to leader. *)
+FollowerHandleNEWLEADER(i, j) ==
+        /\ state[i] = FOLLOWING
+        /\ msgs[j][i] /= << >>
+        /\ msgs[j][i][1].mtype = NEWLEADER
+        /\ LET msg     == msgs[j][i][1]
+               infoOk  == /\ leaderAddr[i] = j
+                          /\ acceptedEpoch[i] = msg.mepoch
+               correct == /\ infoOk
+                          /\ zabState[i] = SYNCHRONIZATION
+                          /\ synced[i]
+                          /\ ZxidEqual(lastZxid[i], msg.mlastZxid)
+           IN /\ infoOk
+              /\ currentEpoch' = [currentEpoch EXCEPT ![i] = msg.mepoch]
+              /\ UpdateProposal(i, j, lastZxid[i], currentEpoch'[i])
+              /\ \/ /\ correct
+                    /\ Reply(i, j, [mtype  |-> ACKLD,
+                                    mepoch |-> msg.mepoch])
+                    /\ UNCHANGED inherentViolated
+                 \/ /\ ~correct
+                    /\ PrintT("Exception: Condition correct is false in FollowerHandleNEWLEADER(" \o ToString(i) \o ", " \o ToString(j) \o ").")
+                    /\ inherentViolated' = TRUE
+                    /\ Discard(j, i)
+        /\ UNCHANGED <<state, lastZxid, zabState, acceptedEpoch, history, commitIndex, logicalClock, receiveVotes, outOfElection, recvQueue, waitNotmsg,
+                       leaderVarsZ, tempVarsZ, followerVarsZ, proposalMsgsLog, epochLeader, electionMsgs, idTable>>
+
+\* In phase l22, leader receives ACK-LD from a quorum of followers, and sends COMMIT-LD(UPTODATE) to followers.
+LeaderHandleACKLD(i, j) ==
+        /\ state[i] = LEADING
+        /\ msgs[j][i] /= << >>
+        /\ msgs[j][i][1].mtype = ACKLD

Review comment:
       ACKLD is not real message type, though I realise using it may simplify the spec. Might be worth noting it in a comment somewhere.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@zookeeper.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] Vanlightly commented on pull request #1690: ZOOKEEPER-3615: Provide formal specification and verification using TLA+ for Zab

Posted by GitBox <gi...@apache.org>.
Vanlightly commented on pull request #1690:
URL: https://github.com/apache/zookeeper/pull/1690#issuecomment-831736892


   I left the new version running on my workstation overnight with Servers {s1,s1} symmetrical and Values {v1,v2} symmetrical and no violations happened. It was still running this morning after 16 hours and I had to stop it. It reached diameter 66, distinct states 2,370,569,088, 127GB storage and interestingly four actions with 0 states: FindCluster, LeaderHandleCEPOCHinPhase3, LeaderHandleACKLDinPhase3 and BecomeFollower.
   
   Again, I haven't had time to dig into the spec itself. I will take some time soon to understand each action and invariant. The numbers above do exceed what you have reported though and the four actions without states could indicate a problem.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] BinyuHuang-nju commented on pull request #1690: ZOOKEEPER-3615: Provide formal specification and verification using TLA+ for Zab

Posted by GitBox <gi...@apache.org>.
BinyuHuang-nju commented on pull request #1690:
URL: https://github.com/apache/zookeeper/pull/1690#issuecomment-975587852


   @Vanlightly Hi. I have updated spec according to your suggestions.
   I have modified Value as type Nat, and replaced its function with recorder.nClientRequest. Everytime when  LeaderProcessRequest is performed, newTxn will use nClientRequest as its value and UpdateRecorder will inc nClientRequest. So we can confirm that any proposal in one trace has unique value.
   And thistime TxnEqual will check both zxid and value equality. If you have any new suggestions, just tell me.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@zookeeper.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] BinyuHuang-nju commented on pull request #1690: ZOOKEEPER-3615: Provide formal specification and verification using TLA+ for Zab

Posted by GitBox <gi...@apache.org>.
BinyuHuang-nju commented on pull request #1690:
URL: https://github.com/apache/zookeeper/pull/1690#issuecomment-834295404


   @Vanlightly Hi! When I set 3servers with parameters MaxTotalRestartNum = 1,MaxElectionNum = 1,MaxTransactionNum = 2, the model cannot stop after 31 hours and I have to stop running. It can be seen that even with 3 servers, the speed of state space growth is unimaginable.
   
   But it reached a diameter of 45 with 9,602,018,536 states. Compared to previous models that reached a diameter of about 21 in 15 hours when setting 3 servers, now the model can reach a diamater of 43 in 15 hours. Obviously the state space has been greatly compressed, but it is still difficult to see the termination.
   
   However, most of states consisted of actions from timeout, recovery and phase1(Discovery) before, and now actions from phase3 (Broadcast) occupy a considerable part. This is what we want to see, because states those reach actions in phase3 are more likely to go wrong. So the current spec may have better error detection capability than the past spec.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] Vanlightly commented on a change in pull request #1690: ZOOKEEPER-3615: Provide formal specification and verification using TLA+ for Zab

Posted by GitBox <gi...@apache.org>.
Vanlightly commented on a change in pull request #1690:
URL: https://github.com/apache/zookeeper/pull/1690#discussion_r705456407



##########
File path: zookeeper-specifications/zab-1.0/ZabWithFLE.tla
##########
@@ -0,0 +1,1020 @@
+(*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *)
+
+----------------------------- MODULE ZabWithFLE -----------------------------
+(* This is the formal specification for the Zab consensus algorithm,
+   which means Zookeeper Atomic Broadcast.*)
+
+(* Reference:
+   FLE: FastLeaderElection.java, Vote.java, QuorumPeer.java in https://github.com/apache/zookeeper.
+   ZAB: QuorumPeer.java, Learner.java, Follower.java, LearnerHandler.java, Leader.java 
+        in https://github.com/apache/zookeeper.
+        https://cwiki.apache.org/confluence/display/ZOOKEEPER/Zab1.0.
+ *)
+EXTENDS FastLeaderElection
+
+-----------------------------------------------------------------------------
+(* Defined in FastLeaderElection.tla:
+\* The set of server identifiers
+CONSTANT Server
+
+\* Server states
+CONSTANTS LOOKING, FOLLOWING, LEADING
+
+\* Message types
+CONSTANTS NOTIFICATION
+
+\* Timeout signal
+CONSTANT NONE
+*)
+\* The set of requests that can go into history
+CONSTANT Value
+
+\* Zab states
+CONSTANTS ELECTION, DISCOVERY, SYNCHRONIZATION, BROADCAST
+
+\* Message types
+CONSTANTS FOLLOWERINFO, LEADERINFO, ACKEPOCH, NEWLEADER, ACKLD, UPTODATE, PROPOSAL, ACK, COMMIT
+(* Additional message types used for recovery in synchronization(TRUNC/DIFF/SNAP) are not needed
+   since we abstract this part.(see action RECOVERYSYNC) *)
+   
+-----------------------------------------------------------------------------
+(* Defined in FastLeaderElection.tla: Quorums, NullPoint *)
+\* Return the maximum value from the set S
+Maximum(S) == IF S = {} THEN -1
+                        ELSE CHOOSE n \in S: \A m \in S: n >= m
+
+\* Return the minimum value from the set S
+Minimum(S) == IF S = {} THEN -1
+                        ELSE CHOOSE n \in S: \A m \in S: n <= m
+
+MAXEPOCH == 10
+
+-----------------------------------------------------------------------------
+(* Defined in FastLeaderElection.tla:
+   serverVars: <<state, currentEpoch, lastZxid>>,
+   electionVars: <<currentVote, logicalClock, receiveVotes, outOfElection, recvQueue, waitNotmsg>>,
+   leaderVars: <<leadingVoteSet>>,
+   electionMsgs,
+   idTable *)
+\* The current phase of server(ELECTION,DISCOVERY,SYNCHRONIZATION,BROADCAST)
+VARIABLE zabState
+
+\* The epoch number of the last NEWEPOCH(LEADERINFO) packet accepted
+\* namely f.p in paper, and currentEpoch in Zab.tla.
+VARIABLE acceptedEpoch
+
+\* The history of servers as the sequence of transactions.
+VARIABLE history
+
+\* commitIndex[i]: The maximum index of transactions that have been saved in a quorum of servers
+\*                 in the perspective of server i.(increases monotonically before restarting)
+VARIABLE commitIndex
+
+(* These transactions whose index \le commitIndex[i] can be applied to state machine immediately.
+   So if we have a variable applyIndex, we can suppose that applyIndex[i] = commitIndex[i] when verifying properties.
+   But in phase SYNC, follower will apply all queued proposals to state machine when receiving NEWLEADER.
+   But follower only serves traffic after receiving UPTODATE, so sequential consistency is not violated.
+   
+   So when we verify properties, we still suppose applyIndex[i] = commitIndex[i], because this is an engineering detail.*)
+
+\* learners[i]: The set of servers which leader i think are connected wich i.
+VARIABLE learners
+
+\* The messages representing requests and responses sent from one server to another.
+\* msgs[i][j] means the input buffer of server j from server i.
+VARIABLE msgs
+
+\* The set of followers who has successfully sent CEPOCH(FOLLOWERINFO) to leader.(equals to connectingFollowers in code)
+VARIABLE cepochRecv
+
+\* The set of followers who has successfully sent ACK-E to leader.(equals to electingFollowers in code)
+VARIABLE ackeRecv
+
+\* The set of followers who has successfully sent ACK-LD to leader in leader.(equals to newLeaderProposal in code)
+VARIABLE ackldRecv
+
+\* The set of servers which leader i broadcasts PROPOSAL and COMMIT to.(equals to forwardingFollowers in code)
+VARIABLE forwarding
+
+\* ackIndex[i][j]: The latest index that leader i has received from follower j via ACK.
+VARIABLE ackIndex
+
+\* currentCounter[i]: The count of transactions that clients request leader i.
+VARIABLE currentCounter
+
+\* sendCounter[i]: The count of transactions that leader i has broadcast in PROPOSAL.
+VARIABLE sendCounter
+
+\* committedIndex[i]: The maximum index of trasactions that leader i has broadcast in COMMIT.
+VARIABLE committedIndex
+
+\* committedCounter[i][j]: The latest counter of transaction that leader i has confirmed that follower j has committed.
+VARIABLE committedCounter
+
+\* initialHistory[i]: The initial history if leader i in epoch acceptedEpoch[i].
+VARIABLE initialHistory
+
+\* the maximum epoch in CEPOCH the prospective leader received from followers.
+VARIABLE tempMaxEpoch
+
+\* cepochSent[i] = TRUE means follower i has sent CEPOCH(FOLLOWERINFO) to leader.
+VARIABLE cepochSent
+
+\* leaderAddr[i]: The leader id of follower i. We use leaderAddr to express whether follower i has connected or lost connection.
+VARIABLE leaderAddr
+
+\* synced[i] = TRUE: follower i has completed sync with leader.
+VARIABLE synced
+
+\* The set of leaders in every epoch, only used in verifying properties.
+VARIABLE epochLeader
+
+\* The set of all broadcast messages, only used in verifying properties.
+VARIABLE proposalMsgsLog
+
+\* A variable used to check whether there are conditions contrary to the facts.
+VARIABLE inherentViolated
+
+serverVarsZ == <<state, currentEpoch, lastZxid, zabState, acceptedEpoch, history, commitIndex>>         \* 7 variables
+
+electionVarsZ == electionVars  \* 6 variables
+
+leaderVarsZ == <<leadingVoteSet, learners, cepochRecv, ackeRecv, ackldRecv, forwarding, 
+                 ackIndex, currentCounter, sendCounter, committedIndex, committedCounter>>              \* 11 variables
+
+tempVarsZ == <<initialHistory, tempMaxEpoch>>     \* 2 variables
+  
+followerVarsZ == <<cepochSent, leaderAddr, synced>>                 \* 3 variables
+
+verifyVarsZ == <<proposalMsgsLog, epochLeader, inherentViolated>>   \* 3 variables
+
+msgVarsZ == <<msgs, electionMsgs>>                \* 2 variables
+
+vars == <<serverVarsZ, electionVarsZ, leaderVarsZ, tempVarsZ, followerVarsZ, verifyVarsZ, msgVarsZ, idTable>>  \* 35 variables
+-----------------------------------------------------------------------------
+\* Add a message to msgs - add a message m to msgs[i][j].
+Send(i, j, m) == msgs' = [msgs EXCEPT ![i][j] = Append(msgs[i][j], m)]
+
+\* Remove a message from msgs - discard head of msgs[i][j].
+Discard(i, j) == msgs' = IF msgs[i][j] /= << >> THEN [msgs EXCEPT ![i][j] = Tail(msgs[i][j])]
+                                                ELSE msgs
+
+\* Leader broadcasts a message(PROPOSAL/COMMIT) to all other servers in forwardingFollowers.
+Broadcast(i, m) == msgs' = [msgs EXCEPT ![i] = [v \in Server |-> IF /\ v \in forwarding[i]
+                                                                    /\ v /= i
+                                                                    /\ \/ /\ m.mtype = PROPOSAL
+                                                                          /\ ackIndex[i][v] < Len(initialHistory[i]) + m.mproposal.counter
+                                                                       \/ /\ m.mtype = COMMIT
+                                                                          /\ committedCounter[i][v] < m.mzxid[2]
+                                                                  THEN Append(msgs[i][v], m)
+                                                                  ELSE msgs[i][v]]]
+
+BroadcastLEADERINFO(i, m) == msgs' = [msgs EXCEPT ![i] = [v \in Server |-> IF /\ v \in cepochRecv[i]
+                                                                              /\ v \in learners[i]
+                                                                              /\ v /= i THEN Append(msgs[i][v], m)
+                                                                                        ELSE msgs[i][v]]]
+
+BroadcastUPTODATE(i, m) == msgs' = [msgs EXCEPT ![i] = [v \in Server |-> IF /\ v \in ackldRecv[i]
+                                                                            /\ v \in learners[i]
+                                                                            /\ v /= i THEN Append(msgs[i][v], m)
+                                                                                      ELSE msgs[i][v]]]
+
+\* Combination of Send and Discard - discard head of msgs[j][i] and add m into msgs[i][j].
+Reply(i, j, m) == msgs' = [msgs EXCEPT ![j][i] = Tail(msgs[j][i]),
+                                       ![i][j] = Append(msgs[i][j], m)]
+
+\* shuffle the input buffer from server j(i) in server i(j).
+Clean(i, j) == msgs' = [msgs EXCEPT ![j][i] = << >>, ![i][j] = << >>]
+
+-----------------------------------------------------------------------------
+PZxidEqual(p, z) == p.epoch = z[1] /\ p.counter = z[2]
+
+TransactionEqual(t1, t2) == /\ t1.epoch   = t2.epoch
+                            /\ t1.counter = t2.counter
+                            
+TransactionPrecede(t1, t2) == \/ t1.epoch < t2.epoch
+                              \/ /\ t1.epoch   = t2.epoch
+                                 /\ t1.counter < t2.counter
+                                 
+-----------------------------------------------------------------------------
+\* Define initial values for all variables 
+InitServerVarsZ == /\ InitServerVars
+                   /\ zabState      = [s \in Server |-> ELECTION]
+                   /\ acceptedEpoch = [s \in Server |-> 0]
+                   /\ history       = [s \in Server |-> << >>]
+                   /\ commitIndex   = [s \in Server |-> 0]
+
+InitLeaderVarsZ == /\ InitLeaderVars
+                   /\ learners         = [s \in Server |-> {}]
+                   /\ cepochRecv       = [s \in Server |-> {}]
+                   /\ ackeRecv         = [s \in Server |-> {}]
+                   /\ ackldRecv        = [s \in Server |-> {}]
+                   /\ ackIndex         = [s \in Server |-> [v \in Server |-> 0]]
+                   /\ currentCounter   = [s \in Server |-> 0]
+                   /\ sendCounter      = [s \in Server |-> 0]
+                   /\ committedIndex   = [s \in Server |-> 0]
+                   /\ committedCounter = [s \in Server |-> [v \in Server |-> 0]]
+                   /\ forwarding       = [s \in Server |-> {}]
+
+InitElectionVarsZ == InitElectionVars
+
+InitTempVarsZ == /\ initialHistory = [s \in Server |-> << >>]
+                 /\ tempMaxEpoch   = [s \in Server |-> 0]
+
+InitFollowerVarsZ == /\ cepochSent = [s \in Server |-> FALSE]
+                     /\ leaderAddr = [s \in Server |-> NullPoint]
+                     /\ synced     = [s \in Server |-> FALSE]
+
+InitVerifyVarsZ == /\ proposalMsgsLog  = {}
+                   /\ epochLeader      = [i \in 1..MAXEPOCH |-> {}]
+                   /\ inherentViolated = FALSE
+                   
+InitMsgVarsZ == /\ msgs         = [s \in Server |-> [v \in Server |-> << >>]]
+                /\ electionMsgs = [s \in Server |-> [v \in Server |-> << >>]]
+
+InitZ == /\ InitServerVarsZ
+         /\ InitLeaderVarsZ
+         /\ InitElectionVarsZ
+         /\ InitTempVarsZ
+         /\ InitFollowerVarsZ
+         /\ InitVerifyVarsZ
+         /\ InitMsgVarsZ
+         /\ idTable = InitializeIdTable(Server)
+         
+-----------------------------------------------------------------------------
+ZabTurnToLeading(i) ==
+        /\ zabState'       = [zabState   EXCEPT ![i] = DISCOVERY]
+        /\ learners'       = [learners   EXCEPT ![i] = {i}]
+        /\ cepochRecv'     = [cepochRecv EXCEPT ![i] = {i}]
+        /\ ackeRecv'       = [ackeRecv   EXCEPT ![i] = {i}]
+        /\ ackldRecv'      = [ackldRecv  EXCEPT ![i] = {i}]
+        /\ forwarding'     = [forwarding EXCEPT ![i] = {}]
+        /\ ackIndex'       = [ackIndex   EXCEPT ![i] = [v \in Server |-> IF v = i THEN Len(history[i])
+                                                                                  ELSE 0]]
+        /\ currentCounter'   = [currentCounter   EXCEPT ![i] = 0]
+        /\ sendCounter'      = [sendCounter      EXCEPT ![i] = 0]
+        /\ commitIndex'      = [commitIndex      EXCEPT ![i] = 0]
+        /\ committedIndex'   = [committedIndex   EXCEPT ![i] = 0]
+        /\ committedCounter' = [committedCounter EXCEPT ![i] = [v \in Server |-> IF v = i THEN Len(history[i])
+                                                                                          ELSE 0]]
+        /\ initialHistory' = [initialHistory EXCEPT ![i] = history[i]]
+        /\ tempMaxEpoch'   = [tempMaxEpoch   EXCEPT ![i] = acceptedEpoch[i]]
+
+ZabTurnToFollowing(i) ==
+        /\ zabState'    = [zabState   EXCEPT ![i] = DISCOVERY]
+        /\ cepochSent'  = [cepochSent EXCEPT ![i] = FALSE]
+        /\ synced'      = [synced     EXCEPT ![i] = FALSE]
+        /\ commitIndex' = [commitIndex      EXCEPT ![i] = 0]
+        
+          
+(* Fast Leader Election *)
+FLEReceiveNotmsg(i, j) ==
+        /\ ReceiveNotmsg(i, j)
+        /\ UNCHANGED <<zabState, acceptedEpoch, history, commitIndex,learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter,
+                       sendCounter, committedIndex, committedCounter, tempVarsZ, followerVarsZ, verifyVarsZ, msgs>>
+
+FLENotmsgTimeout(i) ==
+        /\ NotmsgTimeout(i)
+        /\ UNCHANGED <<zabState, acceptedEpoch, history, commitIndex,learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter,
+                       sendCounter, committedIndex, committedCounter, tempVarsZ, followerVarsZ, verifyVarsZ, msgs>>
+
+FLEHandleNotmsg(i) ==
+        /\ HandleNotmsg(i)
+        /\ LET newState == state'[i]
+           IN
+           \/ /\ newState = LEADING
+              /\ ZabTurnToLeading(i)
+              /\ UNCHANGED <<cepochSent, synced>>
+           \/ /\ newState = FOLLOWING
+              /\ ZabTurnToFollowing(i)
+              /\ UNCHANGED <<learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter, sendCounter, committedIndex, committedCounter, tempVarsZ>>
+           \/ /\ newState = LOOKING
+              /\ UNCHANGED <<zabState, learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter, sendCounter, commitIndex,
+                             committedIndex, committedCounter, tempVarsZ, cepochSent, synced>>
+        /\ UNCHANGED <<acceptedEpoch, history, leaderAddr, verifyVarsZ, msgs>>
+
+\* On the premise that ReceiveVotes.HasQuorums = TRUE, corresponding to logic in line 1050-1055 in LFE.java.
+FLEWaitNewNotmsg(i) ==
+        /\ WaitNewNotmsg(i)
+        /\ UNCHANGED <<zabState, acceptedEpoch, history, commitIndex,learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter,
+                       sendCounter, committedIndex, committedCounter, tempVarsZ, followerVarsZ, verifyVarsZ, msgs>>
+
+\* On the premise that ReceiveVotes.HasQuorums = TRUE, corresponding to logic in line 1061-1066 in LFE.java.
+FLEWaitNewNotmsgEnd(i) ==
+        /\ WaitNewNotmsgEnd(i)
+        /\ LET newState == state'[i]
+           IN
+           \/ /\ newState = LEADING
+              /\ ZabTurnToLeading(i)
+              /\ UNCHANGED <<cepochSent, synced>>
+           \/ /\ newState = FOLLOWING
+              /\ ZabTurnToFollowing(i)
+              /\ UNCHANGED <<learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter, sendCounter, committedIndex, committedCounter, tempVarsZ>>
+           \/ /\ newState = LOOKING
+              /\ PrintT("New state is LOOKING in FLEWaitNewNotmsgEnd, which should not happen.")
+              /\ UNCHANGED <<zabState, learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter, sendCounter, commitIndex,
+                             committedIndex, committedCounter, tempVarsZ, cepochSent, synced>>
+        /\ UNCHANGED <<acceptedEpoch, history, leaderAddr, verifyVarsZ, msgs>>
+              
+-----------------------------------------------------------------------------
+(* A sub-action describing how a server transitions from LEADING/FOLLOWING to LOOKING.
+   Initially I call it 'ZabTimeoutZ', but it will be called not only when timeout, 
+   but also when finding a low epoch from leader.*)
+FollowerShutdown(i) ==
+        /\ ZabTimeout(i)
+        /\ zabState'   = [zabState   EXCEPT ![i] = ELECTION]
+        /\ leaderAddr' = [leaderAddr EXCEPT ![i] = NullPoint]
+
+LeaderShutdown(i) ==
+        /\ ZabTimeout(i)
+        /\ zabState'   = [zabState   EXCEPT ![i] = ELECTION]
+        /\ leaderAddr' = [s \in Server |-> IF s \in learners[i] THEN NullPoint ELSE leaderAddr[s]]
+        /\ learners'   = [learners   EXCEPT ![i] = {}]
+        /\ forwarding' = [forwarding EXCEPT ![i] = {}]
+        /\ msgs'       = [s \in Server |-> [v \in Server |-> IF v \in learners[i] \/ s \in learners[i] THEN << >> ELSE msgs[s][v]]]
+
+FollowerTimout(i) ==
+        /\ state[i]      = FOLLOWING
+        /\ leaderAddr[i] = NullPoint
+        /\ FollowerShutdown(i)
+        /\ msgs' = [s \in Server |-> [v \in Server |-> IF v = i THEN << >> ELSE msgs[s][v]]]
+        /\ UNCHANGED <<acceptedEpoch, history, commitIndex, learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, 
+                       currentCounter, sendCounter, committedIndex, committedCounter, tempVarsZ, cepochSent, synced, verifyVarsZ>>
+
+LeaderTimeout(i) ==
+        /\ state[i] = LEADING
+        /\ learners[i] \notin Quorums
+        /\ LeaderShutdown(i)
+        /\ UNCHANGED <<acceptedEpoch, history, commitIndex, cepochRecv, ackeRecv, ackldRecv, ackIndex, currentCounter, sendCounter, committedIndex, committedCounter,
+                       tempVarsZ, cepochSent, synced, verifyVarsZ>>
+    
+-----------------------------------------------------------------------------
+(* Establish connection between leader i and follower j. 
+   It means i creates a learnerHandler for communicating with j, and j finds i's address.*)
+EstablishConnection(i, j) ==
+        /\ state[i] = LEADING   /\ state[j] = FOLLOWING
+        /\ j \notin learners[i] /\ leaderAddr[j] = NullPoint
+        /\ currentVote[j].proposedLeader = i
+        /\ learners'   = [learners   EXCEPT ![i] = learners[i] \union {j}] \* Leader:   'addLearnerHandler(peer)'
+        /\ leaderAddr' = [leaderAddr EXCEPT ![j] = i]                      \* Follower: 'connectToLeader(addr, hostname)'
+        /\ UNCHANGED <<serverVarsZ, electionVarsZ, leadingVoteSet, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, 
+                       currentCounter, sendCounter, committedIndex, committedCounter, tempVarsZ, cepochSent, synced, verifyVarsZ, msgVarsZ, idTable>>
+        
+(* The leader i finds timeout and TCP connection between i and j closes.*)       
+Timeout(i, j) ==
+        /\ state[i] = LEADING /\ state[j] = FOLLOWING
+        /\ j \in learners[i]  /\ leaderAddr[j] = i
+        (* The action of leader i.(corresponding to function 'removeLearnerHandler(peer)'.) *)
+        /\ learners'   = [learners   EXCEPT ![i] = learners[i] \ {j}] 
+        /\ forwarding' = [forwarding EXCEPT ![i] = IF j \in forwarding[i] THEN forwarding[i] \ {j} ELSE forwarding[i]]
+        /\ cepochRecv' = [cepochRecv EXCEPT ![i] = IF j \in cepochRecv[i] THEN cepochRecv[i] \ {j} ELSE cepochRecv[i]]
+        /\ ackeRecv'   = [ackeRecv   EXCEPT ![i] = IF j \in ackeRecv[i] THEN ackeRecv[i] \ {j} ELSE ackeRecv[i]]
+        /\ ackldRecv'  = [ackldRecv  EXCEPT ![i] = IF j \in ackldRecv[i] THEN ackldRecv[i] \ {j} ELSE ackldRecv[i]]
+        /\ ackIndex'   = [ackIndex   EXCEPT ![i][j] = 0]
+        /\ committedCounter' = [committedCounter EXCEPT ![i][j] = 0]
+        (* The action of follower j. *)
+        /\ FollowerShutdown(j)
+        (* Clean input buffer.*)
+        /\ Clean(i, j)
+        /\ UNCHANGED <<acceptedEpoch, history, commitIndex, currentCounter, sendCounter, committedIndex,
+                       tempVarsZ, cepochSent, synced, verifyVarsZ>>
+-----------------------------------------------------------------------------
+\* In phase f11, follower sends f.p to leader via FOLLOWERINFO(CEPOCH).
+FollowerSendFOLLOWERINFO(i) ==
+        /\ state[i]    = FOLLOWING
+        /\ zabState[i] = DISCOVERY
+        /\ leaderAddr[i] /= NullPoint
+        /\ \lnot cepochSent[i]
+        /\ Send(i, leaderAddr[i], [mtype  |-> FOLLOWERINFO,
+                                   mepoch |-> acceptedEpoch[i]])
+        /\ cepochSent' = [cepochSent EXCEPT ![i] = TRUE]
+        /\ UNCHANGED <<serverVarsZ, leaderVarsZ, electionVarsZ, tempVarsZ, leaderAddr, synced, verifyVarsZ, electionMsgs, idTable>>
+        
+(* In phase l11, leader waits for receiving FOLLOWERINFO from a quorum,
+   and then chooses a new epoch e' as its own epoch and broadcasts LEADERINFO. *)
+LeaderHandleFOLLOWERINFO(i, j) ==
+        /\ state[i] = LEADING
+        /\ msgs[j][i] /= << >>
+        /\ msgs[j][i][1].mtype = FOLLOWERINFO
+        /\ LET msg == msgs[j][i][1]
+           IN \/ /\ NullPoint \notin cepochRecv[i]   \* 1. has not broadcast LEADERINFO - modify tempMaxEpoch
+                 /\ LET newEpoch == Maximum({tempMaxEpoch[i], msg.mepoch})
+                    IN tempMaxEpoch' = [tempMaxEpoch EXCEPT ![i] = newEpoch]
+                 /\ Discard(j, i)
+              \/ /\ NullPoint \in cepochRecv[i]      \* 2. has broadcast LEADERINFO - no need to handle the msg, just send LEADERINFO to corresponding server
+                 /\ Reply(i, j, [mtype  |-> LEADERINFO,
+                                 mepoch |-> acceptedEpoch[i]])
+                 /\ UNCHANGED tempMaxEpoch
+        /\ cepochRecv' = [cepochRecv EXCEPT ![i] = IF j \in cepochRecv[i] THEN cepochRecv[i]
+                                                                          ELSE cepochRecv[i] \union {j}]
+        /\ UNCHANGED <<serverVarsZ, followerVarsZ, electionVarsZ, initialHistory, leadingVoteSet, learners, ackeRecv, ackldRecv, 
+                       forwarding, ackIndex, currentCounter, sendCounter, committedIndex, committedCounter, verifyVarsZ, electionMsgs, idTable>>
+
+LeaderDiscovery1(i) ==
+        /\ state[i]    = LEADING
+        /\ zabState[i] = DISCOVERY
+        /\ cepochRecv[i] \in Quorums
+        /\ acceptedEpoch' = [acceptedEpoch EXCEPT ![i] = tempMaxEpoch[i] + 1]
+        /\ cepochRecv'    = [cepochRecv    EXCEPT ![i] = cepochRecv[i] \union {NullPoint}]
+        /\ BroadcastLEADERINFO(i, [mtype  |-> LEADERINFO,
+                                   mepoch |-> acceptedEpoch'[i]])
+        /\ UNCHANGED <<state, currentEpoch, lastZxid, zabState, history, commitIndex, electionVarsZ, leadingVoteSet, learners, ackeRecv, ackldRecv, 
+                       forwarding, ackIndex, currentCounter, sendCounter, committedIndex, committedCounter, 
+                       tempVarsZ, followerVarsZ, verifyVarsZ, electionMsgs, idTable>>
+
+(* In phase f12, follower receives NEWEPOCH. If e' > f.p, then follower sends ACK-E back,
+   and ACK-E contains f.a and lastZxid to let leader judge whether it is the latest.
+   After handling NEWEPOCH, follower's zabState turns to SYNCHRONIZATION. *)
+FollowerHandleLEADERINFO(i, j) ==
+        /\ state[i] = FOLLOWING
+        /\ msgs[j][i] /= << >>
+        /\ msgs[j][i][1].mtype = LEADERINFO
+        /\ LET msg     == msgs[j][i][1]
+               infoOk  == j = leaderAddr[i]
+               epochOk == /\ infoOk
+                          /\ msg.mepoch >= acceptedEpoch[i]
+               correct == /\ epochOk
+                          /\ zabState[i] = DISCOVERY
+           IN /\ infoOk
+              /\ \/ /\ epochOk                      \* 1. Normal case
+                    /\ \/ /\ correct
+                          /\ acceptedEpoch' = [acceptedEpoch EXCEPT ![i] = msg.mepoch]
+                          /\ Reply(i, j, [mtype      |-> ACKEPOCH,
+                                          mepoch     |-> msg.mepoch,
+                                          mlastEpoch |-> currentEpoch[i],
+                                          mlastZxid  |-> lastZxid[i]])
+                          /\ cepochSent' = [cepochSent EXCEPT ![i] = TRUE]
+                          /\ UNCHANGED inherentViolated
+                       \/ /\ ~correct
+                          /\ PrintT("Exception: Condition correct is false in FollowerHandleLEADERINFO(" \o ToString(i) \o ", " \o ToString(j) \o ").")
+                          /\ inherentViolated' = TRUE
+                          /\ Discard(j, i)
+                          /\ UNCHANGED <<acceptedEpoch, cepochSent>>
+                    /\ zabState' = [zabState EXCEPT ![i] = IF zabState[i] = DISCOVERY THEN SYNCHRONIZATION
+                                                                                      ELSE zabState[i]]
+                    /\ UNCHANGED <<varsL, leaderAddr>>
+                 \/ /\ ~epochOk                    \* 2. Abnormal case - go back to election
+                    /\ FollowerShutdown(i)
+                    /\ Clean(i, j)
+                    /\ UNCHANGED <<acceptedEpoch, cepochSent, inherentViolated>>
+        /\ UNCHANGED <<history, commitIndex, learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter, sendCounter, committedIndex,
+                       committedCounter, tempVarsZ, synced, proposalMsgsLog, epochLeader>>
+
+\* Abstraction of actions making follower synced with leader before leader sending NEWLEADER.
+subRECOVERYSYNC(i, j) ==
+        LET canSync == /\ state[i] = LEADING   /\ zabState[i] /= DISCOVERY      /\ j \in learners[i]  /\ j \in ackeRecv[i]
+                       /\ state[j] = FOLLOWING /\ zabState[j] = SYNCHRONIZATION /\ leaderAddr[j] = i  /\ synced[j] = FALSE
+        IN
+        \/ /\ canSync
+           /\ history'     = [history     EXCEPT ![j] = history[i]]
+           /\ lastZxid'    = [lastZxid    EXCEPT ![j] = lastZxid[i]]
+           /\ UpdateProposal(j, leaderAddr[j], lastZxid'[j], currentEpoch[j])
+           /\ commitIndex' = [commitIndex EXCEPT ![j] = commitIndex[i]]
+           /\ synced'      = [synced      EXCEPT ![j] = TRUE]
+           /\ forwarding'  = [forwarding  EXCEPT ![i] = forwarding[i] \union {j}]    \* j will receive PROPOSAL and COMMIT
+           /\ ackIndex'    = [ackIndex    EXCEPT ![i][j] = Len(history[i])]
+           /\ committedCounter' = [committedCounter EXCEPT ![i][j] = Maximum({commitIndex[i] - Len(initialHistory[i]), 0})]
+           /\ LET ms == [msource|->i, mtype|->"RECOVERYSYNC", mepoch|->acceptedEpoch[i], mproposals|->history[i]]
+              IN proposalMsgsLog' = IF ms \in proposalMsgsLog THEN proposalMsgsLog
+                                                              ELSE proposalMsgsLog \union {ms}
+           /\ Reply(i, j, [mtype     |-> NEWLEADER,
+                           mepoch    |-> acceptedEpoch[i],
+                           mlastZxid |-> lastZxid[i]])
+        \/ /\ ~canSync
+           /\ Discard(j, i)
+           /\ UNCHANGED <<history, lastZxid, currentVote, commitIndex, synced, forwarding, ackIndex, committedCounter, proposalMsgsLog>>
+        
+(* In phase l12, leader waits for receiving ACKEPOPCH from a quorum,
+   and check whether it has the latest history and epoch from them.
+   If so, leader's zabState turns to SYNCHRONIZATION. *)
+LeaderHandleACKEPOCH(i, j) ==
+        /\ state[i] = LEADING
+        /\ msgs[j][i] /= << >>
+        /\ msgs[j][i][1].mtype = ACKEPOCH
+        /\ LET msg == msgs[j][i][1]
+               infoOk == /\ j \in learners[i]            
+                         /\ acceptedEpoch[i] = msg.mepoch
+               logOk  == /\ infoOk                          \* logOk = TRUE means leader is more up-to-date than follower j
+                         /\ \/ currentEpoch[i] > msg.mlastEpoch
+                            \/ /\ currentEpoch[i] = msg.mlastEpoch
+                               /\ \/ lastZxid[i][1] > msg.mlastZxid[1]
+                                  \/ /\ lastZxid[i][1] = msg.mlastZxid[1]
+                                     /\ lastZxid[i][2] >= msg.mlastZxid[2]
+               replyOk == /\ infoOk
+                          /\ NullPoint \in ackeRecv[i]
+           IN /\ infoOk
+              /\ \/ /\ replyOk
+                    /\ subRECOVERYSYNC(i, j)
+                    /\ ackeRecv' = [ackeRecv EXCEPT ![i] = IF j \notin ackeRecv[i] THEN ackeRecv[i] \union {j}
+                                                                                   ELSE ackeRecv[i]]
+                    /\ UNCHANGED <<state, currentEpoch,logicalClock, receiveVotes, outOfElection, recvQueue, waitNotmsg, leadingVoteSet, electionMsgs, idTable,
+                                   zabState, leaderAddr, learners>>
+                 \/ /\ ~replyOk
+                    /\ \/ /\ logOk
+                          /\ ackeRecv' = [ackeRecv EXCEPT ![i] = IF j \notin ackeRecv[i] THEN ackeRecv[i] \union {j}
+                                                                                         ELSE ackeRecv[i]]
+                          /\ Discard(j, i)
+                          /\ UNCHANGED <<varsL, zabState, leaderAddr, learners, forwarding>>
+                       \/ /\ ~logOk            \* go back to election
+                          /\ LeaderShutdown(i)
+                          /\ UNCHANGED ackeRecv
+                    /\ UNCHANGED <<history, commitIndex, synced, forwarding, ackIndex, committedCounter, proposalMsgsLog>>
+        /\ UNCHANGED <<acceptedEpoch, cepochRecv, ackldRecv,currentCounter, sendCounter, committedIndex, tempVarsZ, cepochSent, epochLeader, inherentViolated>>
+                          
+LeaderDiscovery2(i) ==
+        /\ state[i] = LEADING
+        /\ zabState[i] = DISCOVERY
+        /\ ackeRecv[i] \in Quorums
+        /\ zabState'       = [zabState       EXCEPT ![i] = SYNCHRONIZATION]
+        /\ currentEpoch'   = [currentEpoch   EXCEPT ![i] = acceptedEpoch[i]]
+        /\ initialHistory' = [initialHistory EXCEPT ![i] = history[i]]
+        /\ ackeRecv'       = [ackeRecv       EXCEPT ![i] = ackeRecv[i] \union {NullPoint}]
+        /\ ackIndex'       = [ackIndex       EXCEPT ![i][i] = Len(history[i])]
+        /\ UpdateProposal(i, i, lastZxid[i], currentEpoch'[i])
+        /\ LET epoch == acceptedEpoch[i]
+           IN epochLeader' = [epochLeader EXCEPT ![epoch] = epochLeader[epoch] \union {i}]
+        /\ UNCHANGED <<state, lastZxid, acceptedEpoch, history, commitIndex, logicalClock, receiveVotes, outOfElection, recvQueue, waitNotmsg,
+                       leadingVoteSet, learners, cepochRecv, ackldRecv, forwarding, currentCounter, sendCounter, committedIndex, committedCounter,
+                       tempMaxEpoch, followerVarsZ, proposalMsgsLog, inherentViolated, msgVarsZ, idTable>>
+
+(* Note:  Set cepochRecv, ackeRecv, ackldRecv to {NullPoint} in corresponding three actions to
+          make sure that the prospective leader will not broadcast NEWEPOCH/NEWLEADER/COMMITLD twice.*)
+-----------------------------------------------------------------------------
+RECOVERYSYNC(i, j) ==

Review comment:
       Consider in the future replacing this magic atomic synchronization with DIFF based message passing. The current form could hide a defect. It would increase the state space a lot though, so other state space reduction might be worth doing first.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@zookeeper.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] Vanlightly commented on a change in pull request #1690: ZOOKEEPER-3615: Provide formal specification and verification using TLA+ for Zab

Posted by GitBox <gi...@apache.org>.
Vanlightly commented on a change in pull request #1690:
URL: https://github.com/apache/zookeeper/pull/1690#discussion_r705461423



##########
File path: zookeeper-specifications/zab-1.0/ZabWithFLE.tla
##########
@@ -0,0 +1,1020 @@
+(*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *)
+
+----------------------------- MODULE ZabWithFLE -----------------------------
+(* This is the formal specification for the Zab consensus algorithm,
+   which means Zookeeper Atomic Broadcast.*)
+
+(* Reference:
+   FLE: FastLeaderElection.java, Vote.java, QuorumPeer.java in https://github.com/apache/zookeeper.
+   ZAB: QuorumPeer.java, Learner.java, Follower.java, LearnerHandler.java, Leader.java 
+        in https://github.com/apache/zookeeper.
+        https://cwiki.apache.org/confluence/display/ZOOKEEPER/Zab1.0.
+ *)
+EXTENDS FastLeaderElection
+
+-----------------------------------------------------------------------------
+(* Defined in FastLeaderElection.tla:
+\* The set of server identifiers
+CONSTANT Server
+
+\* Server states
+CONSTANTS LOOKING, FOLLOWING, LEADING
+
+\* Message types
+CONSTANTS NOTIFICATION
+
+\* Timeout signal
+CONSTANT NONE
+*)
+\* The set of requests that can go into history
+CONSTANT Value
+
+\* Zab states
+CONSTANTS ELECTION, DISCOVERY, SYNCHRONIZATION, BROADCAST
+
+\* Message types
+CONSTANTS FOLLOWERINFO, LEADERINFO, ACKEPOCH, NEWLEADER, ACKLD, UPTODATE, PROPOSAL, ACK, COMMIT
+(* Additional message types used for recovery in synchronization(TRUNC/DIFF/SNAP) are not needed
+   since we abstract this part.(see action RECOVERYSYNC) *)
+   
+-----------------------------------------------------------------------------
+(* Defined in FastLeaderElection.tla: Quorums, NullPoint *)
+\* Return the maximum value from the set S
+Maximum(S) == IF S = {} THEN -1
+                        ELSE CHOOSE n \in S: \A m \in S: n >= m
+
+\* Return the minimum value from the set S
+Minimum(S) == IF S = {} THEN -1
+                        ELSE CHOOSE n \in S: \A m \in S: n <= m
+
+MAXEPOCH == 10
+
+-----------------------------------------------------------------------------
+(* Defined in FastLeaderElection.tla:
+   serverVars: <<state, currentEpoch, lastZxid>>,
+   electionVars: <<currentVote, logicalClock, receiveVotes, outOfElection, recvQueue, waitNotmsg>>,
+   leaderVars: <<leadingVoteSet>>,
+   electionMsgs,
+   idTable *)
+\* The current phase of server(ELECTION,DISCOVERY,SYNCHRONIZATION,BROADCAST)
+VARIABLE zabState
+
+\* The epoch number of the last NEWEPOCH(LEADERINFO) packet accepted
+\* namely f.p in paper, and currentEpoch in Zab.tla.
+VARIABLE acceptedEpoch
+
+\* The history of servers as the sequence of transactions.
+VARIABLE history
+
+\* commitIndex[i]: The maximum index of transactions that have been saved in a quorum of servers
+\*                 in the perspective of server i.(increases monotonically before restarting)
+VARIABLE commitIndex
+
+(* These transactions whose index \le commitIndex[i] can be applied to state machine immediately.
+   So if we have a variable applyIndex, we can suppose that applyIndex[i] = commitIndex[i] when verifying properties.
+   But in phase SYNC, follower will apply all queued proposals to state machine when receiving NEWLEADER.
+   But follower only serves traffic after receiving UPTODATE, so sequential consistency is not violated.
+   
+   So when we verify properties, we still suppose applyIndex[i] = commitIndex[i], because this is an engineering detail.*)
+
+\* learners[i]: The set of servers which leader i think are connected wich i.
+VARIABLE learners
+
+\* The messages representing requests and responses sent from one server to another.
+\* msgs[i][j] means the input buffer of server j from server i.
+VARIABLE msgs
+
+\* The set of followers who has successfully sent CEPOCH(FOLLOWERINFO) to leader.(equals to connectingFollowers in code)
+VARIABLE cepochRecv
+
+\* The set of followers who has successfully sent ACK-E to leader.(equals to electingFollowers in code)
+VARIABLE ackeRecv
+
+\* The set of followers who has successfully sent ACK-LD to leader in leader.(equals to newLeaderProposal in code)
+VARIABLE ackldRecv
+
+\* The set of servers which leader i broadcasts PROPOSAL and COMMIT to.(equals to forwardingFollowers in code)
+VARIABLE forwarding
+
+\* ackIndex[i][j]: The latest index that leader i has received from follower j via ACK.
+VARIABLE ackIndex
+
+\* currentCounter[i]: The count of transactions that clients request leader i.
+VARIABLE currentCounter
+
+\* sendCounter[i]: The count of transactions that leader i has broadcast in PROPOSAL.
+VARIABLE sendCounter
+
+\* committedIndex[i]: The maximum index of trasactions that leader i has broadcast in COMMIT.
+VARIABLE committedIndex
+
+\* committedCounter[i][j]: The latest counter of transaction that leader i has confirmed that follower j has committed.
+VARIABLE committedCounter
+
+\* initialHistory[i]: The initial history if leader i in epoch acceptedEpoch[i].
+VARIABLE initialHistory
+
+\* the maximum epoch in CEPOCH the prospective leader received from followers.
+VARIABLE tempMaxEpoch
+
+\* cepochSent[i] = TRUE means follower i has sent CEPOCH(FOLLOWERINFO) to leader.
+VARIABLE cepochSent
+
+\* leaderAddr[i]: The leader id of follower i. We use leaderAddr to express whether follower i has connected or lost connection.
+VARIABLE leaderAddr
+
+\* synced[i] = TRUE: follower i has completed sync with leader.
+VARIABLE synced
+
+\* The set of leaders in every epoch, only used in verifying properties.
+VARIABLE epochLeader
+
+\* The set of all broadcast messages, only used in verifying properties.
+VARIABLE proposalMsgsLog
+
+\* A variable used to check whether there are conditions contrary to the facts.
+VARIABLE inherentViolated
+
+serverVarsZ == <<state, currentEpoch, lastZxid, zabState, acceptedEpoch, history, commitIndex>>         \* 7 variables
+
+electionVarsZ == electionVars  \* 6 variables
+
+leaderVarsZ == <<leadingVoteSet, learners, cepochRecv, ackeRecv, ackldRecv, forwarding, 
+                 ackIndex, currentCounter, sendCounter, committedIndex, committedCounter>>              \* 11 variables
+
+tempVarsZ == <<initialHistory, tempMaxEpoch>>     \* 2 variables
+  
+followerVarsZ == <<cepochSent, leaderAddr, synced>>                 \* 3 variables
+
+verifyVarsZ == <<proposalMsgsLog, epochLeader, inherentViolated>>   \* 3 variables
+
+msgVarsZ == <<msgs, electionMsgs>>                \* 2 variables
+
+vars == <<serverVarsZ, electionVarsZ, leaderVarsZ, tempVarsZ, followerVarsZ, verifyVarsZ, msgVarsZ, idTable>>  \* 35 variables
+-----------------------------------------------------------------------------
+\* Add a message to msgs - add a message m to msgs[i][j].
+Send(i, j, m) == msgs' = [msgs EXCEPT ![i][j] = Append(msgs[i][j], m)]
+
+\* Remove a message from msgs - discard head of msgs[i][j].
+Discard(i, j) == msgs' = IF msgs[i][j] /= << >> THEN [msgs EXCEPT ![i][j] = Tail(msgs[i][j])]
+                                                ELSE msgs
+
+\* Leader broadcasts a message(PROPOSAL/COMMIT) to all other servers in forwardingFollowers.
+Broadcast(i, m) == msgs' = [msgs EXCEPT ![i] = [v \in Server |-> IF /\ v \in forwarding[i]
+                                                                    /\ v /= i
+                                                                    /\ \/ /\ m.mtype = PROPOSAL
+                                                                          /\ ackIndex[i][v] < Len(initialHistory[i]) + m.mproposal.counter
+                                                                       \/ /\ m.mtype = COMMIT
+                                                                          /\ committedCounter[i][v] < m.mzxid[2]
+                                                                  THEN Append(msgs[i][v], m)
+                                                                  ELSE msgs[i][v]]]
+
+BroadcastLEADERINFO(i, m) == msgs' = [msgs EXCEPT ![i] = [v \in Server |-> IF /\ v \in cepochRecv[i]
+                                                                              /\ v \in learners[i]
+                                                                              /\ v /= i THEN Append(msgs[i][v], m)
+                                                                                        ELSE msgs[i][v]]]
+
+BroadcastUPTODATE(i, m) == msgs' = [msgs EXCEPT ![i] = [v \in Server |-> IF /\ v \in ackldRecv[i]
+                                                                            /\ v \in learners[i]
+                                                                            /\ v /= i THEN Append(msgs[i][v], m)
+                                                                                      ELSE msgs[i][v]]]
+
+\* Combination of Send and Discard - discard head of msgs[j][i] and add m into msgs[i][j].
+Reply(i, j, m) == msgs' = [msgs EXCEPT ![j][i] = Tail(msgs[j][i]),
+                                       ![i][j] = Append(msgs[i][j], m)]
+
+\* shuffle the input buffer from server j(i) in server i(j).
+Clean(i, j) == msgs' = [msgs EXCEPT ![j][i] = << >>, ![i][j] = << >>]
+
+-----------------------------------------------------------------------------
+PZxidEqual(p, z) == p.epoch = z[1] /\ p.counter = z[2]
+
+TransactionEqual(t1, t2) == /\ t1.epoch   = t2.epoch
+                            /\ t1.counter = t2.counter
+                            
+TransactionPrecede(t1, t2) == \/ t1.epoch < t2.epoch
+                              \/ /\ t1.epoch   = t2.epoch
+                                 /\ t1.counter < t2.counter
+                                 
+-----------------------------------------------------------------------------
+\* Define initial values for all variables 
+InitServerVarsZ == /\ InitServerVars
+                   /\ zabState      = [s \in Server |-> ELECTION]
+                   /\ acceptedEpoch = [s \in Server |-> 0]
+                   /\ history       = [s \in Server |-> << >>]
+                   /\ commitIndex   = [s \in Server |-> 0]
+
+InitLeaderVarsZ == /\ InitLeaderVars
+                   /\ learners         = [s \in Server |-> {}]
+                   /\ cepochRecv       = [s \in Server |-> {}]
+                   /\ ackeRecv         = [s \in Server |-> {}]
+                   /\ ackldRecv        = [s \in Server |-> {}]
+                   /\ ackIndex         = [s \in Server |-> [v \in Server |-> 0]]
+                   /\ currentCounter   = [s \in Server |-> 0]
+                   /\ sendCounter      = [s \in Server |-> 0]
+                   /\ committedIndex   = [s \in Server |-> 0]
+                   /\ committedCounter = [s \in Server |-> [v \in Server |-> 0]]
+                   /\ forwarding       = [s \in Server |-> {}]
+
+InitElectionVarsZ == InitElectionVars
+
+InitTempVarsZ == /\ initialHistory = [s \in Server |-> << >>]
+                 /\ tempMaxEpoch   = [s \in Server |-> 0]
+
+InitFollowerVarsZ == /\ cepochSent = [s \in Server |-> FALSE]
+                     /\ leaderAddr = [s \in Server |-> NullPoint]
+                     /\ synced     = [s \in Server |-> FALSE]
+
+InitVerifyVarsZ == /\ proposalMsgsLog  = {}
+                   /\ epochLeader      = [i \in 1..MAXEPOCH |-> {}]
+                   /\ inherentViolated = FALSE
+                   
+InitMsgVarsZ == /\ msgs         = [s \in Server |-> [v \in Server |-> << >>]]
+                /\ electionMsgs = [s \in Server |-> [v \in Server |-> << >>]]
+
+InitZ == /\ InitServerVarsZ
+         /\ InitLeaderVarsZ
+         /\ InitElectionVarsZ
+         /\ InitTempVarsZ
+         /\ InitFollowerVarsZ
+         /\ InitVerifyVarsZ
+         /\ InitMsgVarsZ
+         /\ idTable = InitializeIdTable(Server)
+         
+-----------------------------------------------------------------------------
+ZabTurnToLeading(i) ==
+        /\ zabState'       = [zabState   EXCEPT ![i] = DISCOVERY]
+        /\ learners'       = [learners   EXCEPT ![i] = {i}]
+        /\ cepochRecv'     = [cepochRecv EXCEPT ![i] = {i}]
+        /\ ackeRecv'       = [ackeRecv   EXCEPT ![i] = {i}]
+        /\ ackldRecv'      = [ackldRecv  EXCEPT ![i] = {i}]
+        /\ forwarding'     = [forwarding EXCEPT ![i] = {}]
+        /\ ackIndex'       = [ackIndex   EXCEPT ![i] = [v \in Server |-> IF v = i THEN Len(history[i])
+                                                                                  ELSE 0]]
+        /\ currentCounter'   = [currentCounter   EXCEPT ![i] = 0]
+        /\ sendCounter'      = [sendCounter      EXCEPT ![i] = 0]
+        /\ commitIndex'      = [commitIndex      EXCEPT ![i] = 0]
+        /\ committedIndex'   = [committedIndex   EXCEPT ![i] = 0]
+        /\ committedCounter' = [committedCounter EXCEPT ![i] = [v \in Server |-> IF v = i THEN Len(history[i])
+                                                                                          ELSE 0]]
+        /\ initialHistory' = [initialHistory EXCEPT ![i] = history[i]]
+        /\ tempMaxEpoch'   = [tempMaxEpoch   EXCEPT ![i] = acceptedEpoch[i]]
+
+ZabTurnToFollowing(i) ==
+        /\ zabState'    = [zabState   EXCEPT ![i] = DISCOVERY]
+        /\ cepochSent'  = [cepochSent EXCEPT ![i] = FALSE]
+        /\ synced'      = [synced     EXCEPT ![i] = FALSE]
+        /\ commitIndex' = [commitIndex      EXCEPT ![i] = 0]
+        
+          
+(* Fast Leader Election *)
+FLEReceiveNotmsg(i, j) ==
+        /\ ReceiveNotmsg(i, j)
+        /\ UNCHANGED <<zabState, acceptedEpoch, history, commitIndex,learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter,
+                       sendCounter, committedIndex, committedCounter, tempVarsZ, followerVarsZ, verifyVarsZ, msgs>>
+
+FLENotmsgTimeout(i) ==
+        /\ NotmsgTimeout(i)
+        /\ UNCHANGED <<zabState, acceptedEpoch, history, commitIndex,learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter,
+                       sendCounter, committedIndex, committedCounter, tempVarsZ, followerVarsZ, verifyVarsZ, msgs>>
+
+FLEHandleNotmsg(i) ==
+        /\ HandleNotmsg(i)
+        /\ LET newState == state'[i]
+           IN
+           \/ /\ newState = LEADING
+              /\ ZabTurnToLeading(i)
+              /\ UNCHANGED <<cepochSent, synced>>
+           \/ /\ newState = FOLLOWING
+              /\ ZabTurnToFollowing(i)
+              /\ UNCHANGED <<learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter, sendCounter, committedIndex, committedCounter, tempVarsZ>>
+           \/ /\ newState = LOOKING
+              /\ UNCHANGED <<zabState, learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter, sendCounter, commitIndex,
+                             committedIndex, committedCounter, tempVarsZ, cepochSent, synced>>
+        /\ UNCHANGED <<acceptedEpoch, history, leaderAddr, verifyVarsZ, msgs>>
+
+\* On the premise that ReceiveVotes.HasQuorums = TRUE, corresponding to logic in line 1050-1055 in LFE.java.
+FLEWaitNewNotmsg(i) ==
+        /\ WaitNewNotmsg(i)
+        /\ UNCHANGED <<zabState, acceptedEpoch, history, commitIndex,learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter,
+                       sendCounter, committedIndex, committedCounter, tempVarsZ, followerVarsZ, verifyVarsZ, msgs>>
+
+\* On the premise that ReceiveVotes.HasQuorums = TRUE, corresponding to logic in line 1061-1066 in LFE.java.
+FLEWaitNewNotmsgEnd(i) ==
+        /\ WaitNewNotmsgEnd(i)
+        /\ LET newState == state'[i]
+           IN
+           \/ /\ newState = LEADING
+              /\ ZabTurnToLeading(i)
+              /\ UNCHANGED <<cepochSent, synced>>
+           \/ /\ newState = FOLLOWING
+              /\ ZabTurnToFollowing(i)
+              /\ UNCHANGED <<learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter, sendCounter, committedIndex, committedCounter, tempVarsZ>>
+           \/ /\ newState = LOOKING
+              /\ PrintT("New state is LOOKING in FLEWaitNewNotmsgEnd, which should not happen.")
+              /\ UNCHANGED <<zabState, learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter, sendCounter, commitIndex,
+                             committedIndex, committedCounter, tempVarsZ, cepochSent, synced>>
+        /\ UNCHANGED <<acceptedEpoch, history, leaderAddr, verifyVarsZ, msgs>>
+              
+-----------------------------------------------------------------------------
+(* A sub-action describing how a server transitions from LEADING/FOLLOWING to LOOKING.
+   Initially I call it 'ZabTimeoutZ', but it will be called not only when timeout, 
+   but also when finding a low epoch from leader.*)
+FollowerShutdown(i) ==
+        /\ ZabTimeout(i)
+        /\ zabState'   = [zabState   EXCEPT ![i] = ELECTION]
+        /\ leaderAddr' = [leaderAddr EXCEPT ![i] = NullPoint]
+
+LeaderShutdown(i) ==
+        /\ ZabTimeout(i)
+        /\ zabState'   = [zabState   EXCEPT ![i] = ELECTION]
+        /\ leaderAddr' = [s \in Server |-> IF s \in learners[i] THEN NullPoint ELSE leaderAddr[s]]
+        /\ learners'   = [learners   EXCEPT ![i] = {}]
+        /\ forwarding' = [forwarding EXCEPT ![i] = {}]
+        /\ msgs'       = [s \in Server |-> [v \in Server |-> IF v \in learners[i] \/ s \in learners[i] THEN << >> ELSE msgs[s][v]]]
+
+FollowerTimout(i) ==
+        /\ state[i]      = FOLLOWING
+        /\ leaderAddr[i] = NullPoint
+        /\ FollowerShutdown(i)
+        /\ msgs' = [s \in Server |-> [v \in Server |-> IF v = i THEN << >> ELSE msgs[s][v]]]
+        /\ UNCHANGED <<acceptedEpoch, history, commitIndex, learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, 
+                       currentCounter, sendCounter, committedIndex, committedCounter, tempVarsZ, cepochSent, synced, verifyVarsZ>>
+
+LeaderTimeout(i) ==
+        /\ state[i] = LEADING
+        /\ learners[i] \notin Quorums
+        /\ LeaderShutdown(i)
+        /\ UNCHANGED <<acceptedEpoch, history, commitIndex, cepochRecv, ackeRecv, ackldRecv, ackIndex, currentCounter, sendCounter, committedIndex, committedCounter,
+                       tempVarsZ, cepochSent, synced, verifyVarsZ>>
+    
+-----------------------------------------------------------------------------
+(* Establish connection between leader i and follower j. 
+   It means i creates a learnerHandler for communicating with j, and j finds i's address.*)
+EstablishConnection(i, j) ==
+        /\ state[i] = LEADING   /\ state[j] = FOLLOWING
+        /\ j \notin learners[i] /\ leaderAddr[j] = NullPoint
+        /\ currentVote[j].proposedLeader = i
+        /\ learners'   = [learners   EXCEPT ![i] = learners[i] \union {j}] \* Leader:   'addLearnerHandler(peer)'
+        /\ leaderAddr' = [leaderAddr EXCEPT ![j] = i]                      \* Follower: 'connectToLeader(addr, hostname)'
+        /\ UNCHANGED <<serverVarsZ, electionVarsZ, leadingVoteSet, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, 
+                       currentCounter, sendCounter, committedIndex, committedCounter, tempVarsZ, cepochSent, synced, verifyVarsZ, msgVarsZ, idTable>>
+        
+(* The leader i finds timeout and TCP connection between i and j closes.*)       
+Timeout(i, j) ==
+        /\ state[i] = LEADING /\ state[j] = FOLLOWING
+        /\ j \in learners[i]  /\ leaderAddr[j] = i
+        (* The action of leader i.(corresponding to function 'removeLearnerHandler(peer)'.) *)
+        /\ learners'   = [learners   EXCEPT ![i] = learners[i] \ {j}] 
+        /\ forwarding' = [forwarding EXCEPT ![i] = IF j \in forwarding[i] THEN forwarding[i] \ {j} ELSE forwarding[i]]
+        /\ cepochRecv' = [cepochRecv EXCEPT ![i] = IF j \in cepochRecv[i] THEN cepochRecv[i] \ {j} ELSE cepochRecv[i]]
+        /\ ackeRecv'   = [ackeRecv   EXCEPT ![i] = IF j \in ackeRecv[i] THEN ackeRecv[i] \ {j} ELSE ackeRecv[i]]
+        /\ ackldRecv'  = [ackldRecv  EXCEPT ![i] = IF j \in ackldRecv[i] THEN ackldRecv[i] \ {j} ELSE ackldRecv[i]]
+        /\ ackIndex'   = [ackIndex   EXCEPT ![i][j] = 0]
+        /\ committedCounter' = [committedCounter EXCEPT ![i][j] = 0]
+        (* The action of follower j. *)
+        /\ FollowerShutdown(j)
+        (* Clean input buffer.*)
+        /\ Clean(i, j)
+        /\ UNCHANGED <<acceptedEpoch, history, commitIndex, currentCounter, sendCounter, committedIndex,
+                       tempVarsZ, cepochSent, synced, verifyVarsZ>>
+-----------------------------------------------------------------------------
+\* In phase f11, follower sends f.p to leader via FOLLOWERINFO(CEPOCH).
+FollowerSendFOLLOWERINFO(i) ==
+        /\ state[i]    = FOLLOWING
+        /\ zabState[i] = DISCOVERY
+        /\ leaderAddr[i] /= NullPoint
+        /\ \lnot cepochSent[i]
+        /\ Send(i, leaderAddr[i], [mtype  |-> FOLLOWERINFO,
+                                   mepoch |-> acceptedEpoch[i]])
+        /\ cepochSent' = [cepochSent EXCEPT ![i] = TRUE]
+        /\ UNCHANGED <<serverVarsZ, leaderVarsZ, electionVarsZ, tempVarsZ, leaderAddr, synced, verifyVarsZ, electionMsgs, idTable>>
+        
+(* In phase l11, leader waits for receiving FOLLOWERINFO from a quorum,
+   and then chooses a new epoch e' as its own epoch and broadcasts LEADERINFO. *)
+LeaderHandleFOLLOWERINFO(i, j) ==
+        /\ state[i] = LEADING
+        /\ msgs[j][i] /= << >>
+        /\ msgs[j][i][1].mtype = FOLLOWERINFO
+        /\ LET msg == msgs[j][i][1]
+           IN \/ /\ NullPoint \notin cepochRecv[i]   \* 1. has not broadcast LEADERINFO - modify tempMaxEpoch
+                 /\ LET newEpoch == Maximum({tempMaxEpoch[i], msg.mepoch})
+                    IN tempMaxEpoch' = [tempMaxEpoch EXCEPT ![i] = newEpoch]
+                 /\ Discard(j, i)
+              \/ /\ NullPoint \in cepochRecv[i]      \* 2. has broadcast LEADERINFO - no need to handle the msg, just send LEADERINFO to corresponding server
+                 /\ Reply(i, j, [mtype  |-> LEADERINFO,
+                                 mepoch |-> acceptedEpoch[i]])
+                 /\ UNCHANGED tempMaxEpoch
+        /\ cepochRecv' = [cepochRecv EXCEPT ![i] = IF j \in cepochRecv[i] THEN cepochRecv[i]
+                                                                          ELSE cepochRecv[i] \union {j}]
+        /\ UNCHANGED <<serverVarsZ, followerVarsZ, electionVarsZ, initialHistory, leadingVoteSet, learners, ackeRecv, ackldRecv, 
+                       forwarding, ackIndex, currentCounter, sendCounter, committedIndex, committedCounter, verifyVarsZ, electionMsgs, idTable>>
+
+LeaderDiscovery1(i) ==
+        /\ state[i]    = LEADING
+        /\ zabState[i] = DISCOVERY
+        /\ cepochRecv[i] \in Quorums
+        /\ acceptedEpoch' = [acceptedEpoch EXCEPT ![i] = tempMaxEpoch[i] + 1]
+        /\ cepochRecv'    = [cepochRecv    EXCEPT ![i] = cepochRecv[i] \union {NullPoint}]
+        /\ BroadcastLEADERINFO(i, [mtype  |-> LEADERINFO,
+                                   mepoch |-> acceptedEpoch'[i]])
+        /\ UNCHANGED <<state, currentEpoch, lastZxid, zabState, history, commitIndex, electionVarsZ, leadingVoteSet, learners, ackeRecv, ackldRecv, 
+                       forwarding, ackIndex, currentCounter, sendCounter, committedIndex, committedCounter, 
+                       tempVarsZ, followerVarsZ, verifyVarsZ, electionMsgs, idTable>>
+
+(* In phase f12, follower receives NEWEPOCH. If e' > f.p, then follower sends ACK-E back,
+   and ACK-E contains f.a and lastZxid to let leader judge whether it is the latest.
+   After handling NEWEPOCH, follower's zabState turns to SYNCHRONIZATION. *)
+FollowerHandleLEADERINFO(i, j) ==
+        /\ state[i] = FOLLOWING
+        /\ msgs[j][i] /= << >>
+        /\ msgs[j][i][1].mtype = LEADERINFO
+        /\ LET msg     == msgs[j][i][1]
+               infoOk  == j = leaderAddr[i]
+               epochOk == /\ infoOk
+                          /\ msg.mepoch >= acceptedEpoch[i]
+               correct == /\ epochOk
+                          /\ zabState[i] = DISCOVERY
+           IN /\ infoOk
+              /\ \/ /\ epochOk                      \* 1. Normal case
+                    /\ \/ /\ correct
+                          /\ acceptedEpoch' = [acceptedEpoch EXCEPT ![i] = msg.mepoch]
+                          /\ Reply(i, j, [mtype      |-> ACKEPOCH,
+                                          mepoch     |-> msg.mepoch,
+                                          mlastEpoch |-> currentEpoch[i],
+                                          mlastZxid  |-> lastZxid[i]])
+                          /\ cepochSent' = [cepochSent EXCEPT ![i] = TRUE]
+                          /\ UNCHANGED inherentViolated
+                       \/ /\ ~correct
+                          /\ PrintT("Exception: Condition correct is false in FollowerHandleLEADERINFO(" \o ToString(i) \o ", " \o ToString(j) \o ").")
+                          /\ inherentViolated' = TRUE
+                          /\ Discard(j, i)
+                          /\ UNCHANGED <<acceptedEpoch, cepochSent>>
+                    /\ zabState' = [zabState EXCEPT ![i] = IF zabState[i] = DISCOVERY THEN SYNCHRONIZATION
+                                                                                      ELSE zabState[i]]
+                    /\ UNCHANGED <<varsL, leaderAddr>>
+                 \/ /\ ~epochOk                    \* 2. Abnormal case - go back to election
+                    /\ FollowerShutdown(i)
+                    /\ Clean(i, j)
+                    /\ UNCHANGED <<acceptedEpoch, cepochSent, inherentViolated>>
+        /\ UNCHANGED <<history, commitIndex, learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter, sendCounter, committedIndex,
+                       committedCounter, tempVarsZ, synced, proposalMsgsLog, epochLeader>>
+
+\* Abstraction of actions making follower synced with leader before leader sending NEWLEADER.
+subRECOVERYSYNC(i, j) ==
+        LET canSync == /\ state[i] = LEADING   /\ zabState[i] /= DISCOVERY      /\ j \in learners[i]  /\ j \in ackeRecv[i]
+                       /\ state[j] = FOLLOWING /\ zabState[j] = SYNCHRONIZATION /\ leaderAddr[j] = i  /\ synced[j] = FALSE
+        IN
+        \/ /\ canSync
+           /\ history'     = [history     EXCEPT ![j] = history[i]]
+           /\ lastZxid'    = [lastZxid    EXCEPT ![j] = lastZxid[i]]
+           /\ UpdateProposal(j, leaderAddr[j], lastZxid'[j], currentEpoch[j])
+           /\ commitIndex' = [commitIndex EXCEPT ![j] = commitIndex[i]]
+           /\ synced'      = [synced      EXCEPT ![j] = TRUE]
+           /\ forwarding'  = [forwarding  EXCEPT ![i] = forwarding[i] \union {j}]    \* j will receive PROPOSAL and COMMIT
+           /\ ackIndex'    = [ackIndex    EXCEPT ![i][j] = Len(history[i])]
+           /\ committedCounter' = [committedCounter EXCEPT ![i][j] = Maximum({commitIndex[i] - Len(initialHistory[i]), 0})]
+           /\ LET ms == [msource|->i, mtype|->"RECOVERYSYNC", mepoch|->acceptedEpoch[i], mproposals|->history[i]]
+              IN proposalMsgsLog' = IF ms \in proposalMsgsLog THEN proposalMsgsLog
+                                                              ELSE proposalMsgsLog \union {ms}
+           /\ Reply(i, j, [mtype     |-> NEWLEADER,
+                           mepoch    |-> acceptedEpoch[i],
+                           mlastZxid |-> lastZxid[i]])
+        \/ /\ ~canSync
+           /\ Discard(j, i)
+           /\ UNCHANGED <<history, lastZxid, currentVote, commitIndex, synced, forwarding, ackIndex, committedCounter, proposalMsgsLog>>
+        
+(* In phase l12, leader waits for receiving ACKEPOPCH from a quorum,
+   and check whether it has the latest history and epoch from them.
+   If so, leader's zabState turns to SYNCHRONIZATION. *)
+LeaderHandleACKEPOCH(i, j) ==
+        /\ state[i] = LEADING
+        /\ msgs[j][i] /= << >>
+        /\ msgs[j][i][1].mtype = ACKEPOCH
+        /\ LET msg == msgs[j][i][1]
+               infoOk == /\ j \in learners[i]            
+                         /\ acceptedEpoch[i] = msg.mepoch
+               logOk  == /\ infoOk                          \* logOk = TRUE means leader is more up-to-date than follower j
+                         /\ \/ currentEpoch[i] > msg.mlastEpoch
+                            \/ /\ currentEpoch[i] = msg.mlastEpoch
+                               /\ \/ lastZxid[i][1] > msg.mlastZxid[1]
+                                  \/ /\ lastZxid[i][1] = msg.mlastZxid[1]
+                                     /\ lastZxid[i][2] >= msg.mlastZxid[2]
+               replyOk == /\ infoOk
+                          /\ NullPoint \in ackeRecv[i]
+           IN /\ infoOk
+              /\ \/ /\ replyOk
+                    /\ subRECOVERYSYNC(i, j)
+                    /\ ackeRecv' = [ackeRecv EXCEPT ![i] = IF j \notin ackeRecv[i] THEN ackeRecv[i] \union {j}
+                                                                                   ELSE ackeRecv[i]]
+                    /\ UNCHANGED <<state, currentEpoch,logicalClock, receiveVotes, outOfElection, recvQueue, waitNotmsg, leadingVoteSet, electionMsgs, idTable,
+                                   zabState, leaderAddr, learners>>
+                 \/ /\ ~replyOk
+                    /\ \/ /\ logOk
+                          /\ ackeRecv' = [ackeRecv EXCEPT ![i] = IF j \notin ackeRecv[i] THEN ackeRecv[i] \union {j}
+                                                                                         ELSE ackeRecv[i]]
+                          /\ Discard(j, i)
+                          /\ UNCHANGED <<varsL, zabState, leaderAddr, learners, forwarding>>
+                       \/ /\ ~logOk            \* go back to election
+                          /\ LeaderShutdown(i)
+                          /\ UNCHANGED ackeRecv
+                    /\ UNCHANGED <<history, commitIndex, synced, forwarding, ackIndex, committedCounter, proposalMsgsLog>>
+        /\ UNCHANGED <<acceptedEpoch, cepochRecv, ackldRecv,currentCounter, sendCounter, committedIndex, tempVarsZ, cepochSent, epochLeader, inherentViolated>>
+                          
+LeaderDiscovery2(i) ==
+        /\ state[i] = LEADING
+        /\ zabState[i] = DISCOVERY
+        /\ ackeRecv[i] \in Quorums
+        /\ zabState'       = [zabState       EXCEPT ![i] = SYNCHRONIZATION]
+        /\ currentEpoch'   = [currentEpoch   EXCEPT ![i] = acceptedEpoch[i]]
+        /\ initialHistory' = [initialHistory EXCEPT ![i] = history[i]]
+        /\ ackeRecv'       = [ackeRecv       EXCEPT ![i] = ackeRecv[i] \union {NullPoint}]
+        /\ ackIndex'       = [ackIndex       EXCEPT ![i][i] = Len(history[i])]
+        /\ UpdateProposal(i, i, lastZxid[i], currentEpoch'[i])
+        /\ LET epoch == acceptedEpoch[i]
+           IN epochLeader' = [epochLeader EXCEPT ![epoch] = epochLeader[epoch] \union {i}]
+        /\ UNCHANGED <<state, lastZxid, acceptedEpoch, history, commitIndex, logicalClock, receiveVotes, outOfElection, recvQueue, waitNotmsg,
+                       leadingVoteSet, learners, cepochRecv, ackldRecv, forwarding, currentCounter, sendCounter, committedIndex, committedCounter,
+                       tempMaxEpoch, followerVarsZ, proposalMsgsLog, inherentViolated, msgVarsZ, idTable>>
+
+(* Note:  Set cepochRecv, ackeRecv, ackldRecv to {NullPoint} in corresponding three actions to
+          make sure that the prospective leader will not broadcast NEWEPOCH/NEWLEADER/COMMITLD twice.*)
+-----------------------------------------------------------------------------
+RECOVERYSYNC(i, j) ==
+        /\ state[i] = LEADING   /\ zabState[i] /= DISCOVERY      /\ j \in learners[i] /\ j \in ackeRecv[i]
+        /\ state[j] = FOLLOWING /\ zabState[j] = SYNCHRONIZATION /\ leaderAddr[j] = i /\ synced[j] = FALSE
+        \* /\ acceptedEpoch[i] = acceptedEpoch[j] \* This condition is unnecessary.
+        /\ history'     = [history     EXCEPT ![j] = history[i]]
+        /\ lastZxid'    = [lastZxid    EXCEPT ![j] = lastZxid[i]]
+        /\ UpdateProposal(j, leaderAddr[j], lastZxid'[j], currentEpoch[j])
+        /\ commitIndex' = [commitIndex EXCEPT ![j] = commitIndex[i]]
+        /\ synced'      = [synced      EXCEPT ![j] = TRUE]
+        /\ forwarding'  = [forwarding  EXCEPT ![i] = forwarding[i] \union {j}]
+        /\ ackIndex'    = [ackIndex    EXCEPT ![i][j] = Len(history[i])]
+        /\ committedCounter' = [committedCounter EXCEPT ![i][j] = Maximum({commitIndex[i] - Len(initialHistory[i]), 0})]
+        /\ LET ms == [msource|->i, mtype|->"RECOVERYSYNC", mepoch|->acceptedEpoch[i], mproposals|->history[i]]
+           IN proposalMsgsLog' = IF ms \in proposalMsgsLog THEN proposalMsgsLog
+                                                           ELSE proposalMsgsLog \union {ms}
+        /\ Send(i, j, [mtype     |-> NEWLEADER,
+                       mepoch    |-> acceptedEpoch[i],
+                       mlastZxid |-> lastZxid[i]])
+        /\ UNCHANGED <<state, zabState, acceptedEpoch, currentEpoch, logicalClock, receiveVotes, outOfElection, recvQueue, waitNotmsg, 
+                       leadingVoteSet, learners, cepochRecv, ackeRecv, ackldRecv, currentCounter, sendCounter, committedIndex, 
+                       tempVarsZ, cepochSent, leaderAddr, epochLeader, inherentViolated, electionMsgs, idTable>>
+
+(* In phase f21, follower receives NEWLEADER. The follower updates its epoch and history,
+   and sends back ACK-LD to leader. *)
+FollowerHandleNEWLEADER(i, j) ==
+        /\ state[i] = FOLLOWING
+        /\ msgs[j][i] /= << >>
+        /\ msgs[j][i][1].mtype = NEWLEADER
+        /\ LET msg     == msgs[j][i][1]
+               infoOk  == /\ leaderAddr[i] = j
+                          /\ acceptedEpoch[i] = msg.mepoch
+               correct == /\ infoOk
+                          /\ zabState[i] = SYNCHRONIZATION
+                          /\ synced[i]
+                          /\ ZxidEqual(lastZxid[i], msg.mlastZxid)
+           IN /\ infoOk
+              /\ currentEpoch' = [currentEpoch EXCEPT ![i] = msg.mepoch]
+              /\ UpdateProposal(i, j, lastZxid[i], currentEpoch'[i])
+              /\ \/ /\ correct
+                    /\ Reply(i, j, [mtype  |-> ACKLD,
+                                    mepoch |-> msg.mepoch])
+                    /\ UNCHANGED inherentViolated
+                 \/ /\ ~correct
+                    /\ PrintT("Exception: Condition correct is false in FollowerHandleNEWLEADER(" \o ToString(i) \o ", " \o ToString(j) \o ").")
+                    /\ inherentViolated' = TRUE
+                    /\ Discard(j, i)
+        /\ UNCHANGED <<state, lastZxid, zabState, acceptedEpoch, history, commitIndex, logicalClock, receiveVotes, outOfElection, recvQueue, waitNotmsg,
+                       leaderVarsZ, tempVarsZ, followerVarsZ, proposalMsgsLog, epochLeader, electionMsgs, idTable>>
+
+\* In phase l22, leader receives ACK-LD from a quorum of followers, and sends COMMIT-LD(UPTODATE) to followers.
+LeaderHandleACKLD(i, j) ==
+        /\ state[i] = LEADING
+        /\ msgs[j][i] /= << >>
+        /\ msgs[j][i][1].mtype = ACKLD
+        /\ LET msg     == msgs[j][i][1]
+               infoOk  == /\ acceptedEpoch[i] = msg.mepoch
+                          /\ j \in learners[i]
+               replyOk == /\ infoOk
+                          /\ NullPoint \in ackldRecv[i]
+           IN /\ infoOk
+              /\ \/ /\ replyOk
+                    /\ Reply(i, j, [mtype   |-> UPTODATE,
+                                    mepoch  |-> acceptedEpoch[i],
+                                    mcommit |-> commitIndex[i]])
+                    /\ committedCounter' = [committedCounter EXCEPT ![i][j] = Maximum({commitIndex[i] - Len(initialHistory[i]), committedCounter[i][j]})]
+                 \/ /\ ~replyOk
+                    /\ Discard(j, i)
+                    /\ UNCHANGED committedCounter
+              /\ ackldRecv' = [ackldRecv EXCEPT ![i] = IF j \notin ackldRecv[i] THEN ackldRecv[i] \union {j}
+                                                                                ELSE ackldRecv[i]]
+        /\ UNCHANGED <<serverVarsZ, electionVarsZ, leadingVoteSet, learners, cepochRecv, ackeRecv, forwarding, 
+                       ackIndex, currentCounter, sendCounter, committedIndex, tempVarsZ, followerVarsZ, verifyVarsZ, electionMsgs, idTable>>
+
+LeaderSync2(i) ==
+        /\ state[i] = LEADING
+        /\ zabState[i] = SYNCHRONIZATION
+        /\ ackldRecv[i] \in Quorums
+        /\ commitIndex'    = [commitIndex    EXCEPT ![i] = Len(history[i])]
+        /\ committedIndex' = [committedIndex EXCEPT ![i] = Len(history[i])]
+        /\ zabState'       = [zabState       EXCEPT ![i] = BROADCAST]
+        /\ currentCounter' = [currentCounter EXCEPT ![i] = 0]
+        /\ sendCounter'    = [sendCounter    EXCEPT ![i] = 0]
+        /\ ackldRecv'      = [ackldRecv      EXCEPT ![i] = ackldRecv[i] \union {NullPoint}]
+        /\ BroadcastUPTODATE(i, [mtype   |-> UPTODATE,
+                                 mepoch  |-> acceptedEpoch[i],
+                                 mcommit |-> Len(history[i])]) \* In actual UPTODATE doesn't carry this info
+        /\ UNCHANGED <<state, currentEpoch, lastZxid, acceptedEpoch, history, electionVarsZ, leadingVoteSet, learners, cepochRecv, ackeRecv, forwarding, ackIndex, 
+                       committedCounter, tempVarsZ, followerVarsZ, verifyVarsZ, electionMsgs, idTable>>
+
+FollowerHandleUPTODATE(i, j) ==
+        /\ state[i] = FOLLOWING
+        /\ msgs[j][i] /= << >>
+        /\ msgs[j][i][1].mtype = UPTODATE
+        /\ LET msg == msgs[j][i][1]
+               infoOk == /\ leaderAddr[i] = j
+                         /\ acceptedEpoch[i] = msg.mepoch
+               correct == /\ infoOk
+                          /\ zabState[i] = SYNCHRONIZATION
+                          /\ currentEpoch[i] = msg.mepoch
+           IN /\ infoOk
+              /\ \/ /\ correct
+                    /\ commitIndex' = [commitIndex EXCEPT ![i] = Maximum({commitIndex[i], msg.mcommit})] \* May have received COMMIT and have a bigger commit index.
+                    /\ zabState'    = [zabState    EXCEPT ![i] = BROADCAST]
+                    /\ UNCHANGED inherentViolated
+                 \/ /\ ~correct
+                    /\ PrintT("Exception: Condition correct is false in FollowerHandleUPTODATE(" \o ToString(i) \o ", " \o ToString(j) \o ").")
+                    /\ inherentViolated' = TRUE
+                    /\ UNCHANGED <<commitIndex, zabState>>
+        /\ Discard(j, i)
+        /\ UNCHANGED <<state, currentEpoch, lastZxid, acceptedEpoch, history, electionVarsZ, leaderVarsZ, tempVarsZ, followerVarsZ,
+                       proposalMsgsLog, epochLeader, electionMsgs, idTable>>
+
+-----------------------------------------------------------------------------
+(* In phase l31, leader receives client request and broadcasts PROPOSAL.
+   Note: In production, any server in traffic can receive requests and forward it to leader if necessary.
+   We choose to let leader be the sole one who can receive requests, to simplify spec and keep correctness at the same time. *)
+ClientRequest(i, v) ==
+        /\ state[i] = LEADING
+        /\ zabState[i] = BROADCAST
+        /\ currentCounter' = [currentCounter EXCEPT ![i] = currentCounter[i] + 1]
+        /\ LET newTransaction == [epoch   |-> acceptedEpoch[i],
+                                  counter |-> currentCounter'[i],
+                                  value   |-> v]
+           IN /\ history'  = [history  EXCEPT ![i] = Append(history[i], newTransaction)]
+              /\ lastZxid' = [lastZxid EXCEPT ![i] = <<acceptedEpoch[i], currentCounter'[i]>>]
+              /\ ackIndex' = [ackIndex EXCEPT ![i][i] = Len(history'[i])]
+              /\ UpdateProposal(i, i, lastZxid'[i], currentEpoch[i])
+        /\ UNCHANGED <<state, currentEpoch, zabState, acceptedEpoch, commitIndex, logicalClock, receiveVotes, outOfElection, recvQueue, waitNotmsg,
+                       leadingVoteSet, learners, cepochRecv, ackeRecv, ackldRecv, forwarding, sendCounter, committedIndex, committedCounter,
+                       tempVarsZ, followerVarsZ, verifyVarsZ, msgVarsZ, idTable>>
+
+LeaderBroadcast1(i) ==

Review comment:
       Action name is not descriptive, perhaps rename to something like LeaderBroadcastProposal




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@zookeeper.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] BinyuHuang-nju commented on pull request #1690: ZOOKEEPER-3615: Provide formal specification and verification using TLA+ for Zab

Posted by GitBox <gi...@apache.org>.
BinyuHuang-nju commented on pull request #1690:
URL: https://github.com/apache/zookeeper/pull/1690#issuecomment-853899721


   @hanm OK, thank you very much! You replies above will help me a lot.
    Currently I am devoted to writing spec for ZAB 1.0, and abstract two modules of election and sync phase before sending NEWLEADER. 
   I think I can implement election module if necessary. And I need to learn more about sync phase if we need to implement this part.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] BinyuHuang-nju edited a comment on pull request #1690: ZOOKEEPER-3615: Provide formal specification and verification using TLA+ for Zab

Posted by GitBox <gi...@apache.org>.
BinyuHuang-nju edited a comment on pull request #1690:
URL: https://github.com/apache/zookeeper/pull/1690#issuecomment-844163499


   @hanm Maybe it would be inappropriate for me to ask the following questions under this PR. Now I am learning ZAB 1.0 and prepare to write spec. Could you please let me ask about details in it?
   
   Currently I have several problems about ZAB 1.0.
   1. For some detailed industial implementation (like message with type PING, connection between leader and followers), do I need to implement these in spec?
   2. I see before sending NEWLEADER, leader will choose a best message from SNAP,TRUNC,DIFF to send according to the corresponding follower's state. Do I need to implement these, or abstract this part and just send NEWLEADER to sync with followers?
   3. I don't understand whether variable 'loracle' is more like a global variable that has the latest leader ID and every server can reach it, or 'loracle' is a local variable like 'votedFor' in Raft. (Actually I used the latter in Zab.tla, which corresponds to 'leaderOracle'. And I think it has no effect on correctness no matter which one is used.)
   4. The last problem what I want to ask is that about recovery, because I didn't see notes about this part in paper and the link. 
   I saw how the leader handles when a new server wants to join them, but I didn't see how a restarted server finds the latest leader. If 'loracle' is the former in question 3, I can understand this part. In Zab.tla, I use the method which the follower broadcasts messages to ack other servers what their local oracle is, and update its oracle when receiving the same oracle and epoch from a quorum(I used this method because I saw servers recover like this in View-stamped Replication).
   A follower must have received corresponding PROPOSAL when receiving COMMIT, if the follower is the initial server that joins Q. But I think this may not always be true when the follower is one that joins Q midway. So I want to know how follower handles to catch up state when receiving COMMIT corresponding to a transaction that not exists in its local history.(You could see this condition in the image at the bottom of README. When I wrote spec for Zab pre-1.0, I choose to let followers keep re-sending CEPOCH to obtain latest transactions until the conflict described above disappers.)
   
   I am very sorry if my problems bother you. Thank you for the various feedbacks and suggestions you have provided before!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] BinyuHuang-nju commented on pull request #1690: Provide formal specification and verification using TLA+ for Zab

Posted by GitBox <gi...@apache.org>.
BinyuHuang-nju commented on pull request #1690:
URL: https://github.com/apache/zookeeper/pull/1690#issuecomment-831296902


   @Vanlightly @fpj I am very glad to see your replies!!  
   
   I have found faults in my spec that resulted in invariant violations about LocalPrimaryOrder and Integrity.
   In writing LocalPrimaryOrder, I wrongly wrote 'A=>B' as 'A /\ B'.
   Invariant Integirty means, if some follower delivers one transaction, then some primary must have broadcast it. But I only considered broadcast in msg with type PROPOSE from phase3(normal case), and ignored broadcast in msg with type NEWLEADER from phase2(Sync).
   
   Now, I have fixed these bugs, and have added this commit. You could download new spec and run models. Until now, by using Servers {s1,s2} and Values {v1,v2}(it is not important to change CONSTANT Value) as sets of model values, and adding all invariants that I defined, I have not found errors.
   
   I will run this model and update data tomorrow. Since it is half past ten in China, I have to go home, and I will check your replies tomorrow.
   
   Thanks again for your attention!! (By the way, since this is a project independent of zookeeper, and I am a rookie in using Travis CI, how can I pass Travis CI building?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] Vanlightly commented on a change in pull request #1690: ZOOKEEPER-3615: Provide formal specification and verification using TLA+ for Zab

Posted by GitBox <gi...@apache.org>.
Vanlightly commented on a change in pull request #1690:
URL: https://github.com/apache/zookeeper/pull/1690#discussion_r705460217



##########
File path: zookeeper-specifications/zab-1.0/ZabWithFLE.tla
##########
@@ -0,0 +1,1020 @@
+(*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *)
+
+----------------------------- MODULE ZabWithFLE -----------------------------
+(* This is the formal specification for the Zab consensus algorithm,
+   which means Zookeeper Atomic Broadcast.*)
+
+(* Reference:
+   FLE: FastLeaderElection.java, Vote.java, QuorumPeer.java in https://github.com/apache/zookeeper.
+   ZAB: QuorumPeer.java, Learner.java, Follower.java, LearnerHandler.java, Leader.java 
+        in https://github.com/apache/zookeeper.
+        https://cwiki.apache.org/confluence/display/ZOOKEEPER/Zab1.0.
+ *)
+EXTENDS FastLeaderElection
+
+-----------------------------------------------------------------------------
+(* Defined in FastLeaderElection.tla:
+\* The set of server identifiers
+CONSTANT Server
+
+\* Server states
+CONSTANTS LOOKING, FOLLOWING, LEADING
+
+\* Message types
+CONSTANTS NOTIFICATION
+
+\* Timeout signal
+CONSTANT NONE
+*)
+\* The set of requests that can go into history
+CONSTANT Value
+
+\* Zab states
+CONSTANTS ELECTION, DISCOVERY, SYNCHRONIZATION, BROADCAST
+
+\* Message types
+CONSTANTS FOLLOWERINFO, LEADERINFO, ACKEPOCH, NEWLEADER, ACKLD, UPTODATE, PROPOSAL, ACK, COMMIT
+(* Additional message types used for recovery in synchronization(TRUNC/DIFF/SNAP) are not needed
+   since we abstract this part.(see action RECOVERYSYNC) *)
+   
+-----------------------------------------------------------------------------
+(* Defined in FastLeaderElection.tla: Quorums, NullPoint *)
+\* Return the maximum value from the set S
+Maximum(S) == IF S = {} THEN -1
+                        ELSE CHOOSE n \in S: \A m \in S: n >= m
+
+\* Return the minimum value from the set S
+Minimum(S) == IF S = {} THEN -1
+                        ELSE CHOOSE n \in S: \A m \in S: n <= m
+
+MAXEPOCH == 10
+
+-----------------------------------------------------------------------------
+(* Defined in FastLeaderElection.tla:
+   serverVars: <<state, currentEpoch, lastZxid>>,
+   electionVars: <<currentVote, logicalClock, receiveVotes, outOfElection, recvQueue, waitNotmsg>>,
+   leaderVars: <<leadingVoteSet>>,
+   electionMsgs,
+   idTable *)
+\* The current phase of server(ELECTION,DISCOVERY,SYNCHRONIZATION,BROADCAST)
+VARIABLE zabState
+
+\* The epoch number of the last NEWEPOCH(LEADERINFO) packet accepted
+\* namely f.p in paper, and currentEpoch in Zab.tla.
+VARIABLE acceptedEpoch
+
+\* The history of servers as the sequence of transactions.
+VARIABLE history
+
+\* commitIndex[i]: The maximum index of transactions that have been saved in a quorum of servers
+\*                 in the perspective of server i.(increases monotonically before restarting)
+VARIABLE commitIndex
+
+(* These transactions whose index \le commitIndex[i] can be applied to state machine immediately.
+   So if we have a variable applyIndex, we can suppose that applyIndex[i] = commitIndex[i] when verifying properties.
+   But in phase SYNC, follower will apply all queued proposals to state machine when receiving NEWLEADER.
+   But follower only serves traffic after receiving UPTODATE, so sequential consistency is not violated.
+   
+   So when we verify properties, we still suppose applyIndex[i] = commitIndex[i], because this is an engineering detail.*)
+
+\* learners[i]: The set of servers which leader i think are connected wich i.
+VARIABLE learners
+
+\* The messages representing requests and responses sent from one server to another.
+\* msgs[i][j] means the input buffer of server j from server i.
+VARIABLE msgs
+
+\* The set of followers who has successfully sent CEPOCH(FOLLOWERINFO) to leader.(equals to connectingFollowers in code)
+VARIABLE cepochRecv
+
+\* The set of followers who has successfully sent ACK-E to leader.(equals to electingFollowers in code)
+VARIABLE ackeRecv
+
+\* The set of followers who has successfully sent ACK-LD to leader in leader.(equals to newLeaderProposal in code)
+VARIABLE ackldRecv
+
+\* The set of servers which leader i broadcasts PROPOSAL and COMMIT to.(equals to forwardingFollowers in code)
+VARIABLE forwarding
+
+\* ackIndex[i][j]: The latest index that leader i has received from follower j via ACK.
+VARIABLE ackIndex
+
+\* currentCounter[i]: The count of transactions that clients request leader i.
+VARIABLE currentCounter
+
+\* sendCounter[i]: The count of transactions that leader i has broadcast in PROPOSAL.
+VARIABLE sendCounter
+
+\* committedIndex[i]: The maximum index of trasactions that leader i has broadcast in COMMIT.
+VARIABLE committedIndex
+
+\* committedCounter[i][j]: The latest counter of transaction that leader i has confirmed that follower j has committed.
+VARIABLE committedCounter
+
+\* initialHistory[i]: The initial history if leader i in epoch acceptedEpoch[i].
+VARIABLE initialHistory
+
+\* the maximum epoch in CEPOCH the prospective leader received from followers.
+VARIABLE tempMaxEpoch
+
+\* cepochSent[i] = TRUE means follower i has sent CEPOCH(FOLLOWERINFO) to leader.
+VARIABLE cepochSent
+
+\* leaderAddr[i]: The leader id of follower i. We use leaderAddr to express whether follower i has connected or lost connection.
+VARIABLE leaderAddr
+
+\* synced[i] = TRUE: follower i has completed sync with leader.
+VARIABLE synced
+
+\* The set of leaders in every epoch, only used in verifying properties.
+VARIABLE epochLeader
+
+\* The set of all broadcast messages, only used in verifying properties.
+VARIABLE proposalMsgsLog
+
+\* A variable used to check whether there are conditions contrary to the facts.
+VARIABLE inherentViolated
+
+serverVarsZ == <<state, currentEpoch, lastZxid, zabState, acceptedEpoch, history, commitIndex>>         \* 7 variables
+
+electionVarsZ == electionVars  \* 6 variables
+
+leaderVarsZ == <<leadingVoteSet, learners, cepochRecv, ackeRecv, ackldRecv, forwarding, 
+                 ackIndex, currentCounter, sendCounter, committedIndex, committedCounter>>              \* 11 variables
+
+tempVarsZ == <<initialHistory, tempMaxEpoch>>     \* 2 variables
+  
+followerVarsZ == <<cepochSent, leaderAddr, synced>>                 \* 3 variables
+
+verifyVarsZ == <<proposalMsgsLog, epochLeader, inherentViolated>>   \* 3 variables
+
+msgVarsZ == <<msgs, electionMsgs>>                \* 2 variables
+
+vars == <<serverVarsZ, electionVarsZ, leaderVarsZ, tempVarsZ, followerVarsZ, verifyVarsZ, msgVarsZ, idTable>>  \* 35 variables
+-----------------------------------------------------------------------------
+\* Add a message to msgs - add a message m to msgs[i][j].
+Send(i, j, m) == msgs' = [msgs EXCEPT ![i][j] = Append(msgs[i][j], m)]
+
+\* Remove a message from msgs - discard head of msgs[i][j].
+Discard(i, j) == msgs' = IF msgs[i][j] /= << >> THEN [msgs EXCEPT ![i][j] = Tail(msgs[i][j])]
+                                                ELSE msgs
+
+\* Leader broadcasts a message(PROPOSAL/COMMIT) to all other servers in forwardingFollowers.
+Broadcast(i, m) == msgs' = [msgs EXCEPT ![i] = [v \in Server |-> IF /\ v \in forwarding[i]
+                                                                    /\ v /= i
+                                                                    /\ \/ /\ m.mtype = PROPOSAL
+                                                                          /\ ackIndex[i][v] < Len(initialHistory[i]) + m.mproposal.counter
+                                                                       \/ /\ m.mtype = COMMIT
+                                                                          /\ committedCounter[i][v] < m.mzxid[2]
+                                                                  THEN Append(msgs[i][v], m)
+                                                                  ELSE msgs[i][v]]]
+
+BroadcastLEADERINFO(i, m) == msgs' = [msgs EXCEPT ![i] = [v \in Server |-> IF /\ v \in cepochRecv[i]
+                                                                              /\ v \in learners[i]
+                                                                              /\ v /= i THEN Append(msgs[i][v], m)
+                                                                                        ELSE msgs[i][v]]]
+
+BroadcastUPTODATE(i, m) == msgs' = [msgs EXCEPT ![i] = [v \in Server |-> IF /\ v \in ackldRecv[i]
+                                                                            /\ v \in learners[i]
+                                                                            /\ v /= i THEN Append(msgs[i][v], m)
+                                                                                      ELSE msgs[i][v]]]
+
+\* Combination of Send and Discard - discard head of msgs[j][i] and add m into msgs[i][j].
+Reply(i, j, m) == msgs' = [msgs EXCEPT ![j][i] = Tail(msgs[j][i]),
+                                       ![i][j] = Append(msgs[i][j], m)]
+
+\* shuffle the input buffer from server j(i) in server i(j).
+Clean(i, j) == msgs' = [msgs EXCEPT ![j][i] = << >>, ![i][j] = << >>]
+
+-----------------------------------------------------------------------------
+PZxidEqual(p, z) == p.epoch = z[1] /\ p.counter = z[2]
+
+TransactionEqual(t1, t2) == /\ t1.epoch   = t2.epoch
+                            /\ t1.counter = t2.counter
+                            
+TransactionPrecede(t1, t2) == \/ t1.epoch < t2.epoch
+                              \/ /\ t1.epoch   = t2.epoch
+                                 /\ t1.counter < t2.counter
+                                 
+-----------------------------------------------------------------------------
+\* Define initial values for all variables 
+InitServerVarsZ == /\ InitServerVars
+                   /\ zabState      = [s \in Server |-> ELECTION]
+                   /\ acceptedEpoch = [s \in Server |-> 0]
+                   /\ history       = [s \in Server |-> << >>]
+                   /\ commitIndex   = [s \in Server |-> 0]
+
+InitLeaderVarsZ == /\ InitLeaderVars
+                   /\ learners         = [s \in Server |-> {}]
+                   /\ cepochRecv       = [s \in Server |-> {}]
+                   /\ ackeRecv         = [s \in Server |-> {}]
+                   /\ ackldRecv        = [s \in Server |-> {}]
+                   /\ ackIndex         = [s \in Server |-> [v \in Server |-> 0]]
+                   /\ currentCounter   = [s \in Server |-> 0]
+                   /\ sendCounter      = [s \in Server |-> 0]
+                   /\ committedIndex   = [s \in Server |-> 0]
+                   /\ committedCounter = [s \in Server |-> [v \in Server |-> 0]]
+                   /\ forwarding       = [s \in Server |-> {}]
+
+InitElectionVarsZ == InitElectionVars
+
+InitTempVarsZ == /\ initialHistory = [s \in Server |-> << >>]
+                 /\ tempMaxEpoch   = [s \in Server |-> 0]
+
+InitFollowerVarsZ == /\ cepochSent = [s \in Server |-> FALSE]
+                     /\ leaderAddr = [s \in Server |-> NullPoint]
+                     /\ synced     = [s \in Server |-> FALSE]
+
+InitVerifyVarsZ == /\ proposalMsgsLog  = {}
+                   /\ epochLeader      = [i \in 1..MAXEPOCH |-> {}]
+                   /\ inherentViolated = FALSE
+                   
+InitMsgVarsZ == /\ msgs         = [s \in Server |-> [v \in Server |-> << >>]]
+                /\ electionMsgs = [s \in Server |-> [v \in Server |-> << >>]]
+
+InitZ == /\ InitServerVarsZ
+         /\ InitLeaderVarsZ
+         /\ InitElectionVarsZ
+         /\ InitTempVarsZ
+         /\ InitFollowerVarsZ
+         /\ InitVerifyVarsZ
+         /\ InitMsgVarsZ
+         /\ idTable = InitializeIdTable(Server)
+         
+-----------------------------------------------------------------------------
+ZabTurnToLeading(i) ==
+        /\ zabState'       = [zabState   EXCEPT ![i] = DISCOVERY]
+        /\ learners'       = [learners   EXCEPT ![i] = {i}]
+        /\ cepochRecv'     = [cepochRecv EXCEPT ![i] = {i}]
+        /\ ackeRecv'       = [ackeRecv   EXCEPT ![i] = {i}]
+        /\ ackldRecv'      = [ackldRecv  EXCEPT ![i] = {i}]
+        /\ forwarding'     = [forwarding EXCEPT ![i] = {}]
+        /\ ackIndex'       = [ackIndex   EXCEPT ![i] = [v \in Server |-> IF v = i THEN Len(history[i])
+                                                                                  ELSE 0]]
+        /\ currentCounter'   = [currentCounter   EXCEPT ![i] = 0]
+        /\ sendCounter'      = [sendCounter      EXCEPT ![i] = 0]
+        /\ commitIndex'      = [commitIndex      EXCEPT ![i] = 0]
+        /\ committedIndex'   = [committedIndex   EXCEPT ![i] = 0]
+        /\ committedCounter' = [committedCounter EXCEPT ![i] = [v \in Server |-> IF v = i THEN Len(history[i])
+                                                                                          ELSE 0]]
+        /\ initialHistory' = [initialHistory EXCEPT ![i] = history[i]]
+        /\ tempMaxEpoch'   = [tempMaxEpoch   EXCEPT ![i] = acceptedEpoch[i]]
+
+ZabTurnToFollowing(i) ==
+        /\ zabState'    = [zabState   EXCEPT ![i] = DISCOVERY]
+        /\ cepochSent'  = [cepochSent EXCEPT ![i] = FALSE]
+        /\ synced'      = [synced     EXCEPT ![i] = FALSE]
+        /\ commitIndex' = [commitIndex      EXCEPT ![i] = 0]
+        
+          
+(* Fast Leader Election *)
+FLEReceiveNotmsg(i, j) ==
+        /\ ReceiveNotmsg(i, j)
+        /\ UNCHANGED <<zabState, acceptedEpoch, history, commitIndex,learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter,
+                       sendCounter, committedIndex, committedCounter, tempVarsZ, followerVarsZ, verifyVarsZ, msgs>>
+
+FLENotmsgTimeout(i) ==
+        /\ NotmsgTimeout(i)
+        /\ UNCHANGED <<zabState, acceptedEpoch, history, commitIndex,learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter,
+                       sendCounter, committedIndex, committedCounter, tempVarsZ, followerVarsZ, verifyVarsZ, msgs>>
+
+FLEHandleNotmsg(i) ==
+        /\ HandleNotmsg(i)
+        /\ LET newState == state'[i]
+           IN
+           \/ /\ newState = LEADING
+              /\ ZabTurnToLeading(i)
+              /\ UNCHANGED <<cepochSent, synced>>
+           \/ /\ newState = FOLLOWING
+              /\ ZabTurnToFollowing(i)
+              /\ UNCHANGED <<learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter, sendCounter, committedIndex, committedCounter, tempVarsZ>>
+           \/ /\ newState = LOOKING
+              /\ UNCHANGED <<zabState, learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter, sendCounter, commitIndex,
+                             committedIndex, committedCounter, tempVarsZ, cepochSent, synced>>
+        /\ UNCHANGED <<acceptedEpoch, history, leaderAddr, verifyVarsZ, msgs>>
+
+\* On the premise that ReceiveVotes.HasQuorums = TRUE, corresponding to logic in line 1050-1055 in LFE.java.
+FLEWaitNewNotmsg(i) ==
+        /\ WaitNewNotmsg(i)
+        /\ UNCHANGED <<zabState, acceptedEpoch, history, commitIndex,learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter,
+                       sendCounter, committedIndex, committedCounter, tempVarsZ, followerVarsZ, verifyVarsZ, msgs>>
+
+\* On the premise that ReceiveVotes.HasQuorums = TRUE, corresponding to logic in line 1061-1066 in LFE.java.
+FLEWaitNewNotmsgEnd(i) ==
+        /\ WaitNewNotmsgEnd(i)
+        /\ LET newState == state'[i]
+           IN
+           \/ /\ newState = LEADING
+              /\ ZabTurnToLeading(i)
+              /\ UNCHANGED <<cepochSent, synced>>
+           \/ /\ newState = FOLLOWING
+              /\ ZabTurnToFollowing(i)
+              /\ UNCHANGED <<learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter, sendCounter, committedIndex, committedCounter, tempVarsZ>>
+           \/ /\ newState = LOOKING
+              /\ PrintT("New state is LOOKING in FLEWaitNewNotmsgEnd, which should not happen.")
+              /\ UNCHANGED <<zabState, learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter, sendCounter, commitIndex,
+                             committedIndex, committedCounter, tempVarsZ, cepochSent, synced>>
+        /\ UNCHANGED <<acceptedEpoch, history, leaderAddr, verifyVarsZ, msgs>>
+              
+-----------------------------------------------------------------------------
+(* A sub-action describing how a server transitions from LEADING/FOLLOWING to LOOKING.
+   Initially I call it 'ZabTimeoutZ', but it will be called not only when timeout, 
+   but also when finding a low epoch from leader.*)
+FollowerShutdown(i) ==
+        /\ ZabTimeout(i)
+        /\ zabState'   = [zabState   EXCEPT ![i] = ELECTION]
+        /\ leaderAddr' = [leaderAddr EXCEPT ![i] = NullPoint]
+
+LeaderShutdown(i) ==
+        /\ ZabTimeout(i)
+        /\ zabState'   = [zabState   EXCEPT ![i] = ELECTION]
+        /\ leaderAddr' = [s \in Server |-> IF s \in learners[i] THEN NullPoint ELSE leaderAddr[s]]
+        /\ learners'   = [learners   EXCEPT ![i] = {}]
+        /\ forwarding' = [forwarding EXCEPT ![i] = {}]
+        /\ msgs'       = [s \in Server |-> [v \in Server |-> IF v \in learners[i] \/ s \in learners[i] THEN << >> ELSE msgs[s][v]]]
+
+FollowerTimout(i) ==
+        /\ state[i]      = FOLLOWING
+        /\ leaderAddr[i] = NullPoint
+        /\ FollowerShutdown(i)
+        /\ msgs' = [s \in Server |-> [v \in Server |-> IF v = i THEN << >> ELSE msgs[s][v]]]
+        /\ UNCHANGED <<acceptedEpoch, history, commitIndex, learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, 
+                       currentCounter, sendCounter, committedIndex, committedCounter, tempVarsZ, cepochSent, synced, verifyVarsZ>>
+
+LeaderTimeout(i) ==
+        /\ state[i] = LEADING
+        /\ learners[i] \notin Quorums
+        /\ LeaderShutdown(i)
+        /\ UNCHANGED <<acceptedEpoch, history, commitIndex, cepochRecv, ackeRecv, ackldRecv, ackIndex, currentCounter, sendCounter, committedIndex, committedCounter,
+                       tempVarsZ, cepochSent, synced, verifyVarsZ>>
+    
+-----------------------------------------------------------------------------
+(* Establish connection between leader i and follower j. 
+   It means i creates a learnerHandler for communicating with j, and j finds i's address.*)
+EstablishConnection(i, j) ==
+        /\ state[i] = LEADING   /\ state[j] = FOLLOWING
+        /\ j \notin learners[i] /\ leaderAddr[j] = NullPoint
+        /\ currentVote[j].proposedLeader = i
+        /\ learners'   = [learners   EXCEPT ![i] = learners[i] \union {j}] \* Leader:   'addLearnerHandler(peer)'
+        /\ leaderAddr' = [leaderAddr EXCEPT ![j] = i]                      \* Follower: 'connectToLeader(addr, hostname)'
+        /\ UNCHANGED <<serverVarsZ, electionVarsZ, leadingVoteSet, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, 
+                       currentCounter, sendCounter, committedIndex, committedCounter, tempVarsZ, cepochSent, synced, verifyVarsZ, msgVarsZ, idTable>>
+        
+(* The leader i finds timeout and TCP connection between i and j closes.*)       
+Timeout(i, j) ==
+        /\ state[i] = LEADING /\ state[j] = FOLLOWING
+        /\ j \in learners[i]  /\ leaderAddr[j] = i
+        (* The action of leader i.(corresponding to function 'removeLearnerHandler(peer)'.) *)
+        /\ learners'   = [learners   EXCEPT ![i] = learners[i] \ {j}] 
+        /\ forwarding' = [forwarding EXCEPT ![i] = IF j \in forwarding[i] THEN forwarding[i] \ {j} ELSE forwarding[i]]
+        /\ cepochRecv' = [cepochRecv EXCEPT ![i] = IF j \in cepochRecv[i] THEN cepochRecv[i] \ {j} ELSE cepochRecv[i]]
+        /\ ackeRecv'   = [ackeRecv   EXCEPT ![i] = IF j \in ackeRecv[i] THEN ackeRecv[i] \ {j} ELSE ackeRecv[i]]
+        /\ ackldRecv'  = [ackldRecv  EXCEPT ![i] = IF j \in ackldRecv[i] THEN ackldRecv[i] \ {j} ELSE ackldRecv[i]]
+        /\ ackIndex'   = [ackIndex   EXCEPT ![i][j] = 0]
+        /\ committedCounter' = [committedCounter EXCEPT ![i][j] = 0]
+        (* The action of follower j. *)
+        /\ FollowerShutdown(j)
+        (* Clean input buffer.*)
+        /\ Clean(i, j)
+        /\ UNCHANGED <<acceptedEpoch, history, commitIndex, currentCounter, sendCounter, committedIndex,
+                       tempVarsZ, cepochSent, synced, verifyVarsZ>>
+-----------------------------------------------------------------------------
+\* In phase f11, follower sends f.p to leader via FOLLOWERINFO(CEPOCH).
+FollowerSendFOLLOWERINFO(i) ==
+        /\ state[i]    = FOLLOWING
+        /\ zabState[i] = DISCOVERY
+        /\ leaderAddr[i] /= NullPoint
+        /\ \lnot cepochSent[i]
+        /\ Send(i, leaderAddr[i], [mtype  |-> FOLLOWERINFO,
+                                   mepoch |-> acceptedEpoch[i]])
+        /\ cepochSent' = [cepochSent EXCEPT ![i] = TRUE]
+        /\ UNCHANGED <<serverVarsZ, leaderVarsZ, electionVarsZ, tempVarsZ, leaderAddr, synced, verifyVarsZ, electionMsgs, idTable>>
+        
+(* In phase l11, leader waits for receiving FOLLOWERINFO from a quorum,
+   and then chooses a new epoch e' as its own epoch and broadcasts LEADERINFO. *)
+LeaderHandleFOLLOWERINFO(i, j) ==
+        /\ state[i] = LEADING
+        /\ msgs[j][i] /= << >>
+        /\ msgs[j][i][1].mtype = FOLLOWERINFO
+        /\ LET msg == msgs[j][i][1]
+           IN \/ /\ NullPoint \notin cepochRecv[i]   \* 1. has not broadcast LEADERINFO - modify tempMaxEpoch
+                 /\ LET newEpoch == Maximum({tempMaxEpoch[i], msg.mepoch})
+                    IN tempMaxEpoch' = [tempMaxEpoch EXCEPT ![i] = newEpoch]
+                 /\ Discard(j, i)
+              \/ /\ NullPoint \in cepochRecv[i]      \* 2. has broadcast LEADERINFO - no need to handle the msg, just send LEADERINFO to corresponding server
+                 /\ Reply(i, j, [mtype  |-> LEADERINFO,
+                                 mepoch |-> acceptedEpoch[i]])
+                 /\ UNCHANGED tempMaxEpoch
+        /\ cepochRecv' = [cepochRecv EXCEPT ![i] = IF j \in cepochRecv[i] THEN cepochRecv[i]
+                                                                          ELSE cepochRecv[i] \union {j}]
+        /\ UNCHANGED <<serverVarsZ, followerVarsZ, electionVarsZ, initialHistory, leadingVoteSet, learners, ackeRecv, ackldRecv, 
+                       forwarding, ackIndex, currentCounter, sendCounter, committedIndex, committedCounter, verifyVarsZ, electionMsgs, idTable>>
+
+LeaderDiscovery1(i) ==
+        /\ state[i]    = LEADING
+        /\ zabState[i] = DISCOVERY
+        /\ cepochRecv[i] \in Quorums
+        /\ acceptedEpoch' = [acceptedEpoch EXCEPT ![i] = tempMaxEpoch[i] + 1]
+        /\ cepochRecv'    = [cepochRecv    EXCEPT ![i] = cepochRecv[i] \union {NullPoint}]
+        /\ BroadcastLEADERINFO(i, [mtype  |-> LEADERINFO,
+                                   mepoch |-> acceptedEpoch'[i]])
+        /\ UNCHANGED <<state, currentEpoch, lastZxid, zabState, history, commitIndex, electionVarsZ, leadingVoteSet, learners, ackeRecv, ackldRecv, 
+                       forwarding, ackIndex, currentCounter, sendCounter, committedIndex, committedCounter, 
+                       tempVarsZ, followerVarsZ, verifyVarsZ, electionMsgs, idTable>>
+
+(* In phase f12, follower receives NEWEPOCH. If e' > f.p, then follower sends ACK-E back,
+   and ACK-E contains f.a and lastZxid to let leader judge whether it is the latest.
+   After handling NEWEPOCH, follower's zabState turns to SYNCHRONIZATION. *)
+FollowerHandleLEADERINFO(i, j) ==
+        /\ state[i] = FOLLOWING
+        /\ msgs[j][i] /= << >>
+        /\ msgs[j][i][1].mtype = LEADERINFO
+        /\ LET msg     == msgs[j][i][1]
+               infoOk  == j = leaderAddr[i]
+               epochOk == /\ infoOk
+                          /\ msg.mepoch >= acceptedEpoch[i]
+               correct == /\ epochOk
+                          /\ zabState[i] = DISCOVERY
+           IN /\ infoOk
+              /\ \/ /\ epochOk                      \* 1. Normal case
+                    /\ \/ /\ correct
+                          /\ acceptedEpoch' = [acceptedEpoch EXCEPT ![i] = msg.mepoch]
+                          /\ Reply(i, j, [mtype      |-> ACKEPOCH,
+                                          mepoch     |-> msg.mepoch,
+                                          mlastEpoch |-> currentEpoch[i],
+                                          mlastZxid  |-> lastZxid[i]])
+                          /\ cepochSent' = [cepochSent EXCEPT ![i] = TRUE]
+                          /\ UNCHANGED inherentViolated
+                       \/ /\ ~correct
+                          /\ PrintT("Exception: Condition correct is false in FollowerHandleLEADERINFO(" \o ToString(i) \o ", " \o ToString(j) \o ").")
+                          /\ inherentViolated' = TRUE
+                          /\ Discard(j, i)
+                          /\ UNCHANGED <<acceptedEpoch, cepochSent>>
+                    /\ zabState' = [zabState EXCEPT ![i] = IF zabState[i] = DISCOVERY THEN SYNCHRONIZATION
+                                                                                      ELSE zabState[i]]
+                    /\ UNCHANGED <<varsL, leaderAddr>>
+                 \/ /\ ~epochOk                    \* 2. Abnormal case - go back to election
+                    /\ FollowerShutdown(i)
+                    /\ Clean(i, j)
+                    /\ UNCHANGED <<acceptedEpoch, cepochSent, inherentViolated>>
+        /\ UNCHANGED <<history, commitIndex, learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter, sendCounter, committedIndex,
+                       committedCounter, tempVarsZ, synced, proposalMsgsLog, epochLeader>>
+
+\* Abstraction of actions making follower synced with leader before leader sending NEWLEADER.
+subRECOVERYSYNC(i, j) ==
+        LET canSync == /\ state[i] = LEADING   /\ zabState[i] /= DISCOVERY      /\ j \in learners[i]  /\ j \in ackeRecv[i]
+                       /\ state[j] = FOLLOWING /\ zabState[j] = SYNCHRONIZATION /\ leaderAddr[j] = i  /\ synced[j] = FALSE
+        IN
+        \/ /\ canSync
+           /\ history'     = [history     EXCEPT ![j] = history[i]]
+           /\ lastZxid'    = [lastZxid    EXCEPT ![j] = lastZxid[i]]
+           /\ UpdateProposal(j, leaderAddr[j], lastZxid'[j], currentEpoch[j])
+           /\ commitIndex' = [commitIndex EXCEPT ![j] = commitIndex[i]]
+           /\ synced'      = [synced      EXCEPT ![j] = TRUE]
+           /\ forwarding'  = [forwarding  EXCEPT ![i] = forwarding[i] \union {j}]    \* j will receive PROPOSAL and COMMIT
+           /\ ackIndex'    = [ackIndex    EXCEPT ![i][j] = Len(history[i])]
+           /\ committedCounter' = [committedCounter EXCEPT ![i][j] = Maximum({commitIndex[i] - Len(initialHistory[i]), 0})]
+           /\ LET ms == [msource|->i, mtype|->"RECOVERYSYNC", mepoch|->acceptedEpoch[i], mproposals|->history[i]]
+              IN proposalMsgsLog' = IF ms \in proposalMsgsLog THEN proposalMsgsLog
+                                                              ELSE proposalMsgsLog \union {ms}
+           /\ Reply(i, j, [mtype     |-> NEWLEADER,
+                           mepoch    |-> acceptedEpoch[i],
+                           mlastZxid |-> lastZxid[i]])
+        \/ /\ ~canSync
+           /\ Discard(j, i)
+           /\ UNCHANGED <<history, lastZxid, currentVote, commitIndex, synced, forwarding, ackIndex, committedCounter, proposalMsgsLog>>
+        
+(* In phase l12, leader waits for receiving ACKEPOPCH from a quorum,
+   and check whether it has the latest history and epoch from them.
+   If so, leader's zabState turns to SYNCHRONIZATION. *)
+LeaderHandleACKEPOCH(i, j) ==
+        /\ state[i] = LEADING
+        /\ msgs[j][i] /= << >>
+        /\ msgs[j][i][1].mtype = ACKEPOCH
+        /\ LET msg == msgs[j][i][1]
+               infoOk == /\ j \in learners[i]            
+                         /\ acceptedEpoch[i] = msg.mepoch
+               logOk  == /\ infoOk                          \* logOk = TRUE means leader is more up-to-date than follower j
+                         /\ \/ currentEpoch[i] > msg.mlastEpoch
+                            \/ /\ currentEpoch[i] = msg.mlastEpoch
+                               /\ \/ lastZxid[i][1] > msg.mlastZxid[1]
+                                  \/ /\ lastZxid[i][1] = msg.mlastZxid[1]
+                                     /\ lastZxid[i][2] >= msg.mlastZxid[2]
+               replyOk == /\ infoOk
+                          /\ NullPoint \in ackeRecv[i]
+           IN /\ infoOk
+              /\ \/ /\ replyOk
+                    /\ subRECOVERYSYNC(i, j)
+                    /\ ackeRecv' = [ackeRecv EXCEPT ![i] = IF j \notin ackeRecv[i] THEN ackeRecv[i] \union {j}
+                                                                                   ELSE ackeRecv[i]]
+                    /\ UNCHANGED <<state, currentEpoch,logicalClock, receiveVotes, outOfElection, recvQueue, waitNotmsg, leadingVoteSet, electionMsgs, idTable,
+                                   zabState, leaderAddr, learners>>
+                 \/ /\ ~replyOk
+                    /\ \/ /\ logOk
+                          /\ ackeRecv' = [ackeRecv EXCEPT ![i] = IF j \notin ackeRecv[i] THEN ackeRecv[i] \union {j}
+                                                                                         ELSE ackeRecv[i]]
+                          /\ Discard(j, i)
+                          /\ UNCHANGED <<varsL, zabState, leaderAddr, learners, forwarding>>
+                       \/ /\ ~logOk            \* go back to election
+                          /\ LeaderShutdown(i)
+                          /\ UNCHANGED ackeRecv
+                    /\ UNCHANGED <<history, commitIndex, synced, forwarding, ackIndex, committedCounter, proposalMsgsLog>>
+        /\ UNCHANGED <<acceptedEpoch, cepochRecv, ackldRecv,currentCounter, sendCounter, committedIndex, tempVarsZ, cepochSent, epochLeader, inherentViolated>>
+                          
+LeaderDiscovery2(i) ==
+        /\ state[i] = LEADING
+        /\ zabState[i] = DISCOVERY
+        /\ ackeRecv[i] \in Quorums
+        /\ zabState'       = [zabState       EXCEPT ![i] = SYNCHRONIZATION]
+        /\ currentEpoch'   = [currentEpoch   EXCEPT ![i] = acceptedEpoch[i]]
+        /\ initialHistory' = [initialHistory EXCEPT ![i] = history[i]]
+        /\ ackeRecv'       = [ackeRecv       EXCEPT ![i] = ackeRecv[i] \union {NullPoint}]
+        /\ ackIndex'       = [ackIndex       EXCEPT ![i][i] = Len(history[i])]
+        /\ UpdateProposal(i, i, lastZxid[i], currentEpoch'[i])
+        /\ LET epoch == acceptedEpoch[i]
+           IN epochLeader' = [epochLeader EXCEPT ![epoch] = epochLeader[epoch] \union {i}]
+        /\ UNCHANGED <<state, lastZxid, acceptedEpoch, history, commitIndex, logicalClock, receiveVotes, outOfElection, recvQueue, waitNotmsg,
+                       leadingVoteSet, learners, cepochRecv, ackldRecv, forwarding, currentCounter, sendCounter, committedIndex, committedCounter,
+                       tempMaxEpoch, followerVarsZ, proposalMsgsLog, inherentViolated, msgVarsZ, idTable>>
+
+(* Note:  Set cepochRecv, ackeRecv, ackldRecv to {NullPoint} in corresponding three actions to
+          make sure that the prospective leader will not broadcast NEWEPOCH/NEWLEADER/COMMITLD twice.*)
+-----------------------------------------------------------------------------
+RECOVERYSYNC(i, j) ==
+        /\ state[i] = LEADING   /\ zabState[i] /= DISCOVERY      /\ j \in learners[i] /\ j \in ackeRecv[i]
+        /\ state[j] = FOLLOWING /\ zabState[j] = SYNCHRONIZATION /\ leaderAddr[j] = i /\ synced[j] = FALSE
+        \* /\ acceptedEpoch[i] = acceptedEpoch[j] \* This condition is unnecessary.
+        /\ history'     = [history     EXCEPT ![j] = history[i]]
+        /\ lastZxid'    = [lastZxid    EXCEPT ![j] = lastZxid[i]]
+        /\ UpdateProposal(j, leaderAddr[j], lastZxid'[j], currentEpoch[j])
+        /\ commitIndex' = [commitIndex EXCEPT ![j] = commitIndex[i]]
+        /\ synced'      = [synced      EXCEPT ![j] = TRUE]
+        /\ forwarding'  = [forwarding  EXCEPT ![i] = forwarding[i] \union {j}]
+        /\ ackIndex'    = [ackIndex    EXCEPT ![i][j] = Len(history[i])]
+        /\ committedCounter' = [committedCounter EXCEPT ![i][j] = Maximum({commitIndex[i] - Len(initialHistory[i]), 0})]
+        /\ LET ms == [msource|->i, mtype|->"RECOVERYSYNC", mepoch|->acceptedEpoch[i], mproposals|->history[i]]
+           IN proposalMsgsLog' = IF ms \in proposalMsgsLog THEN proposalMsgsLog
+                                                           ELSE proposalMsgsLog \union {ms}
+        /\ Send(i, j, [mtype     |-> NEWLEADER,
+                       mepoch    |-> acceptedEpoch[i],
+                       mlastZxid |-> lastZxid[i]])
+        /\ UNCHANGED <<state, zabState, acceptedEpoch, currentEpoch, logicalClock, receiveVotes, outOfElection, recvQueue, waitNotmsg, 
+                       leadingVoteSet, learners, cepochRecv, ackeRecv, ackldRecv, currentCounter, sendCounter, committedIndex, 
+                       tempVarsZ, cepochSent, leaderAddr, epochLeader, inherentViolated, electionMsgs, idTable>>
+
+(* In phase f21, follower receives NEWLEADER. The follower updates its epoch and history,
+   and sends back ACK-LD to leader. *)
+FollowerHandleNEWLEADER(i, j) ==
+        /\ state[i] = FOLLOWING
+        /\ msgs[j][i] /= << >>
+        /\ msgs[j][i][1].mtype = NEWLEADER
+        /\ LET msg     == msgs[j][i][1]
+               infoOk  == /\ leaderAddr[i] = j
+                          /\ acceptedEpoch[i] = msg.mepoch
+               correct == /\ infoOk
+                          /\ zabState[i] = SYNCHRONIZATION
+                          /\ synced[i]
+                          /\ ZxidEqual(lastZxid[i], msg.mlastZxid)
+           IN /\ infoOk
+              /\ currentEpoch' = [currentEpoch EXCEPT ![i] = msg.mepoch]
+              /\ UpdateProposal(i, j, lastZxid[i], currentEpoch'[i])
+              /\ \/ /\ correct
+                    /\ Reply(i, j, [mtype  |-> ACKLD,
+                                    mepoch |-> msg.mepoch])
+                    /\ UNCHANGED inherentViolated
+                 \/ /\ ~correct
+                    /\ PrintT("Exception: Condition correct is false in FollowerHandleNEWLEADER(" \o ToString(i) \o ", " \o ToString(j) \o ").")
+                    /\ inherentViolated' = TRUE
+                    /\ Discard(j, i)
+        /\ UNCHANGED <<state, lastZxid, zabState, acceptedEpoch, history, commitIndex, logicalClock, receiveVotes, outOfElection, recvQueue, waitNotmsg,
+                       leaderVarsZ, tempVarsZ, followerVarsZ, proposalMsgsLog, epochLeader, electionMsgs, idTable>>
+
+\* In phase l22, leader receives ACK-LD from a quorum of followers, and sends COMMIT-LD(UPTODATE) to followers.
+LeaderHandleACKLD(i, j) ==
+        /\ state[i] = LEADING
+        /\ msgs[j][i] /= << >>
+        /\ msgs[j][i][1].mtype = ACKLD
+        /\ LET msg     == msgs[j][i][1]
+               infoOk  == /\ acceptedEpoch[i] = msg.mepoch
+                          /\ j \in learners[i]
+               replyOk == /\ infoOk
+                          /\ NullPoint \in ackldRecv[i]
+           IN /\ infoOk
+              /\ \/ /\ replyOk
+                    /\ Reply(i, j, [mtype   |-> UPTODATE,
+                                    mepoch  |-> acceptedEpoch[i],
+                                    mcommit |-> commitIndex[i]])
+                    /\ committedCounter' = [committedCounter EXCEPT ![i][j] = Maximum({commitIndex[i] - Len(initialHistory[i]), committedCounter[i][j]})]
+                 \/ /\ ~replyOk
+                    /\ Discard(j, i)
+                    /\ UNCHANGED committedCounter
+              /\ ackldRecv' = [ackldRecv EXCEPT ![i] = IF j \notin ackldRecv[i] THEN ackldRecv[i] \union {j}
+                                                                                ELSE ackldRecv[i]]
+        /\ UNCHANGED <<serverVarsZ, electionVarsZ, leadingVoteSet, learners, cepochRecv, ackeRecv, forwarding, 
+                       ackIndex, currentCounter, sendCounter, committedIndex, tempVarsZ, followerVarsZ, verifyVarsZ, electionMsgs, idTable>>
+
+LeaderSync2(i) ==
+        /\ state[i] = LEADING
+        /\ zabState[i] = SYNCHRONIZATION
+        /\ ackldRecv[i] \in Quorums
+        /\ commitIndex'    = [commitIndex    EXCEPT ![i] = Len(history[i])]
+        /\ committedIndex' = [committedIndex EXCEPT ![i] = Len(history[i])]
+        /\ zabState'       = [zabState       EXCEPT ![i] = BROADCAST]
+        /\ currentCounter' = [currentCounter EXCEPT ![i] = 0]
+        /\ sendCounter'    = [sendCounter    EXCEPT ![i] = 0]
+        /\ ackldRecv'      = [ackldRecv      EXCEPT ![i] = ackldRecv[i] \union {NullPoint}]
+        /\ BroadcastUPTODATE(i, [mtype   |-> UPTODATE,
+                                 mepoch  |-> acceptedEpoch[i],
+                                 mcommit |-> Len(history[i])]) \* In actual UPTODATE doesn't carry this info
+        /\ UNCHANGED <<state, currentEpoch, lastZxid, acceptedEpoch, history, electionVarsZ, leadingVoteSet, learners, cepochRecv, ackeRecv, forwarding, ackIndex, 
+                       committedCounter, tempVarsZ, followerVarsZ, verifyVarsZ, electionMsgs, idTable>>
+
+FollowerHandleUPTODATE(i, j) ==
+        /\ state[i] = FOLLOWING
+        /\ msgs[j][i] /= << >>
+        /\ msgs[j][i][1].mtype = UPTODATE
+        /\ LET msg == msgs[j][i][1]
+               infoOk == /\ leaderAddr[i] = j
+                         /\ acceptedEpoch[i] = msg.mepoch
+               correct == /\ infoOk
+                          /\ zabState[i] = SYNCHRONIZATION
+                          /\ currentEpoch[i] = msg.mepoch
+           IN /\ infoOk
+              /\ \/ /\ correct
+                    /\ commitIndex' = [commitIndex EXCEPT ![i] = Maximum({commitIndex[i], msg.mcommit})] \* May have received COMMIT and have a bigger commit index.
+                    /\ zabState'    = [zabState    EXCEPT ![i] = BROADCAST]
+                    /\ UNCHANGED inherentViolated
+                 \/ /\ ~correct
+                    /\ PrintT("Exception: Condition correct is false in FollowerHandleUPTODATE(" \o ToString(i) \o ", " \o ToString(j) \o ").")
+                    /\ inherentViolated' = TRUE
+                    /\ UNCHANGED <<commitIndex, zabState>>
+        /\ Discard(j, i)
+        /\ UNCHANGED <<state, currentEpoch, lastZxid, acceptedEpoch, history, electionVarsZ, leaderVarsZ, tempVarsZ, followerVarsZ,
+                       proposalMsgsLog, epochLeader, electionMsgs, idTable>>
+
+-----------------------------------------------------------------------------
+(* In phase l31, leader receives client request and broadcasts PROPOSAL.
+   Note: In production, any server in traffic can receive requests and forward it to leader if necessary.
+   We choose to let leader be the sole one who can receive requests, to simplify spec and keep correctness at the same time. *)
+ClientRequest(i, v) ==

Review comment:
       Nothing is stopping the same value being sent over and over again. Usually use a set for values when you want to limit one client request per set member. So with {v1, v2}, no history could be longer than 2 and never have duplicates.
   I think you need to add an enabling condition that checks that the value does not already exist in a history.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@zookeeper.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] Vanlightly commented on pull request #1690: ZOOKEEPER-3615: Provide formal specification and verification using TLA+ for Zab

Posted by GitBox <gi...@apache.org>.
Vanlightly commented on pull request #1690:
URL: https://github.com/apache/zookeeper/pull/1690#issuecomment-916196689


   I have finished my review of the spec against the code and I do not find any problems in terms of correctness. There are however, a number of suggestions. 
   
   - Some action names are not very helpful, see my comments in code
   - The FastLeaderElection module is too focused on the production implementation in my opinion. One of the benefits of TLA+ is being free of programming considerations like multi-threading. The fact that the FastLeaderElection implementation uses the internal queues recvqueue and sendqueue is an implementation detail that we don't need to replicate in the TLA+. I think you could combine ReceiveNotmsg with HandleNotmsg. Also the WaitNewNotmsg and WaitNewNotmsgEnd are not optimized for state space and are also perhaps, implementation oriented. I think there are some big opportunities to make FLE simpler and with a smaller state space.
   - Consider adding the test variables of ZabWithFLETest to the main spec, using a view to remove those variables from state space exploration and using state constraints to limit state space exploration using those variables. Then this test spec is not required.
   - Add some additional comments to explain what is going on in each action.
   
   The above are not blockers, just suggestions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@zookeeper.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] BinyuHuang-nju commented on pull request #1690: ZOOKEEPER-3615: Provide formal specification and verification using TLA+ for Zab

Posted by GitBox <gi...@apache.org>.
BinyuHuang-nju commented on pull request #1690:
URL: https://github.com/apache/zookeeper/pull/1690#issuecomment-974830152


   @Vanlightly @hanm Hi! This time I have implemented RECOVERY SYNC according to code and upoad a new version, containing ZabWithFLE which use magic action RECOVERYSYNC, and ZabWithFLEAndSYNC which implements phase of SYNC such as DIFF,TRUNC. I hope you could check it!
   As to your suggestions on modifying logics of Value, I will update it and upload as soon as possible! Thank you!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@zookeeper.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] Vanlightly commented on a change in pull request #1690: ZOOKEEPER-3615: Provide formal specification and verification using TLA+ for Zab

Posted by GitBox <gi...@apache.org>.
Vanlightly commented on a change in pull request #1690:
URL: https://github.com/apache/zookeeper/pull/1690#discussion_r705458717



##########
File path: zookeeper-specifications/zab-1.0/ZabWithFLE.tla
##########
@@ -0,0 +1,1020 @@
+(*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *)
+
+----------------------------- MODULE ZabWithFLE -----------------------------
+(* This is the formal specification for the Zab consensus algorithm,
+   which means Zookeeper Atomic Broadcast.*)
+
+(* Reference:
+   FLE: FastLeaderElection.java, Vote.java, QuorumPeer.java in https://github.com/apache/zookeeper.
+   ZAB: QuorumPeer.java, Learner.java, Follower.java, LearnerHandler.java, Leader.java 
+        in https://github.com/apache/zookeeper.
+        https://cwiki.apache.org/confluence/display/ZOOKEEPER/Zab1.0.
+ *)
+EXTENDS FastLeaderElection
+
+-----------------------------------------------------------------------------
+(* Defined in FastLeaderElection.tla:
+\* The set of server identifiers
+CONSTANT Server
+
+\* Server states
+CONSTANTS LOOKING, FOLLOWING, LEADING
+
+\* Message types
+CONSTANTS NOTIFICATION
+
+\* Timeout signal
+CONSTANT NONE
+*)
+\* The set of requests that can go into history
+CONSTANT Value
+
+\* Zab states
+CONSTANTS ELECTION, DISCOVERY, SYNCHRONIZATION, BROADCAST
+
+\* Message types
+CONSTANTS FOLLOWERINFO, LEADERINFO, ACKEPOCH, NEWLEADER, ACKLD, UPTODATE, PROPOSAL, ACK, COMMIT
+(* Additional message types used for recovery in synchronization(TRUNC/DIFF/SNAP) are not needed
+   since we abstract this part.(see action RECOVERYSYNC) *)
+   
+-----------------------------------------------------------------------------
+(* Defined in FastLeaderElection.tla: Quorums, NullPoint *)
+\* Return the maximum value from the set S
+Maximum(S) == IF S = {} THEN -1
+                        ELSE CHOOSE n \in S: \A m \in S: n >= m
+
+\* Return the minimum value from the set S
+Minimum(S) == IF S = {} THEN -1
+                        ELSE CHOOSE n \in S: \A m \in S: n <= m
+
+MAXEPOCH == 10
+
+-----------------------------------------------------------------------------
+(* Defined in FastLeaderElection.tla:
+   serverVars: <<state, currentEpoch, lastZxid>>,
+   electionVars: <<currentVote, logicalClock, receiveVotes, outOfElection, recvQueue, waitNotmsg>>,
+   leaderVars: <<leadingVoteSet>>,
+   electionMsgs,
+   idTable *)
+\* The current phase of server(ELECTION,DISCOVERY,SYNCHRONIZATION,BROADCAST)
+VARIABLE zabState
+
+\* The epoch number of the last NEWEPOCH(LEADERINFO) packet accepted
+\* namely f.p in paper, and currentEpoch in Zab.tla.
+VARIABLE acceptedEpoch
+
+\* The history of servers as the sequence of transactions.
+VARIABLE history
+
+\* commitIndex[i]: The maximum index of transactions that have been saved in a quorum of servers
+\*                 in the perspective of server i.(increases monotonically before restarting)
+VARIABLE commitIndex
+
+(* These transactions whose index \le commitIndex[i] can be applied to state machine immediately.
+   So if we have a variable applyIndex, we can suppose that applyIndex[i] = commitIndex[i] when verifying properties.
+   But in phase SYNC, follower will apply all queued proposals to state machine when receiving NEWLEADER.
+   But follower only serves traffic after receiving UPTODATE, so sequential consistency is not violated.
+   
+   So when we verify properties, we still suppose applyIndex[i] = commitIndex[i], because this is an engineering detail.*)
+
+\* learners[i]: The set of servers which leader i think are connected wich i.
+VARIABLE learners
+
+\* The messages representing requests and responses sent from one server to another.
+\* msgs[i][j] means the input buffer of server j from server i.
+VARIABLE msgs
+
+\* The set of followers who has successfully sent CEPOCH(FOLLOWERINFO) to leader.(equals to connectingFollowers in code)
+VARIABLE cepochRecv
+
+\* The set of followers who has successfully sent ACK-E to leader.(equals to electingFollowers in code)
+VARIABLE ackeRecv
+
+\* The set of followers who has successfully sent ACK-LD to leader in leader.(equals to newLeaderProposal in code)
+VARIABLE ackldRecv
+
+\* The set of servers which leader i broadcasts PROPOSAL and COMMIT to.(equals to forwardingFollowers in code)
+VARIABLE forwarding
+
+\* ackIndex[i][j]: The latest index that leader i has received from follower j via ACK.
+VARIABLE ackIndex
+
+\* currentCounter[i]: The count of transactions that clients request leader i.
+VARIABLE currentCounter
+
+\* sendCounter[i]: The count of transactions that leader i has broadcast in PROPOSAL.
+VARIABLE sendCounter
+
+\* committedIndex[i]: The maximum index of trasactions that leader i has broadcast in COMMIT.
+VARIABLE committedIndex
+
+\* committedCounter[i][j]: The latest counter of transaction that leader i has confirmed that follower j has committed.
+VARIABLE committedCounter
+
+\* initialHistory[i]: The initial history if leader i in epoch acceptedEpoch[i].
+VARIABLE initialHistory
+
+\* the maximum epoch in CEPOCH the prospective leader received from followers.
+VARIABLE tempMaxEpoch
+
+\* cepochSent[i] = TRUE means follower i has sent CEPOCH(FOLLOWERINFO) to leader.
+VARIABLE cepochSent
+
+\* leaderAddr[i]: The leader id of follower i. We use leaderAddr to express whether follower i has connected or lost connection.
+VARIABLE leaderAddr
+
+\* synced[i] = TRUE: follower i has completed sync with leader.
+VARIABLE synced
+
+\* The set of leaders in every epoch, only used in verifying properties.
+VARIABLE epochLeader
+
+\* The set of all broadcast messages, only used in verifying properties.
+VARIABLE proposalMsgsLog
+
+\* A variable used to check whether there are conditions contrary to the facts.
+VARIABLE inherentViolated
+
+serverVarsZ == <<state, currentEpoch, lastZxid, zabState, acceptedEpoch, history, commitIndex>>         \* 7 variables
+
+electionVarsZ == electionVars  \* 6 variables
+
+leaderVarsZ == <<leadingVoteSet, learners, cepochRecv, ackeRecv, ackldRecv, forwarding, 
+                 ackIndex, currentCounter, sendCounter, committedIndex, committedCounter>>              \* 11 variables
+
+tempVarsZ == <<initialHistory, tempMaxEpoch>>     \* 2 variables
+  
+followerVarsZ == <<cepochSent, leaderAddr, synced>>                 \* 3 variables
+
+verifyVarsZ == <<proposalMsgsLog, epochLeader, inherentViolated>>   \* 3 variables
+
+msgVarsZ == <<msgs, electionMsgs>>                \* 2 variables
+
+vars == <<serverVarsZ, electionVarsZ, leaderVarsZ, tempVarsZ, followerVarsZ, verifyVarsZ, msgVarsZ, idTable>>  \* 35 variables
+-----------------------------------------------------------------------------
+\* Add a message to msgs - add a message m to msgs[i][j].
+Send(i, j, m) == msgs' = [msgs EXCEPT ![i][j] = Append(msgs[i][j], m)]
+
+\* Remove a message from msgs - discard head of msgs[i][j].
+Discard(i, j) == msgs' = IF msgs[i][j] /= << >> THEN [msgs EXCEPT ![i][j] = Tail(msgs[i][j])]
+                                                ELSE msgs
+
+\* Leader broadcasts a message(PROPOSAL/COMMIT) to all other servers in forwardingFollowers.
+Broadcast(i, m) == msgs' = [msgs EXCEPT ![i] = [v \in Server |-> IF /\ v \in forwarding[i]
+                                                                    /\ v /= i
+                                                                    /\ \/ /\ m.mtype = PROPOSAL
+                                                                          /\ ackIndex[i][v] < Len(initialHistory[i]) + m.mproposal.counter
+                                                                       \/ /\ m.mtype = COMMIT
+                                                                          /\ committedCounter[i][v] < m.mzxid[2]
+                                                                  THEN Append(msgs[i][v], m)
+                                                                  ELSE msgs[i][v]]]
+
+BroadcastLEADERINFO(i, m) == msgs' = [msgs EXCEPT ![i] = [v \in Server |-> IF /\ v \in cepochRecv[i]
+                                                                              /\ v \in learners[i]
+                                                                              /\ v /= i THEN Append(msgs[i][v], m)
+                                                                                        ELSE msgs[i][v]]]
+
+BroadcastUPTODATE(i, m) == msgs' = [msgs EXCEPT ![i] = [v \in Server |-> IF /\ v \in ackldRecv[i]
+                                                                            /\ v \in learners[i]
+                                                                            /\ v /= i THEN Append(msgs[i][v], m)
+                                                                                      ELSE msgs[i][v]]]
+
+\* Combination of Send and Discard - discard head of msgs[j][i] and add m into msgs[i][j].
+Reply(i, j, m) == msgs' = [msgs EXCEPT ![j][i] = Tail(msgs[j][i]),
+                                       ![i][j] = Append(msgs[i][j], m)]
+
+\* shuffle the input buffer from server j(i) in server i(j).
+Clean(i, j) == msgs' = [msgs EXCEPT ![j][i] = << >>, ![i][j] = << >>]
+
+-----------------------------------------------------------------------------
+PZxidEqual(p, z) == p.epoch = z[1] /\ p.counter = z[2]
+
+TransactionEqual(t1, t2) == /\ t1.epoch   = t2.epoch
+                            /\ t1.counter = t2.counter
+                            
+TransactionPrecede(t1, t2) == \/ t1.epoch < t2.epoch
+                              \/ /\ t1.epoch   = t2.epoch
+                                 /\ t1.counter < t2.counter
+                                 
+-----------------------------------------------------------------------------
+\* Define initial values for all variables 
+InitServerVarsZ == /\ InitServerVars
+                   /\ zabState      = [s \in Server |-> ELECTION]
+                   /\ acceptedEpoch = [s \in Server |-> 0]
+                   /\ history       = [s \in Server |-> << >>]
+                   /\ commitIndex   = [s \in Server |-> 0]
+
+InitLeaderVarsZ == /\ InitLeaderVars
+                   /\ learners         = [s \in Server |-> {}]
+                   /\ cepochRecv       = [s \in Server |-> {}]
+                   /\ ackeRecv         = [s \in Server |-> {}]
+                   /\ ackldRecv        = [s \in Server |-> {}]
+                   /\ ackIndex         = [s \in Server |-> [v \in Server |-> 0]]
+                   /\ currentCounter   = [s \in Server |-> 0]
+                   /\ sendCounter      = [s \in Server |-> 0]
+                   /\ committedIndex   = [s \in Server |-> 0]
+                   /\ committedCounter = [s \in Server |-> [v \in Server |-> 0]]
+                   /\ forwarding       = [s \in Server |-> {}]
+
+InitElectionVarsZ == InitElectionVars
+
+InitTempVarsZ == /\ initialHistory = [s \in Server |-> << >>]
+                 /\ tempMaxEpoch   = [s \in Server |-> 0]
+
+InitFollowerVarsZ == /\ cepochSent = [s \in Server |-> FALSE]
+                     /\ leaderAddr = [s \in Server |-> NullPoint]
+                     /\ synced     = [s \in Server |-> FALSE]
+
+InitVerifyVarsZ == /\ proposalMsgsLog  = {}
+                   /\ epochLeader      = [i \in 1..MAXEPOCH |-> {}]
+                   /\ inherentViolated = FALSE
+                   
+InitMsgVarsZ == /\ msgs         = [s \in Server |-> [v \in Server |-> << >>]]
+                /\ electionMsgs = [s \in Server |-> [v \in Server |-> << >>]]
+
+InitZ == /\ InitServerVarsZ
+         /\ InitLeaderVarsZ
+         /\ InitElectionVarsZ
+         /\ InitTempVarsZ
+         /\ InitFollowerVarsZ
+         /\ InitVerifyVarsZ
+         /\ InitMsgVarsZ
+         /\ idTable = InitializeIdTable(Server)
+         
+-----------------------------------------------------------------------------
+ZabTurnToLeading(i) ==
+        /\ zabState'       = [zabState   EXCEPT ![i] = DISCOVERY]
+        /\ learners'       = [learners   EXCEPT ![i] = {i}]
+        /\ cepochRecv'     = [cepochRecv EXCEPT ![i] = {i}]
+        /\ ackeRecv'       = [ackeRecv   EXCEPT ![i] = {i}]
+        /\ ackldRecv'      = [ackldRecv  EXCEPT ![i] = {i}]
+        /\ forwarding'     = [forwarding EXCEPT ![i] = {}]
+        /\ ackIndex'       = [ackIndex   EXCEPT ![i] = [v \in Server |-> IF v = i THEN Len(history[i])
+                                                                                  ELSE 0]]
+        /\ currentCounter'   = [currentCounter   EXCEPT ![i] = 0]
+        /\ sendCounter'      = [sendCounter      EXCEPT ![i] = 0]
+        /\ commitIndex'      = [commitIndex      EXCEPT ![i] = 0]
+        /\ committedIndex'   = [committedIndex   EXCEPT ![i] = 0]
+        /\ committedCounter' = [committedCounter EXCEPT ![i] = [v \in Server |-> IF v = i THEN Len(history[i])
+                                                                                          ELSE 0]]
+        /\ initialHistory' = [initialHistory EXCEPT ![i] = history[i]]
+        /\ tempMaxEpoch'   = [tempMaxEpoch   EXCEPT ![i] = acceptedEpoch[i]]
+
+ZabTurnToFollowing(i) ==
+        /\ zabState'    = [zabState   EXCEPT ![i] = DISCOVERY]
+        /\ cepochSent'  = [cepochSent EXCEPT ![i] = FALSE]
+        /\ synced'      = [synced     EXCEPT ![i] = FALSE]
+        /\ commitIndex' = [commitIndex      EXCEPT ![i] = 0]
+        
+          
+(* Fast Leader Election *)
+FLEReceiveNotmsg(i, j) ==
+        /\ ReceiveNotmsg(i, j)
+        /\ UNCHANGED <<zabState, acceptedEpoch, history, commitIndex,learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter,
+                       sendCounter, committedIndex, committedCounter, tempVarsZ, followerVarsZ, verifyVarsZ, msgs>>
+
+FLENotmsgTimeout(i) ==
+        /\ NotmsgTimeout(i)
+        /\ UNCHANGED <<zabState, acceptedEpoch, history, commitIndex,learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter,
+                       sendCounter, committedIndex, committedCounter, tempVarsZ, followerVarsZ, verifyVarsZ, msgs>>
+
+FLEHandleNotmsg(i) ==
+        /\ HandleNotmsg(i)
+        /\ LET newState == state'[i]
+           IN
+           \/ /\ newState = LEADING
+              /\ ZabTurnToLeading(i)
+              /\ UNCHANGED <<cepochSent, synced>>
+           \/ /\ newState = FOLLOWING
+              /\ ZabTurnToFollowing(i)
+              /\ UNCHANGED <<learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter, sendCounter, committedIndex, committedCounter, tempVarsZ>>
+           \/ /\ newState = LOOKING
+              /\ UNCHANGED <<zabState, learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter, sendCounter, commitIndex,
+                             committedIndex, committedCounter, tempVarsZ, cepochSent, synced>>
+        /\ UNCHANGED <<acceptedEpoch, history, leaderAddr, verifyVarsZ, msgs>>
+
+\* On the premise that ReceiveVotes.HasQuorums = TRUE, corresponding to logic in line 1050-1055 in LFE.java.
+FLEWaitNewNotmsg(i) ==
+        /\ WaitNewNotmsg(i)
+        /\ UNCHANGED <<zabState, acceptedEpoch, history, commitIndex,learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter,
+                       sendCounter, committedIndex, committedCounter, tempVarsZ, followerVarsZ, verifyVarsZ, msgs>>
+
+\* On the premise that ReceiveVotes.HasQuorums = TRUE, corresponding to logic in line 1061-1066 in LFE.java.
+FLEWaitNewNotmsgEnd(i) ==
+        /\ WaitNewNotmsgEnd(i)
+        /\ LET newState == state'[i]
+           IN
+           \/ /\ newState = LEADING
+              /\ ZabTurnToLeading(i)
+              /\ UNCHANGED <<cepochSent, synced>>
+           \/ /\ newState = FOLLOWING
+              /\ ZabTurnToFollowing(i)
+              /\ UNCHANGED <<learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter, sendCounter, committedIndex, committedCounter, tempVarsZ>>
+           \/ /\ newState = LOOKING
+              /\ PrintT("New state is LOOKING in FLEWaitNewNotmsgEnd, which should not happen.")
+              /\ UNCHANGED <<zabState, learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter, sendCounter, commitIndex,
+                             committedIndex, committedCounter, tempVarsZ, cepochSent, synced>>
+        /\ UNCHANGED <<acceptedEpoch, history, leaderAddr, verifyVarsZ, msgs>>
+              
+-----------------------------------------------------------------------------
+(* A sub-action describing how a server transitions from LEADING/FOLLOWING to LOOKING.
+   Initially I call it 'ZabTimeoutZ', but it will be called not only when timeout, 
+   but also when finding a low epoch from leader.*)
+FollowerShutdown(i) ==
+        /\ ZabTimeout(i)
+        /\ zabState'   = [zabState   EXCEPT ![i] = ELECTION]
+        /\ leaderAddr' = [leaderAddr EXCEPT ![i] = NullPoint]
+
+LeaderShutdown(i) ==
+        /\ ZabTimeout(i)
+        /\ zabState'   = [zabState   EXCEPT ![i] = ELECTION]
+        /\ leaderAddr' = [s \in Server |-> IF s \in learners[i] THEN NullPoint ELSE leaderAddr[s]]
+        /\ learners'   = [learners   EXCEPT ![i] = {}]
+        /\ forwarding' = [forwarding EXCEPT ![i] = {}]
+        /\ msgs'       = [s \in Server |-> [v \in Server |-> IF v \in learners[i] \/ s \in learners[i] THEN << >> ELSE msgs[s][v]]]
+
+FollowerTimout(i) ==
+        /\ state[i]      = FOLLOWING
+        /\ leaderAddr[i] = NullPoint
+        /\ FollowerShutdown(i)
+        /\ msgs' = [s \in Server |-> [v \in Server |-> IF v = i THEN << >> ELSE msgs[s][v]]]
+        /\ UNCHANGED <<acceptedEpoch, history, commitIndex, learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, 
+                       currentCounter, sendCounter, committedIndex, committedCounter, tempVarsZ, cepochSent, synced, verifyVarsZ>>
+
+LeaderTimeout(i) ==
+        /\ state[i] = LEADING
+        /\ learners[i] \notin Quorums
+        /\ LeaderShutdown(i)
+        /\ UNCHANGED <<acceptedEpoch, history, commitIndex, cepochRecv, ackeRecv, ackldRecv, ackIndex, currentCounter, sendCounter, committedIndex, committedCounter,
+                       tempVarsZ, cepochSent, synced, verifyVarsZ>>
+    
+-----------------------------------------------------------------------------
+(* Establish connection between leader i and follower j. 
+   It means i creates a learnerHandler for communicating with j, and j finds i's address.*)
+EstablishConnection(i, j) ==
+        /\ state[i] = LEADING   /\ state[j] = FOLLOWING
+        /\ j \notin learners[i] /\ leaderAddr[j] = NullPoint
+        /\ currentVote[j].proposedLeader = i
+        /\ learners'   = [learners   EXCEPT ![i] = learners[i] \union {j}] \* Leader:   'addLearnerHandler(peer)'
+        /\ leaderAddr' = [leaderAddr EXCEPT ![j] = i]                      \* Follower: 'connectToLeader(addr, hostname)'
+        /\ UNCHANGED <<serverVarsZ, electionVarsZ, leadingVoteSet, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, 
+                       currentCounter, sendCounter, committedIndex, committedCounter, tempVarsZ, cepochSent, synced, verifyVarsZ, msgVarsZ, idTable>>
+        
+(* The leader i finds timeout and TCP connection between i and j closes.*)       
+Timeout(i, j) ==
+        /\ state[i] = LEADING /\ state[j] = FOLLOWING
+        /\ j \in learners[i]  /\ leaderAddr[j] = i
+        (* The action of leader i.(corresponding to function 'removeLearnerHandler(peer)'.) *)
+        /\ learners'   = [learners   EXCEPT ![i] = learners[i] \ {j}] 
+        /\ forwarding' = [forwarding EXCEPT ![i] = IF j \in forwarding[i] THEN forwarding[i] \ {j} ELSE forwarding[i]]
+        /\ cepochRecv' = [cepochRecv EXCEPT ![i] = IF j \in cepochRecv[i] THEN cepochRecv[i] \ {j} ELSE cepochRecv[i]]
+        /\ ackeRecv'   = [ackeRecv   EXCEPT ![i] = IF j \in ackeRecv[i] THEN ackeRecv[i] \ {j} ELSE ackeRecv[i]]
+        /\ ackldRecv'  = [ackldRecv  EXCEPT ![i] = IF j \in ackldRecv[i] THEN ackldRecv[i] \ {j} ELSE ackldRecv[i]]
+        /\ ackIndex'   = [ackIndex   EXCEPT ![i][j] = 0]
+        /\ committedCounter' = [committedCounter EXCEPT ![i][j] = 0]
+        (* The action of follower j. *)
+        /\ FollowerShutdown(j)
+        (* Clean input buffer.*)
+        /\ Clean(i, j)
+        /\ UNCHANGED <<acceptedEpoch, history, commitIndex, currentCounter, sendCounter, committedIndex,
+                       tempVarsZ, cepochSent, synced, verifyVarsZ>>
+-----------------------------------------------------------------------------
+\* In phase f11, follower sends f.p to leader via FOLLOWERINFO(CEPOCH).
+FollowerSendFOLLOWERINFO(i) ==
+        /\ state[i]    = FOLLOWING
+        /\ zabState[i] = DISCOVERY
+        /\ leaderAddr[i] /= NullPoint
+        /\ \lnot cepochSent[i]
+        /\ Send(i, leaderAddr[i], [mtype  |-> FOLLOWERINFO,
+                                   mepoch |-> acceptedEpoch[i]])
+        /\ cepochSent' = [cepochSent EXCEPT ![i] = TRUE]
+        /\ UNCHANGED <<serverVarsZ, leaderVarsZ, electionVarsZ, tempVarsZ, leaderAddr, synced, verifyVarsZ, electionMsgs, idTable>>
+        
+(* In phase l11, leader waits for receiving FOLLOWERINFO from a quorum,
+   and then chooses a new epoch e' as its own epoch and broadcasts LEADERINFO. *)
+LeaderHandleFOLLOWERINFO(i, j) ==
+        /\ state[i] = LEADING
+        /\ msgs[j][i] /= << >>
+        /\ msgs[j][i][1].mtype = FOLLOWERINFO
+        /\ LET msg == msgs[j][i][1]
+           IN \/ /\ NullPoint \notin cepochRecv[i]   \* 1. has not broadcast LEADERINFO - modify tempMaxEpoch
+                 /\ LET newEpoch == Maximum({tempMaxEpoch[i], msg.mepoch})
+                    IN tempMaxEpoch' = [tempMaxEpoch EXCEPT ![i] = newEpoch]
+                 /\ Discard(j, i)
+              \/ /\ NullPoint \in cepochRecv[i]      \* 2. has broadcast LEADERINFO - no need to handle the msg, just send LEADERINFO to corresponding server
+                 /\ Reply(i, j, [mtype  |-> LEADERINFO,
+                                 mepoch |-> acceptedEpoch[i]])
+                 /\ UNCHANGED tempMaxEpoch
+        /\ cepochRecv' = [cepochRecv EXCEPT ![i] = IF j \in cepochRecv[i] THEN cepochRecv[i]
+                                                                          ELSE cepochRecv[i] \union {j}]
+        /\ UNCHANGED <<serverVarsZ, followerVarsZ, electionVarsZ, initialHistory, leadingVoteSet, learners, ackeRecv, ackldRecv, 
+                       forwarding, ackIndex, currentCounter, sendCounter, committedIndex, committedCounter, verifyVarsZ, electionMsgs, idTable>>
+
+LeaderDiscovery1(i) ==
+        /\ state[i]    = LEADING
+        /\ zabState[i] = DISCOVERY
+        /\ cepochRecv[i] \in Quorums
+        /\ acceptedEpoch' = [acceptedEpoch EXCEPT ![i] = tempMaxEpoch[i] + 1]
+        /\ cepochRecv'    = [cepochRecv    EXCEPT ![i] = cepochRecv[i] \union {NullPoint}]
+        /\ BroadcastLEADERINFO(i, [mtype  |-> LEADERINFO,
+                                   mepoch |-> acceptedEpoch'[i]])
+        /\ UNCHANGED <<state, currentEpoch, lastZxid, zabState, history, commitIndex, electionVarsZ, leadingVoteSet, learners, ackeRecv, ackldRecv, 
+                       forwarding, ackIndex, currentCounter, sendCounter, committedIndex, committedCounter, 
+                       tempVarsZ, followerVarsZ, verifyVarsZ, electionMsgs, idTable>>
+
+(* In phase f12, follower receives NEWEPOCH. If e' > f.p, then follower sends ACK-E back,
+   and ACK-E contains f.a and lastZxid to let leader judge whether it is the latest.
+   After handling NEWEPOCH, follower's zabState turns to SYNCHRONIZATION. *)
+FollowerHandleLEADERINFO(i, j) ==
+        /\ state[i] = FOLLOWING
+        /\ msgs[j][i] /= << >>
+        /\ msgs[j][i][1].mtype = LEADERINFO
+        /\ LET msg     == msgs[j][i][1]
+               infoOk  == j = leaderAddr[i]
+               epochOk == /\ infoOk
+                          /\ msg.mepoch >= acceptedEpoch[i]
+               correct == /\ epochOk
+                          /\ zabState[i] = DISCOVERY
+           IN /\ infoOk
+              /\ \/ /\ epochOk                      \* 1. Normal case
+                    /\ \/ /\ correct
+                          /\ acceptedEpoch' = [acceptedEpoch EXCEPT ![i] = msg.mepoch]
+                          /\ Reply(i, j, [mtype      |-> ACKEPOCH,
+                                          mepoch     |-> msg.mepoch,
+                                          mlastEpoch |-> currentEpoch[i],
+                                          mlastZxid  |-> lastZxid[i]])
+                          /\ cepochSent' = [cepochSent EXCEPT ![i] = TRUE]
+                          /\ UNCHANGED inherentViolated
+                       \/ /\ ~correct
+                          /\ PrintT("Exception: Condition correct is false in FollowerHandleLEADERINFO(" \o ToString(i) \o ", " \o ToString(j) \o ").")
+                          /\ inherentViolated' = TRUE
+                          /\ Discard(j, i)
+                          /\ UNCHANGED <<acceptedEpoch, cepochSent>>
+                    /\ zabState' = [zabState EXCEPT ![i] = IF zabState[i] = DISCOVERY THEN SYNCHRONIZATION
+                                                                                      ELSE zabState[i]]
+                    /\ UNCHANGED <<varsL, leaderAddr>>
+                 \/ /\ ~epochOk                    \* 2. Abnormal case - go back to election
+                    /\ FollowerShutdown(i)
+                    /\ Clean(i, j)
+                    /\ UNCHANGED <<acceptedEpoch, cepochSent, inherentViolated>>
+        /\ UNCHANGED <<history, commitIndex, learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter, sendCounter, committedIndex,
+                       committedCounter, tempVarsZ, synced, proposalMsgsLog, epochLeader>>
+
+\* Abstraction of actions making follower synced with leader before leader sending NEWLEADER.
+subRECOVERYSYNC(i, j) ==
+        LET canSync == /\ state[i] = LEADING   /\ zabState[i] /= DISCOVERY      /\ j \in learners[i]  /\ j \in ackeRecv[i]
+                       /\ state[j] = FOLLOWING /\ zabState[j] = SYNCHRONIZATION /\ leaderAddr[j] = i  /\ synced[j] = FALSE
+        IN
+        \/ /\ canSync
+           /\ history'     = [history     EXCEPT ![j] = history[i]]
+           /\ lastZxid'    = [lastZxid    EXCEPT ![j] = lastZxid[i]]
+           /\ UpdateProposal(j, leaderAddr[j], lastZxid'[j], currentEpoch[j])
+           /\ commitIndex' = [commitIndex EXCEPT ![j] = commitIndex[i]]
+           /\ synced'      = [synced      EXCEPT ![j] = TRUE]
+           /\ forwarding'  = [forwarding  EXCEPT ![i] = forwarding[i] \union {j}]    \* j will receive PROPOSAL and COMMIT
+           /\ ackIndex'    = [ackIndex    EXCEPT ![i][j] = Len(history[i])]
+           /\ committedCounter' = [committedCounter EXCEPT ![i][j] = Maximum({commitIndex[i] - Len(initialHistory[i]), 0})]
+           /\ LET ms == [msource|->i, mtype|->"RECOVERYSYNC", mepoch|->acceptedEpoch[i], mproposals|->history[i]]
+              IN proposalMsgsLog' = IF ms \in proposalMsgsLog THEN proposalMsgsLog
+                                                              ELSE proposalMsgsLog \union {ms}
+           /\ Reply(i, j, [mtype     |-> NEWLEADER,
+                           mepoch    |-> acceptedEpoch[i],
+                           mlastZxid |-> lastZxid[i]])
+        \/ /\ ~canSync
+           /\ Discard(j, i)
+           /\ UNCHANGED <<history, lastZxid, currentVote, commitIndex, synced, forwarding, ackIndex, committedCounter, proposalMsgsLog>>
+        
+(* In phase l12, leader waits for receiving ACKEPOPCH from a quorum,
+   and check whether it has the latest history and epoch from them.
+   If so, leader's zabState turns to SYNCHRONIZATION. *)
+LeaderHandleACKEPOCH(i, j) ==
+        /\ state[i] = LEADING
+        /\ msgs[j][i] /= << >>
+        /\ msgs[j][i][1].mtype = ACKEPOCH
+        /\ LET msg == msgs[j][i][1]
+               infoOk == /\ j \in learners[i]            
+                         /\ acceptedEpoch[i] = msg.mepoch
+               logOk  == /\ infoOk                          \* logOk = TRUE means leader is more up-to-date than follower j
+                         /\ \/ currentEpoch[i] > msg.mlastEpoch
+                            \/ /\ currentEpoch[i] = msg.mlastEpoch
+                               /\ \/ lastZxid[i][1] > msg.mlastZxid[1]
+                                  \/ /\ lastZxid[i][1] = msg.mlastZxid[1]
+                                     /\ lastZxid[i][2] >= msg.mlastZxid[2]
+               replyOk == /\ infoOk
+                          /\ NullPoint \in ackeRecv[i]
+           IN /\ infoOk
+              /\ \/ /\ replyOk
+                    /\ subRECOVERYSYNC(i, j)
+                    /\ ackeRecv' = [ackeRecv EXCEPT ![i] = IF j \notin ackeRecv[i] THEN ackeRecv[i] \union {j}
+                                                                                   ELSE ackeRecv[i]]
+                    /\ UNCHANGED <<state, currentEpoch,logicalClock, receiveVotes, outOfElection, recvQueue, waitNotmsg, leadingVoteSet, electionMsgs, idTable,
+                                   zabState, leaderAddr, learners>>
+                 \/ /\ ~replyOk
+                    /\ \/ /\ logOk
+                          /\ ackeRecv' = [ackeRecv EXCEPT ![i] = IF j \notin ackeRecv[i] THEN ackeRecv[i] \union {j}
+                                                                                         ELSE ackeRecv[i]]
+                          /\ Discard(j, i)
+                          /\ UNCHANGED <<varsL, zabState, leaderAddr, learners, forwarding>>
+                       \/ /\ ~logOk            \* go back to election
+                          /\ LeaderShutdown(i)
+                          /\ UNCHANGED ackeRecv
+                    /\ UNCHANGED <<history, commitIndex, synced, forwarding, ackIndex, committedCounter, proposalMsgsLog>>
+        /\ UNCHANGED <<acceptedEpoch, cepochRecv, ackldRecv,currentCounter, sendCounter, committedIndex, tempVarsZ, cepochSent, epochLeader, inherentViolated>>
+                          
+LeaderDiscovery2(i) ==
+        /\ state[i] = LEADING
+        /\ zabState[i] = DISCOVERY
+        /\ ackeRecv[i] \in Quorums
+        /\ zabState'       = [zabState       EXCEPT ![i] = SYNCHRONIZATION]
+        /\ currentEpoch'   = [currentEpoch   EXCEPT ![i] = acceptedEpoch[i]]
+        /\ initialHistory' = [initialHistory EXCEPT ![i] = history[i]]
+        /\ ackeRecv'       = [ackeRecv       EXCEPT ![i] = ackeRecv[i] \union {NullPoint}]
+        /\ ackIndex'       = [ackIndex       EXCEPT ![i][i] = Len(history[i])]
+        /\ UpdateProposal(i, i, lastZxid[i], currentEpoch'[i])
+        /\ LET epoch == acceptedEpoch[i]
+           IN epochLeader' = [epochLeader EXCEPT ![epoch] = epochLeader[epoch] \union {i}]
+        /\ UNCHANGED <<state, lastZxid, acceptedEpoch, history, commitIndex, logicalClock, receiveVotes, outOfElection, recvQueue, waitNotmsg,
+                       leadingVoteSet, learners, cepochRecv, ackldRecv, forwarding, currentCounter, sendCounter, committedIndex, committedCounter,
+                       tempMaxEpoch, followerVarsZ, proposalMsgsLog, inherentViolated, msgVarsZ, idTable>>
+
+(* Note:  Set cepochRecv, ackeRecv, ackldRecv to {NullPoint} in corresponding three actions to
+          make sure that the prospective leader will not broadcast NEWEPOCH/NEWLEADER/COMMITLD twice.*)
+-----------------------------------------------------------------------------
+RECOVERYSYNC(i, j) ==
+        /\ state[i] = LEADING   /\ zabState[i] /= DISCOVERY      /\ j \in learners[i] /\ j \in ackeRecv[i]
+        /\ state[j] = FOLLOWING /\ zabState[j] = SYNCHRONIZATION /\ leaderAddr[j] = i /\ synced[j] = FALSE
+        \* /\ acceptedEpoch[i] = acceptedEpoch[j] \* This condition is unnecessary.
+        /\ history'     = [history     EXCEPT ![j] = history[i]]
+        /\ lastZxid'    = [lastZxid    EXCEPT ![j] = lastZxid[i]]
+        /\ UpdateProposal(j, leaderAddr[j], lastZxid'[j], currentEpoch[j])
+        /\ commitIndex' = [commitIndex EXCEPT ![j] = commitIndex[i]]
+        /\ synced'      = [synced      EXCEPT ![j] = TRUE]
+        /\ forwarding'  = [forwarding  EXCEPT ![i] = forwarding[i] \union {j}]
+        /\ ackIndex'    = [ackIndex    EXCEPT ![i][j] = Len(history[i])]
+        /\ committedCounter' = [committedCounter EXCEPT ![i][j] = Maximum({commitIndex[i] - Len(initialHistory[i]), 0})]
+        /\ LET ms == [msource|->i, mtype|->"RECOVERYSYNC", mepoch|->acceptedEpoch[i], mproposals|->history[i]]
+           IN proposalMsgsLog' = IF ms \in proposalMsgsLog THEN proposalMsgsLog
+                                                           ELSE proposalMsgsLog \union {ms}
+        /\ Send(i, j, [mtype     |-> NEWLEADER,
+                       mepoch    |-> acceptedEpoch[i],
+                       mlastZxid |-> lastZxid[i]])
+        /\ UNCHANGED <<state, zabState, acceptedEpoch, currentEpoch, logicalClock, receiveVotes, outOfElection, recvQueue, waitNotmsg, 
+                       leadingVoteSet, learners, cepochRecv, ackeRecv, ackldRecv, currentCounter, sendCounter, committedIndex, 
+                       tempVarsZ, cepochSent, leaderAddr, epochLeader, inherentViolated, electionMsgs, idTable>>
+
+(* In phase f21, follower receives NEWLEADER. The follower updates its epoch and history,
+   and sends back ACK-LD to leader. *)
+FollowerHandleNEWLEADER(i, j) ==
+        /\ state[i] = FOLLOWING
+        /\ msgs[j][i] /= << >>
+        /\ msgs[j][i][1].mtype = NEWLEADER
+        /\ LET msg     == msgs[j][i][1]
+               infoOk  == /\ leaderAddr[i] = j
+                          /\ acceptedEpoch[i] = msg.mepoch
+               correct == /\ infoOk
+                          /\ zabState[i] = SYNCHRONIZATION
+                          /\ synced[i]
+                          /\ ZxidEqual(lastZxid[i], msg.mlastZxid)
+           IN /\ infoOk
+              /\ currentEpoch' = [currentEpoch EXCEPT ![i] = msg.mepoch]
+              /\ UpdateProposal(i, j, lastZxid[i], currentEpoch'[i])
+              /\ \/ /\ correct
+                    /\ Reply(i, j, [mtype  |-> ACKLD,
+                                    mepoch |-> msg.mepoch])
+                    /\ UNCHANGED inherentViolated
+                 \/ /\ ~correct
+                    /\ PrintT("Exception: Condition correct is false in FollowerHandleNEWLEADER(" \o ToString(i) \o ", " \o ToString(j) \o ").")
+                    /\ inherentViolated' = TRUE
+                    /\ Discard(j, i)
+        /\ UNCHANGED <<state, lastZxid, zabState, acceptedEpoch, history, commitIndex, logicalClock, receiveVotes, outOfElection, recvQueue, waitNotmsg,
+                       leaderVarsZ, tempVarsZ, followerVarsZ, proposalMsgsLog, epochLeader, electionMsgs, idTable>>
+
+\* In phase l22, leader receives ACK-LD from a quorum of followers, and sends COMMIT-LD(UPTODATE) to followers.
+LeaderHandleACKLD(i, j) ==
+        /\ state[i] = LEADING
+        /\ msgs[j][i] /= << >>
+        /\ msgs[j][i][1].mtype = ACKLD
+        /\ LET msg     == msgs[j][i][1]
+               infoOk  == /\ acceptedEpoch[i] = msg.mepoch
+                          /\ j \in learners[i]
+               replyOk == /\ infoOk
+                          /\ NullPoint \in ackldRecv[i]
+           IN /\ infoOk
+              /\ \/ /\ replyOk
+                    /\ Reply(i, j, [mtype   |-> UPTODATE,
+                                    mepoch  |-> acceptedEpoch[i],
+                                    mcommit |-> commitIndex[i]])
+                    /\ committedCounter' = [committedCounter EXCEPT ![i][j] = Maximum({commitIndex[i] - Len(initialHistory[i]), committedCounter[i][j]})]
+                 \/ /\ ~replyOk
+                    /\ Discard(j, i)
+                    /\ UNCHANGED committedCounter
+              /\ ackldRecv' = [ackldRecv EXCEPT ![i] = IF j \notin ackldRecv[i] THEN ackldRecv[i] \union {j}
+                                                                                ELSE ackldRecv[i]]
+        /\ UNCHANGED <<serverVarsZ, electionVarsZ, leadingVoteSet, learners, cepochRecv, ackeRecv, forwarding, 
+                       ackIndex, currentCounter, sendCounter, committedIndex, tempVarsZ, followerVarsZ, verifyVarsZ, electionMsgs, idTable>>
+
+LeaderSync2(i) ==

Review comment:
       Action name is not descriptive, perhaps rename to something like LeaderTransitionsToBroadcast




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@zookeeper.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] hanm commented on a change in pull request #1690: ZOOKEEPER-3615: Provide formal specification and verification using TLA+ for Zab

Posted by GitBox <gi...@apache.org>.
hanm commented on a change in pull request #1690:
URL: https://github.com/apache/zookeeper/pull/1690#discussion_r630726805



##########
File path: zookeeper-contrib/zookeeper-contrib-zabtla/experiment/ZabWithQTest.tla
##########
@@ -0,0 +1,994 @@
+(*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *)
+-------------------------------- MODULE ZabWithQTest --------------------------------
+\* This is the test for formal specification for the Zab consensus algorithm,

Review comment:
       Can we make this module extends Zab module to avoid copy / paste?

##########
File path: zookeeper-contrib/zookeeper-contrib-zabtla/README.md
##########
@@ -0,0 +1,71 @@
+# Zab-tla
+
+## Overview
+This project is devoted to providing formal specification and verification using TLA+ for the Zookeeper Atomic Broadcast(Zab) consensus protocol proposed in *Junqueira F P, Reed B C, Serafini M. Zab: High-performance broadcast for primary-backup systems[C]//2011 IEEE/IFIP 41st International Conference on Dependable Systems & Networks (DSN). IEEE, 2011: 245-256*.  

Review comment:
       Sounds like this PR is specifically scoped to the paper (which is kind of out of date), which is totally fine. Just want to point out that the actual ZAB running in ZooKeeper production today is ZAB 1.0 while the algorithm described in the paper is pre-1.0 and there are a few changes between the two. There is a canonical description of ZAB 1.0 here: https://cwiki.apache.org/confluence/display/ZOOKEEPER/Zab1.0. It would be more practically impactful if we have a specification checking the modern version of protocol but that can certainly be done after this exercise.

##########
File path: zookeeper-contrib/zookeeper-contrib-zabtla/README.md
##########
@@ -0,0 +1,71 @@
+# Zab-tla
+
+## Overview
+This project is devoted to providing formal specification and verification using TLA+ for the Zookeeper Atomic Broadcast(Zab) consensus protocol proposed in *Junqueira F P, Reed B C, Serafini M. Zab: High-performance broadcast for primary-backup systems[C]//2011 IEEE/IFIP 41st International Conference on Dependable Systems & Networks (DSN). IEEE, 2011: 245-256*.  
+
+We have made a formal [specification](Zab.tla) for Zab using TLA+ toolbox, and we have done a certain scale of model checking to verify the correctness of Zab.
+
+Due to the simplification of Zab algorithm description in the paper, some details in specification were modified and added. If you have any question, please let us know.
+
+You can find this document in chinese in [doc-in-chinsese](doc-in-chinese/README.md).
+
+## Requirements
+TLA+ toolbox version 1.7.0
+
+## Run
+Create specification and run models in the usual way.  

Review comment:
       I would recommend we put more descriptions on the steps to run the TLC model checker on this specification. In particular, the configurations like invariants, constraints, etc, so everyone who is interested can trivially reproduce (I see the previous discussions in this PR largely related to figure out this). What some other projects do is to also commit the toolbox configuration files (the .toolbox folder), maybe we can also consider doing the same.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] BinyuHuang-nju commented on pull request #1690: ZOOKEEPER-3615: Provide formal specification and verification using TLA+ for Zab

Posted by GitBox <gi...@apache.org>.
BinyuHuang-nju commented on pull request #1690:
URL: https://github.com/apache/zookeeper/pull/1690#issuecomment-860063584


   @hanm Thank you! I am now writing this spec and I will push it here.  At that time, I hope you can help me check whether my spec is logically deviated from zab 1.0 in production, especially in SYNC part.
   I will implement those parts now we choose to abstract like FLE if you think it is necessary!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] BinyuHuang-nju commented on pull request #1690: ZOOKEEPER-3615: Provide formal specification and verification using TLA+ for Zab

Posted by GitBox <gi...@apache.org>.
BinyuHuang-nju commented on pull request #1690:
URL: https://github.com/apache/zookeeper/pull/1690#issuecomment-831741370


   @Vanlightly It is correct to find these actions not executed. Because when there are only two servers, the quorum is just these two servers. And these actions should be executed when server number is larger than 2. Like FindCluster, this action means a server after restart wants to join a cluster, but it will never receive responses from a quorum, because there is only one left server and it can not construct a quorum.
   
   Thank you for your attention! I promise you will find these actions reached when you set server number like 3,4,5.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] BinyuHuang-nju commented on pull request #1690: ZOOKEEPER-3615: Provide formal specification and verification using TLA+ for Zab

Posted by GitBox <gi...@apache.org>.
BinyuHuang-nju commented on pull request #1690:
URL: https://github.com/apache/zookeeper/pull/1690#issuecomment-836631229


   @Vanlightly Thank you! There exists differences between zab in paper and zab in my spec. Please forgive me if it causes confusion in understanding spec. We are thinking about other ways to compress state space of model checking more significantly without affecting correctness. But it will take a longer time when we can achive a good result if we choose to do it. We will be very happy if you could provide some comments or suggestions in spec and model checking!!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] BinyuHuang-nju commented on pull request #1690: ZOOKEEPER-3615: Provide formal specification and verification using TLA+ for Zab

Posted by GitBox <gi...@apache.org>.
BinyuHuang-nju commented on pull request #1690:
URL: https://github.com/apache/zookeeper/pull/1690#issuecomment-880483752


   @hanm @Vanlightly Hi! Now I have pushed spec of zab-1.0 here.
   In spec, I use action RECOVERYSYNC to abstract the phase of SYNC before leader sending NEWLEADER.
   Due to the differences between formal specification and engineering implementation, and the purposes of reducing state space, there are differences in my expression and code in some places. But I think this does not affect the equivalence between the two.
   For example, in spec, only the leader in broadcast phase will accept the requests, which is to omit the extra space brought by the follower’s acceptance of client’s request.
   I will be very glad to see your feedbacks! And I will reply to you as soon as possible if you have any question!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@zookeeper.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] hanm commented on pull request #1690: ZOOKEEPER-3615: Provide formal specification and verification using TLA+ for Zab

Posted by GitBox <gi...@apache.org>.
hanm commented on pull request #1690:
URL: https://github.com/apache/zookeeper/pull/1690#issuecomment-852694844


   
   > 
   > 1. At line 630 in function 'syncWithLeader' In [Learner.java](https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/quorum/Learner.java), I see follower in phase SYNCHRONIZATION may receive PROPOSAL and COMMIT. But in paper, leader will add a new server into its 'forwardingFollowers'(Q in paper) only when receiving ack of NEWLEADER from it, so in paper this new server should not receive PROPOSAL or COMMIT because in phase SYNCHRONIZATION it has not sent ACK-NEWLEADER.
   >    So here, my question is in ZAB 1.0, when does leader add a newly joined server into its 'forwardingFollowers'?(I am sorry that I didn't find this in code)
   
   You can't find one, because there is no such thing in production. In production, a newly joined server will connect to leader first, the leader will spin up a dedicated handler thread to sync this follower. You can check LearnerHandler implementation for details.
   
   The other important detail here is - in paper, the NEWLEADER does not have the same semantic as the actual production implementation. In paper, the NEWLEADER(e, Ie) carries the epoch AND the leader history, but the actual implementation of NEWLEADER is a pure control operation rather than a mixed control + data operation as described in paper. 
   
   > 2. My understanding of question above is: in phase 2, leader syncs with each follower and ''sends'' NEWLEADER only to this corresponding follower that has completed sync, not ''broadcasts'' NEWLEADER. So when leader is in phase 3, there may exist some follower in phase 2, and has not completed sync with leader since it hasn't received NEWLEADER, but it is allowed to receive PROPOSAL and COMMIT.
   
   As previously mentioned, because each follower sync is handled by a dedicated handler thread, there is no such state as "leader is in phase X". We can say that the "handler of follower X is in phase Y", though. Because of this, we don't need worry about "So when leader is in phase 3, there may exist some follower in phase 2", which will not happen, because its the handler thread, instead of leader, that's in different phases; and because one to one mapping between handler thread and follower, their phase always match.
   
   >    I am not sure whether my understanding is right. If so, I think my spec about phase 2 is that, for each connected follower: first, leader syncs with  the follower by an abstract action  RECOVERY_SYNC; second, in the action, follower will have the latest history of leader, and leader sends NEWLEADER to it; then, leader and follower communicate using ACK-NEWLEADER and UPTODATE and follower turns to phase 3. Leader turns to phase 3 upon receiving ACK-NEWLEADER from a quorum.
   
   The sequences of events  is correct.
   
   > 3. At line 766 in function 'syncWithLeader' in [Learner.java](https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/quorum/Learner.java), I see when receiving NEWLEADER, the variable 'packetsNotCommitted' is set as empty, and when receiving UPTODATE, there is no action handling transactions left to commit them.
   
   When follower receives NEWLEADER, it will apply all previous queued proposals from leader atomically to its state machine. The UPTODATE is merely a control operation to signal follower that the leadership and quorum is established, and this specific follower can start serving traffic.
   
   >    Does that mean follower thinks all transactions committed upon receiving NEWLEADER not UPTODATE? And it seems like that NEWLEADER, ACK-LEADER, UPTODATE carries little data except epoch, just used for acknowledgement between leader and followers.
   
   I don't quite get your first question here. 
   The remaining part is correct, these are control operations that mark the milestones of the sync phase, and they don't carry replicated data.
   
   > 4. In phase 1 in manual tha we mentioned above, there's one word: "Note, if a follower is connecting, but the leader is already established (in phase 3) the follower follows the phases, but the leader ignores any ACKs."
   >    It seems like that leader in phase 3(BROADCAST) will ignre ACKEPOCH in phase1 and ACK of NEWLEADER in phase 2, then how dose this newly connecting follower receive UPTODATE to go to phase BROADCAST?
   
   As previously mentioned, it's the handler thread in leader that handles the sync phase. Each handler thread will have different state but this state will be kept in sync with the corresponding follower where the handler thread is designated to.
   
   > 
   > Since these details are related to how to let follower sync with leader in my spec of ZAB 1.0, I want to have a further understanding of them. Thank you!
   
   I would recommend read https://cwiki.apache.org/confluence/display/ZOOKEEPER/Zab1.0 - which is exactly the same as production implementation. Maybe treat this as source of truth and use the paper as advisory only.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] BinyuHuang-nju commented on pull request #1690: ZOOKEEPER-3615: Provide formal specification and verification using TLA+ for Zab

Posted by GitBox <gi...@apache.org>.
BinyuHuang-nju commented on pull request #1690:
URL: https://github.com/apache/zookeeper/pull/1690#issuecomment-850846721


   @Vanlightly @hanm Hi! I have read some codes about zab. But I still cannot understand some details. I want to ask you about them.
   
   1. At line 630 in function 'syncWithLeader' In [Learner.java](https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/quorum/Learner.java), I see follower in phase SYNCHRONIZATION may receive PROPOSAL and COMMIT. But in paper, leader will add a new server into its 'forwardingFollowers'(Q in paper) only when receiving ack of NEWLEADER from it, so in paper this new server should not receive PROPOSAL or COMMIT because in phase SYNCHRONIZATION it has not sent ACK-NEWLEADER.
   So here, my question is in ZAB 1.0, when does leader add a newly joined server into its 'forwardingFollowers'?(I am sorry that I didn't find this in code)
   
   2. At line 766 in function 'syncWithLeader' in [Learner.java](https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/quorum/Learner.java), I see when receiving NEWLEADER, the variable 'packetsNotCommitted' is set as empty, and when receiving UPTODATE, there is no action handling transactions left to commit them.
   Does that mean follower thinks all transactions committed upon receiving NEWLEADER not UPTODATE?(Actually in papaer and my spec of zab pre-1.0, follower thinks them committed upon receiving UPTODATE)
   
   Since these details are related to how to let follower sync with leader in my spec of ZAB 1.0, I want to have a further understanding of them. Thank you!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] BinyuHuang-nju commented on pull request #1690: Provide formal specification and verification using TLA+ for Zab

Posted by GitBox <gi...@apache.org>.
BinyuHuang-nju commented on pull request #1690:
URL: https://github.com/apache/zookeeper/pull/1690#issuecomment-831720581


   @hanm Thank you. I will update the title and try this building way later.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] Vanlightly commented on a change in pull request #1690: ZOOKEEPER-3615: Provide formal specification and verification using TLA+ for Zab

Posted by GitBox <gi...@apache.org>.
Vanlightly commented on a change in pull request #1690:
URL: https://github.com/apache/zookeeper/pull/1690#discussion_r705452806



##########
File path: zookeeper-specifications/zab-1.0/ZabWithFLE.tla
##########
@@ -0,0 +1,1020 @@
+(*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *)
+
+----------------------------- MODULE ZabWithFLE -----------------------------
+(* This is the formal specification for the Zab consensus algorithm,
+   which means Zookeeper Atomic Broadcast.*)
+
+(* Reference:
+   FLE: FastLeaderElection.java, Vote.java, QuorumPeer.java in https://github.com/apache/zookeeper.
+   ZAB: QuorumPeer.java, Learner.java, Follower.java, LearnerHandler.java, Leader.java 
+        in https://github.com/apache/zookeeper.
+        https://cwiki.apache.org/confluence/display/ZOOKEEPER/Zab1.0.
+ *)
+EXTENDS FastLeaderElection
+
+-----------------------------------------------------------------------------
+(* Defined in FastLeaderElection.tla:
+\* The set of server identifiers
+CONSTANT Server
+
+\* Server states
+CONSTANTS LOOKING, FOLLOWING, LEADING
+
+\* Message types
+CONSTANTS NOTIFICATION
+
+\* Timeout signal
+CONSTANT NONE
+*)
+\* The set of requests that can go into history
+CONSTANT Value
+
+\* Zab states
+CONSTANTS ELECTION, DISCOVERY, SYNCHRONIZATION, BROADCAST
+
+\* Message types
+CONSTANTS FOLLOWERINFO, LEADERINFO, ACKEPOCH, NEWLEADER, ACKLD, UPTODATE, PROPOSAL, ACK, COMMIT
+(* Additional message types used for recovery in synchronization(TRUNC/DIFF/SNAP) are not needed
+   since we abstract this part.(see action RECOVERYSYNC) *)
+   
+-----------------------------------------------------------------------------
+(* Defined in FastLeaderElection.tla: Quorums, NullPoint *)
+\* Return the maximum value from the set S
+Maximum(S) == IF S = {} THEN -1
+                        ELSE CHOOSE n \in S: \A m \in S: n >= m
+
+\* Return the minimum value from the set S
+Minimum(S) == IF S = {} THEN -1
+                        ELSE CHOOSE n \in S: \A m \in S: n <= m
+
+MAXEPOCH == 10
+
+-----------------------------------------------------------------------------
+(* Defined in FastLeaderElection.tla:
+   serverVars: <<state, currentEpoch, lastZxid>>,
+   electionVars: <<currentVote, logicalClock, receiveVotes, outOfElection, recvQueue, waitNotmsg>>,
+   leaderVars: <<leadingVoteSet>>,
+   electionMsgs,
+   idTable *)
+\* The current phase of server(ELECTION,DISCOVERY,SYNCHRONIZATION,BROADCAST)
+VARIABLE zabState
+
+\* The epoch number of the last NEWEPOCH(LEADERINFO) packet accepted
+\* namely f.p in paper, and currentEpoch in Zab.tla.
+VARIABLE acceptedEpoch
+
+\* The history of servers as the sequence of transactions.
+VARIABLE history
+
+\* commitIndex[i]: The maximum index of transactions that have been saved in a quorum of servers
+\*                 in the perspective of server i.(increases monotonically before restarting)
+VARIABLE commitIndex
+
+(* These transactions whose index \le commitIndex[i] can be applied to state machine immediately.
+   So if we have a variable applyIndex, we can suppose that applyIndex[i] = commitIndex[i] when verifying properties.
+   But in phase SYNC, follower will apply all queued proposals to state machine when receiving NEWLEADER.
+   But follower only serves traffic after receiving UPTODATE, so sequential consistency is not violated.
+   
+   So when we verify properties, we still suppose applyIndex[i] = commitIndex[i], because this is an engineering detail.*)
+
+\* learners[i]: The set of servers which leader i think are connected wich i.
+VARIABLE learners
+
+\* The messages representing requests and responses sent from one server to another.
+\* msgs[i][j] means the input buffer of server j from server i.
+VARIABLE msgs
+
+\* The set of followers who has successfully sent CEPOCH(FOLLOWERINFO) to leader.(equals to connectingFollowers in code)
+VARIABLE cepochRecv
+
+\* The set of followers who has successfully sent ACK-E to leader.(equals to electingFollowers in code)
+VARIABLE ackeRecv
+
+\* The set of followers who has successfully sent ACK-LD to leader in leader.(equals to newLeaderProposal in code)
+VARIABLE ackldRecv
+
+\* The set of servers which leader i broadcasts PROPOSAL and COMMIT to.(equals to forwardingFollowers in code)
+VARIABLE forwarding
+
+\* ackIndex[i][j]: The latest index that leader i has received from follower j via ACK.
+VARIABLE ackIndex
+
+\* currentCounter[i]: The count of transactions that clients request leader i.
+VARIABLE currentCounter
+
+\* sendCounter[i]: The count of transactions that leader i has broadcast in PROPOSAL.
+VARIABLE sendCounter
+
+\* committedIndex[i]: The maximum index of trasactions that leader i has broadcast in COMMIT.
+VARIABLE committedIndex
+
+\* committedCounter[i][j]: The latest counter of transaction that leader i has confirmed that follower j has committed.
+VARIABLE committedCounter
+
+\* initialHistory[i]: The initial history if leader i in epoch acceptedEpoch[i].
+VARIABLE initialHistory
+
+\* the maximum epoch in CEPOCH the prospective leader received from followers.
+VARIABLE tempMaxEpoch
+
+\* cepochSent[i] = TRUE means follower i has sent CEPOCH(FOLLOWERINFO) to leader.
+VARIABLE cepochSent
+
+\* leaderAddr[i]: The leader id of follower i. We use leaderAddr to express whether follower i has connected or lost connection.
+VARIABLE leaderAddr
+
+\* synced[i] = TRUE: follower i has completed sync with leader.
+VARIABLE synced
+
+\* The set of leaders in every epoch, only used in verifying properties.
+VARIABLE epochLeader
+
+\* The set of all broadcast messages, only used in verifying properties.
+VARIABLE proposalMsgsLog
+
+\* A variable used to check whether there are conditions contrary to the facts.
+VARIABLE inherentViolated
+
+serverVarsZ == <<state, currentEpoch, lastZxid, zabState, acceptedEpoch, history, commitIndex>>         \* 7 variables
+
+electionVarsZ == electionVars  \* 6 variables
+
+leaderVarsZ == <<leadingVoteSet, learners, cepochRecv, ackeRecv, ackldRecv, forwarding, 
+                 ackIndex, currentCounter, sendCounter, committedIndex, committedCounter>>              \* 11 variables
+
+tempVarsZ == <<initialHistory, tempMaxEpoch>>     \* 2 variables
+  
+followerVarsZ == <<cepochSent, leaderAddr, synced>>                 \* 3 variables
+
+verifyVarsZ == <<proposalMsgsLog, epochLeader, inherentViolated>>   \* 3 variables
+
+msgVarsZ == <<msgs, electionMsgs>>                \* 2 variables
+
+vars == <<serverVarsZ, electionVarsZ, leaderVarsZ, tempVarsZ, followerVarsZ, verifyVarsZ, msgVarsZ, idTable>>  \* 35 variables
+-----------------------------------------------------------------------------
+\* Add a message to msgs - add a message m to msgs[i][j].
+Send(i, j, m) == msgs' = [msgs EXCEPT ![i][j] = Append(msgs[i][j], m)]
+
+\* Remove a message from msgs - discard head of msgs[i][j].
+Discard(i, j) == msgs' = IF msgs[i][j] /= << >> THEN [msgs EXCEPT ![i][j] = Tail(msgs[i][j])]
+                                                ELSE msgs
+
+\* Leader broadcasts a message(PROPOSAL/COMMIT) to all other servers in forwardingFollowers.
+Broadcast(i, m) == msgs' = [msgs EXCEPT ![i] = [v \in Server |-> IF /\ v \in forwarding[i]
+                                                                    /\ v /= i
+                                                                    /\ \/ /\ m.mtype = PROPOSAL
+                                                                          /\ ackIndex[i][v] < Len(initialHistory[i]) + m.mproposal.counter
+                                                                       \/ /\ m.mtype = COMMIT
+                                                                          /\ committedCounter[i][v] < m.mzxid[2]
+                                                                  THEN Append(msgs[i][v], m)
+                                                                  ELSE msgs[i][v]]]
+
+BroadcastLEADERINFO(i, m) == msgs' = [msgs EXCEPT ![i] = [v \in Server |-> IF /\ v \in cepochRecv[i]
+                                                                              /\ v \in learners[i]
+                                                                              /\ v /= i THEN Append(msgs[i][v], m)
+                                                                                        ELSE msgs[i][v]]]
+
+BroadcastUPTODATE(i, m) == msgs' = [msgs EXCEPT ![i] = [v \in Server |-> IF /\ v \in ackldRecv[i]
+                                                                            /\ v \in learners[i]
+                                                                            /\ v /= i THEN Append(msgs[i][v], m)
+                                                                                      ELSE msgs[i][v]]]
+
+\* Combination of Send and Discard - discard head of msgs[j][i] and add m into msgs[i][j].
+Reply(i, j, m) == msgs' = [msgs EXCEPT ![j][i] = Tail(msgs[j][i]),
+                                       ![i][j] = Append(msgs[i][j], m)]
+
+\* shuffle the input buffer from server j(i) in server i(j).
+Clean(i, j) == msgs' = [msgs EXCEPT ![j][i] = << >>, ![i][j] = << >>]
+
+-----------------------------------------------------------------------------
+PZxidEqual(p, z) == p.epoch = z[1] /\ p.counter = z[2]
+
+TransactionEqual(t1, t2) == /\ t1.epoch   = t2.epoch
+                            /\ t1.counter = t2.counter
+                            
+TransactionPrecede(t1, t2) == \/ t1.epoch < t2.epoch
+                              \/ /\ t1.epoch   = t2.epoch
+                                 /\ t1.counter < t2.counter
+                                 
+-----------------------------------------------------------------------------
+\* Define initial values for all variables 
+InitServerVarsZ == /\ InitServerVars
+                   /\ zabState      = [s \in Server |-> ELECTION]
+                   /\ acceptedEpoch = [s \in Server |-> 0]
+                   /\ history       = [s \in Server |-> << >>]
+                   /\ commitIndex   = [s \in Server |-> 0]
+
+InitLeaderVarsZ == /\ InitLeaderVars
+                   /\ learners         = [s \in Server |-> {}]
+                   /\ cepochRecv       = [s \in Server |-> {}]
+                   /\ ackeRecv         = [s \in Server |-> {}]
+                   /\ ackldRecv        = [s \in Server |-> {}]
+                   /\ ackIndex         = [s \in Server |-> [v \in Server |-> 0]]
+                   /\ currentCounter   = [s \in Server |-> 0]
+                   /\ sendCounter      = [s \in Server |-> 0]
+                   /\ committedIndex   = [s \in Server |-> 0]
+                   /\ committedCounter = [s \in Server |-> [v \in Server |-> 0]]
+                   /\ forwarding       = [s \in Server |-> {}]
+
+InitElectionVarsZ == InitElectionVars
+
+InitTempVarsZ == /\ initialHistory = [s \in Server |-> << >>]
+                 /\ tempMaxEpoch   = [s \in Server |-> 0]
+
+InitFollowerVarsZ == /\ cepochSent = [s \in Server |-> FALSE]
+                     /\ leaderAddr = [s \in Server |-> NullPoint]
+                     /\ synced     = [s \in Server |-> FALSE]
+
+InitVerifyVarsZ == /\ proposalMsgsLog  = {}
+                   /\ epochLeader      = [i \in 1..MAXEPOCH |-> {}]
+                   /\ inherentViolated = FALSE
+                   
+InitMsgVarsZ == /\ msgs         = [s \in Server |-> [v \in Server |-> << >>]]
+                /\ electionMsgs = [s \in Server |-> [v \in Server |-> << >>]]
+
+InitZ == /\ InitServerVarsZ
+         /\ InitLeaderVarsZ
+         /\ InitElectionVarsZ
+         /\ InitTempVarsZ
+         /\ InitFollowerVarsZ
+         /\ InitVerifyVarsZ
+         /\ InitMsgVarsZ
+         /\ idTable = InitializeIdTable(Server)
+         
+-----------------------------------------------------------------------------
+ZabTurnToLeading(i) ==
+        /\ zabState'       = [zabState   EXCEPT ![i] = DISCOVERY]
+        /\ learners'       = [learners   EXCEPT ![i] = {i}]
+        /\ cepochRecv'     = [cepochRecv EXCEPT ![i] = {i}]
+        /\ ackeRecv'       = [ackeRecv   EXCEPT ![i] = {i}]
+        /\ ackldRecv'      = [ackldRecv  EXCEPT ![i] = {i}]
+        /\ forwarding'     = [forwarding EXCEPT ![i] = {}]
+        /\ ackIndex'       = [ackIndex   EXCEPT ![i] = [v \in Server |-> IF v = i THEN Len(history[i])
+                                                                                  ELSE 0]]
+        /\ currentCounter'   = [currentCounter   EXCEPT ![i] = 0]
+        /\ sendCounter'      = [sendCounter      EXCEPT ![i] = 0]
+        /\ commitIndex'      = [commitIndex      EXCEPT ![i] = 0]
+        /\ committedIndex'   = [committedIndex   EXCEPT ![i] = 0]
+        /\ committedCounter' = [committedCounter EXCEPT ![i] = [v \in Server |-> IF v = i THEN Len(history[i])
+                                                                                          ELSE 0]]
+        /\ initialHistory' = [initialHistory EXCEPT ![i] = history[i]]
+        /\ tempMaxEpoch'   = [tempMaxEpoch   EXCEPT ![i] = acceptedEpoch[i]]
+
+ZabTurnToFollowing(i) ==
+        /\ zabState'    = [zabState   EXCEPT ![i] = DISCOVERY]
+        /\ cepochSent'  = [cepochSent EXCEPT ![i] = FALSE]
+        /\ synced'      = [synced     EXCEPT ![i] = FALSE]
+        /\ commitIndex' = [commitIndex      EXCEPT ![i] = 0]
+        
+          
+(* Fast Leader Election *)
+FLEReceiveNotmsg(i, j) ==
+        /\ ReceiveNotmsg(i, j)
+        /\ UNCHANGED <<zabState, acceptedEpoch, history, commitIndex,learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter,
+                       sendCounter, committedIndex, committedCounter, tempVarsZ, followerVarsZ, verifyVarsZ, msgs>>
+
+FLENotmsgTimeout(i) ==
+        /\ NotmsgTimeout(i)
+        /\ UNCHANGED <<zabState, acceptedEpoch, history, commitIndex,learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter,
+                       sendCounter, committedIndex, committedCounter, tempVarsZ, followerVarsZ, verifyVarsZ, msgs>>
+
+FLEHandleNotmsg(i) ==
+        /\ HandleNotmsg(i)
+        /\ LET newState == state'[i]
+           IN
+           \/ /\ newState = LEADING
+              /\ ZabTurnToLeading(i)
+              /\ UNCHANGED <<cepochSent, synced>>
+           \/ /\ newState = FOLLOWING
+              /\ ZabTurnToFollowing(i)
+              /\ UNCHANGED <<learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter, sendCounter, committedIndex, committedCounter, tempVarsZ>>
+           \/ /\ newState = LOOKING
+              /\ UNCHANGED <<zabState, learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter, sendCounter, commitIndex,
+                             committedIndex, committedCounter, tempVarsZ, cepochSent, synced>>
+        /\ UNCHANGED <<acceptedEpoch, history, leaderAddr, verifyVarsZ, msgs>>
+
+\* On the premise that ReceiveVotes.HasQuorums = TRUE, corresponding to logic in line 1050-1055 in LFE.java.
+FLEWaitNewNotmsg(i) ==
+        /\ WaitNewNotmsg(i)
+        /\ UNCHANGED <<zabState, acceptedEpoch, history, commitIndex,learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter,
+                       sendCounter, committedIndex, committedCounter, tempVarsZ, followerVarsZ, verifyVarsZ, msgs>>
+
+\* On the premise that ReceiveVotes.HasQuorums = TRUE, corresponding to logic in line 1061-1066 in LFE.java.
+FLEWaitNewNotmsgEnd(i) ==
+        /\ WaitNewNotmsgEnd(i)
+        /\ LET newState == state'[i]
+           IN
+           \/ /\ newState = LEADING
+              /\ ZabTurnToLeading(i)
+              /\ UNCHANGED <<cepochSent, synced>>
+           \/ /\ newState = FOLLOWING
+              /\ ZabTurnToFollowing(i)
+              /\ UNCHANGED <<learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter, sendCounter, committedIndex, committedCounter, tempVarsZ>>
+           \/ /\ newState = LOOKING
+              /\ PrintT("New state is LOOKING in FLEWaitNewNotmsgEnd, which should not happen.")
+              /\ UNCHANGED <<zabState, learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter, sendCounter, commitIndex,
+                             committedIndex, committedCounter, tempVarsZ, cepochSent, synced>>
+        /\ UNCHANGED <<acceptedEpoch, history, leaderAddr, verifyVarsZ, msgs>>
+              
+-----------------------------------------------------------------------------
+(* A sub-action describing how a server transitions from LEADING/FOLLOWING to LOOKING.
+   Initially I call it 'ZabTimeoutZ', but it will be called not only when timeout, 
+   but also when finding a low epoch from leader.*)
+FollowerShutdown(i) ==
+        /\ ZabTimeout(i)
+        /\ zabState'   = [zabState   EXCEPT ![i] = ELECTION]
+        /\ leaderAddr' = [leaderAddr EXCEPT ![i] = NullPoint]
+
+LeaderShutdown(i) ==
+        /\ ZabTimeout(i)
+        /\ zabState'   = [zabState   EXCEPT ![i] = ELECTION]
+        /\ leaderAddr' = [s \in Server |-> IF s \in learners[i] THEN NullPoint ELSE leaderAddr[s]]
+        /\ learners'   = [learners   EXCEPT ![i] = {}]
+        /\ forwarding' = [forwarding EXCEPT ![i] = {}]
+        /\ msgs'       = [s \in Server |-> [v \in Server |-> IF v \in learners[i] \/ s \in learners[i] THEN << >> ELSE msgs[s][v]]]
+
+FollowerTimout(i) ==
+        /\ state[i]      = FOLLOWING
+        /\ leaderAddr[i] = NullPoint
+        /\ FollowerShutdown(i)
+        /\ msgs' = [s \in Server |-> [v \in Server |-> IF v = i THEN << >> ELSE msgs[s][v]]]
+        /\ UNCHANGED <<acceptedEpoch, history, commitIndex, learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, 
+                       currentCounter, sendCounter, committedIndex, committedCounter, tempVarsZ, cepochSent, synced, verifyVarsZ>>
+
+LeaderTimeout(i) ==
+        /\ state[i] = LEADING
+        /\ learners[i] \notin Quorums
+        /\ LeaderShutdown(i)
+        /\ UNCHANGED <<acceptedEpoch, history, commitIndex, cepochRecv, ackeRecv, ackldRecv, ackIndex, currentCounter, sendCounter, committedIndex, committedCounter,
+                       tempVarsZ, cepochSent, synced, verifyVarsZ>>
+    
+-----------------------------------------------------------------------------
+(* Establish connection between leader i and follower j. 
+   It means i creates a learnerHandler for communicating with j, and j finds i's address.*)
+EstablishConnection(i, j) ==
+        /\ state[i] = LEADING   /\ state[j] = FOLLOWING
+        /\ j \notin learners[i] /\ leaderAddr[j] = NullPoint
+        /\ currentVote[j].proposedLeader = i
+        /\ learners'   = [learners   EXCEPT ![i] = learners[i] \union {j}] \* Leader:   'addLearnerHandler(peer)'
+        /\ leaderAddr' = [leaderAddr EXCEPT ![j] = i]                      \* Follower: 'connectToLeader(addr, hostname)'
+        /\ UNCHANGED <<serverVarsZ, electionVarsZ, leadingVoteSet, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, 
+                       currentCounter, sendCounter, committedIndex, committedCounter, tempVarsZ, cepochSent, synced, verifyVarsZ, msgVarsZ, idTable>>
+        
+(* The leader i finds timeout and TCP connection between i and j closes.*)       
+Timeout(i, j) ==
+        /\ state[i] = LEADING /\ state[j] = FOLLOWING
+        /\ j \in learners[i]  /\ leaderAddr[j] = i
+        (* The action of leader i.(corresponding to function 'removeLearnerHandler(peer)'.) *)
+        /\ learners'   = [learners   EXCEPT ![i] = learners[i] \ {j}] 
+        /\ forwarding' = [forwarding EXCEPT ![i] = IF j \in forwarding[i] THEN forwarding[i] \ {j} ELSE forwarding[i]]
+        /\ cepochRecv' = [cepochRecv EXCEPT ![i] = IF j \in cepochRecv[i] THEN cepochRecv[i] \ {j} ELSE cepochRecv[i]]
+        /\ ackeRecv'   = [ackeRecv   EXCEPT ![i] = IF j \in ackeRecv[i] THEN ackeRecv[i] \ {j} ELSE ackeRecv[i]]
+        /\ ackldRecv'  = [ackldRecv  EXCEPT ![i] = IF j \in ackldRecv[i] THEN ackldRecv[i] \ {j} ELSE ackldRecv[i]]
+        /\ ackIndex'   = [ackIndex   EXCEPT ![i][j] = 0]
+        /\ committedCounter' = [committedCounter EXCEPT ![i][j] = 0]
+        (* The action of follower j. *)
+        /\ FollowerShutdown(j)
+        (* Clean input buffer.*)
+        /\ Clean(i, j)
+        /\ UNCHANGED <<acceptedEpoch, history, commitIndex, currentCounter, sendCounter, committedIndex,
+                       tempVarsZ, cepochSent, synced, verifyVarsZ>>
+-----------------------------------------------------------------------------
+\* In phase f11, follower sends f.p to leader via FOLLOWERINFO(CEPOCH).
+FollowerSendFOLLOWERINFO(i) ==
+        /\ state[i]    = FOLLOWING
+        /\ zabState[i] = DISCOVERY
+        /\ leaderAddr[i] /= NullPoint
+        /\ \lnot cepochSent[i]
+        /\ Send(i, leaderAddr[i], [mtype  |-> FOLLOWERINFO,
+                                   mepoch |-> acceptedEpoch[i]])
+        /\ cepochSent' = [cepochSent EXCEPT ![i] = TRUE]
+        /\ UNCHANGED <<serverVarsZ, leaderVarsZ, electionVarsZ, tempVarsZ, leaderAddr, synced, verifyVarsZ, electionMsgs, idTable>>
+        
+(* In phase l11, leader waits for receiving FOLLOWERINFO from a quorum,
+   and then chooses a new epoch e' as its own epoch and broadcasts LEADERINFO. *)
+LeaderHandleFOLLOWERINFO(i, j) ==
+        /\ state[i] = LEADING
+        /\ msgs[j][i] /= << >>
+        /\ msgs[j][i][1].mtype = FOLLOWERINFO
+        /\ LET msg == msgs[j][i][1]
+           IN \/ /\ NullPoint \notin cepochRecv[i]   \* 1. has not broadcast LEADERINFO - modify tempMaxEpoch
+                 /\ LET newEpoch == Maximum({tempMaxEpoch[i], msg.mepoch})
+                    IN tempMaxEpoch' = [tempMaxEpoch EXCEPT ![i] = newEpoch]
+                 /\ Discard(j, i)
+              \/ /\ NullPoint \in cepochRecv[i]      \* 2. has broadcast LEADERINFO - no need to handle the msg, just send LEADERINFO to corresponding server
+                 /\ Reply(i, j, [mtype  |-> LEADERINFO,
+                                 mepoch |-> acceptedEpoch[i]])
+                 /\ UNCHANGED tempMaxEpoch
+        /\ cepochRecv' = [cepochRecv EXCEPT ![i] = IF j \in cepochRecv[i] THEN cepochRecv[i]
+                                                                          ELSE cepochRecv[i] \union {j}]
+        /\ UNCHANGED <<serverVarsZ, followerVarsZ, electionVarsZ, initialHistory, leadingVoteSet, learners, ackeRecv, ackldRecv, 
+                       forwarding, ackIndex, currentCounter, sendCounter, committedIndex, committedCounter, verifyVarsZ, electionMsgs, idTable>>
+
+LeaderDiscovery1(i) ==
+        /\ state[i]    = LEADING
+        /\ zabState[i] = DISCOVERY
+        /\ cepochRecv[i] \in Quorums
+        /\ acceptedEpoch' = [acceptedEpoch EXCEPT ![i] = tempMaxEpoch[i] + 1]
+        /\ cepochRecv'    = [cepochRecv    EXCEPT ![i] = cepochRecv[i] \union {NullPoint}]
+        /\ BroadcastLEADERINFO(i, [mtype  |-> LEADERINFO,
+                                   mepoch |-> acceptedEpoch'[i]])
+        /\ UNCHANGED <<state, currentEpoch, lastZxid, zabState, history, commitIndex, electionVarsZ, leadingVoteSet, learners, ackeRecv, ackldRecv, 
+                       forwarding, ackIndex, currentCounter, sendCounter, committedIndex, committedCounter, 
+                       tempVarsZ, followerVarsZ, verifyVarsZ, electionMsgs, idTable>>
+
+(* In phase f12, follower receives NEWEPOCH. If e' > f.p, then follower sends ACK-E back,
+   and ACK-E contains f.a and lastZxid to let leader judge whether it is the latest.
+   After handling NEWEPOCH, follower's zabState turns to SYNCHRONIZATION. *)
+FollowerHandleLEADERINFO(i, j) ==
+        /\ state[i] = FOLLOWING
+        /\ msgs[j][i] /= << >>
+        /\ msgs[j][i][1].mtype = LEADERINFO
+        /\ LET msg     == msgs[j][i][1]
+               infoOk  == j = leaderAddr[i]
+               epochOk == /\ infoOk
+                          /\ msg.mepoch >= acceptedEpoch[i]
+               correct == /\ epochOk
+                          /\ zabState[i] = DISCOVERY
+           IN /\ infoOk
+              /\ \/ /\ epochOk                      \* 1. Normal case
+                    /\ \/ /\ correct
+                          /\ acceptedEpoch' = [acceptedEpoch EXCEPT ![i] = msg.mepoch]
+                          /\ Reply(i, j, [mtype      |-> ACKEPOCH,
+                                          mepoch     |-> msg.mepoch,
+                                          mlastEpoch |-> currentEpoch[i],
+                                          mlastZxid  |-> lastZxid[i]])
+                          /\ cepochSent' = [cepochSent EXCEPT ![i] = TRUE]
+                          /\ UNCHANGED inherentViolated
+                       \/ /\ ~correct
+                          /\ PrintT("Exception: Condition correct is false in FollowerHandleLEADERINFO(" \o ToString(i) \o ", " \o ToString(j) \o ").")
+                          /\ inherentViolated' = TRUE
+                          /\ Discard(j, i)
+                          /\ UNCHANGED <<acceptedEpoch, cepochSent>>
+                    /\ zabState' = [zabState EXCEPT ![i] = IF zabState[i] = DISCOVERY THEN SYNCHRONIZATION
+                                                                                      ELSE zabState[i]]
+                    /\ UNCHANGED <<varsL, leaderAddr>>
+                 \/ /\ ~epochOk                    \* 2. Abnormal case - go back to election
+                    /\ FollowerShutdown(i)
+                    /\ Clean(i, j)
+                    /\ UNCHANGED <<acceptedEpoch, cepochSent, inherentViolated>>
+        /\ UNCHANGED <<history, commitIndex, learners, cepochRecv, ackeRecv, ackldRecv, forwarding, ackIndex, currentCounter, sendCounter, committedIndex,
+                       committedCounter, tempVarsZ, synced, proposalMsgsLog, epochLeader>>
+
+\* Abstraction of actions making follower synced with leader before leader sending NEWLEADER.
+subRECOVERYSYNC(i, j) ==

Review comment:
       Consider sharing some of this code with the RECOVERYSYNC action to avoid repetition.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@zookeeper.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] BinyuHuang-nju edited a comment on pull request #1690: ZOOKEEPER-3615: Provide formal specification and verification using TLA+ for Zab

Posted by GitBox <gi...@apache.org>.
BinyuHuang-nju edited a comment on pull request #1690:
URL: https://github.com/apache/zookeeper/pull/1690#issuecomment-938689405


   @Vanlightly @hanm Hi! I have pushed a new version of specification of zab-1.0 here. I have modified most of codes mentioned in your suggestions. And I merged all parameters that used to compress space into one variable 'Parameters'. I think we can add more parameters into it to optimize state space better.
    I hope you could have time to check it. And I will update this version later to optimize some parts of spec like compressing state space. Thank you!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@zookeeper.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] BinyuHuang-nju commented on a change in pull request #1690: ZOOKEEPER-3615: Provide formal specification and verification using TLA+ for Zab

Posted by GitBox <gi...@apache.org>.
BinyuHuang-nju commented on a change in pull request #1690:
URL: https://github.com/apache/zookeeper/pull/1690#discussion_r630744828



##########
File path: zookeeper-contrib/zookeeper-contrib-zabtla/experiment/ZabWithQTest.tla
##########
@@ -0,0 +1,994 @@
+(*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *)
+-------------------------------- MODULE ZabWithQTest --------------------------------
+\* This is the test for formal specification for the Zab consensus algorithm,

Review comment:
       Yes, we can. We need to ensure that Zab.tla and ZabWithQTest.tla are in the same directory in this situation. So here we can put these two files in the same directory or copy Zab.tla into experiment/. I think what you suggest will make the experimental framework look clearer. Thank you!




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] Vanlightly commented on pull request #1690: ZOOKEEPER-3615: Provide formal specification and verification using TLA+ for Zab

Posted by GitBox <gi...@apache.org>.
Vanlightly commented on pull request #1690:
URL: https://github.com/apache/zookeeper/pull/1690#issuecomment-974609840


   @BinyuHuang-nju I have checked it over and it's looking good. There's just one last point from my perspective.
   
   The Value constant is not really that useful as it stands right now. It is still possible for a client to write the same value more than once. More importantly the values are not checked in any invariants, so the values do not serve any purpose right now other than to increase the state space. 
   
   I would recommend adding a value equality check to `TransactionEqual` to ensure that not only the zxid matches but the value also. Then also prevent the same value from being written twice in `ClientRequestAndLeaderBroadcastProposal` and remove the check for transaction num.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@zookeeper.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] Vanlightly commented on a change in pull request #1690: ZOOKEEPER-3615: Provide formal specification and verification using TLA+ for Zab

Posted by GitBox <gi...@apache.org>.
Vanlightly commented on a change in pull request #1690:
URL: https://github.com/apache/zookeeper/pull/1690#discussion_r699459102



##########
File path: zookeeper-specifications/zab-1.0/FastLeaderElection.tla
##########
@@ -0,0 +1,518 @@
+(*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *)
+------------------------- MODULE FastLeaderElection -------------------------
+\* This is the formal specification for Fast Leader Election in Zab protocol.
+(* Reference:
+   FastLeaderElection.java, Vote.java, QuorumPeer.java in https://github.com/apache/zookeeper.
+   Medeiros A. ZooKeeper��s atomic broadcast protocol: Theory and practice[J]. Aalto University School of Science, 2012.
+*)
+EXTENDS Integers, FiniteSets, Sequences, Naturals, TLC
+
+-----------------------------------------------------------------------------
+\* The set of server identifiers
+CONSTANT Server
+
+\* Server states
+CONSTANTS LOOKING, FOLLOWING, LEADING
+
+\* Message types
+CONSTANTS NOTIFICATION
+
+\* Timeout signal
+CONSTANT NONE
+-----------------------------------------------------------------------------
+Quorums == {Q \in SUBSET Server: Cardinality(Q)*2 > Cardinality(Server)}
+
+NullPoint == CHOOSE p: p \notin Server
+
+-----------------------------------------------------------------------------
+\* Server's state(LOOKING, FOLLOWING, LEADING).
+VARIABLE state
+
+\* The epoch number of the last NEWLEADER packet accepted, used for comparing.
+VARIABLE currentEpoch
+
+\* The zxid of the last transaction in history.
+VARIABLE lastZxid
+
+\* currentVote[i]: The server who i thinks is the current leader(id,zxid,peerEpoch,...).
+VARIABLE currentVote
+
+\* Election instance.(logicalClock in code)
+VARIABLE logicalClock
+
+\* The votes from the current leader election are stored in ReceiveVotes.
+VARIABLE receiveVotes
+
+(* The votes from previous leader elections, as well as the votes from the current leader election are
+   stored in outofelection. Note that notifications in a LOOKING state are not stored in outofelection.
+   Only FOLLOWING or LEADING notifications are stored in outofelection.  *)
+VARIABLE outOfElection
+
+\* recvQueue[i]: The queue of received notifications or timeout signals in server i.
+VARIABLE recvQueue
+
+\* A veriable to wait for new notifications, corresponding to line 1050 in FastLeaderElection.java.
+VARIABLE waitNotmsg
+
+\* leadingVoteSet[i]: The set of voters that follow i.
+VARIABLE leadingVoteSet
+
+(* The messages about election sent from one server to another.
+   electionMsgs[i][j] means the input buffer of server j from server i. *)
+VARIABLE electionMsgs
+
+\* Set used for mapping Server to Integers, to compare ids from different servers.
+VARIABLE idTable
+
+serverVars == <<state, currentEpoch, lastZxid>>
+
+electionVars == <<currentVote, logicalClock, receiveVotes, outOfElection, recvQueue, waitNotmsg>>
+
+leaderVars == <<leadingVoteSet>>
+
+varsL == <<serverVars, electionVars, leaderVars, electionMsgs, idTable>>
+-----------------------------------------------------------------------------
+\* Processing of electionMsgs
+BroadcastNotmsg(i, m) == electionMsgs' = [electionMsgs EXCEPT ![i] = [v \in Server |-> IF v /= i
+                                                                                       THEN Append(electionMsgs[i][v], m)
+                                                                                       ELSE electionMsgs[i][v]]]
+
+DiscardNotmsg(i, j) == electionMsgs' = [electionMsgs EXCEPT ![i][j] = IF electionMsgs[i][j] /= << >>
+                                                                      THEN Tail(electionMsgs[i][j])
+                                                                      ELSE << >>]
+
+ReplyNotmsg(i, j, m) == electionMsgs' = [electionMsgs EXCEPT ![i][j] = Append(electionMsgs[i][j], m),
+                                                             ![j][i] = Tail(electionMsgs[j][i])]
+                                               
+-----------------------------------------------------------------------------
+\* Processing of recvQueue
+RECURSIVE RemoveNone(_)
+RemoveNone(seq) == CASE seq =  << >> -> << >>
+                   []   seq /= << >> -> IF Head(seq).mtype = NONE THEN RemoveNone(Tail(seq))
+                                                                  ELSE <<Head(seq)>> \o RemoveNone(Tail(seq)) 
+
+\* Processing of idTable and order comparing
+RECURSIVE InitializeIdTable(_)
+InitializeIdTable(Remaining) == IF Remaining = {} THEN {}
+                                ELSE LET chosen == CHOOSE i \in Remaining: TRUE
+                                         re     == Remaining \ {chosen}
+                                     IN {<<chosen, Cardinality(Remaining)>>} \union InitializeIdTable(re)
+                                     
+\* FALSE: id1 < id2; TRUE: id1 > id2
+IdCompare(id1,id2) == LET item1 == CHOOSE item \in idTable: item[1] = id1
+                          item2 == CHOOSE item \in idTable: item[1] = id2
+                      IN item1[2] > item2[2]
+
+\* FALSE: zxid1 <= zxid2; TRUE: zxid1 > zxid2
+ZxidCompare(zxid1, zxid2) == \/ zxid1[1] > zxid2[1]
+                             \/ /\ zxid1[1] = zxid2[1]
+                                /\ zxid1[2] > zxid2[2]
+
+ZxidEqual(zxid1, zxid2) == zxid1[1] = zxid2[1] /\ zxid1[2] = zxid2[2]
+
+\* FALSE: vote1 <= vote2; TRUE: vote1 > vote2
+TotalOrderPredicate(vote1, vote2) == \/ vote1.proposedEpoch > vote2.proposedEpoch
+                                     \/ /\ vote1.proposedEpoch = vote2.proposedEpoch
+                                        /\ \/ ZxidCompare(vote1.proposedZxid, vote2.proposedZxid)
+                                           \/ /\ ZxidEqual(vote1.proposedZxid, vote2.proposedZxid)
+                                              /\ IdCompare(vote1.proposedLeader, vote2.proposedLeader)
+
+VoteEqual(vote1, round1, vote2, round2) == /\ vote1.proposedLeader = vote2.proposedLeader
+                                           /\ ZxidEqual(vote1.proposedZxid, vote2.proposedZxid)
+                                           /\ vote1.proposedEpoch  = vote2.proposedEpoch
+                                           /\ round1 = round2
+
+-----------------------------------------------------------------------------
+\* Processing of currentVote
+InitialVote == [proposedLeader |-> NullPoint,
+                proposedZxid   |-> <<0, 0>>,
+                proposedEpoch  |-> 0]
+
+SelfVote(i) == [proposedLeader |-> i,
+                proposedZxid   |-> lastZxid[i],
+                proposedEpoch  |-> currentEpoch[i]]
+
+UpdateProposal(i, nid, nzxid, nepoch) == currentVote' = [currentVote EXCEPT ![i].proposedLeader = nid, \* no need to record state in LOOKING
+                                                                            ![i].proposedZxid   = nzxid,
+                                                                            ![i].proposedEpoch  = nepoch]  
+                                                                            
+-----------------------------------------------------------------------------
+\* Processing of receiveVotes and outOfElection
+RvClear(i) == receiveVotes'  = [receiveVotes  EXCEPT ![i] = [v \in Server |-> [vote    |-> InitialVote,
+                                                                               round   |-> 0,
+                                                                               state   |-> LOOKING,
+                                                                               version |-> 0]]]
+
+RvPut(i, id, mvote, mround, mstate) == receiveVotes' = CASE receiveVotes[i][id].round < mround -> [receiveVotes EXCEPT ![i][id].vote    = mvote,
+                                                                                                                       ![i][id].round   = mround,
+                                                                                                                       ![i][id].state   = mstate,
+                                                                                                                       ![i][id].version = 1]
+                                                       []   receiveVotes[i][id].round = mround -> [receiveVotes EXCEPT ![i][id].vote    = mvote,
+                                                                                                                       ![i][id].state   = mstate,
+                                                                                                                       ![i][id].version = @ + 1]
+                                                       []   receiveVotes[i][id].round > mround -> receiveVotes
+
+Put(i, id, rcvset, mvote, mround, mstate) == CASE rcvset[id].round < mround -> [rcvset EXCEPT ![id].vote    = mvote,
+                                                                                              ![id].round   = mround,
+                                                                                              ![id].state   = mstate,
+                                                                                              ![id].version = 1]
+                                             []   rcvset[id].round = mround -> [rcvset EXCEPT ![id].vote    = mvote,
+                                                                                              ![id].state   = mstate,
+                                                                                              ![id].version = @ + 1]
+                                             []   rcvset[id].round > mround -> rcvset
+
+RvClearAndPut(i, id, vote, round) == receiveVotes' = LET oneVote == [vote    |-> vote, 
+                                                                     round   |-> round, 
+                                                                     state   |-> LOOKING,
+                                                                     version |-> 1]
+                                                     IN [receiveVotes EXCEPT ![i] = [v \in Server |-> IF v = id THEN oneVote
+                                                                                                                ELSE [vote    |-> InitialVote,
+                                                                                                                      round   |-> 0,
+                                                                                                                      state   |-> LOOKING,
+                                                                                                                      version |-> 0]]]                     
+
+VoteSet(i, msource, rcvset, thisvote, thisround) == {msource} \union {s \in (Server \ {msource}): VoteEqual(rcvset[s].vote, 

Review comment:
       The argument i is not used in VoteSet body.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@zookeeper.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] hanm commented on a change in pull request #1690: ZOOKEEPER-3615: Provide formal specification and verification using TLA+ for Zab

Posted by GitBox <gi...@apache.org>.
hanm commented on a change in pull request #1690:
URL: https://github.com/apache/zookeeper/pull/1690#discussion_r634909509



##########
File path: zookeeper-contrib/zookeeper-contrib-zabtla/README.md
##########
@@ -0,0 +1,71 @@
+# Zab-tla
+
+## Overview
+This project is devoted to providing formal specification and verification using TLA+ for the Zookeeper Atomic Broadcast(Zab) consensus protocol proposed in *Junqueira F P, Reed B C, Serafini M. Zab: High-performance broadcast for primary-backup systems[C]//2011 IEEE/IFIP 41st International Conference on Dependable Systems & Networks (DSN). IEEE, 2011: 245-256*.  
+
+We have made a formal [specification](Zab.tla) for Zab using TLA+ toolbox, and we have done a certain scale of model checking to verify the correctness of Zab.
+
+Due to the simplification of Zab algorithm description in the paper, some details in specification were modified and added. If you have any question, please let us know.
+
+You can find this document in chinese in [doc-in-chinsese](doc-in-chinese/README.md).
+
+## Requirements
+TLA+ toolbox version 1.7.0
+
+## Run
+Create specification and run models in the usual way.  

Review comment:
       New folder structure looks awesome.
   
   Travis CI failed because some newly introduced files missing Apache license (same reason Travis failed the first time for this pull request):
   `[WARNING] Files with unapproved licenses:
     zookeeper-specifications/zab/Zab.toolbox/Model_1/MC.cfg
     zookeeper-specifications/zab/Zab.toolbox/Zab.aux
     zookeeper-specifications/zab/Zab.toolbox/Zab.tex
     zookeeper-specifications/zab/experiment/ZabWithQTest2.toolbox/ZabWithQTest2.tex
     zookeeper-specifications/zab/experiment/ZabWithQTest2.toolbox/ZabWithQTest2.aux
     zookeeper-specifications/zab/experiment/ZabWithQTest2.toolbox/Model_1/MC.cfg
     zookeeper-specifications/zab/experiment/ZabWithQTest.toolbox/ZabWithQTest.tex
     zookeeper-specifications/zab/experiment/ZabWithQTest.toolbox/ZabWithQTest.aux
     zookeeper-specifications/zab/experiment/ZabWithQTest.toolbox/Model_1/MC.cfg
   
   [ERROR] Failed to execute goal org.apache.rat:apache-rat-plugin:0.13:check (default-cli) on project parent: Too many files with unapproved license: 9 See RAT report in: /home/travis/build/apache/zookeeper/target/rat.txt -> [Help 1]
   `
   
   Add license header to these files should make the build pass.
   
   To debug Travis issue - simply click the "Details" link and then go to the job page (in this case it's https://travis-ci.com/github/apache/zookeeper/jobs/506036712), and check the log for specific errors.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] BinyuHuang-nju commented on pull request #1690: ZOOKEEPER-3615: Provide formal specification and verification using TLA+ for Zab

Posted by GitBox <gi...@apache.org>.
BinyuHuang-nju commented on pull request #1690:
URL: https://github.com/apache/zookeeper/pull/1690#issuecomment-832510959


   @Vanlightly Sorry, I didn't make it clear that I didn't complete running. Actually, It is almost impossible to traverse all states due to state space explosion, and this is a disadvantage of "model checking". So it is very normal when you see the model cannot stop.
   
   We can assume situations when a server halts and restarts for multiple times, or re-elect a leader  for multiple times but not increase epoch because the last round of elections ended before the prospective leader broadcasts NEWEPOCH. So in principle, there are infinite large state spaces when running models.
   On the other hand, there are a lot of repetitive actions like restart, re-election happen continuously, which make state space expand rapidly.
   
   And the experience imparted to me by the senior’s work like Parallel Raft, TPaxos shows that it is quite normal that models cannot be run completely. So in actual, I run it for a long time to judge whether there's invariant violation or not. 
   
   Also, I have run models with excluding the restart situation(modify "currentEpoch[i] < 0" in Restart and RecoveryAfterRestart), and the result will produce a lot of actions we care more about. Until now, the results appears good. I think you could have a try if you have enough time.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] BinyuHuang-nju commented on pull request #1690: ZOOKEEPER-3615: Provide formal specification and verification using TLA+ for Zab

Posted by GitBox <gi...@apache.org>.
BinyuHuang-nju commented on pull request #1690:
URL: https://github.com/apache/zookeeper/pull/1690#issuecomment-853899721


   @hanm OK, thank you very much! You replies above will help me a lot.
    Currently I am devoted to writing spec for ZAB 1.0, and abstract two modules of election and sync phase before sending NEWLEADER. 
   I think I can implement election module if necessary. And I need to learn more about sync phase if we need to implement this part.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] hanm commented on pull request #1690: ZOOKEEPER-3615: Provide formal specification and verification using TLA+ for Zab

Posted by GitBox <gi...@apache.org>.
hanm commented on pull request #1690:
URL: https://github.com/apache/zookeeper/pull/1690#issuecomment-859265319


   > @hanm There's one point I don't get.
   > 
   > > When follower receives NEWLEADER, it will apply all previous queued proposals from leader atomically to its state machine.
   > 
   > It seems like that when some learnerMaster sends NEWLEADER to the corresponding follower, the leader may have not completed sync with a quorum of followers. So some transaction saved by this follower may have not been saved by a quorum, which is not committed and can not be applied from my perspective. Here I don't understand why these queued proposals can be applied. Thank you!
   
   You are correct that NEWLEADER does not imply the transactions synced from leader on a follower is quorum committed. That's fine because the follower is not serving traffic (or visible to clients) so sequential consistency is not violated. Later, when leader gets a quorum of ACK of the NEWLEADER from followers, leader will send UPTODATE to the followers and after this the quorum and leadership is established. Followers can serve traffic after receiving UPTODATE.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] BinyuHuang-nju edited a comment on pull request #1690: ZOOKEEPER-3615: Provide formal specification and verification using TLA+ for Zab

Posted by GitBox <gi...@apache.org>.
BinyuHuang-nju edited a comment on pull request #1690:
URL: https://github.com/apache/zookeeper/pull/1690#issuecomment-855615675


   @hanm There's one point I don't get.
   > When follower receives NEWLEADER, it will apply all previous queued proposals from leader atomically to its state machine. 
   
   It seems like that when some learnerMaster sends NEWLEADER to the corresponding follower, the leader may have not completed sync with a quorum of followers. So some transaction saved by this follower may have not been saved by a quorum, which is not committed and can not be applied from my perspective. Here I don't understand why these queued proposals can be applied. Thank you!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] hanm commented on a change in pull request #1690: ZOOKEEPER-3615: Provide formal specification and verification using TLA+ for Zab

Posted by GitBox <gi...@apache.org>.
hanm commented on a change in pull request #1690:
URL: https://github.com/apache/zookeeper/pull/1690#discussion_r631471863



##########
File path: zookeeper-contrib/zookeeper-contrib-zabtla/experiment/ZabWithQTest.tla
##########
@@ -0,0 +1,994 @@
+(*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *)
+-------------------------------- MODULE ZabWithQTest --------------------------------
+\* This is the test for formal specification for the Zab consensus algorithm,

Review comment:
       Thinking more about the folder structure of the  specification files, I am tempted to make a proposal that instead of putting them under zookeeper-contrib, we create a folder under root level called zookeeper-specifications, and put ZAB specifications there. I feel these specifications have more affinities to the zookeeper core project comparing to zookeeper-contrib. 
   
   The new root folder structures would look like:
   ├── bin
   ├── conf
   ├── dev
   ├── target
   ├── tools
   ├── zookeeper-assembly
   ├── zookeeper-client
   ├── zookeeper-compatibility-tests
   ├── zookeeper-contrib
   ├── zookeeper-docs
   ├── zookeeper-it
   ├── zookeeper-jute
   ├── zookeeper-metrics-providers
   ├── zookeeper-recipes
   ├── zookeeper-server
   └── zookeeper-specifications
   
   And the spec folder looks like
   └── zookeeper-specifications
         ├── leader-election
         ├── reconfig
        └── zab
   
   Here we put the zab spec, toolbox configs, tests, etc in zab folder.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] BinyuHuang-nju edited a comment on pull request #1690: ZOOKEEPER-3615: Provide formal specification and verification using TLA+ for Zab

Posted by GitBox <gi...@apache.org>.
BinyuHuang-nju edited a comment on pull request #1690:
URL: https://github.com/apache/zookeeper/pull/1690#issuecomment-832510959


   @Vanlightly Sorry, I didn't make it clear that I didn't do complete running. Actually, It is almost impossible to traverse all states due to state space explosion, and this is a disadvantage of "model checking". So it is very normal when you see the model cannot stop.
   
   We can assume situations when a server halts and restarts for multiple times, or re-elect a leader  for multiple times but not increase epoch because the last round of elections ended before the prospective leader broadcasts NEWEPOCH. So in principle, there are infinite large state spaces when running models.
   On the other hand, there are a lot of repetitive actions like restart, re-election happen continuously, which make state space expand rapidly.
   
   So in actual, I run it for a long time to judge whether there's invariant violation or not. 
   
   Also, I have run models with excluding the restart situation(modify "currentEpoch[i] < 0" in Restart and RecoveryAfterRestart), and the result will produce a lot of actions we care more about. Until now, the results appears good.
   
   If you have any needs on model's scale, we will actively cooperate within our capacity. Thank you for your investment in verification!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] BinyuHuang-nju commented on pull request #1690: ZOOKEEPER-3615: Provide formal specification and verification using TLA+ for Zab

Posted by GitBox <gi...@apache.org>.
BinyuHuang-nju commented on pull request #1690:
URL: https://github.com/apache/zookeeper/pull/1690#issuecomment-938689405


   @Vanlightly @hanm Hi! I have pushed a new version of specification of zab-1.0 here. I hope you could have time to check it. And I will update this version later to optimize some parts of spec like compressing state space. Thank you!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@zookeeper.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org