[jira] [Created] (ZOOKEEPER-4643) Committed txns may be improperly truncated when node crashes right after updating currentEpoch

["Sirius (Jira)" <ji...@apache.org>]

Sirius created ZOOKEEPER-4643:
---------------------------------

             Summary: Committed txns may be improperly truncated when node crashes right after updating currentEpoch 
                 Key: ZOOKEEPER-4643
                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-4643
             Project: ZooKeeper
          Issue Type: Bug
          Components: quorum, server
    Affects Versions: 3.7.1, 3.8.0
            Reporter: Sirius


When a follower is processing the NEWLEADER message in SYNC phase, it will update its currentEpoch to the currentEpoch file *before* writing the uncommitted txns to the log file. Such order may lead to improper truncation of *committed* txns in later rounds. 

 

The critical step is to make a follower node crash right after it updates its currentEpoch to the file but before writing the uncommitted txns to the log file. 

 
h2. Trace

Here is an example to trigger the bug. (Focus on the currentEpoch and the lastLoggedZxid)

*Initial condition:*

- Start the ensemble with three nodes: A, B and C. 

- Node C is elected leader. 

- For all of them, acceptedEpoch=1, currentEpoch=1. 

- Besides, all of them have lastLoggedZxid = <1, 3>, lastProcessedZxid = <1, 3>. 

*Step 1:* Node A crashes.

*Step 2:* A new txn is logged and committed by Node B & C. Then, B & C have lastLoggedZxid = <1, 4>, lastProcessedZxid = <1, 4> ( Clients can read the datatree with latest zxid <1, 4>).

*Step 3:* Node A restarts, Node C restarts and Node B crashes. Again, C is elected leader. During the DISCOVERY phase, both A and C update their acceptedEpoch to 2. Then, during the SYNC phase, the leader C (maxCommittedLog = <1, 4>) uses DIFF to sync with the follower A (lastLoggedZxid = <1, 3>). and their currentEpoch will be set to 2 (and written to disk). Note that the follower A updates its currentEpoch file before writing the uncommitted txns to the log file when receiving NEWLEADER message. *Unfortunately, right after the follower A finishes updating its currentEpoch file, it crashes.*

*Step 4:* Node A and B restarts and Node C crashes. Since Node A has currentEpoch=2, Node C has currentEpoch=1, Node A will be elected leader. During the SYNC phase, the leader A (maxCommittedLog = <1, 3>) will use TRUNC to sync with B (lastLoggedZxid = <1, 4>). Then, B removes txn <1, 4>. 

 

However, <1, 4> was committed and visible by clients before, and is not supposed to be truncated!

 

(Note: With careful time tuning of crash & restart, the trace can be constructed with quorum nodes alive at any moment.)

 

The above trace has been triggered by our testing tools. I will provide more materials to further demonstrate above case soon. Besides, I think the affected versions might be more. 

 
h2. Analysis

The critical step here is to interrupt two updates that should be done together. In the Zab paper, a follower updates its current epoch and history during the SYNC phase in an *atomic* action, and the correctness of Zab is based on that. However, in actual environment, a node crash (or other environment failures) may occur at any time, breaking the procedures that will not be interrupted at the protocol level. This is the gap between the theory and the reality. As the system evolves, the implementation of the SYNC phase becomes much more complicated compared to the original Zab protocol. We think it important to keep ZooKeeper still in a correct condition even under such type of environment circumstances.

 
h2. Possible Fix

Intuitively, this issue can be avoided by exchanging the order of writing uncommitted txns to the log file and writing currentEpoch to the currentEpoch file when the follower is processing NEWLEADER message. (See the code of Learner.java)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)