[jira] [Commented] (KAFKA-7164) Follower should truncate after every leader epoch change

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-7164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16563169#comment-16563169 ] 

ASF GitHub Bot commented on KAFKA-7164:
---------------------------------------

bob-barrett opened a new pull request #5436: KAFKA-7164: Follower should truncate after every leader epoch change
URL: https://github.com/apache/kafka/pull/5436
 
 
   Currently, we skip the steps to make a replica a follower if the leader does not change, inlcuding truncating the follower log if necessary. This can cause problems if the follower has missed one or more leader updates. Change the logic to only skip the steps if the new epoch is the same or one greater than the old epoch. Tested with unit tests that verify the behavior of Partition.scala and that show log truncation when the follower's log is ahead of the leader's, the follower has missed an epoch update, and the follower receives a LeaderAndIsrRequest making it a follower
   
   ### Committer Checklist (excluded from commit message)
   - [ ] Verify design and implementation 
   - [ ] Verify test coverage and CI build status
   - [ ] Verify documentation (including upgrade notes)
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> Follower should truncate after every leader epoch change
> --------------------------------------------------------
>
>                 Key: KAFKA-7164
>                 URL: https://issues.apache.org/jira/browse/KAFKA-7164
>             Project: Kafka
>          Issue Type: Bug
>            Reporter: Jason Gustafson
>            Assignee: Bob Barrett
>            Priority: Major
>
> Currently we skip log truncation for followers if a LeaderAndIsr request is received, but the leader does not change. This can lead to log divergence if the follower missed a leader change before the current known leader was reelected. Basically the problem is that the leader may truncate its own log prior to becoming leader again, so the follower would need to reconcile its log again.
> For example, suppose that we have three replicas: r1, r2, and r3. Initially, r1 is the leader in epoch 0 and writes one record at offset 0. r3 replicates this successfully.
> {code}
> r1: 
>   status: leader
>   epoch: 0
>   log: [{id: 0, offset: 0, epoch:0}]
> r2: 
>   status: follower
>   epoch: 0
>   log: []
> r3: 
>   status: follower
>   epoch: 0
>   log: [{id: 0, offset: 0, epoch:0}]
> {code}
> Suppose then that r2 becomes leader in epoch 1. r1 notices the leader change and truncates, but r3 for whatever reason, does not.
> {code}
> r1: 
>   status: follower
>   epoch: 1
>   log: []
> r2: 
>   status: leader
>   epoch: 1
>   log: []
> r3: 
>   status: follower
>   epoch: 0
>   log: [{offset: 0, epoch:0}]
> {code}
> Now suppose that r2 fails and r1 becomes the leader in epoch 2. Immediately it writes a new record:
> {code}
> r1: 
>   status: leader
>   epoch: 2
>   log: [{id: 1, offset: 0, epoch:2}]
> r2: 
>   status: follower
>   epoch: 2
>   log: []
> r3: 
>   status: follower
>   epoch: 0
>   log: [{id: 0, offset: 0, epoch:0}]
> {code}
> If the replica continues fetching with the old epoch, we can have log divergence as noted in KAFKA-6880. However, if r3 successfully receives the new LeaderAndIsr request which updates the epoch to 2, but skips the truncation, then the logs will stay inconsistent.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)