[jira] [Commented] (KAFKA-1097) Race condition while reassigning low throughput partition leads to incorrect ISR information in zookeeper

["Neha Narkhede (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-1097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13802108#comment-13802108 ] 

Neha Narkhede commented on KAFKA-1097:
--------------------------------------

This is a timing issue between when the controller executes related state changes and when those state changes actually get executed on the broker. In this case, the right time to remove the replica from ISR is actually on receiving the StopReplicaResponse from the broker. This ensures that the broker will not send any more fetch requests to the leader and add itself back to the ISR. This is the right fix, but is a non-trivial change to the controller and is dependent on KAFKA-1099. 

A short-term fix which will alleviate the issue, but not completely fix it is to send the stop replica request to the broker on the OfflineReplica state change, then shrink the ISR on the controller. In addition, send the stop replica (with delete) request on the NonExistentReplica state change and shrink the ISR again. There is a small chance that the broker hasn't acted on the 2 stop replica requests and still adds itself back to the ISR, after the controller shrinks the ISR during the NonExistentReplica state change. But this is unlikely. 

So the options for this bug fix are -

1. Take the short term fix for 0.8 and leave the larger change for trunk
2. Don't even take the short term fix and just fix it properly on trunk

> Race condition while reassigning low throughput partition leads to incorrect ISR information in zookeeper 
> ----------------------------------------------------------------------------------------------------------
>
>                 Key: KAFKA-1097
>                 URL: https://issues.apache.org/jira/browse/KAFKA-1097
>             Project: Kafka
>          Issue Type: Bug
>          Components: controller
>    Affects Versions: 0.8
>            Reporter: Neha Narkhede
>            Assignee: Neha Narkhede
>            Priority: Critical
>             Fix For: 0.8
>
>
> While moving partitions, the controller moves the old replicas through the following state changes -
> ONLINE -> OFFLINE -> NON_EXISTENT
> During the offline state change, the controller removes the old replica and writes the updated ISR to zookeeper and notifies the leader. Note that it doesn't notify the old replicas to stop fetching from the leader (to be fixed in KAFKA-1032). During the non-existent state change, the controller does not write the updated ISR or replica list to zookeeper. Right after the non-existent state change, the controller writes the new replica list to zookeeper, but does not update the ISR. So an old replica can send a fetch request after the offline state change, essentially letting the leader add it back to the ISR. The problem is that if there is no new data coming in for the partition and the old replica is fully caught up, the leader cannot remove it from the ISR. That lets a non existent replica live in the ISR at least until new data comes in to the partition



--
This message was sent by Atlassian JIRA
(v6.1#6144)