[jira] [Updated] (KAFKA-532) Multiple controllers can co-exist during soft failures

["Neha Narkhede (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Neha Narkhede updated KAFKA-532:
--------------------------------

    Attachment: kafka-532-v2.patch

Thanks for the review, Jun ! 

1, 3 That's a bug, fixed it
2. Changed it to be a case class instead of tuple
4. While adding comments, realized that there was a bug in the way we computed the size of the leader and isr request. The size had an extra 1 byte in the be
ginning, not sure if its required or not. This is probably a bug introduce in the very first version of the controller that we didn't catch during testing.

6. I'm afraid that will not solve the problem. The whole point of the controller generation is to prevent the brokers from following requests sent by a stale controller. It doesn't matter whether the controller is re-publishing the old controller's decision or making its own, once it sends the decision to the brokers, it is effectively certifying that decision to be the right one. Hence, both the leader and isr request as well as the stop replica request needs to contain the epoch of the controller sending the request.With the above semantics, the new controller should re-write the leader and isr path with its epoch after sending the leader and isr request to the brokers. However, re-writing the path during the controller failover will have performance implications on the controller failover latency. An alternative is to do this in the leader and isr response callback. Currently, we rely on asynchronous leader election to work correctly. Ideally, we need to be able to act on the event that the leader and isr response is either negative or lost. When this happens, leader election needs to be triggered again. Since this is asynchronous, we can also update the leader and isr path with the new controller's epoch on receiving a successful leader and isr response. If this sounds good, I can either make the changes in patch v3 or file another JIRA. Let me know what you prefer. Until then, the broker will re-write the zk path with the latest controller epoch, which is theoretically correct, but not semantically.    

5, 7. With the semantics mentioned above, the brokers should just write the isr with the controller epoch that it knows.

                
> Multiple controllers can co-exist during soft failures
> ------------------------------------------------------
>
>                 Key: KAFKA-532
>                 URL: https://issues.apache.org/jira/browse/KAFKA-532
>             Project: Kafka
>          Issue Type: Bug
>    Affects Versions: 0.8
>            Reporter: Neha Narkhede
>            Assignee: Neha Narkhede
>            Priority: Blocker
>              Labels: bugs
>         Attachments: kafka-532-v1.patch, kafka-532-v2.patch
>
>   Original Estimate: 48h
>  Remaining Estimate: 48h
>
> If the current controller experiences an intermittent soft failure (GC pause) in the middle of leader election or partition reassignment, a new controller might get elected and start communicating new state change decisions to the brokers. After recovering from the soft failure, the old controller might continue sending some stale state change decisions to the brokers, resulting in unexpected failures. We need to introduce a controller generation id that increments with controller election. The brokers should reject any state change requests by a controller with an older generation id.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira