[jira] [Updated] (KAFKA-911) Bug in controlled shutdown logic in controller leads to controller not sending out some state change request

["Neha Narkhede (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Neha Narkhede updated KAFKA-911:
--------------------------------

    Attachment: kafka-911-v1.patch

The root cause of this bug was a race condition while using the ControllerBrokerRequestBatch that assumes synchronization at the caller. This patch synchronizes access to the ControllerBrokerRequestBatch while sending out the StopReplicaRequest. 

While working on this and testing the patch, I noticed some inefficiency with the controller shutdown API. When we move the leader for a partition one at a time, we also remove the broker being shutdown from the ISR. This involves at least one read and one write per partition to zookeeper, sometimes more. Besides being slow, this operation is not effective. This is because the broker being shutdown still has alive fetchers to the new leader before it receives and acts on the StopReplicaRequest. So the new leader adds it back to the ISR anyways. 

Then I thought I could move it to the end of the controlled shutdown API where we check if all partitions are successfully moved, then send StopReplicaRequest to the broker being shut down. Right after this, we could move the replica to Offline and remove it from ISR as part of that operation. Even if we can invoke this operation as a batch, it will still be slow since the zookeeper reads/writes will happen serially. Also, I realized this is not required as well for 2 reasons -

1. Since the shutting broker is sent the StopReplicaRequest, it will stop the fetcher and fall out of ISR. Until then, as long as the controller doesn't failover, it will not be elected as the leader, even if it is in the ISR, since it is one of the shuttingDownBrokers.

2. Even if the controller fails over, by the time the new controller has initialized and is ready to serve, the StopReplicaRequest would've ensured that the shutdown broker is no longer in the ISR. And until the controller fails over, there cannot be any leader election anyway.

I've tested this patch on a 7 node cluster that continuously gets rolling bounced and has ~100 producers sending production data to it with ~1000 consumers consuming data from it. 
                
> Bug in controlled shutdown logic in controller leads to controller not sending out some state change request 
> -------------------------------------------------------------------------------------------------------------
>
>                 Key: KAFKA-911
>                 URL: https://issues.apache.org/jira/browse/KAFKA-911
>             Project: Kafka
>          Issue Type: Bug
>          Components: controller
>    Affects Versions: 0.8
>            Reporter: Neha Narkhede
>            Assignee: Neha Narkhede
>            Priority: Blocker
>              Labels: kafka-0.8, p1
>         Attachments: kafka-911-v1.patch
>
>
> The controlled shutdown logic in the controller first tries to move the leaders from the broker being shutdown. Then it tries to remove the broker from the isr list. During that operation, it does not synchronize on the controllerLock. This causes a race condition while dispatching data using the controller's channel manager.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira