[jira] [Created] (KAFKA-6051) ReplicaFetcherThread should close the ReplicaFetcherBlockingSend earlier on shutdown

["Maytee Chinavanichkit (JIRA)" <ji...@apache.org>]

Maytee Chinavanichkit created KAFKA-6051:
--------------------------------------------

             Summary: ReplicaFetcherThread should close the ReplicaFetcherBlockingSend earlier on shutdown
                 Key: KAFKA-6051
                 URL: https://issues.apache.org/jira/browse/KAFKA-6051
             Project: Kafka
          Issue Type: Bug
            Reporter: Maytee Chinavanichkit


The ReplicaFetcherBlockingSend works as designed and will blocks until it is able to get data. This becomes a problem when we are gracefully shutting down a broker. The controller will attempt to shutdown the fetchers and elect new leaders. When the last fetch of partition is removed, as part of the {replicaManager.becomeLeaderOrFollower} call will proceed to shut down any idle ReplicaFetcherThread. The shutdown process here can block up to until the last fetch request completes. This blocking delay is a big problem because the {replicaStateChangeLock}, and {mapLock} in {AbstractFetcherManager} is still locked causing latency spikes on multiple brokers.

At this point in time, we do not need the last response as the fetcher is shutting down. We should close the leaderEndpoint early during {initiateShutdown()} instead of after {super.shutdown()}.


For example we see here the shutdown blocked the broker from processing more replica changes for ~500 ms 

{code}
[2017-09-01 18:11:42,879] INFO [ReplicaFetcherThread-0-2], Shutting down (kafka.server.ReplicaFetcherThread) 
[2017-09-01 18:11:43,314] INFO [ReplicaFetcherThread-0-2], Stopped (kafka.server.ReplicaFetcherThread) 
[2017-09-01 18:11:43,314] INFO [ReplicaFetcherThread-0-2], Shutdown completed (kafka.server.ReplicaFetcherThread)
{code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)