[jira] [Created] (KAFKA-6375) Follower replicas can never catch up to be ISR due to creating ReplicaFetcherThread failed.

["Rong Tang (JIRA)" <ji...@apache.org>]

Rong Tang created KAFKA-6375:
--------------------------------

             Summary: Follower replicas can never catch up to be ISR due to creating ReplicaFetcherThread failed.
                 Key: KAFKA-6375
                 URL: https://issues.apache.org/jira/browse/KAFKA-6375
             Project: Kafka
          Issue Type: Bug
          Components: core
    Affects Versions: 0.10.2.0
         Environment: Windows,  23 brokers KafkaCluster
            Reporter: Rong Tang


Hi, I met with a case that in one broker, the out of sync replicas never catch up.
When the broker starts up, it receives LeaderAndISR requests from controller, which will call createFetcherThread, the thread creation failed, with exceptions below.

And then, there is no fetcher for these follower replicas, and it is out of sync forever. Unless, later, it receives LeaderAndISR requests that has higher leader EPOCH. 

Restart the broker can mitigate the issue.

I have 2 questions.  
First, Why NEW ReplicaFetcherThread failed?
*Second, shouldn't Kafka do something to fail over, instead of letting the broker in abnormal state.*

It is a 23 brokers Kafka cluster running on Windows. each broker has 330 replicas.

[2017-12-13 16:29:21,317] ERROR Error on broker 1000 while processing LeaderAndIsr request with correlationId 1 received from controller 427703487 epoch 22 (state.change.logger)
org.apache.kafka.common.KafkaException: java.io.IOException: *Unable to establish loopback connection
	at org.apache.kafka.common.network.Selector.<init>(Selector.java:124)
	at kafka.server.ReplicaFetcherThread.<init>(ReplicaFetcherThread.scala:87)
	at *kafka.server.ReplicaFetcherManager.createFetcherThread(ReplicaFetcherManager.scala:35)
	at kafka.server.AbstractFetcherManager$$anonfun$addFetcherForPartitions$2.apply(AbstractFetcherManager.scala:83)
	at kafka.server.AbstractFetcherManager$$anonfun$addFetcherForPartitions$2.apply(AbstractFetcherManager.scala:78)
	at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
	at scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:221)
	at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:428)
	at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:428)
	at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
	at kafka.server.AbstractFetcherManager.addFetcherForPartitions(AbstractFetcherManager.scala:78)
	at kafka.server.ReplicaManager.makeFollowers(ReplicaManager.scala:869)
	at kafka.server.ReplicaManager.becomeLeaderOrFollower(ReplicaManager.scala:689)
	at kafka.server.KafkaApis.handleLeaderAndIsrRequest(KafkaApis.scala:149)
	at kafka.server.KafkaApis.handle(KafkaApis.scala:83)
	at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Unable to establish loopback connection
	at sun.nio.ch.PipeImpl$Initializer.run(PipeImpl.java:94)
	at sun.nio.ch.PipeImpl$Initializer.run(PipeImpl.java:61)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.nio.ch.PipeImpl.<init>(PipeImpl.java:171)
	at sun.nio.ch.SelectorProviderImpl.openPipe(SelectorProviderImpl.java:50)
	at java.nio.channels.Pipe.open(Pipe.java:155)
	at sun.nio.ch.WindowsSelectorImpl.<init>(WindowsSelectorImpl.java:127)
	at sun.nio.ch.WindowsSelectorProvider.openSelector(WindowsSelectorProvider.java:44)
	at java.nio.channels.Selector.open(Selector.java:227)
	at org.apache.kafka.common.network.Selector.<init>(Selector.java:122)
	... 16 more
Caused by: java.net.ConnectException: Connection timed out: connect
	at sun.nio.ch.Net.connect0(Native Method)
	at sun.nio.ch.Net.connect(Net.java:454)
	at sun.nio.ch.Net.connect(Net.java:446)
	at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648)
	at java.nio.channels.SocketChannel.open(SocketChannel.java:189)
	at sun.nio.ch.PipeImpl$Initializer$LoopbackConnector.run(PipeImpl.java:127)
	at sun.nio.ch.PipeImpl$Initializer.run(PipeImpl.java:76)
	... 25 more




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)