You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "huxihx (JIRA)" <ji...@apache.org> on 2017/12/16 02:28:02 UTC
[jira] [Commented] (KAFKA-6375) Follower replicas can never catch
up to be ISR due to creating ReplicaFetcherThread failed.
[ https://issues.apache.org/jira/browse/KAFKA-6375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16293576#comment-16293576 ]
huxihx commented on KAFKA-6375:
-------------------------------
“ Unable to establish loopback connection” Did you disable the firewall?
> Follower replicas can never catch up to be ISR due to creating ReplicaFetcherThread failed.
> -------------------------------------------------------------------------------------------
>
> Key: KAFKA-6375
> URL: https://issues.apache.org/jira/browse/KAFKA-6375
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.10.1.0
> Environment: Windows, 23 brokers KafkaCluster
> Reporter: Rong Tang
>
> Hi, I met with a case that in one broker, the out of sync replicas never catch up.
> When the broker starts up, it receives LeaderAndISR requests from controller, which will call createFetcherThread, the thread creation failed, with exceptions below.
> And then, there is no fetcher for these follower replicas, and it is out of sync forever. Unless, later, it receives LeaderAndISR requests that has higher leader EPOCH. The broker had 260 out of 330 replicas out of sync for one day, until I restarted it.
> Restart the broker can mitigate the issue.
> I have 2 questions.
> First, Why NEW ReplicaFetcherThread failed?
> *Second, should Kafka do something to fail over, instead of letting the broker in abnormal state.*
> It is a 23 brokers Kafka cluster running on Windows. each broker has 330 replicas.
> [2017-12-13 16:29:21,317] ERROR Error on broker 1000 while processing LeaderAndIsr request with correlationId 1 received from controller 427703487 epoch 22 (state.change.logger)
> org.apache.kafka.common.KafkaException: java.io.IOException: Unable to establish loopback connection
> at org.apache.kafka.common.network.Selector.<init>(Selector.java:124)
> at kafka.server.ReplicaFetcherThread.<init>(ReplicaFetcherThread.scala:87)
> at kafka.server.ReplicaFetcherManager.createFetcherThread(ReplicaFetcherManager.scala:35)
> at kafka.server.AbstractFetcherManager$$anonfun$addFetcherForPartitions$2.apply(AbstractFetcherManager.scala:83)
> at kafka.server.AbstractFetcherManager$$anonfun$addFetcherForPartitions$2.apply(AbstractFetcherManager.scala:78)
> at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
> at scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:221)
> at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:428)
> at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:428)
> at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
> at kafka.server.AbstractFetcherManager.addFetcherForPartitions(AbstractFetcherManager.scala:78)
> at kafka.server.ReplicaManager.makeFollowers(ReplicaManager.scala:869)
> at kafka.server.ReplicaManager.becomeLeaderOrFollower(ReplicaManager.scala:689)
> at kafka.server.KafkaApis.handleLeaderAndIsrRequest(KafkaApis.scala:149)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:83)
> at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: Unable to establish loopback connection
> at sun.nio.ch.PipeImpl$Initializer.run(PipeImpl.java:94)
> at sun.nio.ch.PipeImpl$Initializer.run(PipeImpl.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.nio.ch.PipeImpl.<init>(PipeImpl.java:171)
> at sun.nio.ch.SelectorProviderImpl.openPipe(SelectorProviderImpl.java:50)
> at java.nio.channels.Pipe.open(Pipe.java:155)
> at sun.nio.ch.WindowsSelectorImpl.<init>(WindowsSelectorImpl.java:127)
> at sun.nio.ch.WindowsSelectorProvider.openSelector(WindowsSelectorProvider.java:44)
> at java.nio.channels.Selector.open(Selector.java:227)
> at org.apache.kafka.common.network.Selector.<init>(Selector.java:122)
> ... 16 more
> Caused by: java.net.ConnectException: Connection timed out: connect
> at sun.nio.ch.Net.connect0(Native Method)
> at sun.nio.ch.Net.connect(Net.java:454)
> at sun.nio.ch.Net.connect(Net.java:446)
> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648)
> at java.nio.channels.SocketChannel.open(SocketChannel.java:189)
> at sun.nio.ch.PipeImpl$Initializer$LoopbackConnector.run(PipeImpl.java:127)
> at sun.nio.ch.PipeImpl$Initializer.run(PipeImpl.java:76)
> ... 25 more
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)