You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Suri Nuthalapati (Jira)" <ji...@apache.org> on 2019/08/23 14:57:00 UTC
[jira] [Comment Edited] (KAFKA-4674) Frequent ISR shrinking and
expanding and disconnects among brokers
[ https://issues.apache.org/jira/browse/KAFKA-4674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16914314#comment-16914314 ]
Suri Nuthalapati edited comment on KAFKA-4674 at 8/23/19 2:56 PM:
------------------------------------------------------------------
*Kafka v0.10.2*
Is there any ETA for the resolution of this issue?
We have been seeing this problem in very high volume which is causing the producer to fail on messages when they have acks=all and also some messages being lost.
Our Environment has 13 Brokers (8G heap), 3 zookeepers (2G heap), We have tuned a lot of parameters so far and we are still seeing this issue. This is creating untrust in our tenants (we have so many teams) about Kafka Performance and Reliability.
{code:java}
// Frequent ISR shrinking and expanding for Topics -- server.log
[2019-08-23 11:40:40,200] INFO Partition [topic-name,5] on broker 1012: Shrinking ISR for partition [topic-name,5] from 1012,1011,1001 to 1012 (kafka.cluster.Partition)
[2019-08-23 11:40:41,641] INFO Partition [topic-name,101] on broker 1012: Expanding ISR for partition topic-name-101 from 1012 to 1012,1007 (kafka.cluster.Partition)
{code}
{code:java}
// disconnects among brokers -- controller.log
[2019-07-09 18:53:38,918] WARN [Controller-1012-to-broker-1012-send-thread], Controller 1012's connection to broker broker.fqdn.com:6668 (
id: 1012 rack: null) was unsuccessful (kafka.controller.RequestSendThread)
java.io.IOException: Connection to broker.fqdn.com:6668 (id: 1012 rack: null) failed
at kafka.utils.NetworkClientBlockingOps$.awaitReady$1(NetworkClientBlockingOps.scala:84)
at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:94)
at kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:236)
at kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:189)
at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:188)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
{code}
Parameters we Tried:
{code:java}
//Broker Settings:
num.network.threads=10
num.replica.fetchers=10
replica.socket.timeout.ms=60000
controller.socket.timeout.ms to 60000
zookeeper.connection.timeout.ms to 60000
zookeeper.session.timeout.ms to 60000
replica.fetch.wait.max.ms to 750 {code}
{code:java}
// System Settings:
net.core.somaxconn=8192
net.ipv4.tcp_max_syn_backlog=4096
{code}
Any pointers or suggestions will be very helpful.
Thanks,
Suri
was (Author: surinuthalapati@live.com):
Is there any ETA for the resolution of this issue?
We have been seeing this problem in very high volume which is causing the producer to fail on messages when they have acks=all and also some messages being lost.
Our Environment has 13 Brokers (8G heap), 3 zookeepers (2G heap), We have tuned a lot of parameters so far and we are still seeing this issue. This is creating untrust in our tenants (we have so many teams) about Kafka Performance and Reliability.
{code:java}
// Frequent ISR shrinking and expanding for Topics -- server.log
[2019-08-23 11:40:40,200] INFO Partition [topic-name,5] on broker 1012: Shrinking ISR for partition [topic-name,5] from 1012,1011,1001 to 1012 (kafka.cluster.Partition)
[2019-08-23 11:40:41,641] INFO Partition [topic-name,101] on broker 1012: Expanding ISR for partition topic-name-101 from 1012 to 1012,1007 (kafka.cluster.Partition)
{code}
{code:java}
// disconnects among brokers -- controller.log
[2019-07-09 18:53:38,918] WARN [Controller-1012-to-broker-1012-send-thread], Controller 1012's connection to broker broker.fqdn.com:6668 (
id: 1012 rack: null) was unsuccessful (kafka.controller.RequestSendThread)
java.io.IOException: Connection to broker.fqdn.com:6668 (id: 1012 rack: null) failed
at kafka.utils.NetworkClientBlockingOps$.awaitReady$1(NetworkClientBlockingOps.scala:84)
at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:94)
at kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:236)
at kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:189)
at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:188)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
{code}
Parameters we Tried:
{code:java}
//Broker Settings:
num.network.threads=10
num.replica.fetchers=10
replica.socket.timeout.ms=60000
controller.socket.timeout.ms to 60000
zookeeper.connection.timeout.ms to 60000
zookeeper.session.timeout.ms to 60000
replica.fetch.wait.max.ms to 750 {code}
{code:java}
// System Settings:
net.core.somaxconn=8192
net.ipv4.tcp_max_syn_backlog=4096
{code}
Any pointers or suggestions will be very helpful.
Thanks,
Suri
> Frequent ISR shrinking and expanding and disconnects among brokers
> ------------------------------------------------------------------
>
> Key: KAFKA-4674
> URL: https://issues.apache.org/jira/browse/KAFKA-4674
> Project: Kafka
> Issue Type: Bug
> Components: controller, core
> Affects Versions: 0.10.0.1
> Environment: OS: Redhat Linux 2.6.32-431.el6.x86_64
> JDK: 1.8.0_45
> Reporter: Kaiming Wan
> Priority: Major
> Attachments: controller.log.rar, kafkabroker.20170221.log.zip, server.log.2017-01-11-14, zookeeper.out.2017-01-11.log
>
>
> We use a kafka cluster with 3 brokers in production environment. It works well for several month. Recently, we get the UnderReplicatedPartitions>0 warning mail. When we check the log, we find that the partition is always experience ISR shrinking and expanding. And the disconnection exception can be found in controller's log.
> We also found some deviant output in zookeeper's log which point to a consumer(using old API depends on zookeeper ) which has stopped its work with many lags.
> Actually, it is not the first time we encounter this problem. When we first met this problem, we also found the same phenomenon and the log output. We solve the problem by deleting the consumer node info in zookeeper. Then everything goes well.
> However, this time, after we deleting the consumer which already have large lag, the frequent ISR shrinking and expanding didn't stop for a very long time(serveral hours). Though, the issue didn't affect our consumer and producer, we think it will make our cluster unstable. So at last, we solve this problem by restart the controller broker.
> And now I wander what cause this problem. I check the source code and only know poll timeout will cause disconnection and ISR shrinking. Is the issue related to zookeeper because it will not hold too many metadata modification and make the replication fetch thread take more time?
> I upload the log file in the attachment.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)