You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Xiaobing Bu <bu...@gmail.com> on 2019/08/22 05:48:32 UTC

Kafka gc remark time very long

Hi all,

Our kafka cluster on production environment have some problem, some brokers
connect to zookeeper timeout almost everyday. There are four brokers, each
has 10 core CPU and 8G memory.
The following is the server.log , it said broker can not connect to
zookeeper. I had capture packets using tcpdump , and found it was broker
side problem, broker not sent heartbeats to zookeeper, then timeout.

[2019-08-07 02:30:01,626] WARN Client session timed out, have not heard
from server in 6628ms for sessionid 0x36c2faa9d5f3cf3
(org.apache.zookeeper.ClientCnxn)
[2019-08-07 02:30:01,639] INFO Client session timed out, have not heard
from server in 6628ms for sessionid 0x36c2faa9d5f3cf3, closing socket
connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
[2019-08-07 02:30:04,709] WARN Attempting to send response via channel for
which there is no open connection, connection id 10.97.133.17:9092
-10.97.165.60:43578-52592642 (kafka.network.Processor)
[2019-08-07 02:30:04,802] WARN Attempting to send response via channel for
which there is no open connection, connection id 10.97.133.17:9092
-10.97.200.19:58674-52592642 (kafka.network.Processor)

And the kafkaServer-gc.log:

2019-08-07T02:29:56.746+0800: 12540338.986: [GC concurrent-mark-end,
0.7387587 secs]
2019-08-07T02:29:56.749+0800: 12540338.990: [GC remark
2019-08-07T02:29:56.749+0800: 12540338.990: [Finalize Marking, 0.0080879
secs]
2019-08-07T02:29:56.758+0800: 12540338.998: [GC ref-proc, 0.0011597 secs]
2019-08-07T02:29:56.759+0800: 12540338.999: [Unloading, 4.7635856 secs],
4.7984788 secs]
[Times: user=0.00 sys=0.00, real=4.80 secs]

gc Unloading time take 4.8 secs, and I used jstack, it appeared full gc
count is zero.

JDK version : jdk1.8.0_73
kafka version: kafka_2.11-1.1.1 and running in compatible mode:
inter.broker.protocol.version=1.1
log.message.format.version=0.9.0

java running parameters:

/usr/java/jdk1.8.0_73/bin/java -XX:PermSize=128m -XX:MaxPermSize=128m

-Xms4096m -Xmx4096m -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20
-XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCInvokesConcurrent

-Djava.awt.headless=true
-Xloggc:/usr/local/kafka/bin/../logs/kafkaServer-gc.log -verbose:gc
-XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps
-XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=100M


Anyone who can help me figure out the problem, thanks.

Re: Kafka gc remark time very long

Posted by Mark Anderson <ma...@gmail.com>.
The first thing I would do is update to the latest Java 8 release. Just in
case you are hitting any G1GC bugs in such an old version.

Mark

On Thu, 22 Aug 2019, 07:17 Xiaobing Bu, <bu...@gmail.com> wrote:

> it not a network issues, since i had capture the network packets.
> when the GC remark and unloading class, we guess the java application is in
> stop the world mode, and can not send heartbeat to zookeeper, and after
> some time, the broker is disconnected from the cluster,
> so anther clients and brokers seems this broker is crash, and doing
> failover.
>
> On Thu, Aug 22, 2019 at 2:00 PM Eliza <el...@chinabuckets.com> wrote:
>
> > Hi
> >
> > on 2019/8/22 13:48, Xiaobing Bu wrote:
> > > [2019-08-07 02:30:04,802] WARN Attempting to send response via channel
> > for
> > > which there is no open connection, connection id 10.97.133.17:9092
> > > -10.97.200.19:58674-52592642 (kafka.network.Processor)
> >
> > It seems the network issues.
> > Did you double check the network connection? and firewall stuff?
> >
> > regards.
> >
>

Re: Kafka gc remark time very long

Posted by Xiaobing Bu <bu...@gmail.com>.
it not a network issues, since i had capture the network packets.
when the GC remark and unloading class, we guess the java application is in
stop the world mode, and can not send heartbeat to zookeeper, and after
some time, the broker is disconnected from the cluster,
so anther clients and brokers seems this broker is crash, and doing
failover.

On Thu, Aug 22, 2019 at 2:00 PM Eliza <el...@chinabuckets.com> wrote:

> Hi
>
> on 2019/8/22 13:48, Xiaobing Bu wrote:
> > [2019-08-07 02:30:04,802] WARN Attempting to send response via channel
> for
> > which there is no open connection, connection id 10.97.133.17:9092
> > -10.97.200.19:58674-52592642 (kafka.network.Processor)
>
> It seems the network issues.
> Did you double check the network connection? and firewall stuff?
>
> regards.
>

Re: Kafka gc remark time very long

Posted by Eliza <el...@ChinaBuckets.com>.
Hi

on 2019/8/22 13:48, Xiaobing Bu wrote:
> [2019-08-07 02:30:04,802] WARN Attempting to send response via channel for
> which there is no open connection, connection id 10.97.133.17:9092
> -10.97.200.19:58674-52592642 (kafka.network.Processor)

It seems the network issues.
Did you double check the network connection? and firewall stuff?

regards.