You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by JsonTu <ka...@126.com> on 2017/05/09 06:38:37 UTC
Kafka 0.9.0.1 Direct Memory OOM
Hi All,
We have a cluster with 6 nodes. we have meet a direct buffer memory OOM in our prod enviroment. Default config of Kafka’s JVM is used in our cluster.
some error is like below,
java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Bits.java:658)
at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)
at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:174)
at sun.nio.ch.IOUtil.read(IOUtil.java:195)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
at org.apache.kafka.common.network.PlaintextTransportLayer.read(PlaintextTransportLayer.java:108)
at org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:97)
at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:71)
at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:153)
at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:134)
at org.apache.kafka.common.network.Selector.poll(Selector.java:286)
at kafka.network.Processor.run(SocketServer.scala:413)
at java.lang.Thread.run(Thread.java:745)
There are two suspicious place.
1、we have a topic have sent 100millions messages in one minute at that time,so I think whether this is may be the cause.
2、we have over 4000 close_wait socket on our broker.
Can someone help to analyse it. Thanks in advance.
答复: Kafka 0.9.0.1 Direct Memory OOM
Posted by Hu Xi <hu...@hotmail.com>.
Could you set -XX:MaxDirectMemorySize to a larger value and retry?
________________________________
发件人: JsonTu <ka...@126.com>
发送时间: 2017年5月9日 14:38
收件人: users@kafka.apache.org; dev@kafka.apache.org; wangguoz@gmail.com; becket.qin@gmail.com
主题: Kafka 0.9.0.1 Direct Memory OOM
Hi All,
We have a cluster with 6 nodes. we have meet a direct buffer memory OOM in our prod enviroment. Default config of Kafka’s JVM is used in our cluster.
some error is like below,
java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Bits.java:658)
at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)
at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:174)
at sun.nio.ch.IOUtil.read(IOUtil.java:195)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
at org.apache.kafka.common.network.PlaintextTransportLayer.read(PlaintextTransportLayer.java:108)
at org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:97)
at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:71)
at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:153)
at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:134)
at org.apache.kafka.common.network.Selector.poll(Selector.java:286)
at kafka.network.Processor.run(SocketServer.scala:413)
at java.lang.Thread.run(Thread.java:745)
There are two suspicious place.
1、we have a topic have sent 100millions messages in one minute at that time,so I think whether this is may be the cause.
2、we have over 4000 close_wait socket on our broker.
Can someone help to analyse it. Thanks in advance.