You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Jiangjie Qin (JIRA)" <ji...@apache.org> on 2015/07/22 19:13:04 UTC
[jira] [Commented] (KAFKA-2352) Possible memory leak in MirrorMaker
and/or new Producer
[ https://issues.apache.org/jira/browse/KAFKA-2352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14637208#comment-14637208 ]
Jiangjie Qin commented on KAFKA-2352:
-------------------------------------
Can you verify whether the messages you passed to producer has been sent or not? In mirror maker the default configuration was set to avoid data loss. So it producer cannot send a message it will retry infinitely. In that case, you might exhaust the producer's buffer if you keep appending messages to producer.
> Possible memory leak in MirrorMaker and/or new Producer
> -------------------------------------------------------
>
> Key: KAFKA-2352
> URL: https://issues.apache.org/jira/browse/KAFKA-2352
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8.2.1
> Reporter: Kostya Golikov
> Attachments: consumer.conf, output.log, producer.conf
>
>
> I've been playing around with Mirror Maker (version from trunk, dated July 7th) and got a few problems, most noticeable of which is that MirrorMaker exhausts it's memory pool, even though it's size set to relatively huge value of 132 MB, and individual messages are around 2 KB. Batch size is set to just 2 messages (full configs are attached).
> {code}
> [2015-07-21 15:19:52,915] FATAL [mirrormaker-thread-1] Mirror maker thread failure due to (kafka.tools.MirrorMaker$MirrorMakerThread)
> org.apache.kafka.clients.producer.BufferExhaustedException: You have exhausted the 134217728 bytes of memory you configured for the client and the client is configured to error rather than block when memory is exhausted.
> at org.apache.kafka.clients.producer.internals.BufferPool.allocate(BufferPool.java:124)
> at org.apache.kafka.clients.producer.internals.RecordAccumulator.append(RecordAccumulator.java:172)
> at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:388)
> at kafka.tools.MirrorMaker$MirrorMakerProducer.send(MirrorMaker.scala:380)
> at kafka.tools.MirrorMaker$MirrorMakerThread$$anonfun$run$3.apply(MirrorMaker.scala:311)
> at kafka.tools.MirrorMaker$MirrorMakerThread$$anonfun$run$3.apply(MirrorMaker.scala:311)
> at scala.collection.Iterator$class.foreach(Iterator.scala:727)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
> at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> at kafka.tools.MirrorMaker$MirrorMakerThread.run(MirrorMaker.scala:311)
> {code}
> Am I doing something wrong? Any help in further diagnosing of this problem might be handy.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)