You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@kafka.apache.org by Check Peck <co...@gmail.com> on 2017/01/11 22:59:59 UTC

java.lang.OutOfMemoryError: Java heap space while running kafka-consumer-perf-test.sh

I am trying to run kafka performance script on my linux box. Whenever I run
"kafka-consumer-perf-test.sh", I always get an error. In the same box, I am
running "kafka-producer-perf-test.sh" as well and that is not failing at
all. Looks like something is wrong with "kafka-consumer-perf-test.sh".

I am running Kafka version 0.10.1.0.

Command I ran:
./bin/kafka-consumer-perf-test.sh --zookeeper 110.27.14.10:2181 --messages
50 --topic test-topic --threads 1

Error I got:
    [2017-01-11 22:34:09,785] WARN [ConsumerFetcherThread-perf-
consumer-14195_kafka-cluster-3098529006 <(309)%20852-9006>
-zeidk-1484174043509-46a51434-2-0], Error in fetch kafka.consumer.
ConsumerFetcherThread$FetchRequest@54fb48b6 (kafka.consumer.
ConsumerFetcherThread)
java.lang.OutOfMemoryError: Java heap space
        at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
        at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
        at org.apache.kafka.common.network.NetworkReceive.
readFromReadableChannel(NetworkReceive.java:93)
        at kafka.network.BlockingChannel.readCompletely(
BlockingChannel.scala:129)
        at kafka.network.BlockingChannel.receive(BlockingChannel.scala:120)
        at kafka.consumer.SimpleConsumer.liftedTree1$1(SimpleConsumer.
scala:99)
        at kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$
$sendRequest(SimpleConsumer.scala:83)
        at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$
apply$mcV$sp$1.apply$mcV$sp(SimpleConsumer.scala:132)
        at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$
apply$mcV$sp$1.apply(SimpleConsumer.scala:132)
        at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$
apply$mcV$sp$1.apply(SimpleConsumer.scala:132)
        at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
        at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply$mcV$sp(
SimpleConsumer.scala:131)
        at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(
SimpleConsumer.scala:131)
        at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(
SimpleConsumer.scala:131)
        at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
        at kafka.consumer.SimpleConsumer.fetch(SimpleConsumer.scala:130)
        at kafka.consumer.ConsumerFetcherThread.fetch(
ConsumerFetcherThread.scala:109)
        at kafka.consumer.ConsumerFetcherThread.fetch(
ConsumerFetcherThread.scala:29)
        at kafka.server.AbstractFetcherThread.processFetchRequest(
AbstractFetcherThread.scala:118)
        at kafka.server.AbstractFetcherThread.doWork(
AbstractFetcherThread.scala:103)
        at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)

Re: java.lang.OutOfMemoryError: Java heap space while running kafka-consumer-perf-test.sh

Posted by Ewen Cheslack-Postava <ew...@confluent.io>.

Perhaps the default heap options aren't sufficient for your particular use
of the tool. The consumer perf test defaults to 512MB. I'd simply try
increasing the max heap usage: KAFKA_HEAP_OPTS="-Xmx1024M" to bump it up a
bit.

-Ewen

On Wed, Jan 11, 2017 at 2:59 PM, Check Peck <co...@gmail.com> wrote:

> I am trying to run kafka performance script on my linux box. Whenever I run
> "kafka-consumer-perf-test.sh", I always get an error. In the same box, I am
> running "kafka-producer-perf-test.sh" as well and that is not failing at
> all. Looks like something is wrong with "kafka-consumer-perf-test.sh".
>
> I am running Kafka version 0.10.1.0.
>
> Command I ran:
> ./bin/kafka-consumer-perf-test.sh --zookeeper 110.27.14.10:2181 --messages
> 50 --topic test-topic --threads 1
>
> Error I got:
>     [2017-01-11 22:34:09,785] WARN [ConsumerFetcherThread-perf-
> consumer-14195_kafka-cluster-3098529006 <(309)%20852-9006>
> -zeidk-1484174043509-46a51434-2-0], Error in fetch kafka.consumer.
> ConsumerFetcherThread$FetchRequest@54fb48b6 (kafka.consumer.
> ConsumerFetcherThread)
> java.lang.OutOfMemoryError: Java heap space
>         at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
>         at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
>         at org.apache.kafka.common.network.NetworkReceive.
> readFromReadableChannel(NetworkReceive.java:93)
>         at kafka.network.BlockingChannel.readCompletely(
> BlockingChannel.scala:129)
>         at kafka.network.BlockingChannel.receive(BlockingChannel.scala:
> 120)
>         at kafka.consumer.SimpleConsumer.liftedTree1$1(SimpleConsumer.
> scala:99)
>         at kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$
> $sendRequest(SimpleConsumer.scala:83)
>         at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$
> apply$mcV$sp$1.apply$mcV$sp(SimpleConsumer.scala:132)
>         at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$
> apply$mcV$sp$1.apply(SimpleConsumer.scala:132)
>         at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$
> apply$mcV$sp$1.apply(SimpleConsumer.scala:132)
>         at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
>         at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply$mcV$sp(
> SimpleConsumer.scala:131)
>         at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(
> SimpleConsumer.scala:131)
>         at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(
> SimpleConsumer.scala:131)
>         at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
>         at kafka.consumer.SimpleConsumer.fetch(SimpleConsumer.scala:130)
>         at kafka.consumer.ConsumerFetcherThread.fetch(
> ConsumerFetcherThread.scala:109)
>         at kafka.consumer.ConsumerFetcherThread.fetch(
> ConsumerFetcherThread.scala:29)
>         at kafka.server.AbstractFetcherThread.processFetchRequest(
> AbstractFetcherThread.scala:118)
>         at kafka.server.AbstractFetcherThread.doWork(
> AbstractFetcherThread.scala:103)
>         at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
>

Re: java.lang.OutOfMemoryError: Java heap space while running kafka-consumer-perf-test.sh

Posted by Check Peck <co...@gmail.com>.

Can anyone shed some light on this?

On Wed, Jan 11, 2017 at 2:59 PM, Check Peck <co...@gmail.com> wrote:

> I am trying to run kafka performance script on my linux box. Whenever I
> run "kafka-consumer-perf-test.sh", I always get an error. In the same box,
> I am running "kafka-producer-perf-test.sh" as well and that is not failing
> at all. Looks like something is wrong with "kafka-consumer-perf-test.sh".
>
> I am running Kafka version 0.10.1.0.
>
> Command I ran:
> ./bin/kafka-consumer-perf-test.sh --zookeeper 110.27.14.10:2181
> --messages 50 --topic test-topic --threads 1
>
> Error I got:
>     [2017-01-11 22:34:09,785] WARN [ConsumerFetcherThread-perf-co
> nsumer-14195_kafka-cluster-3098529006 <(309)%20852-9006>
> -zeidk-1484174043509-46a51434-2-0], Error in fetch
> kafka.consumer.ConsumerFetcherThread$FetchRequest@54fb48b6
> (kafka.consumer.ConsumerFetcherThread)
> java.lang.OutOfMemoryError: Java heap space
>         at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
>         at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
>         at org.apache.kafka.common.network.NetworkReceive.readFromReada
> bleChannel(NetworkReceive.java:93)
>         at kafka.network.BlockingChannel.readCompletely(BlockingChannel
> .scala:129)
>         at kafka.network.BlockingChannel.receive(BlockingChannel.scala:
> 120)
>         at kafka.consumer.SimpleConsumer.liftedTree1$1(SimpleConsumer.s
> cala:99)
>         at kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$
> $sendRequest(SimpleConsumer.scala:83)
>         at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$appl
> y$mcV$sp$1.apply$mcV$sp(SimpleConsumer.scala:132)
>         at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$appl
> y$mcV$sp$1.apply(SimpleConsumer.scala:132)
>         at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$appl
> y$mcV$sp$1.apply(SimpleConsumer.scala:132)
>         at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
>         at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply$mcV$sp(
> SimpleConsumer.scala:131)
>         at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleC
> onsumer.scala:131)
>         at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleC
> onsumer.scala:131)
>         at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
>         at kafka.consumer.SimpleConsumer.fetch(SimpleConsumer.scala:130)
>         at kafka.consumer.ConsumerFetcherThread.fetch(ConsumerFetcherTh
> read.scala:109)
>         at kafka.consumer.ConsumerFetcherThread.fetch(ConsumerFetcherTh
> read.scala:29)
>         at kafka.server.AbstractFetcherThread.processFetchRequest(Abstr
> actFetcherThread.scala:118)
>         at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThr
> ead.scala:103)
>         at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
>
>