[jira] [Commented] (KAFKA-1795) OOME - high level kafka consumer

["Umesh Batra (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-1795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234738#comment-14234738 ] 

Umesh Batra commented on KAFKA-1795:
------------------------------------

Please ignore; I figured it out and leak was actually post consumer downstream component not related to kafka. 

> OOME - high level kafka consumer
> --------------------------------
>
>                 Key: KAFKA-1795
>                 URL: https://issues.apache.org/jira/browse/KAFKA-1795
>             Project: Kafka
>          Issue Type: Bug
>          Components: consumer
>    Affects Versions: 0.8.1.1, 0.9.0
>            Reporter: Umesh Batra
>            Assignee: Neha Narkhede
>            Priority: Blocker
>              Labels: newbie
>
> I am using Kafka High Level Consumer, version kafka_2.8.0 - 0.8.1.1 with zkclient - 0.3 and I get OOME just after 10/15 minutes, My volume test setup has just one topic with 10 partitions with continuous message (size ~500KB) flow and below are my configuration; 
>  
> zookeeper.connect=localhost:2181,localhost:2181
> group.id=tc
> consumer.id=tc
> zookeeper.sync.time.ms=200
> zookeeper.connection.timeout.ms=10000
> zookeeper.session.timeout.ms=10000
> fetch.size=500000
> fetch.message.max.bytes=1000000
> auto.commit.enable=true
> auto.commit.interval.ms=100
> auto.offset.reset=largest
> queued.max.message.chunks=1
> backoff.increment.ms=1000
> rebalance.max.retries=10
> rebalance.retries.max=10
> rebalance.backoff.ms=10000
> refresh.leader.backoff.ms=20000
> consumer.timeout.ms=50000 
>  
> Memory histogram shows 60% of memory consumed by byte[] and most of the remaining by char[] and HashMap$Node. In my various tries to recover from the situation. I observed "metric-*" thread live even after I shutdown Kafka connector? 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)