You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Shantanu Deshmukh <sh...@gmail.com> on 2018/09/18 12:06:21 UTC
Kafka producer huge memory usage (leak?)
Hello,
We have a 3 broker Kafka 0.10.1.0 deployment in production. There are some
applications which have Kafka Producers embedded in them which send
application logs to a topic. This topic has 10 partitions with replication
factor of 3.
We are observing that memory usage on some of these application servers
keep shooting through the roof intermittently. After taking heapdump we
found out that top suspects were:
*---------------------*
*org.apache.kafka.common.network.Selector -*
occupies *352,519,104 (24.96%)* bytes. The memory is accumulated in one
instance of *"byte[]"* loaded by *"<system class loader>"*.
*org.apache.kafka.common.network.KafkaChannel -*
occupies *352,527,424 (24.96%)* bytes. The memory is accumulated in one
instance of *"byte[]"* loaded by *"<system class loader>"*
* --------------------- *
Both of these were holding about 352MB of space. 3 such instances, so they
were consuming about 1.2GB of memory.
Now regarding usage of producers. Not a huge amount of logs are being sent
to Kafka cluster. It is about 200 msgs/sec. Only one producer object is
being used throughout application. Async send function is used.
What could be the cause of such huge memory usage? Is this some sort of
memory leak in this specific Kafka version?
Re: Kafka producer huge memory usage (leak?)
Posted by Manikumar <ma...@gmail.com>.
Hi,
In that case, it is not related to KAFKA-4493. I see StackOverFlowException
in the uploaded heapdump.
I am pretty sure this is related to StackOverFlowException. Some objects
are dangling after this exception.
You need to check/find out when this exception is occuring. check your
application server logs for any errors.
try increasing stack size by adding -Xss2048k.
thanks,
On Fri, Sep 21, 2018 at 2:44 PM Shantanu Deshmukh <sh...@gmail.com>
wrote:
> Hi Manikumar,
> I am using correct protocol. SSL and truststore is also correctly
> configured and there is a single port which is for SSL. Otherwise it simply
> wouldn't work. Data is getting produced all fine, it's just that producer
> object is consuming massive amount of memory.
>
> On Fri, Sep 21, 2018 at 2:36 PM Manikumar <ma...@gmail.com>
> wrote:
>
> > Hi,
> > Instead trying the PR, make sure you are setting valid security protocol
> > and connecting to valid broker port.
> > also looks for any errors in producer logs.
> >
> > Thanks,
> >
> >
> >
> >
> >
> > On Fri, Sep 21, 2018 at 12:35 PM Shantanu Deshmukh <
> shantanu88d@gmail.com>
> > wrote:
> >
> > > Hi Manikumar,
> > >
> > > I checked this issue. There is this patch available.
> > > https://github.com/apache/kafka/pull/2408.patch
> > >
> > > I pulled Kafka 0.10.1.0 from github. Then tried applying this patch.
> But
> > > several places I am getting error that patch doesn't apply.
> > > I am new to git and patching process. Can you guide me here?
> > >
> > > On Wed, Sep 19, 2018 at 1:02 PM Manikumar <ma...@gmail.com>
> > > wrote:
> > >
> > > > Similar issue reported here:KAFKA-7304, but on broker side. maybe
> you
> > > can
> > > > create a JIRA and upload the heap dump for analysis.
> > > >
> > > > On Wed, Sep 19, 2018 at 11:59 AM Shantanu Deshmukh <
> > > shantanu88d@gmail.com>
> > > > wrote:
> > > >
> > > > > Any thoughts on this matter? Someone, please help.
> > > > >
> > > > > On Tue, Sep 18, 2018 at 6:05 PM Shantanu Deshmukh <
> > > shantanu88d@gmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Additionally, here's the producer config
> > > > > >
> > > > > > kafka.bootstrap.servers=x.x.x.x:9092,x.x.x.x:9092,x.x.x.x:9092
> > > > > > kafka.acks=0
> > > > > >
> > > > >
> > > >
> > >
> >
> kafka.key.serializer=org.apache.kafka.common.serialization.StringSerializer
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> kafka.value.serializer=org.apache.kafka.common.serialization.StringSerializer
> > > > > > kafka.max.block.ms=1000
> > > > > > kafka.request.timeout.ms=1000
> > > > > > kafka.max.in.flight.requests.per.connection=1
> > > > > > kafka.retries=0
> > > > > > kafka.compression.type=gzip
> > > > > > kafka.security.protocol=SSL
> > > > > >
> > kafka.ssl.truststore.location=/data/kafka/kafka-server-truststore.jks
> > > > > > kafka.ssl.truststore.password=XXXXXX
> > > > > > kafka.linger.ms=300
> > > > > > logger.level=INFO
> > > > > >
> > > > > > On Tue, Sep 18, 2018 at 5:36 PM Shantanu Deshmukh <
> > > > shantanu88d@gmail.com
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > >> Hello,
> > > > > >>
> > > > > >> We have a 3 broker Kafka 0.10.1.0 deployment in production.
> There
> > > are
> > > > > >> some applications which have Kafka Producers embedded in them
> > which
> > > > send
> > > > > >> application logs to a topic. This topic has 10 partitions with
> > > > > replication
> > > > > >> factor of 3.
> > > > > >>
> > > > > >> We are observing that memory usage on some of these application
> > > > servers
> > > > > >> keep shooting through the roof intermittently. After taking
> > heapdump
> > > > we
> > > > > >> found out that top suspects were:
> > > > > >> *---------------------*
> > > > > >>
> > > > > >>
> > > > > >> *org.apache.kafka.common.network.Selector -*
> > > > > >>
> > > > > >> occupies *352,519,104 (24.96%)* bytes. The memory is accumulated
> > in
> > > > one
> > > > > >> instance of *"byte[]"* loaded by *"<system class loader>"*.
> > > > > >>
> > > > > >> *org.apache.kafka.common.network.KafkaChannel -*
> > > > > >>
> > > > > >> occupies *352,527,424 (24.96%)* bytes. The memory is accumulated
> > in
> > > > one
> > > > > >> instance of *"byte[]"* loaded by *"<system class loader>"*
> > > > > >>
> > > > > >> * --------------------- *
> > > > > >>
> > > > > >> Both of these were holding about 352MB of space. 3 such
> instances,
> > > so
> > > > > >> they were consuming about 1.2GB of memory.
> > > > > >>
> > > > > >> Now regarding usage of producers. Not a huge amount of logs are
> > > being
> > > > > >> sent to Kafka cluster. It is about 200 msgs/sec. Only one
> producer
> > > > > object
> > > > > >> is being used throughout application. Async send function is
> used.
> > > > > >>
> > > > > >> What could be the cause of such huge memory usage? Is this some
> > sort
> > > > of
> > > > > >> memory leak in this specific Kafka version?
> > > > > >>
> > > > > >>
> > > > >
> > > >
> > >
> >
>
Re: Kafka producer huge memory usage (leak?)
Posted by Shantanu Deshmukh <sh...@gmail.com>.
Hi Manikumar,
I am using correct protocol. SSL and truststore is also correctly
configured and there is a single port which is for SSL. Otherwise it simply
wouldn't work. Data is getting produced all fine, it's just that producer
object is consuming massive amount of memory.
On Fri, Sep 21, 2018 at 2:36 PM Manikumar <ma...@gmail.com> wrote:
> Hi,
> Instead trying the PR, make sure you are setting valid security protocol
> and connecting to valid broker port.
> also looks for any errors in producer logs.
>
> Thanks,
>
>
>
>
>
> On Fri, Sep 21, 2018 at 12:35 PM Shantanu Deshmukh <sh...@gmail.com>
> wrote:
>
> > Hi Manikumar,
> >
> > I checked this issue. There is this patch available.
> > https://github.com/apache/kafka/pull/2408.patch
> >
> > I pulled Kafka 0.10.1.0 from github. Then tried applying this patch. But
> > several places I am getting error that patch doesn't apply.
> > I am new to git and patching process. Can you guide me here?
> >
> > On Wed, Sep 19, 2018 at 1:02 PM Manikumar <ma...@gmail.com>
> > wrote:
> >
> > > Similar issue reported here:KAFKA-7304, but on broker side. maybe you
> > can
> > > create a JIRA and upload the heap dump for analysis.
> > >
> > > On Wed, Sep 19, 2018 at 11:59 AM Shantanu Deshmukh <
> > shantanu88d@gmail.com>
> > > wrote:
> > >
> > > > Any thoughts on this matter? Someone, please help.
> > > >
> > > > On Tue, Sep 18, 2018 at 6:05 PM Shantanu Deshmukh <
> > shantanu88d@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > Additionally, here's the producer config
> > > > >
> > > > > kafka.bootstrap.servers=x.x.x.x:9092,x.x.x.x:9092,x.x.x.x:9092
> > > > > kafka.acks=0
> > > > >
> > > >
> > >
> >
> kafka.key.serializer=org.apache.kafka.common.serialization.StringSerializer
> > > > >
> > > > >
> > > >
> > >
> >
> kafka.value.serializer=org.apache.kafka.common.serialization.StringSerializer
> > > > > kafka.max.block.ms=1000
> > > > > kafka.request.timeout.ms=1000
> > > > > kafka.max.in.flight.requests.per.connection=1
> > > > > kafka.retries=0
> > > > > kafka.compression.type=gzip
> > > > > kafka.security.protocol=SSL
> > > > >
> kafka.ssl.truststore.location=/data/kafka/kafka-server-truststore.jks
> > > > > kafka.ssl.truststore.password=XXXXXX
> > > > > kafka.linger.ms=300
> > > > > logger.level=INFO
> > > > >
> > > > > On Tue, Sep 18, 2018 at 5:36 PM Shantanu Deshmukh <
> > > shantanu88d@gmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > > >> Hello,
> > > > >>
> > > > >> We have a 3 broker Kafka 0.10.1.0 deployment in production. There
> > are
> > > > >> some applications which have Kafka Producers embedded in them
> which
> > > send
> > > > >> application logs to a topic. This topic has 10 partitions with
> > > > replication
> > > > >> factor of 3.
> > > > >>
> > > > >> We are observing that memory usage on some of these application
> > > servers
> > > > >> keep shooting through the roof intermittently. After taking
> heapdump
> > > we
> > > > >> found out that top suspects were:
> > > > >> *---------------------*
> > > > >>
> > > > >>
> > > > >> *org.apache.kafka.common.network.Selector -*
> > > > >>
> > > > >> occupies *352,519,104 (24.96%)* bytes. The memory is accumulated
> in
> > > one
> > > > >> instance of *"byte[]"* loaded by *"<system class loader>"*.
> > > > >>
> > > > >> *org.apache.kafka.common.network.KafkaChannel -*
> > > > >>
> > > > >> occupies *352,527,424 (24.96%)* bytes. The memory is accumulated
> in
> > > one
> > > > >> instance of *"byte[]"* loaded by *"<system class loader>"*
> > > > >>
> > > > >> * --------------------- *
> > > > >>
> > > > >> Both of these were holding about 352MB of space. 3 such instances,
> > so
> > > > >> they were consuming about 1.2GB of memory.
> > > > >>
> > > > >> Now regarding usage of producers. Not a huge amount of logs are
> > being
> > > > >> sent to Kafka cluster. It is about 200 msgs/sec. Only one producer
> > > > object
> > > > >> is being used throughout application. Async send function is used.
> > > > >>
> > > > >> What could be the cause of such huge memory usage? Is this some
> sort
> > > of
> > > > >> memory leak in this specific Kafka version?
> > > > >>
> > > > >>
> > > >
> > >
> >
>
Re: Kafka producer huge memory usage (leak?)
Posted by Manikumar <ma...@gmail.com>.
Hi,
Instead trying the PR, make sure you are setting valid security protocol
and connecting to valid broker port.
also looks for any errors in producer logs.
Thanks,
On Fri, Sep 21, 2018 at 12:35 PM Shantanu Deshmukh <sh...@gmail.com>
wrote:
> Hi Manikumar,
>
> I checked this issue. There is this patch available.
> https://github.com/apache/kafka/pull/2408.patch
>
> I pulled Kafka 0.10.1.0 from github. Then tried applying this patch. But
> several places I am getting error that patch doesn't apply.
> I am new to git and patching process. Can you guide me here?
>
> On Wed, Sep 19, 2018 at 1:02 PM Manikumar <ma...@gmail.com>
> wrote:
>
> > Similar issue reported here:KAFKA-7304, but on broker side. maybe you
> can
> > create a JIRA and upload the heap dump for analysis.
> >
> > On Wed, Sep 19, 2018 at 11:59 AM Shantanu Deshmukh <
> shantanu88d@gmail.com>
> > wrote:
> >
> > > Any thoughts on this matter? Someone, please help.
> > >
> > > On Tue, Sep 18, 2018 at 6:05 PM Shantanu Deshmukh <
> shantanu88d@gmail.com
> > >
> > > wrote:
> > >
> > > > Additionally, here's the producer config
> > > >
> > > > kafka.bootstrap.servers=x.x.x.x:9092,x.x.x.x:9092,x.x.x.x:9092
> > > > kafka.acks=0
> > > >
> > >
> >
> kafka.key.serializer=org.apache.kafka.common.serialization.StringSerializer
> > > >
> > > >
> > >
> >
> kafka.value.serializer=org.apache.kafka.common.serialization.StringSerializer
> > > > kafka.max.block.ms=1000
> > > > kafka.request.timeout.ms=1000
> > > > kafka.max.in.flight.requests.per.connection=1
> > > > kafka.retries=0
> > > > kafka.compression.type=gzip
> > > > kafka.security.protocol=SSL
> > > > kafka.ssl.truststore.location=/data/kafka/kafka-server-truststore.jks
> > > > kafka.ssl.truststore.password=XXXXXX
> > > > kafka.linger.ms=300
> > > > logger.level=INFO
> > > >
> > > > On Tue, Sep 18, 2018 at 5:36 PM Shantanu Deshmukh <
> > shantanu88d@gmail.com
> > > >
> > > > wrote:
> > > >
> > > >> Hello,
> > > >>
> > > >> We have a 3 broker Kafka 0.10.1.0 deployment in production. There
> are
> > > >> some applications which have Kafka Producers embedded in them which
> > send
> > > >> application logs to a topic. This topic has 10 partitions with
> > > replication
> > > >> factor of 3.
> > > >>
> > > >> We are observing that memory usage on some of these application
> > servers
> > > >> keep shooting through the roof intermittently. After taking heapdump
> > we
> > > >> found out that top suspects were:
> > > >> *---------------------*
> > > >>
> > > >>
> > > >> *org.apache.kafka.common.network.Selector -*
> > > >>
> > > >> occupies *352,519,104 (24.96%)* bytes. The memory is accumulated in
> > one
> > > >> instance of *"byte[]"* loaded by *"<system class loader>"*.
> > > >>
> > > >> *org.apache.kafka.common.network.KafkaChannel -*
> > > >>
> > > >> occupies *352,527,424 (24.96%)* bytes. The memory is accumulated in
> > one
> > > >> instance of *"byte[]"* loaded by *"<system class loader>"*
> > > >>
> > > >> * --------------------- *
> > > >>
> > > >> Both of these were holding about 352MB of space. 3 such instances,
> so
> > > >> they were consuming about 1.2GB of memory.
> > > >>
> > > >> Now regarding usage of producers. Not a huge amount of logs are
> being
> > > >> sent to Kafka cluster. It is about 200 msgs/sec. Only one producer
> > > object
> > > >> is being used throughout application. Async send function is used.
> > > >>
> > > >> What could be the cause of such huge memory usage? Is this some sort
> > of
> > > >> memory leak in this specific Kafka version?
> > > >>
> > > >>
> > >
> >
>
Re: Kafka producer huge memory usage (leak?)
Posted by Shantanu Deshmukh <sh...@gmail.com>.
Hi Manikumar,
I checked this issue. There is this patch available.
https://github.com/apache/kafka/pull/2408.patch
I pulled Kafka 0.10.1.0 from github. Then tried applying this patch. But
several places I am getting error that patch doesn't apply.
I am new to git and patching process. Can you guide me here?
On Wed, Sep 19, 2018 at 1:02 PM Manikumar <ma...@gmail.com> wrote:
> Similar issue reported here:KAFKA-7304, but on broker side. maybe you can
> create a JIRA and upload the heap dump for analysis.
>
> On Wed, Sep 19, 2018 at 11:59 AM Shantanu Deshmukh <sh...@gmail.com>
> wrote:
>
> > Any thoughts on this matter? Someone, please help.
> >
> > On Tue, Sep 18, 2018 at 6:05 PM Shantanu Deshmukh <shantanu88d@gmail.com
> >
> > wrote:
> >
> > > Additionally, here's the producer config
> > >
> > > kafka.bootstrap.servers=x.x.x.x:9092,x.x.x.x:9092,x.x.x.x:9092
> > > kafka.acks=0
> > >
> >
> kafka.key.serializer=org.apache.kafka.common.serialization.StringSerializer
> > >
> > >
> >
> kafka.value.serializer=org.apache.kafka.common.serialization.StringSerializer
> > > kafka.max.block.ms=1000
> > > kafka.request.timeout.ms=1000
> > > kafka.max.in.flight.requests.per.connection=1
> > > kafka.retries=0
> > > kafka.compression.type=gzip
> > > kafka.security.protocol=SSL
> > > kafka.ssl.truststore.location=/data/kafka/kafka-server-truststore.jks
> > > kafka.ssl.truststore.password=XXXXXX
> > > kafka.linger.ms=300
> > > logger.level=INFO
> > >
> > > On Tue, Sep 18, 2018 at 5:36 PM Shantanu Deshmukh <
> shantanu88d@gmail.com
> > >
> > > wrote:
> > >
> > >> Hello,
> > >>
> > >> We have a 3 broker Kafka 0.10.1.0 deployment in production. There are
> > >> some applications which have Kafka Producers embedded in them which
> send
> > >> application logs to a topic. This topic has 10 partitions with
> > replication
> > >> factor of 3.
> > >>
> > >> We are observing that memory usage on some of these application
> servers
> > >> keep shooting through the roof intermittently. After taking heapdump
> we
> > >> found out that top suspects were:
> > >> *---------------------*
> > >>
> > >>
> > >> *org.apache.kafka.common.network.Selector -*
> > >>
> > >> occupies *352,519,104 (24.96%)* bytes. The memory is accumulated in
> one
> > >> instance of *"byte[]"* loaded by *"<system class loader>"*.
> > >>
> > >> *org.apache.kafka.common.network.KafkaChannel -*
> > >>
> > >> occupies *352,527,424 (24.96%)* bytes. The memory is accumulated in
> one
> > >> instance of *"byte[]"* loaded by *"<system class loader>"*
> > >>
> > >> * --------------------- *
> > >>
> > >> Both of these were holding about 352MB of space. 3 such instances, so
> > >> they were consuming about 1.2GB of memory.
> > >>
> > >> Now regarding usage of producers. Not a huge amount of logs are being
> > >> sent to Kafka cluster. It is about 200 msgs/sec. Only one producer
> > object
> > >> is being used throughout application. Async send function is used.
> > >>
> > >> What could be the cause of such huge memory usage? Is this some sort
> of
> > >> memory leak in this specific Kafka version?
> > >>
> > >>
> >
>
Re: Kafka producer huge memory usage (leak?)
Posted by Manikumar <ma...@gmail.com>.
Similar issue reported here:KAFKA-7304, but on broker side. maybe you can
create a JIRA and upload the heap dump for analysis.
On Wed, Sep 19, 2018 at 11:59 AM Shantanu Deshmukh <sh...@gmail.com>
wrote:
> Any thoughts on this matter? Someone, please help.
>
> On Tue, Sep 18, 2018 at 6:05 PM Shantanu Deshmukh <sh...@gmail.com>
> wrote:
>
> > Additionally, here's the producer config
> >
> > kafka.bootstrap.servers=x.x.x.x:9092,x.x.x.x:9092,x.x.x.x:9092
> > kafka.acks=0
> >
> kafka.key.serializer=org.apache.kafka.common.serialization.StringSerializer
> >
> >
> kafka.value.serializer=org.apache.kafka.common.serialization.StringSerializer
> > kafka.max.block.ms=1000
> > kafka.request.timeout.ms=1000
> > kafka.max.in.flight.requests.per.connection=1
> > kafka.retries=0
> > kafka.compression.type=gzip
> > kafka.security.protocol=SSL
> > kafka.ssl.truststore.location=/data/kafka/kafka-server-truststore.jks
> > kafka.ssl.truststore.password=XXXXXX
> > kafka.linger.ms=300
> > logger.level=INFO
> >
> > On Tue, Sep 18, 2018 at 5:36 PM Shantanu Deshmukh <shantanu88d@gmail.com
> >
> > wrote:
> >
> >> Hello,
> >>
> >> We have a 3 broker Kafka 0.10.1.0 deployment in production. There are
> >> some applications which have Kafka Producers embedded in them which send
> >> application logs to a topic. This topic has 10 partitions with
> replication
> >> factor of 3.
> >>
> >> We are observing that memory usage on some of these application servers
> >> keep shooting through the roof intermittently. After taking heapdump we
> >> found out that top suspects were:
> >> *---------------------*
> >>
> >>
> >> *org.apache.kafka.common.network.Selector -*
> >>
> >> occupies *352,519,104 (24.96%)* bytes. The memory is accumulated in one
> >> instance of *"byte[]"* loaded by *"<system class loader>"*.
> >>
> >> *org.apache.kafka.common.network.KafkaChannel -*
> >>
> >> occupies *352,527,424 (24.96%)* bytes. The memory is accumulated in one
> >> instance of *"byte[]"* loaded by *"<system class loader>"*
> >>
> >> * --------------------- *
> >>
> >> Both of these were holding about 352MB of space. 3 such instances, so
> >> they were consuming about 1.2GB of memory.
> >>
> >> Now regarding usage of producers. Not a huge amount of logs are being
> >> sent to Kafka cluster. It is about 200 msgs/sec. Only one producer
> object
> >> is being used throughout application. Async send function is used.
> >>
> >> What could be the cause of such huge memory usage? Is this some sort of
> >> memory leak in this specific Kafka version?
> >>
> >>
>
Re: Kafka producer huge memory usage (leak?)
Posted by Subash Konar <su...@gmail.com>.
Hi Shantanu,
Please refer this
<https://stackoverflow.com/questions/52386300/kafka-producer-huge-memory-usage-leak/52386713#52386713>.
Hope it helps.
Thanks,
Subash
On Wed, Sep 19, 2018 at 11:59 AM Shantanu Deshmukh <sh...@gmail.com>
wrote:
> Any thoughts on this matter? Someone, please help.
>
> On Tue, Sep 18, 2018 at 6:05 PM Shantanu Deshmukh <sh...@gmail.com>
> wrote:
>
> > Additionally, here's the producer config
> >
> > kafka.bootstrap.servers=x.x.x.x:9092,x.x.x.x:9092,x.x.x.x:9092
> > kafka.acks=0
> >
> kafka.key.serializer=org.apache.kafka.common.serialization.StringSerializer
> >
> >
> kafka.value.serializer=org.apache.kafka.common.serialization.StringSerializer
> > kafka.max.block.ms=1000
> > kafka.request.timeout.ms=1000
> > kafka.max.in.flight.requests.per.connection=1
> > kafka.retries=0
> > kafka.compression.type=gzip
> > kafka.security.protocol=SSL
> > kafka.ssl.truststore.location=/data/kafka/kafka-server-truststore.jks
> > kafka.ssl.truststore.password=XXXXXX
> > kafka.linger.ms=300
> > logger.level=INFO
> >
> > On Tue, Sep 18, 2018 at 5:36 PM Shantanu Deshmukh <shantanu88d@gmail.com
> >
> > wrote:
> >
> >> Hello,
> >>
> >> We have a 3 broker Kafka 0.10.1.0 deployment in production. There are
> >> some applications which have Kafka Producers embedded in them which send
> >> application logs to a topic. This topic has 10 partitions with
> replication
> >> factor of 3.
> >>
> >> We are observing that memory usage on some of these application servers
> >> keep shooting through the roof intermittently. After taking heapdump we
> >> found out that top suspects were:
> >> *---------------------*
> >>
> >>
> >> *org.apache.kafka.common.network.Selector -*
> >>
> >> occupies *352,519,104 (24.96%)* bytes. The memory is accumulated in one
> >> instance of *"byte[]"* loaded by *"<system class loader>"*.
> >>
> >> *org.apache.kafka.common.network.KafkaChannel -*
> >>
> >> occupies *352,527,424 (24.96%)* bytes. The memory is accumulated in one
> >> instance of *"byte[]"* loaded by *"<system class loader>"*
> >>
> >> * --------------------- *
> >>
> >> Both of these were holding about 352MB of space. 3 such instances, so
> >> they were consuming about 1.2GB of memory.
> >>
> >> Now regarding usage of producers. Not a huge amount of logs are being
> >> sent to Kafka cluster. It is about 200 msgs/sec. Only one producer
> object
> >> is being used throughout application. Async send function is used.
> >>
> >> What could be the cause of such huge memory usage? Is this some sort of
> >> memory leak in this specific Kafka version?
> >>
> >>
>
--
Thanks and Regards,
Subash Konar
Re: Kafka producer huge memory usage (leak?)
Posted by Shantanu Deshmukh <sh...@gmail.com>.
Any thoughts on this matter? Someone, please help.
On Tue, Sep 18, 2018 at 6:05 PM Shantanu Deshmukh <sh...@gmail.com>
wrote:
> Additionally, here's the producer config
>
> kafka.bootstrap.servers=x.x.x.x:9092,x.x.x.x:9092,x.x.x.x:9092
> kafka.acks=0
> kafka.key.serializer=org.apache.kafka.common.serialization.StringSerializer
>
> kafka.value.serializer=org.apache.kafka.common.serialization.StringSerializer
> kafka.max.block.ms=1000
> kafka.request.timeout.ms=1000
> kafka.max.in.flight.requests.per.connection=1
> kafka.retries=0
> kafka.compression.type=gzip
> kafka.security.protocol=SSL
> kafka.ssl.truststore.location=/data/kafka/kafka-server-truststore.jks
> kafka.ssl.truststore.password=XXXXXX
> kafka.linger.ms=300
> logger.level=INFO
>
> On Tue, Sep 18, 2018 at 5:36 PM Shantanu Deshmukh <sh...@gmail.com>
> wrote:
>
>> Hello,
>>
>> We have a 3 broker Kafka 0.10.1.0 deployment in production. There are
>> some applications which have Kafka Producers embedded in them which send
>> application logs to a topic. This topic has 10 partitions with replication
>> factor of 3.
>>
>> We are observing that memory usage on some of these application servers
>> keep shooting through the roof intermittently. After taking heapdump we
>> found out that top suspects were:
>> *---------------------*
>>
>>
>> *org.apache.kafka.common.network.Selector -*
>>
>> occupies *352,519,104 (24.96%)* bytes. The memory is accumulated in one
>> instance of *"byte[]"* loaded by *"<system class loader>"*.
>>
>> *org.apache.kafka.common.network.KafkaChannel -*
>>
>> occupies *352,527,424 (24.96%)* bytes. The memory is accumulated in one
>> instance of *"byte[]"* loaded by *"<system class loader>"*
>>
>> * --------------------- *
>>
>> Both of these were holding about 352MB of space. 3 such instances, so
>> they were consuming about 1.2GB of memory.
>>
>> Now regarding usage of producers. Not a huge amount of logs are being
>> sent to Kafka cluster. It is about 200 msgs/sec. Only one producer object
>> is being used throughout application. Async send function is used.
>>
>> What could be the cause of such huge memory usage? Is this some sort of
>> memory leak in this specific Kafka version?
>>
>>
Re: Kafka producer huge memory usage (leak?)
Posted by Shantanu Deshmukh <sh...@gmail.com>.
Additionally, here's the producer config
kafka.bootstrap.servers=x.x.x.x:9092,x.x.x.x:9092,x.x.x.x:9092
kafka.acks=0
kafka.key.serializer=org.apache.kafka.common.serialization.StringSerializer
kafka.value.serializer=org.apache.kafka.common.serialization.StringSerializer
kafka.max.block.ms=1000
kafka.request.timeout.ms=1000
kafka.max.in.flight.requests.per.connection=1
kafka.retries=0
kafka.compression.type=gzip
kafka.security.protocol=SSL
kafka.ssl.truststore.location=/data/kafka/kafka-server-truststore.jks
kafka.ssl.truststore.password=XXXXXX
kafka.linger.ms=300
logger.level=INFO
On Tue, Sep 18, 2018 at 5:36 PM Shantanu Deshmukh <sh...@gmail.com>
wrote:
> Hello,
>
> We have a 3 broker Kafka 0.10.1.0 deployment in production. There are some
> applications which have Kafka Producers embedded in them which send
> application logs to a topic. This topic has 10 partitions with replication
> factor of 3.
>
> We are observing that memory usage on some of these application servers
> keep shooting through the roof intermittently. After taking heapdump we
> found out that top suspects were:
> *---------------------*
>
>
> *org.apache.kafka.common.network.Selector -*
>
> occupies *352,519,104 (24.96%)* bytes. The memory is accumulated in one
> instance of *"byte[]"* loaded by *"<system class loader>"*.
>
> *org.apache.kafka.common.network.KafkaChannel -*
>
> occupies *352,527,424 (24.96%)* bytes. The memory is accumulated in one
> instance of *"byte[]"* loaded by *"<system class loader>"*
>
> * --------------------- *
>
> Both of these were holding about 352MB of space. 3 such instances, so they
> were consuming about 1.2GB of memory.
>
> Now regarding usage of producers. Not a huge amount of logs are being sent
> to Kafka cluster. It is about 200 msgs/sec. Only one producer object is
> being used throughout application. Async send function is used.
>
> What could be the cause of such huge memory usage? Is this some sort of
> memory leak in this specific Kafka version?
>
>