You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Ara Ebrahimi <ar...@argyledata.com> on 2016/11/16 22:51:47 UTC

kafka streams rocksdb tuning (or memory leak?)

Hi,

I have a few KTables in my application. Some of them have unlimited windows. If I leave the application to run for a few hours, I see the java process consume more and more memory, way above the -Xmx limit. I understand this is due to the rocksdb native lib used by kafka streams. What I don’t understand is how I can control how much rocksdb data should be kept in memory. I know I can use the RocksDBConfigSetter class, but I need more information on why this unbounded memory allocation happens and how to tune it.

Thanks,
Ara.



________________________________

This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Thank you in advance for your cooperation.

________________________________

Re: kafka streams rocksdb tuning (or memory leak?)

Posted by Guozhang Wang <wa...@gmail.com>.
Hello Ara,

I would love to learn if your rebalance issue mentioned in KAFKA-4392 still
exists, since I think what you observed maybe a combination of various
issues as we discussed offline, which could be related to KIP-62 as well.
Let me know and I'd love to help investigate further.


Guozhang

On Wed, Nov 16, 2016 at 6:25 PM, Ara Ebrahimi <ar...@argyledata.com>
wrote:

> This indeed was the issue and cp2 did fix it! Thank you!
>
> Ara.
>
> > On Nov 16, 2016, at 4:11 PM, Damian Guy <da...@gmail.com> wrote:
> >
> > Hi Ara,
> >
> > Are you running Kafka Streams v0.10.1? If so, there is a resource leak in
> > the window store. See: https://github.com/apache/kafka/pull/2122
> > You can either try checking out the Apache Kafka 0.10.1 branch (which has
> > the fix) and building it manually, or you can try using the latest
> > confluent packaged version of the library, which you can get from
> > confluents maven repository:
> >
> > <repositories>
> >    <repository>
> >        <id>confluent</id>
> >        <url>http://packages.confluent.io/maven/</url>
> >    </repository></repositories>
> >
> > <dependency>
> >    <groupId>org.apache.kafka</groupId>
> >    <artifactId>kafka-streams</artifactId>
> >    <version>0.10.1.0-cp2</version></dependency><dependency>
> >    <groupId>org.apache.kafka</groupId>
> >    <artifactId>kafka-clients</artifactId>
> >    <version>0.10.1.0-cp2</version></dependency>
> >
> > Thanks,
> >
> > Damian
> >
> >
> > On Wed, 16 Nov 2016 at 14:52 Ara Ebrahimi <ar...@argyledata.com>
> > wrote:
> >
> >> Hi,
> >>
> >> I have a few KTables in my application. Some of them have unlimited
> >> windows. If I leave the application to run for a few hours, I see the
> java
> >> process consume more and more memory, way above the -Xmx limit. I
> >> understand this is due to the rocksdb native lib used by kafka streams.
> >> What I don’t understand is how I can control how much rocksdb data
> should
> >> be kept in memory. I know I can use the RocksDBConfigSetter class, but I
> >> need more information on why this unbounded memory allocation happens
> and
> >> how to tune it.
> >>
> >> Thanks,
> >> Ara.
> >>
> >>
> >>
> >> ________________________________
> >>
> >> This message is for the designated recipient only and may contain
> >> privileged, proprietary, or otherwise confidential information. If you
> have
> >> received it in error, please notify the sender immediately and delete
> the
> >> original. Any other use of the e-mail by you is prohibited. Thank you in
> >> advance for your cooperation.
> >>
> >> ________________________________
> >>
> >
> >
> >
> > ________________________________
> >
> > This message is for the designated recipient only and may contain
> privileged, proprietary, or otherwise confidential information. If you have
> received it in error, please notify the sender immediately and delete the
> original. Any other use of the e-mail by you is prohibited. Thank you in
> advance for your cooperation.
> >
> > ________________________________
>
>
>
>
> ________________________________
>
> This message is for the designated recipient only and may contain
> privileged, proprietary, or otherwise confidential information. If you have
> received it in error, please notify the sender immediately and delete the
> original. Any other use of the e-mail by you is prohibited. Thank you in
> advance for your cooperation.
>
> ________________________________
>



-- 
-- Guozhang

Re: kafka streams rocksdb tuning (or memory leak?)

Posted by Ara Ebrahimi <ar...@argyledata.com>.
This indeed was the issue and cp2 did fix it! Thank you!

Ara.

> On Nov 16, 2016, at 4:11 PM, Damian Guy <da...@gmail.com> wrote:
>
> Hi Ara,
>
> Are you running Kafka Streams v0.10.1? If so, there is a resource leak in
> the window store. See: https://github.com/apache/kafka/pull/2122
> You can either try checking out the Apache Kafka 0.10.1 branch (which has
> the fix) and building it manually, or you can try using the latest
> confluent packaged version of the library, which you can get from
> confluents maven repository:
>
> <repositories>
>    <repository>
>        <id>confluent</id>
>        <url>http://packages.confluent.io/maven/</url>
>    </repository></repositories>
>
> <dependency>
>    <groupId>org.apache.kafka</groupId>
>    <artifactId>kafka-streams</artifactId>
>    <version>0.10.1.0-cp2</version></dependency><dependency>
>    <groupId>org.apache.kafka</groupId>
>    <artifactId>kafka-clients</artifactId>
>    <version>0.10.1.0-cp2</version></dependency>
>
> Thanks,
>
> Damian
>
>
> On Wed, 16 Nov 2016 at 14:52 Ara Ebrahimi <ar...@argyledata.com>
> wrote:
>
>> Hi,
>>
>> I have a few KTables in my application. Some of them have unlimited
>> windows. If I leave the application to run for a few hours, I see the java
>> process consume more and more memory, way above the -Xmx limit. I
>> understand this is due to the rocksdb native lib used by kafka streams.
>> What I don’t understand is how I can control how much rocksdb data should
>> be kept in memory. I know I can use the RocksDBConfigSetter class, but I
>> need more information on why this unbounded memory allocation happens and
>> how to tune it.
>>
>> Thanks,
>> Ara.
>>
>>
>>
>> ________________________________
>>
>> This message is for the designated recipient only and may contain
>> privileged, proprietary, or otherwise confidential information. If you have
>> received it in error, please notify the sender immediately and delete the
>> original. Any other use of the e-mail by you is prohibited. Thank you in
>> advance for your cooperation.
>>
>> ________________________________
>>
>
>
>
> ________________________________
>
> This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Thank you in advance for your cooperation.
>
> ________________________________




________________________________

This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Thank you in advance for your cooperation.

________________________________

Re: kafka streams rocksdb tuning (or memory leak?)

Posted by Damian Guy <da...@gmail.com>.
Hi Ara,

Are you running Kafka Streams v0.10.1? If so, there is a resource leak in
the window store. See: https://github.com/apache/kafka/pull/2122
You can either try checking out the Apache Kafka 0.10.1 branch (which has
the fix) and building it manually, or you can try using the latest
confluent packaged version of the library, which you can get from
confluents maven repository:

<repositories>
    <repository>
        <id>confluent</id>
        <url>http://packages.confluent.io/maven/</url>
    </repository></repositories>

<dependency>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka-streams</artifactId>
    <version>0.10.1.0-cp2</version></dependency><dependency>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka-clients</artifactId>
    <version>0.10.1.0-cp2</version></dependency>

Thanks,

Damian


On Wed, 16 Nov 2016 at 14:52 Ara Ebrahimi <ar...@argyledata.com>
wrote:

> Hi,
>
> I have a few KTables in my application. Some of them have unlimited
> windows. If I leave the application to run for a few hours, I see the java
> process consume more and more memory, way above the -Xmx limit. I
> understand this is due to the rocksdb native lib used by kafka streams.
> What I don’t understand is how I can control how much rocksdb data should
> be kept in memory. I know I can use the RocksDBConfigSetter class, but I
> need more information on why this unbounded memory allocation happens and
> how to tune it.
>
> Thanks,
> Ara.
>
>
>
> ________________________________
>
> This message is for the designated recipient only and may contain
> privileged, proprietary, or otherwise confidential information. If you have
> received it in error, please notify the sender immediately and delete the
> original. Any other use of the e-mail by you is prohibited. Thank you in
> advance for your cooperation.
>
> ________________________________
>