You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Ярослав Когут <sl...@gmail.com> on 2019/04/11 07:59:20 UTC
Stream caching
Hi guys.
I’ve deployed spring cloud application using docker, service is using KStream functionality, and I have problems with /tmp folder.
On some instances of application /tmp folder is from 3 to 6 GB. Maybe I missed some property?
I will be very grateful for the help.
Thanks, best regards.
Kohut Yaroslav
Re: Stream caching
Posted by Ярослав Когут <sl...@gmail.com>.
Hello Guozhang!
I’m talking about disk usage, about /tmp/kafka-streams.
My application merges 3 topics, one of them has 120 partitions.
As i understood from documentation(https://docs.confluent.io/current/streams/sizing.html), disk usage depends on partition count. And I really want to decrease size of the tmp folder or try to control it.
Thank you.
Kohut Yaroslav
> On Apr 12, 2019, at 7:08 PM, Guozhang Wang <wa...@gmail.com> wrote:
>
> Hello Kohut,
>
> Is your issue about disk usage, or memory? The first email seems to
> indicate disks (there's no direct way for bounding the disk usage of state
> stores, though you can set e.g. retention period to indirectly control disk
> usage via specifying how long do you want to retain windowed stores), while
> the later seems to indicate memory (btw maybe this can help a bit:
> https://kafka.apache.org/documentation/streams/developer-guide/memory-mgmt.html <https://kafka.apache.org/documentation/streams/developer-guide/memory-mgmt.html>
> )
>
>
> Guozhang
>
> On Fri, Apr 12, 2019 at 2:49 AM Ярослав Когут <slavka.kogut@gmail.com <ma...@gmail.com>>
> wrote:
>
>> And I have a lot of warning that describe in this lira
>>
>> https://issues.apache.org/jira/browse/KAFKA-5998?attachmentOrder=asc <https://issues.apache.org/jira/browse/KAFKA-5998?attachmentOrder=asc> <
>> https://issues.apache.org/jira/browse/KAFKA-5998?attachmentOrder=asc <https://issues.apache.org/jira/browse/KAFKA-5998?attachmentOrder=asc>>
>>
>> Thanks.
>>
>> Kohut Yaroslav
>>
>>> On Apr 12, 2019, at 10:59 AM, Ярослав Когут <sl...@gmail.com>
>> wrote:
>>>
>>> Hello, yes I know it, I read about rocks db.
>>>
>>> But do we have ability to make some size limitation for this db? Because
>> we have alerts in grafana, that application is using more then 80% of full
>> app memory.
>>>
>>> I’ll be very grateful for the help.
>>>
>>> Thanks.
>>>
>>> Kohut Yaroslav
>>>
>>>> On Apr 12, 2019, at 4:02 AM, Guozhang Wang <wa...@gmail.com> wrote:
>>>>
>>>> Hello Kohut,
>>>>
>>>> Streams's default state.dir location is /tmp/kafka-streams, that may
>>>> explained it.
>>>>
>>>> For more info about state management of Kafka Streams, you can read
>>>>
>> https://kafka.apache.org/21/documentation/streams/architecture#streams_architecture_state
>>>>
>>>>
>>>> Guozhang
>>>>
>>>> On Thu, Apr 11, 2019 at 1:09 AM Ярослав Когут <sl...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi guys.
>>>>>
>>>>> I’ve deployed spring cloud application using docker, service is using
>>>>> KStream functionality, and I have problems with /tmp folder.
>>>>>
>>>>> On some instances of application /tmp folder is from 3 to 6 GB. Maybe I
>>>>> missed some property?
>>>>>
>>>>> I will be very grateful for the help.
>>>>>
>>>>> Thanks, best regards.
>>>>>
>>>>> Kohut Yaroslav
>>>>
>>>>
>>>>
>>>> --
>>>> -- Guozhang
>>>
>>
>>
>
> --
> -- Guozhang
Re: Stream caching
Posted by Guozhang Wang <wa...@gmail.com>.
Hello Kohut,
Is your issue about disk usage, or memory? The first email seems to
indicate disks (there's no direct way for bounding the disk usage of state
stores, though you can set e.g. retention period to indirectly control disk
usage via specifying how long do you want to retain windowed stores), while
the later seems to indicate memory (btw maybe this can help a bit:
https://kafka.apache.org/documentation/streams/developer-guide/memory-mgmt.html
)
Guozhang
On Fri, Apr 12, 2019 at 2:49 AM Ярослав Когут <sl...@gmail.com>
wrote:
> And I have a lot of warning that describe in this lira
>
> https://issues.apache.org/jira/browse/KAFKA-5998?attachmentOrder=asc <
> https://issues.apache.org/jira/browse/KAFKA-5998?attachmentOrder=asc>
>
> Thanks.
>
> Kohut Yaroslav
>
> > On Apr 12, 2019, at 10:59 AM, Ярослав Когут <sl...@gmail.com>
> wrote:
> >
> > Hello, yes I know it, I read about rocks db.
> >
> > But do we have ability to make some size limitation for this db? Because
> we have alerts in grafana, that application is using more then 80% of full
> app memory.
> >
> > I’ll be very grateful for the help.
> >
> > Thanks.
> >
> > Kohut Yaroslav
> >
> >> On Apr 12, 2019, at 4:02 AM, Guozhang Wang <wa...@gmail.com> wrote:
> >>
> >> Hello Kohut,
> >>
> >> Streams's default state.dir location is /tmp/kafka-streams, that may
> >> explained it.
> >>
> >> For more info about state management of Kafka Streams, you can read
> >>
> https://kafka.apache.org/21/documentation/streams/architecture#streams_architecture_state
> >>
> >>
> >> Guozhang
> >>
> >> On Thu, Apr 11, 2019 at 1:09 AM Ярослав Когут <sl...@gmail.com>
> >> wrote:
> >>
> >>> Hi guys.
> >>>
> >>> I’ve deployed spring cloud application using docker, service is using
> >>> KStream functionality, and I have problems with /tmp folder.
> >>>
> >>> On some instances of application /tmp folder is from 3 to 6 GB. Maybe I
> >>> missed some property?
> >>>
> >>> I will be very grateful for the help.
> >>>
> >>> Thanks, best regards.
> >>>
> >>> Kohut Yaroslav
> >>
> >>
> >>
> >> --
> >> -- Guozhang
> >
>
>
--
-- Guozhang
Re: Stream caching
Posted by Ярослав Когут <sl...@gmail.com>.
And I have a lot of warning that describe in this lira
https://issues.apache.org/jira/browse/KAFKA-5998?attachmentOrder=asc <https://issues.apache.org/jira/browse/KAFKA-5998?attachmentOrder=asc>
Thanks.
Kohut Yaroslav
> On Apr 12, 2019, at 10:59 AM, Ярослав Когут <sl...@gmail.com> wrote:
>
> Hello, yes I know it, I read about rocks db.
>
> But do we have ability to make some size limitation for this db? Because we have alerts in grafana, that application is using more then 80% of full app memory.
>
> I’ll be very grateful for the help.
>
> Thanks.
>
> Kohut Yaroslav
>
>> On Apr 12, 2019, at 4:02 AM, Guozhang Wang <wa...@gmail.com> wrote:
>>
>> Hello Kohut,
>>
>> Streams's default state.dir location is /tmp/kafka-streams, that may
>> explained it.
>>
>> For more info about state management of Kafka Streams, you can read
>> https://kafka.apache.org/21/documentation/streams/architecture#streams_architecture_state
>>
>>
>> Guozhang
>>
>> On Thu, Apr 11, 2019 at 1:09 AM Ярослав Когут <sl...@gmail.com>
>> wrote:
>>
>>> Hi guys.
>>>
>>> I’ve deployed spring cloud application using docker, service is using
>>> KStream functionality, and I have problems with /tmp folder.
>>>
>>> On some instances of application /tmp folder is from 3 to 6 GB. Maybe I
>>> missed some property?
>>>
>>> I will be very grateful for the help.
>>>
>>> Thanks, best regards.
>>>
>>> Kohut Yaroslav
>>
>>
>>
>> --
>> -- Guozhang
>
Re: Stream caching
Posted by Ярослав Когут <sl...@gmail.com>.
Hello, yes I know it, I read about rocks db.
But do we have ability to make some size limitation for this db? Because we have alerts in grafana, that application is using more then 80% of full app memory.
I’ll be very grateful for the help.
Thanks.
Kohut Yaroslav
> On Apr 12, 2019, at 4:02 AM, Guozhang Wang <wa...@gmail.com> wrote:
>
> Hello Kohut,
>
> Streams's default state.dir location is /tmp/kafka-streams, that may
> explained it.
>
> For more info about state management of Kafka Streams, you can read
> https://kafka.apache.org/21/documentation/streams/architecture#streams_architecture_state
>
>
> Guozhang
>
> On Thu, Apr 11, 2019 at 1:09 AM Ярослав Когут <sl...@gmail.com>
> wrote:
>
>> Hi guys.
>>
>> I’ve deployed spring cloud application using docker, service is using
>> KStream functionality, and I have problems with /tmp folder.
>>
>> On some instances of application /tmp folder is from 3 to 6 GB. Maybe I
>> missed some property?
>>
>> I will be very grateful for the help.
>>
>> Thanks, best regards.
>>
>> Kohut Yaroslav
>
>
>
> --
> -- Guozhang
Re: Stream caching
Posted by Guozhang Wang <wa...@gmail.com>.
Hello Kohut,
Streams's default state.dir location is /tmp/kafka-streams, that may
explained it.
For more info about state management of Kafka Streams, you can read
https://kafka.apache.org/21/documentation/streams/architecture#streams_architecture_state
Guozhang
On Thu, Apr 11, 2019 at 1:09 AM Ярослав Когут <sl...@gmail.com>
wrote:
> Hi guys.
>
> I’ve deployed spring cloud application using docker, service is using
> KStream functionality, and I have problems with /tmp folder.
>
> On some instances of application /tmp folder is from 3 to 6 GB. Maybe I
> missed some property?
>
> I will be very grateful for the help.
>
> Thanks, best regards.
>
> Kohut Yaroslav
--
-- Guozhang