You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Kris K <sq...@gmail.com> on 2015/07/16 22:40:15 UTC

consumer memory footprint

Hi All,

Is there a way to calculate the amount of memory used per thread in case of
a high level consumer?

I am particularly interested in calculating the memory required by a
process running 10 high level consumer threads for 15 topics with max. file
size set to 100 MB.

Thanks,
Kris

Re: consumer memory footprint

Posted by Jaikiran Pai <ja...@gmail.com>.
On Friday 17 July 2015 10:14 AM, Jiangjie Qin wrote:
> I think the rough calculation of max memory footprint for each high level
> consumer would be:
>
> (Number Of Partitions For All Topics) * fetch.message.max.bytes *
> queued.max.message.chunks + (some decompression memory cost for a message)

Is this "Number of Partitions for All Topics in the system" or "Number 
of partitions for the topics that the consumer is requesting to consume 
from" (for example via regex on a topic name filter)?

-Jaikiran



>
> In your case, it would be 10 times above.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
>
> On 7/16/15, 1:40 PM, "Kris K" <sq...@gmail.com> wrote:
>
>> Hi All,
>>
>> Is there a way to calculate the amount of memory used per thread in case
>> of
>> a high level consumer?
>>
>> I am particularly interested in calculating the memory required by a
>> process running 10 high level consumer threads for 15 topics with max.
>> file
>> size set to 100 MB.
>>
>> Thanks,
>> Kris

Re: consumer memory footprint

Posted by Jiangjie Qin <jq...@linkedin.com.INVALID>.
I think the rough calculation of max memory footprint for each high level
consumer would be:

(Number Of Partitions For All Topics) * fetch.message.max.bytes *
queued.max.message.chunks + (some decompression memory cost for a message)

In your case, it would be 10 times above.

Thanks,

Jiangjie (Becket) Qin


On 7/16/15, 1:40 PM, "Kris K" <sq...@gmail.com> wrote:

>Hi All,
>
>Is there a way to calculate the amount of memory used per thread in case
>of
>a high level consumer?
>
>I am particularly interested in calculating the memory required by a
>process running 10 high level consumer threads for 15 topics with max.
>file
>size set to 100 MB.
>
>Thanks,
>Kris

Re: consumer memory footprint

Posted by Jiangjie Qin <jq...@linkedin.com.INVALID>.
I think the rough calculation of max memory footprint for each high level
consumer would be:

(Number Of Partitions For All Topics) * fetch.message.max.bytes *
queued.max.message.chunks + (some decompression memory cost for a message)

In your case, it would be 10 times above.

Thanks,

Jiangjie (Becket) Qin


On 7/16/15, 1:40 PM, "Kris K" <sq...@gmail.com> wrote:

>Hi All,
>
>Is there a way to calculate the amount of memory used per thread in case
>of
>a high level consumer?
>
>I am particularly interested in calculating the memory required by a
>process running 10 high level consumer threads for 15 topics with max.
>file
>size set to 100 MB.
>
>Thanks,
>Kris