You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by aravind s <su...@gmail.com> on 2019/11/14 07:03:29 UTC

Kakfa Broker scaling tips for high number of consumers for a single topic

Hi,

We have a use-case where there are close to 40 producers to a topic with 2
replicas and consumers for this topic are around 2000. We have seen that
producer latency goes by 3 times when consumers grew from 500 to 2000. We
changed the following properties. The machine has sufficient memory. The
rate is around 100k we are targeting.

   - queued.max.requests
   - num.io.threads
   - num.network.threads

But there is no impact. Please, could you suggest how to go about this?

Regards,
Aravind

Re: Kakfa Broker scaling tips for high number of consumers for a single topic

Posted by Xiangyuan LI <fl...@gmail.com>.
hi
I want to know what's the kafka server & client version you're using?
And I want to know ,you said you have 2000 consumer,is that mean 2000
consumer groups or 2000 consumer in one consumer group?


aravind s <su...@gmail.com> 于2019年11月15日周五 上午3:12写道：

> Hi,
>
> We have a use-case where there are close to 40 producers to a topic with 2
> replicas and consumers for this topic are around 2000. We have seen that
> producer latency goes by 3 times when consumers grew from 500 to 2000. We
> changed the following properties. The machine has sufficient memory. The
> rate is around 100k we are targeting.
>
>    - queued.max.requests
>    - num.io.threads
>    - num.network.threads
>
> But there is no impact. Please, could you suggest how to go about this?
>
> Regards,
> Aravind
>

RE: Kakfa Broker scaling tips for high number of consumers for a single topic

Posted by Koushik Chitta <kc...@microsoft.com.INVALID>.
What change did you observe in broker latency metrics 'totaltimems' ?


-----Original Message-----
From: aravind s <su...@gmail.com> 
Sent: Wednesday, November 13, 2019 11:03 PM
To: users@kafka.apache.org
Subject: Kakfa Broker scaling tips for high number of consumers for a single topic

Hi,

We have a use-case where there are close to 40 producers to a topic with 2 replicas and consumers for this topic are around 2000. We have seen that producer latency goes by 3 times when consumers grew from 500 to 2000. We changed the following properties. The machine has sufficient memory. The rate is around 100k we are targeting.

   - queued.max.requests
   - num.io.threads
   - num.network.threads

But there is no impact. Please, could you suggest how to go about this?

Regards,
Aravind