You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@kafka.apache.org by ? ? <fu...@live.com> on 2018/02/08 08:30:59 UTC

how to enhance Kafka streaming Consumer rate ?

Hi:
I used kafka streaming for real time analysis.
and I put stream_thread_num same with partitions of topic
and set ConsumerConfig.max_poll_records =500
I use foreach  method only in this
but find with large records in kafka. the cosumer LAG is big some times and trigger kafka topic rebalance.
How to how to enhance Kafka streaming Consumer rate ?
________________________________
funkyyj@live.com

Re: how to enhance Kafka streaming Consumer rate ?

Posted by "Matthias J. Sax" <ma...@confluent.io>.

Is your network saturated? If yes, you can try to start more instances
of Kafka Streams instead of running with multiple thread within one
instance to increase available network capacity.


-Matthias

On 2/8/18 12:30 AM, ? ? wrote:
> Hi:
> I used kafka streaming for real time analysis.
> and I put stream_thread_num same with partitions of topic
> and set ConsumerConfig.max_poll_records =500
> I use foreach  method only in this
> but find with large records in kafka. the cosumer LAG is big some times and trigger kafka topic rebalance.
> How to how to enhance Kafka streaming Consumer rate ?
> ________________________________
> funkyyj@live.com
>