High-level consumer load-balancing problem

["hsy541@gmail.com" <hs...@gmail.com>]

Hi,

I have questions about the load balancing of kafka high-level consumer
Suppose I have 4 partition
And the producer throughput to these 4 partitions are like this
0                    1                   2                3
10MB/s          10MB/s          1MB/s        1MB/s
1kMsg/s,        10kMsg/s       1kMsg/s     10kMsg/s

And I have only 2 consumers, what partitions will the 2 consumers pick up?

If the throughput is changing dynamically, will the consumer re-balancing
accordingly?

Thanks,
Siyuan

Re: High-level consumer load-balancing problem

[Jun Rao <ju...@gmail.com>]

The consumer load balancing logic today is pretty simple. It just tries to
divide the partitions evenly among the consumers. It doesn't try to balance
by load.

Thanks,

Jun


On Thu, Nov 14, 2013 at 3:34 PM, hsy541@gmail.com <hs...@gmail.com> wrote:

> Hi,
>
> I have questions about the load balancing of kafka high-level consumer
> Suppose I have 4 partition
> And the producer throughput to these 4 partitions are like this
> 0                    1                   2                3
> 10MB/s          10MB/s          1MB/s        1MB/s
> 1kMsg/s,        10kMsg/s       1kMsg/s     10kMsg/s
>
> And I have only 2 consumers, what partitions will the 2 consumers pick up?
>
> If the throughput is changing dynamically, will the consumer re-balancing
> accordingly?
>
> Thanks,
> Siyuan
>