Query regarding Kafka partitions and Consumer rebalancing

["Madhavi Gokana (Vizury)" <ma...@vizury.com>]

Hi,

Currently we trying to configure Kafka in our system for pulling messages
from Queues.

We have multiple consumers( we might want to add consumers if load on one
consumer increases) which need to receive and process messages from a Kafka
queue. Based on my understanding, under a single consumer group, one
partition can be read by only one consumer.

So if we want to make the setup such that no consumer gets over loaded in
any case, what would be the best way to do it.

If we have 6 partitions and 3 consumers which are equally efficient, then
load seems to be distributed equally. Suppose one of the consumers say
Consumer-3, for some reason processes the data 10 times slower, the we
would want to reduce the load on Consumer-3 and equally distribute load on
Consumer-1 and Counsumer-2. We wanted a pull based system which would help
us in reducing the load on a slow consumer.

Please let us know if there is any way to do this? Does kafka have any
alternate implementation in such cases?

Thanks,
Madhavi.

Re: Query regarding Kafka partitions and Consumer rebalancing

[Guozhang Wang <wa...@gmail.com>]

Hi Madhavi,

Dynamically re-balance partitions based on processing efficiency and load
is a bit tricky to do in the current consumer since rebalances will only be
triggered by consumer membership change or topic/partition change. For your
case you would probably stop the slow consumer so that a rebalance will be
triggered to re-distribute partitions to the rest of the consumers.

Guozhang


On Tue, Jul 15, 2014 at 4:35 AM, Madhavi Gokana (Vizury) <
madhavi.gokana@vizury.com> wrote:

> Hi,
>
> Currently we trying to configure Kafka in our system for pulling messages
> from Queues.
>
> We have multiple consumers( we might want to add consumers if load on one
> consumer increases) which need to receive and process messages from a Kafka
> queue. Based on my understanding, under a single consumer group, one
> partition can be read by only one consumer.
>
> So if we want to make the setup such that no consumer gets over loaded in
> any case, what would be the best way to do it.
>
>
If we have 6 partitions and 3 consumers which are equally efficient, then
> load seems to be distributed equally. Suppose one of the consumers say
> Consumer-3, for some reason processes the data 10 times slower, the we
> would want to reduce the load on Consumer-3 and equally distribute load on
> Consumer-1 and Counsumer-2. We wanted a pull based system which would help
> us in reducing the load on a slow consumer.
>
> Please let us know if there is any way to do this? Does kafka have any
> alternate implementation in such cases?
>
> Thanks,
> Madhavi.
>



-- 
-- Guozhang