You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Kam Kasravi <ka...@yahoo.com> on 2013/09/10 22:03:30 UTC
consumer partition rebalancing
I wasreviewingtheconsumer partition rebalancingalgorithm and had a fewrelatedquestions
* Assuming 1) theconsumerdoesn'tcontrolthe partition allocation within a topic and 2) theconstraintthat a single consumer C(i) within a consumer group C(g) must be the only reader of that partition:
* Are there ways to scale partition consumption if C(i) cannot keep up?
* A way to designate particular consumers should consume particular partitions based on consumer capability (similar to container allocation in YARN)?
* A way to designate sticky consumer / partition allocation so that a new consumer C(j) or new broker B(k) or new topic T(l) does not cause C(i) to lose particular partitions during rebalancing?
Thanks
Kam
Re: consumer partition rebalancing
Posted by Kam Kasravi <ka...@yahoo.com>.
Actually an intermediate consumer would be useful for new topics, in which case it would probably need to watch the relevant zookeeper nodes. I could see a number of use cases where consumption strategies across a consumer group would deviate from the resident partition balancing algorithm. Perhaps this starts to get into Samza territory.
On Sep 11, 2013, at 1:14 PM, Joel Koshy <jj...@gmail.com> wrote:
> Correct - but since you wanted sticky allocation rebalancing wouldn't
> really be necessary.
>
> Thanks,
>
> Joel
>
> On Wed, Sep 11, 2013 at 10:08 AM, Kam Kasravi <ka...@yahoo.com> wrote:
>> Thanks Joel. Just to be sure - SimpleConsumer (or an IntermediateConsumer as suggested below) does not join in partition rebalancing when another consumer in it's group joins or a new broker joins or a new topic is created?
>>
>>
>> ________________________________
>> From: Joel Koshy <jj...@gmail.com>
>> To: "users@kafka.apache.org" <us...@kafka.apache.org>; Kam Kasravi <ka...@yahoo.com>
>> Sent: Tuesday, September 10, 2013 4:36 PM
>> Subject: Re: consumer partition rebalancing
>>
>>
>>> * Assuming 1) theconsumerdoesn'tcontrolthe partition allocation within a topic and 2) theconstraintthat a single consumer C(i) within a consumer group C(g) must be the only reader of that partition:
>>> * Are there ways to scale partition consumption if C(i) cannot keep up?
>>
>> To some degree - by either reducing the number of streams that C(i) is
>> configured with; or by removing it altogether.
>>
>>> * A way to designate particular consumers should consume particular partitions based on consumer capability (similar to container allocation in YARN)?
>>> * A way to designate sticky consumer / partition allocation so that a new consumer C(j) or new broker B(k) or new topic T(l) does not cause C(i) to lose particular partitions during rebalancing?
>>
>> Right now, no. I think what you may be effectively suggesting is to
>> support some form of pluggable logic for partition ownership which
>> should work as long as it is consistent across all consumer instances.
>> However, a designated assignment list could be achieved by using
>> SimpleConsumer - although you would then need to write a little code
>> to react to leader changes. (We currently don't have a
>> intermediate-level consumer - i.e., a consumer that can be designated
>> to consume from a given partition and automatically react to leader
>> changes.)
>>
>> Joel
Re: consumer partition rebalancing
Posted by Joel Koshy <jj...@gmail.com>.
Correct - but since you wanted sticky allocation rebalancing wouldn't
really be necessary.
Thanks,
Joel
On Wed, Sep 11, 2013 at 10:08 AM, Kam Kasravi <ka...@yahoo.com> wrote:
> Thanks Joel. Just to be sure - SimpleConsumer (or an IntermediateConsumer as suggested below) does not join in partition rebalancing when another consumer in it's group joins or a new broker joins or a new topic is created?
>
>
> ________________________________
> From: Joel Koshy <jj...@gmail.com>
> To: "users@kafka.apache.org" <us...@kafka.apache.org>; Kam Kasravi <ka...@yahoo.com>
> Sent: Tuesday, September 10, 2013 4:36 PM
> Subject: Re: consumer partition rebalancing
>
>
>> * Assuming 1) theconsumerdoesn'tcontrolthe partition allocation within a topic and 2) theconstraintthat a single consumer C(i) within a consumer group C(g) must be the only reader of that partition:
>> * Are there ways to scale partition consumption if C(i) cannot keep up?
>
> To some degree - by either reducing the number of streams that C(i) is
> configured with; or by removing it altogether.
>
>> * A way to designate particular consumers should consume particular partitions based on consumer capability (similar to container allocation in YARN)?
>> * A way to designate sticky consumer / partition allocation so that a new consumer C(j) or new broker B(k) or new topic T(l) does not cause C(i) to lose particular partitions during rebalancing?
>
> Right now, no. I think what you may be effectively suggesting is to
> support some form of pluggable logic for partition ownership which
> should work as long as it is consistent across all consumer instances.
> However, a designated assignment list could be achieved by using
> SimpleConsumer - although you would then need to write a little code
> to react to leader changes. (We currently don't have a
> intermediate-level consumer - i.e., a consumer that can be designated
> to consume from a given partition and automatically react to leader
> changes.)
>
> Joel
Re: consumer partition rebalancing
Posted by Kam Kasravi <ka...@yahoo.com>.
Thanks Joel. Just to be sure - SimpleConsumer (or an IntermediateConsumer as suggested below) does not join in partition rebalancing when another consumer in it's group joins or a new broker joins or a new topic is created?
________________________________
From: Joel Koshy <jj...@gmail.com>
To: "users@kafka.apache.org" <us...@kafka.apache.org>; Kam Kasravi <ka...@yahoo.com>
Sent: Tuesday, September 10, 2013 4:36 PM
Subject: Re: consumer partition rebalancing
> * Assuming 1) theconsumerdoesn'tcontrolthe partition allocation within a topic and 2) theconstraintthat a single consumer C(i) within a consumer group C(g) must be the only reader of that partition:
> * Are there ways to scale partition consumption if C(i) cannot keep up?
To some degree - by either reducing the number of streams that C(i) is
configured with; or by removing it altogether.
> * A way to designate particular consumers should consume particular partitions based on consumer capability (similar to container allocation in YARN)?
> * A way to designate sticky consumer / partition allocation so that a new consumer C(j) or new broker B(k) or new topic T(l) does not cause C(i) to lose particular partitions during rebalancing?
Right now, no. I think what you may be effectively suggesting is to
support some form of pluggable logic for partition ownership which
should work as long as it is consistent across all consumer instances.
However, a designated assignment list could be achieved by using
SimpleConsumer - although you would then need to write a little code
to react to leader changes. (We currently don't have a
intermediate-level consumer - i.e., a consumer that can be designated
to consume from a given partition and automatically react to leader
changes.)
Joel
Re: consumer partition rebalancing
Posted by Joel Koshy <jj...@gmail.com>.
> * Assuming 1) theconsumerdoesn'tcontrolthe partition allocation within a topic and 2) theconstraintthat a single consumer C(i) within a consumer group C(g) must be the only reader of that partition:
> * Are there ways to scale partition consumption if C(i) cannot keep up?
To some degree - by either reducing the number of streams that C(i) is
configured with; or by removing it altogether.
> * A way to designate particular consumers should consume particular partitions based on consumer capability (similar to container allocation in YARN)?
> * A way to designate sticky consumer / partition allocation so that a new consumer C(j) or new broker B(k) or new topic T(l) does not cause C(i) to lose particular partitions during rebalancing?
Right now, no. I think what you may be effectively suggesting is to
support some form of pluggable logic for partition ownership which
should work as long as it is consistent across all consumer instances.
However, a designated assignment list could be achieved by using
SimpleConsumer - although you would then need to write a little code
to react to leader changes. (We currently don't have a
intermediate-level consumer - i.e., a consumer that can be designated
to consume from a given partition and automatically react to leader
changes.)
Joel