You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by "Johansson, Olof" <Ol...@thingworx.com> on 2015/06/10 18:32:10 UTC
Kafka 0.8.3 - New Consumer - user implemented
partition.assignment.strategies?
Is it possible to have a consumer rebalancing partition assignment strategy that will rebalance with a minimal number of reassignments?
Per the "Kafka 0.9 Consumer Rewrite Design" it should be possible to define my own partition assignment strategy: "partition.assignment.strategies - may take a comma separated list of properties that map the strategy's friendly name to to the class that implements the strategy. This is used for any strategy implemented by the user and released to the Kafka cluster. By default, Kafka will include a set of strategies that can be used by the consumer."
Is there a Jira ticket that tracks adding user defined partitions.assignment.strategies? In the latest source, range, and roundrobin are still the only possible values (hard-coded).
I assume that any user implemented strategy would have to implement the PartitionAssignor trait. If so, by naively looking at the 0.8.3 source, a strategy that should do a minimal number of partition reassignments would need the ConsumerMetaData. That's not currently available in the PartitionAssignor contract - assign(topicsPerConsumer, partitionsPerTopic). Have there been any discussion to change the contract to pass ConsumerMetaData?
Re: Kafka 0.8.3 - New Consumer - user implemented
partition.assignment.strategies?
Posted by Jiangjie Qin <jq...@linkedin.com.INVALID>.
Hi Olof,
I am just wondering what is the benefit of rebalancing with minimal number
of reassignments here?
I am asking this because in new consumer, the rebalance actually is quite
cheap on the consumer side - just updating a topic partition set. That
means the actually rebalance time on consumer side is almost ignorable no
matter how many partitions are reassigned.
Is it because the consumer has some sort of sticking partition
requirements? If that is the case, that seems need an immutable partition
assignment policy.
Just curious about the motivation behind this.
Thanks,
Jiangjie (Becket) Qin
On 6/11/15, 9:30 AM, "Johansson, Olof" <Ol...@thingworx.com>
wrote:
>Thanks Guozhang,
>
>I agree that it seems to be a common reassignment strategy that should be
>one of the pre-defined strategies. Do you have a Jira ticket for this
>specific case, and/or a Jira ticket to add user defined
>partitions.assignment.strategies that I can watch?
>
>/ Olof
>
>On 10/06/2015 14:35, "Guozhang Wang" <wa...@gmail.com> wrote:
>
>>Hi Olof,
>>
>>Yes we have plans to allow user defined partitions.assignment strategy to
>>pass in to the consumer coordinator; I am not sure if this feature will
>>not
>>be available in the first release of the new consumer in 0.8.3 though.
>>Currently users still have to choose one from the server-defined strategy
>>upon registering themselves.
>>
>>On the other hand, I think "rebalance with a minimal number of
>>reassignment" should be quite a common reassignment strategy, and I think
>>it is possible to just add it into the server-defined strategies list.
>>
>>Guozhang
>>
>>
>>On Wed, Jun 10, 2015 at 9:32 AM, Johansson, Olof <
>>Olof.Johansson@thingworx.com> wrote:
>>
>>> Is it possible to have a consumer rebalancing partition assignment
>>> strategy that will rebalance with a minimal number of reassignments?
>>>
>>> Per the "Kafka 0.9 Consumer Rewrite Design" it should be possible to
>>> define my own partition assignment strategy:
>>> "partition.assignment.strategies - may take a comma separated list of
>>> properties that map the strategy's friendly name to to the class that
>>> implements the strategy. This is used for any strategy implemented by
>>>the
>>> user and released to the Kafka cluster. By default, Kafka will include
>>>a
>>> set of strategies that can be used by the consumer."
>>>
>>> Is there a Jira ticket that tracks adding user defined
>>> partitions.assignment.strategies? In the latest source, range, and
>>> roundrobin are still the only possible values (hard-coded).
>>>
>>> I assume that any user implemented strategy would have to implement the
>>> PartitionAssignor trait. If so, by naively looking at the 0.8.3 source,
>>>a
>>> strategy that should do a minimal number of partition reassignments
>>>would
>>> need the ConsumerMetaData. That's not currently available in the
>>> PartitionAssignor contract - assign(topicsPerConsumer,
>>>partitionsPerTopic).
>>> Have there been any discussion to change the contract to pass
>>> ConsumerMetaData?
>>>
>>
>>
>>
>>--
>>-- Guozhang
>
Re: Kafka 0.8.3 - New Consumer - user implemented
partition.assignment.strategies?
Posted by "Johansson, Olof" <Ol...@thingworx.com>.
Thanks Guozhang,
I agree that it seems to be a common reassignment strategy that should be
one of the pre-defined strategies. Do you have a Jira ticket for this
specific case, and/or a Jira ticket to add user defined
partitions.assignment.strategies that I can watch?
/ Olof
On 10/06/2015 14:35, "Guozhang Wang" <wa...@gmail.com> wrote:
>Hi Olof,
>
>Yes we have plans to allow user defined partitions.assignment strategy to
>pass in to the consumer coordinator; I am not sure if this feature will
>not
>be available in the first release of the new consumer in 0.8.3 though.
>Currently users still have to choose one from the server-defined strategy
>upon registering themselves.
>
>On the other hand, I think "rebalance with a minimal number of
>reassignment" should be quite a common reassignment strategy, and I think
>it is possible to just add it into the server-defined strategies list.
>
>Guozhang
>
>
>On Wed, Jun 10, 2015 at 9:32 AM, Johansson, Olof <
>Olof.Johansson@thingworx.com> wrote:
>
>> Is it possible to have a consumer rebalancing partition assignment
>> strategy that will rebalance with a minimal number of reassignments?
>>
>> Per the "Kafka 0.9 Consumer Rewrite Design" it should be possible to
>> define my own partition assignment strategy:
>> "partition.assignment.strategies - may take a comma separated list of
>> properties that map the strategy's friendly name to to the class that
>> implements the strategy. This is used for any strategy implemented by
>>the
>> user and released to the Kafka cluster. By default, Kafka will include a
>> set of strategies that can be used by the consumer."
>>
>> Is there a Jira ticket that tracks adding user defined
>> partitions.assignment.strategies? In the latest source, range, and
>> roundrobin are still the only possible values (hard-coded).
>>
>> I assume that any user implemented strategy would have to implement the
>> PartitionAssignor trait. If so, by naively looking at the 0.8.3 source,
>>a
>> strategy that should do a minimal number of partition reassignments
>>would
>> need the ConsumerMetaData. That's not currently available in the
>> PartitionAssignor contract - assign(topicsPerConsumer,
>>partitionsPerTopic).
>> Have there been any discussion to change the contract to pass
>> ConsumerMetaData?
>>
>
>
>
>--
>-- Guozhang
Re: Kafka 0.8.3 - New Consumer - user implemented partition.assignment.strategies?
Posted by Guozhang Wang <wa...@gmail.com>.
Hi Olof,
Yes we have plans to allow user defined partitions.assignment strategy to
pass in to the consumer coordinator; I am not sure if this feature will not
be available in the first release of the new consumer in 0.8.3 though.
Currently users still have to choose one from the server-defined strategy
upon registering themselves.
On the other hand, I think "rebalance with a minimal number of
reassignment" should be quite a common reassignment strategy, and I think
it is possible to just add it into the server-defined strategies list.
Guozhang
On Wed, Jun 10, 2015 at 9:32 AM, Johansson, Olof <
Olof.Johansson@thingworx.com> wrote:
> Is it possible to have a consumer rebalancing partition assignment
> strategy that will rebalance with a minimal number of reassignments?
>
> Per the "Kafka 0.9 Consumer Rewrite Design" it should be possible to
> define my own partition assignment strategy:
> "partition.assignment.strategies - may take a comma separated list of
> properties that map the strategy's friendly name to to the class that
> implements the strategy. This is used for any strategy implemented by the
> user and released to the Kafka cluster. By default, Kafka will include a
> set of strategies that can be used by the consumer."
>
> Is there a Jira ticket that tracks adding user defined
> partitions.assignment.strategies? In the latest source, range, and
> roundrobin are still the only possible values (hard-coded).
>
> I assume that any user implemented strategy would have to implement the
> PartitionAssignor trait. If so, by naively looking at the 0.8.3 source, a
> strategy that should do a minimal number of partition reassignments would
> need the ConsumerMetaData. That's not currently available in the
> PartitionAssignor contract - assign(topicsPerConsumer, partitionsPerTopic).
> Have there been any discussion to change the contract to pass
> ConsumerMetaData?
>
--
-- Guozhang