Round Robin Partition Assignment

[Bryan Baugher <bj...@gmail.com>]

Hi everyone,

We recently switched to round robin partition assignment after we noticed
that range partition assignment (default) will only make use of the first X
consumers were X is the number of partitions for a topic our consumers are
interested in. We then noticed the caveat in round robin,

"Round-robin assignment is permitted only if: (a) Every topic has the same
number of streams within a consumer instance (b) The set of subscribed
topics is identical for every consumer instance within the group."

We tried this out and found if all consumers don't agree on topic
subscription they basically stop consuming until things get figured out.
This is bad for us since our consumers change their topic subscription
based on config they load from a REST service periodically.

Is there something we can do on our side to avoid this? The best thing I
can think of is to try and use something like zookeeper to coordinate
changing the topic filters.

Would it be possible to see the round robin assignment updated to not
require identical topic subscriptions?

Bryan

Re: Round Robin Partition Assignment

[Jason Rosenberg <jb...@squareup.com>]

I asked about this same issue in a previous thread.  Thanks for reminding
me, I've added this Jira:  https://issues.apache.org/jira/browse/KAFKA-2172

I think this is a great new feature, but it's unfortunately the "all
consumers must be the same" is just a bit too restrictive.

Jason

On Tue, May 5, 2015 at 5:20 PM, Bryan Baugher <bj...@gmail.com> wrote:

> Hi everyone,
>
> We recently switched to round robin partition assignment after we noticed
> that range partition assignment (default) will only make use of the first X
> consumers were X is the number of partitions for a topic our consumers are
> interested in. We then noticed the caveat in round robin,
>
> "Round-robin assignment is permitted only if: (a) Every topic has the same
> number of streams within a consumer instance (b) The set of subscribed
> topics is identical for every consumer instance within the group."
>
> We tried this out and found if all consumers don't agree on topic
> subscription they basically stop consuming until things get figured out.
> This is bad for us since our consumers change their topic subscription
> based on config they load from a REST service periodically.
>
> Is there something we can do on our side to avoid this? The best thing I
> can think of is to try and use something like zookeeper to coordinate
> changing the topic filters.
>
> Would it be possible to see the round robin assignment updated to not
> require identical topic subscriptions?
>
> Bryan
>