Why does consumer.subscribe(Pattern) require a ConsumerRebalanceListener?

[James Cheng <wu...@gmail.com>]

Hi,

I was looking at the docs for the consumer, and noticed that when calling subscribe() with a regex Pattern, that you are required to pass in a ConsumerRebalanceListener. On the other hand, when you use a fixed set of topic names (Collection<String>), the ConsumerRebalanceListener is optional (that is, there is a subscribe(Collection<String>) that does not require a ConsumerRebalanceListener)

http://kafka.apache.org/0101/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#subscribe(java.util.regex.Pattern,%20org.apache.kafka.clients.consumer.ConsumerRebalanceListener) <http://kafka.apache.org/0101/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#subscribe(java.util.regex.Pattern, org.apache.kafka.clients.consumer.ConsumerRebalanceListener)>

Why does the regex one require a rebalance listener, whereas the fixed-topic one does not? Is it to force the user to think through what happens as new topic/partitions appear and disappear?

-James

Re: Why does consumer.subscribe(Pattern) require a ConsumerRebalanceListener?

[Ewen Cheslack-Postava <ew...@confluent.io>]

Tbh, I can't remember the exact details around the discussion of the
addition of this API, but I think this was to minimize API bloat. It's easy
to end up with 83 overloads of methods to handle all the different
combinations of parameters, but just a couple of shorthand overrides cover
the vast majority of use cases. Only providing 2 or 3 shorthand versions of
methods is good enough and requiring the full parameter list for the rest
isn't overly-burdensome for users.

The assumption here is that anyone using regex subscriptions is an "expert"
user that will also understand rebalance listeners. You still don't need to
provide a real implementation if you don't need it -- the
NoOpConsumerRebalanceListener used for the overrides of subscribe that
don't take rebalance listener arguments should work fine here as well. I
think the only concern is that they are technically in the non-public,
internal package, although an equivalent (empty) implementation would work
just as well if you're worried about API compatibility/stability.

-Ewen

On Tue, Jan 3, 2017 at 6:30 PM, James Cheng <wu...@gmail.com> wrote:

> Hi,
>
> I was looking at the docs for the consumer, and noticed that when calling
> subscribe() with a regex Pattern, that you are required to pass in a
> ConsumerRebalanceListener. On the other hand, when you use a fixed set of
> topic names (Collection<String>), the ConsumerRebalanceListener is optional
> (that is, there is a subscribe(Collection<String>) that does not require a
> ConsumerRebalanceListener)
>
> http://kafka.apache.org/0101/javadoc/org/apache/kafka/clients/consumer/
> KafkaConsumer.html#subscribe(java.util.regex.Pattern,%
> 20org.apache.kafka.clients.consumer.ConsumerRebalanceListener) <
> http://kafka.apache.org/0101/javadoc/org/apache/kafka/clients/consumer/
> KafkaConsumer.html#subscribe(java.util.regex.Pattern,
> org.apache.kafka.clients.consumer.ConsumerRebalanceListener)>
>
> Why does the regex one require a rebalance listener, whereas the
> fixed-topic one does not? Is it to force the user to think through what
> happens as new topic/partitions appear and disappear?
>
> -James
>
>