Multi-topic consumer configuration

[Robert Quinlivan <rq...@signal.co>]

Hello,

We are using a multi-topic approach for partitioning data. Each topic will
follow a pattern naming convention. We want to design our consumer
configuration so that a slow topic will not block a fast topic, as each
topic will consume at a different rate.

However I am seeing an issue where all topics matching the pattern are
assigned to a single consumer. When there are several consumers in the
group with a single topic with many partitions, the partitions are
automatically distributed among the available consumers in the group. I
expected that calling KafkaConsumer#subscribe(Pattern,
ConsumerRebalanceListener) would follow a similar behavior by distributing
the assigned topics among all consumers in the group.

Is this not the case? What is the expected behavior and how would you
recommend implementing this design?

Thank you

-- 
Robert Quinlivan
Software Engineer, Signal

Re: Multi-topic consumer configuration

[Ewen Cheslack-Postava <ew...@confluent.io>]

Do each of your topics have only 1 partition? The default partition
assignor is org.apache.kafka.clients.consumer.RangeAssignor which will give
the behavior you describe if each topic only has a single partition.

You probably want to switch it to RoundRobinAssignor which guarantees a
more even distribution of topic partitions to consumers.

-Ewen

On Wed, Dec 21, 2016 at 10:15 AM, Robert Quinlivan <rq...@signal.co>
wrote:

> Hello,
>
> We are using a multi-topic approach for partitioning data. Each topic will
> follow a pattern naming convention. We want to design our consumer
> configuration so that a slow topic will not block a fast topic, as each
> topic will consume at a different rate.
>
> However I am seeing an issue where all topics matching the pattern are
> assigned to a single consumer. When there are several consumers in the
> group with a single topic with many partitions, the partitions are
> automatically distributed among the available consumers in the group. I
> expected that calling KafkaConsumer#subscribe(Pattern,
> ConsumerRebalanceListener) would follow a similar behavior by distributing
> the assigned topics among all consumers in the group.
>
> Is this not the case? What is the expected behavior and how would you
> recommend implementing this design?
>
> Thank you
>
> --
> Robert Quinlivan
> Software Engineer, Signal
>