You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Franz van Betteraey <fv...@web.de> on 2019/02/27 09:54:09 UTC

Load balance Kafka record consumption when using topic pattern

Hi all,

I habe a question: If two KafkaConsumer which uses the same consumer group (group.id) subscribe to the same topicPattern - will Kafka than load balance the resolved topics (! - not only partitions) between the two processes?

In my case about 400 topics match the pattern and each topic has one partion. I wonder if I can load balance the consumption of the topics via different processes/threads with consumers that belong to the same consumer group but it seems that only one consumer handles all the topics (and thus all the partitions). Is this because only partitions are load balanced - or maybe because a topic pattern is used by the subscription?

I made some deeper debugging. My test setup is two threads with an own Kafka Consumer. The Kafka Consumer share the same goupId.
Debug printing the topic description outputs something like this (IPs are obfuscated):

(groupId=foo-bar-group, isSimpleConsumerGroup=false,
  members=
   (memberId=KafkaTestApplication:KafkaTestApplication-1-20126d55-678d-46f5-bc73-3769db2c8901,
     clientId=KafkaTestApplication:KafkaTestApplication-1,
     host=/aa.bb.c.dd,
     assignment=(topicPartitions= <all topics here>)),
   (memberId=KafkaTestApplication:KafkaTestApplication-2-8d34e81a-8f57-4d4e-bd9e-b56edc4e706c,
     clientId=KafkaTestApplication:KafkaTestApplication-2,
     host=/aa.bb.c.dd,
     assignment=(topicPartitions=)),
     partitionAssignor=range,
     state=Stable,
     coordinator=aaa.bb.ccc.d:9092 (id: 9 rack: null)
)

I can see that I have two different members in the consumer group with two different clientIds. Nevertheless all topics are assigned to the first consumer and the second consumer has no topics assigned. I expected that both members get some topics. Both members poll every few seconds.

I also ask this question on StaockOverflow:
https://stackoverflow.com/questions/54594544/load-balance-kafka-record-consumption-when-using-topic-pattern

Any information to understand the behaviour would be of great help for me.

Thanks in advance and best regards
  Franz

Aw: Load balance Kafka record consumption when using topic pattern

Posted by Franz van Betteraey <fv...@web.de>.
Hi again,

I finally found out what to change to get the desired topic/partition balancing. I had to set the Consumer property

> partition.assignment.strategy=org.apache.kafka.clients.consumer.RoundRobinAssignor

As I found out this is the recommended setting when using MirrorMaker and in fact I have a MirrorMaker Job with this configuration which worked as expected. So does my application now ;-)

Best regards,
  Franz
 

Gesendet: Mittwoch, 27. Februar 2019 um 10:54 Uhr
Von: "Franz van Betteraey" <fv...@web.de>
An: users@kafka.apache.org
Betreff: Load balance Kafka record consumption when using topic pattern
Hi all,

I habe a question: If two KafkaConsumer which uses the same consumer group (group.id) subscribe to the same topicPattern - will Kafka than load balance the resolved topics (! - not only partitions) between the two processes?

In my case about 400 topics match the pattern and each topic has one partion. I wonder if I can load balance the consumption of the topics via different processes/threads with consumers that belong to the same consumer group but it seems that only one consumer handles all the topics (and thus all the partitions). Is this because only partitions are load balanced - or maybe because a topic pattern is used by the subscription?

I made some deeper debugging. My test setup is two threads with an own Kafka Consumer. The Kafka Consumer share the same goupId.
Debug printing the topic description outputs something like this (IPs are obfuscated):

(groupId=foo-bar-group, isSimpleConsumerGroup=false,
members=
(memberId=KafkaTestApplication:KafkaTestApplication-1-20126d55-678d-46f5-bc73-3769db2c8901,
clientId=KafkaTestApplication:KafkaTestApplication-1,
host=/aa.bb.c.dd,
assignment=(topicPartitions= <all topics here>)),
(memberId=KafkaTestApplication:KafkaTestApplication-2-8d34e81a-8f57-4d4e-bd9e-b56edc4e706c,
clientId=KafkaTestApplication:KafkaTestApplication-2,
host=/aa.bb.c.dd,
assignment=(topicPartitions=)),
partitionAssignor=range,
state=Stable,
coordinator=aaa.bb.ccc.d:9092 (id: 9 rack: null)
)

I can see that I have two different members in the consumer group with two different clientIds. Nevertheless all topics are assigned to the first consumer and the second consumer has no topics assigned. I expected that both members get some topics. Both members poll every few seconds.

I also ask this question on StaockOverflow:
https://stackoverflow.com/questions/54594544/load-balance-kafka-record-consumption-when-using-topic-pattern

Any information to understand the behaviour would be of great help for me.

Thanks in advance and best regards
Franz