You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@beam.apache.org by "Ka Wah WONG (Jira)" <ji...@apache.org> on 2020/05/08 08:19:00 UTC

[jira] [Commented] (BEAM-9760) KafkaIO supports consumer group?

    [ https://issues.apache.org/jira/browse/BEAM-9760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17102363#comment-17102363 ] 

Ka Wah WONG commented on BEAM-9760:
-----------------------------------

Hi [~aromanenko] , my proposal applies on the case when there are multiple partitions in a topic, and multiple consumers in a consumer group that subscribe the same topic.

From my understanding, to get a dynamically assigned partition with group management by Kafka coordinator, it needs to use KafkaConsumer's subscribe method. Using KafkaConsumer's assign method would manually assign a partition to the consumer and does not use the consumer's group management functionality. (Reference: Javaodc of org.apache.kafka.clients.consumer.KafkaConsumer).

If I want to have two separate Java applications both using Apache Beam subscribing to the same Kafka topic through KafkaIO with same consumer group defined, I would like there can be failover feature supported.

For example, when topic-partition-0 is assigned to App-0, and topic-partition-1 is assigned to App-1, if App-0 is down, the topic-partition-0 is then assigned to the App-1 by the Kafka coordinator / broker. Then App-1 subscribes message from both of topic-partition-0 and topic-partition-1 when App-0 is down. Note that App-0 and App-1 defines same consumer group and subscribes same topic with the 2 partitions.

 

> KafkaIO supports consumer group?
> --------------------------------
>
>                 Key: BEAM-9760
>                 URL: https://issues.apache.org/jira/browse/BEAM-9760
>             Project: Beam
>          Issue Type: Improvement
>          Components: io-java-kafka
>            Reporter: Ka Wah WONG
>            Priority: Minor
>
> It seems only assign method of Kafka Consumer class is called in org.apache.beam.sdk.io.kafka.ConsumerSpEL class. According to documentation of org.apache.kafka.clients.consumer.KafkaConsumer,  manual topic assignment through this assign method does not use the consumer's group management functionality.
> May I ask if KafkaIO will be enhanced to support consumer's group management with using Kafka consumer's subscribe method?
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)