[jira] [Commented] (BEAM-12995) Consumer group with random prefix

["Hubert S (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BEAM-12995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17425496#comment-17425496 ] 

Hubert S commented on BEAM-12995:
---------------------------------

Our observatioin shows that two consumers from the same consumer group can consume the same message from topic when we use assign() method instead of subscribe().

Can we achieve the same behaviour like for subscribe() method in or beam pipeline using KafkaIO?

> Consumer group with random prefix
> ---------------------------------
>
>                 Key: BEAM-12995
>                 URL: https://issues.apache.org/jira/browse/BEAM-12995
>             Project: Beam
>          Issue Type: Bug
>          Components: io-java-kafka
>    Affects Versions: 2.32.0
>            Reporter: Hubert S
>            Priority: P2
>
> I'm running two beam pipelines on Apache Flink which read messages from the same Kafka topic by using KafkaIO. Both of them are in the same consumer group, so in my understanding each message should be consumed by either pipeline (but not both) but I observed that the same set of messages is consumed by each pipeline, meaning each message is processed at least twice. 
> I can see in the log that additional consumer group with prefix "Reader-1_offset_consumer_<random_number>_" was created and also consume messages. 
> Could you explain what should I do to consume messages only once by one consumer group and why that additional group was created?
> Similar issue: BEAM-12880



--
This message was sent by Atlassian Jira
(v8.3.4#803005)