You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Tzu-Li (Gordon) Tai (JIRA)" <ji...@apache.org> on 2016/10/03 11:01:20 UTC
[jira] [Comment Edited] (FLINK-4722) Consumer group concept not
working properly with FlinkKafkaConsumer09
[ https://issues.apache.org/jira/browse/FLINK-4722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15542145#comment-15542145 ]
Tzu-Li (Gordon) Tai edited comment on FLINK-4722 at 10/3/16 11:00 AM:
----------------------------------------------------------------------
Hi [~sudhanshulenka],
I think this is expected behaviour.
Internally, each source subtask of {{FlinkKafkaConsumer09}} uses {{KafkaConsumer#assign()}} (no consumer group functionality) instead of {{KafkaConsumer#subscribe()}} (has consumer group functionality) to read from topic partitions. So that's why all 3 FlinkKafkaConsumer09 are getting all records.
Right now, we need to internally use {{KafkaConsumer#assign()}} because partition-to-subtask assignment must be determinate to achieve exactly-once guarantees using Flink's checkpointing mechanism.
was (Author: tzulitai):
Hi [~sudhanshulenka],
I think this is expected behaviour.
Internally, each source subtask of {{FlinkKafkaConsumer09}} uses {{KafkaConsumer#assign()}} (no consumer group functionality) instead of {{KafkaConsumer#subscribe()}} (has consumer group functionality). So that's why all 3 FlinkKafkaConsumer09 are getting all records.
Right now, we need to internally use {{KafkaConsumer#assign()}} because partition-to-subtask assignment must be determinate to achieve exactly-once guarantees using Flink's checkpointing mechanism.
> Consumer group concept not working properly with FlinkKafkaConsumer09
> -----------------------------------------------------------------------
>
> Key: FLINK-4722
> URL: https://issues.apache.org/jira/browse/FLINK-4722
> Project: Flink
> Issue Type: Bug
> Components: Kafka Connector
> Affects Versions: 1.1.2
> Reporter: Sudhanshu Sekhar Lenka
>
> When Kafka one Topic has 3 partition and 3 FlinkKafkaConsumer09 connected to that same topic using "group.id" ,"myGroup" property . Still flink consumer get all data which are push to each 3 partition . While it work properly with normal java consumer. each consumer get specific data.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)