[jira] [Commented] (FLINK-4789) Avoid Kafka partition discovery on restore and share consumer instance for discovery and data consumption

["Tzu-Li (Gordon) Tai (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-4789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16107050#comment-16107050 ] 

Tzu-Li (Gordon) Tai commented on FLINK-4789:
--------------------------------------------

I would like to close this issue now.
Since Flink 1.4, with partition discovery we will always try to discover new partitions on restore, and continuous partition discovery will require a dedicated consumer independent of the one used for record fetching.

> Avoid Kafka partition discovery on restore and share consumer instance for discovery and data consumption
> ---------------------------------------------------------------------------------------------------------
>
>                 Key: FLINK-4789
>                 URL: https://issues.apache.org/jira/browse/FLINK-4789
>             Project: Flink
>          Issue Type: Improvement
>          Components: Kafka Connector
>    Affects Versions: 1.2.0
>            Reporter: Robert Metzger
>
> As part of FLINK-4379, the Kafka partition discovery was moved from the Constructor to the open() method. This is in general a good change, as outlined in FLINK-4155, as it allows us to detect new partitions and topics based on regex on the fly.
> However, currently the partitions are discovered on restore as well. 
> Also, the {{FlinkKafkaConsumer09.getKafkaPartitions()}} is creating a separate {{KafkaConsumer}} just for the partition discovery.
> Since the partition discovery happens on the task managers now, we can use the regular {{KafkaConsumer}} instance, which is used for data retrieval as well.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)