[jira] [Commented] (FLINK-5704) Deprecate FlinkKafkaConsumer constructors in favor of improvements to decoupling from Kafka offset committing

["Stephan Ewen (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-5704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15851273#comment-15851273 ] 

Stephan Ewen commented on FLINK-5704:
-------------------------------------

+1 sounds good

> Deprecate FlinkKafkaConsumer constructors in favor of improvements to decoupling from Kafka offset committing
> -------------------------------------------------------------------------------------------------------------
>
>                 Key: FLINK-5704
>                 URL: https://issues.apache.org/jira/browse/FLINK-5704
>             Project: Flink
>          Issue Type: Improvement
>          Components: Kafka Connector, Streaming Connectors
>            Reporter: Tzu-Li (Gordon) Tai
>
> With FLINK-3398 and FLINK-4280, the {{FlinkKafkaConsumer}} will be able to completely operate independently of committed offsets in Kafka.
> I.e.,
> (1) *Starting position*: when starting, the consumer can choose to not use any committed offsets in Kafka as the starting position
> (2) *Committing offsets back to Kafka*: the consumer can completely opt-out of committing offsets back to Kafka
> However, our current default behaviour for (1) is to respect committed offsets, and (2) is to always have offset committing. Users still have to call the respective setter configuration methods to change this.
> I think we should deprecate the current constructors in favor of new ones with default behaviours (1) start from the latest record, without respecting Kafka offsets, and (2) don't commit offsets.
> With this change, users explicitly call the config methods of FLINK-3398 and FLINK-4280 to *enable* respecting committed offsets for Kafka, instead of _disabling_ it. They would want to / need to enable it, only when perhaps to migrate from a non-Flink consuming application, or they wish to expose the internal checkpointed offsets to measure consumer lag using Kafka toolings.
> The main advantage for this change is that the API of {{FlinkKafkaConsumer}} can speak for itself that it does not depend on committed offsets in Kafka (this is a misconception that users frequently have), and that exactly-once depends solely on offsets checkpointed internally using Flink's checkpointing mechanics.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)