[jira] [Created] (FLINK-5704) Deprecate FlinkKafkaConsumer constructors in favor of improvements to decoupling from Kafka offset committing

["Tzu-Li (Gordon) Tai (JIRA)" <ji...@apache.org>]

Tzu-Li (Gordon) Tai created FLINK-5704:
------------------------------------------

             Summary: Deprecate FlinkKafkaConsumer constructors in favor of improvements to decoupling from Kafka offset committing
                 Key: FLINK-5704
                 URL: https://issues.apache.org/jira/browse/FLINK-5704
             Project: Flink
          Issue Type: Improvement
          Components: Kafka Connector, Streaming Connectors
            Reporter: Tzu-Li (Gordon) Tai


With FLINK-3398 and FLINK-4280, the {{FlinkKafkaConsumer}} will be able to completely operate independently of committed offsets in Kafka.
I.e.,
1) when starting, the consumer can choose to not use any committed offsets in Kafka as the starting position
2) the consumer can completely opt-out of committing offsets back to Kafka

However, our current default behaviour for 1) is to respect committed offsets, and 2) is to always have offset committing. Users still have to call the respective setter configuration methods to change this.

I think we should deprecate the current constructors in favor of new ones with default behaviours 1) start from the latest record, without respecting Kafka offsets, and 2) don't commit offsets.

With this change, users explicitly call the config methods of FLINK-3398 and FLINK-4280 to *enable* respecting committed offsets for Kafka, instead of **disabling** it. They would want to / need to enable it, only when perhaps to migrate from a non-Flink consuming application, or they wish to expose the internal checkpointed offsets to measure consumer lag using Kafka toolings.

The main advantage for this change is that the API of {{FlinkKafkaConsumer}} can speak for itself that it does not depend on committed offsets in Kafka (this is a misconception that users frequently have), and that exactly-once depends solely on offsets checkpointed internally using Flink's checkpointing mechanics.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)