You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Tzu-Li (Gordon) Tai (JIRA)" <ji...@apache.org> on 2017/02/02 09:49:52 UTC
[jira] [Commented] (FLINK-3398) Flink Kafka consumer should support
auto-commit opt-outs
[ https://issues.apache.org/jira/browse/FLINK-3398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15849733#comment-15849733 ]
Tzu-Li (Gordon) Tai commented on FLINK-3398:
--------------------------------------------
I think we should keep this JIRA instead of merging it into FLINK-4280.
They try to address different new features: FLINK-4280 is for more flexible start position configuration independent from Kafka offsets.
This JIRA is for opt-out from offset committing on Flink's checkpoints.
I'll move the respective proposed API for commit opt-out here:
{code}
Properties props = new Properties();
...
FlinkKafkaConsumer09 kafka = new FlinkKafkaConsumer09("topic", schema, props);
kafka.setEnableOffsetCommittingOnCheckpoints(boolean); // if true (checkpointing should be enabled), overrides periodic checkpointing if "enable.auto.commit" is set in props.
{code}
So the scenarios breaks down into:
1. {{props.put("auto.commit.enable", "true")}} & {{setEnableOffsetCommittingOnCheckpoints(false)}}:
Perform auto committing with the internal client (or in 0.8 case our own periodic committer).
2. {{props.put("auto.commit.enable", "true")}} & {{setEnableOffsetCommittingOnCheckpoints(true)}}:
Overrides whatever value in props for "auto.commit.enable", only commit on checkpoints. If checkpointing isn't enabled, the consumer fails due to conflicting configuration.
> Flink Kafka consumer should support auto-commit opt-outs
> --------------------------------------------------------
>
> Key: FLINK-3398
> URL: https://issues.apache.org/jira/browse/FLINK-3398
> Project: Flink
> Issue Type: Improvement
> Components: Kafka Connector
> Reporter: Shikhar Bhushan
> Assignee: Tzu-Li (Gordon) Tai
> Priority: Critical
> Fix For: 1.3.0
>
>
> Currently the Kafka source will commit consumer offsets to Zookeeper, either upon a checkpoint if checkpointing is enabled, otherwise periodically based on {{auto.commit.interval.ms}}
> It should be possible to opt-out of committing consumer offsets to Zookeeper. Kafka has this config as {{auto.commit.enable}} (0.8) and {{enable.auto.commit}} (0.9).
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)