You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by tzulitai <gi...@git.apache.org> on 2017/03/13 15:19:10 UTC
[GitHub] flink pull request #3527: [FLINK-3398] [kafka] Allow disabling offset commit...
GitHub user tzulitai opened a pull request:
https://github.com/apache/flink/pull/3527
[FLINK-3398] [kafka] Allow disabling offset committing for FlinkKafkaConsumer
This PR allows users to completely disable offset committing back to Kafka.
The way configuration works depends on whether checkpointing was enabled for the job:
- **Checkpointing disabled:** when checkpointing is disabled, the consumer relies on the auto commit functionality of internal Kafka clients. Therefore, to disable / enable offset committing, users simply need to set appropriate values for `auto.commit.enable` / `auto.commit.interval.ms`.
- **Checkpointing enabled:** in this scenario, the `auto.commit.enable` / `auto.commit.interval.ms` is completely ignored. To disable / enable offset committing, users use a new `setCommiOffsetsOnCheckpoints(boolean)` method. By default, this is `true`, so that without any additional configuration, the committing behaviour sticks with the original consumer prior to this change. No user code / behaviour is broken.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tzulitai/flink FLINK-3398
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/3527.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3527
----
commit 1163d82c908423c941f6b8d71c9019c7c33ba6ab
Author: Tzu-Li (Gordon) Tai <tz...@apache.org>
Date: 2017-03-13T05:49:08Z
[FLINK-3398] [kafka] Allow disabling offset committing for FlinkKafkaConsumer
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3527: [FLINK-3398] [kafka] Allow disabling offset committing fo...
Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on the issue:
https://github.com/apache/flink/pull/3527
Thank you for opening a PR for this issue!
I know I've +1ed a lot of PRs of you recently, and I think I'm going to +1 this one without any further comments .. because I did not find anything :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #3527: [FLINK-3398] [kafka] Allow disabling offset commit...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/3527
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3527: [FLINK-3398] [kafka] Allow disabling offset committing fo...
Posted by tzulitai <gi...@git.apache.org>.
Github user tzulitai commented on the issue:
https://github.com/apache/flink/pull/3527
Did a final test run on a Kafka installation, and things worked as expected.
One minor improvement would be to add logs for what exactly the commit mode is used when it is determined in `open()`.
I think it's a safe call to add the log and then merge this :-) Will proceed to merge for `master`.
Thanks for all the recent reviews @rmetzger :-D
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---