You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Tzu-Li (Gordon) Tai (JIRA)" <ji...@apache.org> on 2016/11/11 18:18:58 UTC

[jira] [Comment Edited] (FLINK-3123) Allow setting custom start-offsets for the Kafka consumer

    [ https://issues.apache.org/jira/browse/FLINK-3123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15657704#comment-15657704 ] 

Tzu-Li (Gordon) Tai edited comment on FLINK-3123 at 11/11/16 6:18 PM:
----------------------------------------------------------------------

Hi [~juho.autio.r],

{quote}
My use case is skipping to the latest offsets while keeping the same consumer group id.
{quote}

That's actually also addressed in FLINK-4280, which also has a PR pending (the PR for this issue is actually based on that one). With FLINK-4280, you'd call {{consumer.setStartFromEarliest()}} or {{consumer.setStartFromLatest()}} which completely ignores any committed offsets in Kafka on the initial startup, but the consumer group id is still used if you have configured to commit offsets back to Kafka. So, the proposed {{consumer.setStartFromSpecificOffsets(---)}} is basically a third startup option for max flexibility.

{quote}
It would be nice if this reset the offsets even when a Flink snapshot state exists with some offset positions.
{quote}

Flink state snapshots are only used on automatic restored checkpoints from failures, or from manual savepoint start. So, on a fresh job startup, the startup position can be freely configured using the proposed methods in FLINK-4280 and this issue.


was (Author: tzulitai):
Hi [~juho.autio.r],

{quote}
My use case is skipping to the latest offsets while keeping the same consumer group id.
{quote}

That's actually also addressed in FLINK-4280, which also has a PR pending (the PR for this issue is actually based on that one). With FLINK-4280, you'd call {{consumer.setStartFromEarliest()}} or {{consumer.setStartFromLatest()}} which completely ignores any committed offsets in Kafka on the initial startup, but the consumer group id is still used if you have configured to commit offsets back to Kafka. This issue is basically a third startup option for max flexibility.

{quote}
It would be nice if this reset the offsets even when a Flink snapshot state exists with some offset positions.
{quote}

Flink state snapshots are only used on automatic restored checkpoints from failures, or from manual savepoint start. So, on a fresh job startup, the startup position can be freely configured using the proposed methods in FLINK-4280 and this issue.

> Allow setting custom start-offsets for the Kafka consumer
> ---------------------------------------------------------
>
>                 Key: FLINK-3123
>                 URL: https://issues.apache.org/jira/browse/FLINK-3123
>             Project: Flink
>          Issue Type: Improvement
>          Components: Kafka Connector
>            Reporter: Robert Metzger
>            Assignee: Tzu-Li (Gordon) Tai
>             Fix For: 1.0.0, 1.2.0
>
>
> Currently, the Kafka consumer only allows to start reading from the earliest available offset or the current offset.
> Sometimes, users want to set a specific start offset themselves.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)