flink-kafka-connector offset management

[Arun Balan <ar...@gmail.com>]

Hi, I am trying to use the flink-kafka-connector and I notice that every
time I restart my application it re-reads the last message on the kafka
topic. So if the latest offset on the topic is 10, then when the
application is restarted, kafka-connector will re-read message 10. Why is
this the behavior? I would assume that the last message has already been
read and offset committed. I require that messages that are already
processed from the topic not be reprocessed. Any insight would be helpful.

Thanks
Arun Balan

Re: flink-kafka-connector offset management

[Ufuk Celebi <uc...@apache.org>]

Hey Arun!

How did you configure your Kafka source? If the offset has been
committed and you configured the source to read from the latest
offset, the message should not be re-processed.

– Ufuk


On Fri, May 13, 2016 at 2:19 PM, Arun Balan <ar...@gmail.com> wrote:
> Hi, I am trying to use the flink-kafka-connector and I notice that every
> time I restart my application it re-reads the last message on the kafka
> topic. So if the latest offset on the topic is 10, then when the application
> is restarted, kafka-connector will re-read message 10. Why is this the
> behavior? I would assume that the last message has already been read and
> offset committed. I require that messages that are already processed from
> the topic not be reprocessed. Any insight would be helpful.
>
> Thanks
> Arun Balan