You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Vishal Santoshi <vi...@gmail.com> on 2019/02/04 23:31:39 UTC
KakfaConsumer
A simple query
Does Flink' KafkaConnector flush the current offsets to kafka on a SP ?
Note that the I do koow that Flink consumes data from Kafka topics and
periodically checkpoints using Flink's distributed checkpointing mechanism.
In case of failure, Flink will restore the records from checkpoint
directory and will start reading data from Kafka offset after that. In this
case however I want to restart a radically different pipe but from an
offset that is part of the SP and the hope is that I could just depend on
the kafka offsets of the consumer group.
Thanks
Re: KakfaConsumer
Posted by Vishal Santoshi <vi...@gmail.com>.
In fact "*Checkpointing enabled:* if checkpointing is enabled, the Flink
Kafka Consumer will commit the offsets stored in the checkpointed states
when the checkpoints are completed. This ensures that the committed offsets
in Kafka brokers is consistent with the offsets in the checkpointed
states." confirms that.
On Mon, Feb 4, 2019 at 7:29 PM Vishal Santoshi <vi...@gmail.com>
wrote:
> So it does that also commit the offsets to *kafka* on*
> checkpoint/savepoint *as well to it's own distributed state ? Just wanted
> to confirm.
>
> On Mon, Feb 4, 2019 at 6:56 PM Nagarjun Guraja <na...@gmail.com> wrote:
>
>> Hi Vishal,
>>
>> Flink does checkpoint to Kafka(Offset commits) by default which could be
>> disabled. Look here
>> <https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/kafka.html#kafka-consumers-offset-committing-behaviour-configuration>
>> for more information.
>>
>> Regards,
>> Nagarjun
>>
>> *Success is not final, failure is not fatal: it is the courage to
>> continue that counts. *
>> *- Winston Churchill - *
>>
>>
>> On Mon, Feb 4, 2019 at 3:31 PM Vishal Santoshi <vi...@gmail.com>
>> wrote:
>>
>>> A simple query
>>>
>>> Does Flink' KafkaConnector flush the current offsets to kafka on a SP ?
>>> Note that the I do koow that Flink consumes data from Kafka topics and
>>> periodically checkpoints using Flink's distributed checkpointing mechanism.
>>> In case of failure, Flink will restore the records from checkpoint
>>> directory and will start reading data from Kafka offset after that. In this
>>> case however I want to restart a radically different pipe but from an
>>> offset that is part of the SP and the hope is that I could just depend on
>>> the kafka offsets of the consumer group.
>>>
>>> Thanks
>>>
>>
Re: KakfaConsumer
Posted by Vishal Santoshi <vi...@gmail.com>.
So it does that also commit the offsets to *kafka* on*
checkpoint/savepoint *as well to it's own distributed state ? Just wanted
to confirm.
On Mon, Feb 4, 2019 at 6:56 PM Nagarjun Guraja <na...@gmail.com> wrote:
> Hi Vishal,
>
> Flink does checkpoint to Kafka(Offset commits) by default which could be
> disabled. Look here
> <https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/kafka.html#kafka-consumers-offset-committing-behaviour-configuration>
> for more information.
>
> Regards,
> Nagarjun
>
> *Success is not final, failure is not fatal: it is the courage to continue
> that counts. *
> *- Winston Churchill - *
>
>
> On Mon, Feb 4, 2019 at 3:31 PM Vishal Santoshi <vi...@gmail.com>
> wrote:
>
>> A simple query
>>
>> Does Flink' KafkaConnector flush the current offsets to kafka on a SP ?
>> Note that the I do koow that Flink consumes data from Kafka topics and
>> periodically checkpoints using Flink's distributed checkpointing mechanism.
>> In case of failure, Flink will restore the records from checkpoint
>> directory and will start reading data from Kafka offset after that. In this
>> case however I want to restart a radically different pipe but from an
>> offset that is part of the SP and the hope is that I could just depend on
>> the kafka offsets of the consumer group.
>>
>> Thanks
>>
>
Re: KakfaConsumer
Posted by Nagarjun Guraja <na...@gmail.com>.
Hi Vishal,
Flink does checkpoint to Kafka(Offset commits) by default which could be
disabled. Look here
<https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/kafka.html#kafka-consumers-offset-committing-behaviour-configuration>
for more information.
Regards,
Nagarjun
*Success is not final, failure is not fatal: it is the courage to continue
that counts. *
*- Winston Churchill - *
On Mon, Feb 4, 2019 at 3:31 PM Vishal Santoshi <vi...@gmail.com>
wrote:
> A simple query
>
> Does Flink' KafkaConnector flush the current offsets to kafka on a SP ?
> Note that the I do koow that Flink consumes data from Kafka topics and
> periodically checkpoints using Flink's distributed checkpointing mechanism.
> In case of failure, Flink will restore the records from checkpoint
> directory and will start reading data from Kafka offset after that. In this
> case however I want to restart a radically different pipe but from an
> offset that is part of the SP and the hope is that I could just depend on
> the kafka offsets of the consumer group.
>
> Thanks
>