You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Vishal Santoshi <vi...@gmail.com> on 2021/12/03 18:11:22 UTC
enable.auto.commit=true and checkpointing turned on
Hello folks,
2 questions
1. If we have enabled enable.auto.commit and enabled checkpointing and we
restart a flink application ( without checkpoint or savepoint ) , would the
kafka consumer start consuming from the last offset committed to kafka.
2. What if in the above scenario, we have "auto.offset.reset" set to
"latest". ? Would that ignore the consumer group offset in kafka ?
Regards.
Re: enable.auto.commit=true and checkpointing turned on
Posted by Vishal Santoshi <vi...@gmail.com>.
perfect. Thanks. That is what I imagined.
On Mon, Dec 6, 2021 at 2:04 AM Hang Ruan <ru...@gmail.com> wrote:
> Hi,
>
> 1. Yes, the kafka source will use the Kafka committed offset for the group
> id to start the job.
>
> 2. No, the auto.offset.reset
> <https://kafka.apache.org/documentation/#consumerconfigs_auto.offset.reset> is
> from Kafka consumer config, which defines what to do when there is no
> initial offset in Kafka or if the current offset does not exist any more on
> the server. If the offset exists on the server, the consumer will still
> start from the committed offset.
>
> ps: If you enabled checkpointing, there is no need to enable
> enable.auto.commit. The offset will be committed to Kafka when checkpoints
> complete, which is the default behavior.
>
> Vishal Santoshi <vi...@gmail.com> 于2021年12月4日周六 02:11写道：
>
>> Hello folks,
>>
>> 2 questions
>> 1. If we have enabled enable.auto.commit and enabled checkpointing and
>> we restart a flink application ( without checkpoint or savepoint ) , would
>> the kafka consumer start consuming from the last offset committed to kafka.
>>
>> 2. What if in the above scenario, we have "auto.offset.reset" set to
>> "latest". ? Would that ignore the consumer group offset in kafka ?
>>
>>
>> Regards.
>>
>
Re: enable.auto.commit=true and checkpointing turned on
Posted by Hang Ruan <ru...@gmail.com>.
Hi,
1. Yes, the kafka source will use the Kafka committed offset for the group
id to start the job.
2. No, the auto.offset.reset
<https://kafka.apache.org/documentation/#consumerconfigs_auto.offset.reset> is
from Kafka consumer config, which defines what to do when there is no
initial offset in Kafka or if the current offset does not exist any more on
the server. If the offset exists on the server, the consumer will still
start from the committed offset.
ps: If you enabled checkpointing, there is no need to enable
enable.auto.commit. The offset will be committed to Kafka when checkpoints
complete, which is the default behavior.
Vishal Santoshi <vi...@gmail.com> 于2021年12月4日周六 02:11写道：
> Hello folks,
>
> 2 questions
> 1. If we have enabled enable.auto.commit and enabled checkpointing and we
> restart a flink application ( without checkpoint or savepoint ) , would the
> kafka consumer start consuming from the last offset committed to kafka.
>
> 2. What if in the above scenario, we have "auto.offset.reset" set to
> "latest". ? Would that ignore the consumer group offset in kafka ?
>
>
> Regards.
>