You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Vishal Santoshi <vi...@gmail.com> on 2018/11/20 17:48:39 UTC
Reset kafka offets to latest on restart
Is it possible to have checkpointing but reset the kafka offsets to latest
on restart on failure ?
Re: Reset kafka offets to latest on restart
Posted by Tony Wei <to...@gmail.com>.
Hi Vishal,
Sorry, I didn't notice this requirement, but I can't come up with another
solution, unless disable checkpointing or customize your own
kafka source function. For the first case, you may have to give up storing
states in flink's state backend. For the second one, write
your own implementation with kafka client and always seek to the latest
position when the job begin to run.
Best,
Tony Wei
Vishal Santoshi <vi...@gmail.com> 於 2018年11月25日 週日 上午4:51寫道：
> I think I can set . a new uuid but it seems `allowNonRestoreState` is a
> CLI hint. I need the "automatic" restart on failure to use the new uuid.
> Our use case has no use of data on Kafka that is not current.
>
> On Thu, Nov 22, 2018 at 11:16 PM Tony Wei <to...@gmail.com> wrote:
>
>> Hi Vishal,
>>
>> AFAIK, the current behavior of kafka source will always use checkpoint
>> state as the start position, ignoring other configuration.
>> A workaround solution I can come up with is to set a new uuid to your
>> kafka source and restore your job with `allowNonRestoreState`.
>> Therefore, you can use the way that Rong provided to set the initial
>> start position.
>>
>> cc. Gordon who know more about the details of kafka source.
>>
>> Best,
>> Tony Wei
>>
>> Rong Rong <wa...@gmail.com> 於 2018年11月22日 週四 上午8:23寫道：
>>
>>> Hi Vishal,
>>>
>>> You can probably try using similar offset configuration as a service
>>> consumer.
>>> Maybe this will be useful to look at [1]
>>>
>>> Thanks,
>>> Rong
>>>
>>> [1]
>>> https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/kafka.html#kafka-consumers-start-position-configuration
>>>
>>> On Wed, Nov 21, 2018 at 1:32 PM Jamie Grier <jg...@lyft.com> wrote:
>>>
>>>> Hi Vishal,
>>>>
>>>> No, there is no way to do this currently.
>>>>
>>>>
>>>> On Wed, Nov 21, 2018 at 10:22 AM Vishal Santoshi <
>>>> vishal.santoshi@gmail.com> wrote:
>>>>
>>>>> Any one ?
>>>>>
>>>>> On Tue, Nov 20, 2018 at 12:48 PM Vishal Santoshi <
>>>>> vishal.santoshi@gmail.com> wrote:
>>>>>
>>>>>> Is it possible to have checkpointing but reset the kafka offsets to
>>>>>> latest on restart on failure ?
>>>>>>
>>>>>
Re: Reset kafka offets to latest on restart
Posted by Vishal Santoshi <vi...@gmail.com>.
I think I can set . a new uuid but it seems `allowNonRestoreState` is a CLI
hint. I need the "automatic" restart on failure to use the new uuid.
Our use case has no use of data on Kafka that is not current.
On Thu, Nov 22, 2018 at 11:16 PM Tony Wei <to...@gmail.com> wrote:
> Hi Vishal,
>
> AFAIK, the current behavior of kafka source will always use checkpoint
> state as the start position, ignoring other configuration.
> A workaround solution I can come up with is to set a new uuid to your
> kafka source and restore your job with `allowNonRestoreState`.
> Therefore, you can use the way that Rong provided to set the initial start
> position.
>
> cc. Gordon who know more about the details of kafka source.
>
> Best,
> Tony Wei
>
> Rong Rong <wa...@gmail.com> 於 2018年11月22日 週四 上午8:23寫道：
>
>> Hi Vishal,
>>
>> You can probably try using similar offset configuration as a service
>> consumer.
>> Maybe this will be useful to look at [1]
>>
>> Thanks,
>> Rong
>>
>> [1]
>> https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/kafka.html#kafka-consumers-start-position-configuration
>>
>> On Wed, Nov 21, 2018 at 1:32 PM Jamie Grier <jg...@lyft.com> wrote:
>>
>>> Hi Vishal,
>>>
>>> No, there is no way to do this currently.
>>>
>>>
>>> On Wed, Nov 21, 2018 at 10:22 AM Vishal Santoshi <
>>> vishal.santoshi@gmail.com> wrote:
>>>
>>>> Any one ?
>>>>
>>>> On Tue, Nov 20, 2018 at 12:48 PM Vishal Santoshi <
>>>> vishal.santoshi@gmail.com> wrote:
>>>>
>>>>> Is it possible to have checkpointing but reset the kafka offsets to
>>>>> latest on restart on failure ?
>>>>>
>>>>
Re: Reset kafka offets to latest on restart
Posted by Tony Wei <to...@gmail.com>.
Hi Vishal,
AFAIK, the current behavior of kafka source will always use checkpoint
state as the start position, ignoring other configuration.
A workaround solution I can come up with is to set a new uuid to your kafka
source and restore your job with `allowNonRestoreState`.
Therefore, you can use the way that Rong provided to set the initial start
position.
cc. Gordon who know more about the details of kafka source.
Best,
Tony Wei
Rong Rong <wa...@gmail.com> 於 2018年11月22日 週四 上午8:23寫道：
> Hi Vishal,
>
> You can probably try using similar offset configuration as a service
> consumer.
> Maybe this will be useful to look at [1]
>
> Thanks,
> Rong
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/kafka.html#kafka-consumers-start-position-configuration
>
> On Wed, Nov 21, 2018 at 1:32 PM Jamie Grier <jg...@lyft.com> wrote:
>
>> Hi Vishal,
>>
>> No, there is no way to do this currently.
>>
>>
>> On Wed, Nov 21, 2018 at 10:22 AM Vishal Santoshi <
>> vishal.santoshi@gmail.com> wrote:
>>
>>> Any one ?
>>>
>>> On Tue, Nov 20, 2018 at 12:48 PM Vishal Santoshi <
>>> vishal.santoshi@gmail.com> wrote:
>>>
>>>> Is it possible to have checkpointing but reset the kafka offsets to
>>>> latest on restart on failure ?
>>>>
>>>
Re: Reset kafka offets to latest on restart
Posted by Rong Rong <wa...@gmail.com>.
Hi Vishal,
You can probably try using similar offset configuration as a service
consumer.
Maybe this will be useful to look at [1]
Thanks,
Rong
[1]
https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/kafka.html#kafka-consumers-start-position-configuration
On Wed, Nov 21, 2018 at 1:32 PM Jamie Grier <jg...@lyft.com> wrote:
> Hi Vishal,
>
> No, there is no way to do this currently.
>
>
> On Wed, Nov 21, 2018 at 10:22 AM Vishal Santoshi <
> vishal.santoshi@gmail.com> wrote:
>
>> Any one ?
>>
>> On Tue, Nov 20, 2018 at 12:48 PM Vishal Santoshi <
>> vishal.santoshi@gmail.com> wrote:
>>
>>> Is it possible to have checkpointing but reset the kafka offsets to
>>> latest on restart on failure ?
>>>
>>
Re: Reset kafka offets to latest on restart
Posted by Jamie Grier <jg...@lyft.com>.
Hi Vishal,
No, there is no way to do this currently.
On Wed, Nov 21, 2018 at 10:22 AM Vishal Santoshi <vi...@gmail.com>
wrote:
> Any one ?
>
> On Tue, Nov 20, 2018 at 12:48 PM Vishal Santoshi <
> vishal.santoshi@gmail.com> wrote:
>
>> Is it possible to have checkpointing but reset the kafka offsets to
>> latest on restart on failure ?
>>
>
Re: Reset kafka offets to latest on restart
Posted by Vishal Santoshi <vi...@gmail.com>.
Any one ?
On Tue, Nov 20, 2018 at 12:48 PM Vishal Santoshi <vi...@gmail.com>
wrote:
> Is it possible to have checkpointing but reset the kafka offsets to
> latest on restart on failure ?
>