You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Yaroslav Tkachenko <ya...@goldsky.com> on 2022/10/13 05:34:07 UTC
allowNonRestoredState doesn't seem to be working
Hey everyone,
I'm trying to redeploy an application using a savepoint. The new version of
the application has a few operators with new uids and a few operators with
the old uids. I'd like to keep the state for the old ones.
I passed the allowNonRestoredState flag (using Apache Kubernetes Operator
actually) and I can confirm that
"execution.savepoint.ignore-unclaimed-state" is "true" after that.
However, the application still fails with the following exception:
"java.lang.IllegalStateException: Failed to rollback to
checkpoint/savepoint s3p://<REDACTED>. Cannot map checkpoint/savepoint
state for operator d9ea0f9654a3395802138c72c1bfd35b to the new program,
because the operator is not available in the new program. If you want to
allow to skip this, you can set the --allowNonRestoredState option on the
CLI."
Is there a situation where allowNonRestoredState may not work? Thanks.
Re: allowNonRestoredState doesn't seem to be working
Posted by Yaroslav Tkachenko <ya...@goldsky.com>.
I'm confident I'm hitting a bug, I guess I'm the first one trying this
recovery in the standalone mode :-D
Created https://issues.apache.org/jira/browse/FLINK-29633
On Thu, Oct 13, 2022 at 8:45 AM Yaroslav Tkachenko <ya...@goldsky.com>
wrote:
> Thanks folks, I understand this can be a limitation when redeploying. I
> did try to delete my job and start it from scratch using
> "initialSavepointPath"... and I got the same issue. Going to investigate
> this more today.
>
> On Thu, Oct 13, 2022 at 12:18 AM Evgeniy Lyutikov <eb...@avito.ru>
> wrote:
>
>> The problem is that changing the FlinkDeployment specification (new jar
>> version, changing pod resources, etc.) for JobManager is just a restart.
>>
>> 2022-09-16 09:30:52,526 INFO
>> org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Restoring
>> job 00000000000000000000000000000000 from Checkpoint 34 @ 1663320593326 for
>> 00000000000000000000000000000000 located at
>> s3p://flink-checkpoints/k8s-checkpoint-test-k8s-deploy/00000000000000000000000000000000/chk-34.
>> 2022-09-16 09:30:52,624 INFO
>> org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] - Job
>> 00000000000000000000000000000000 reached terminal state FAILED.
>> org.apache.flink.runtime.client.JobInitializationException: Could not
>> start the JobMaster.
>> Caused by: java.util.concurrent.CompletionException:
>> java.lang.IllegalStateException: There is no operator for the state
>> f215196137eeb29b6f14c1ac14a1dc9f
>> Caused by: java.lang.IllegalStateException: There is no operator for the
>> state f215196137eeb29b6f14c1ac14a1dc9f
>>
>> After starting, it restores everything from the saved HA metadata saved
>> in the configmap (jobgraph, etc.).
>> The only correct method for us was to completely delete the
>> FlinkDeployment object and create a new one with initialSavepointPath and
>> allowNonRestoredState.
>> After that, the startup log looks a little different
>>
>> 2022-09-16 10:30:52,624 INFO
>> org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Restoring
>> job 00000000000000000000000000000000 from Savepoint 34 @ 0 for
>> 00000000000000000000000000000000 located at
>> s3p://flink-checkpoints/k8s-checkpoint-test-k8s-deploy/00000000000000000000000000000000/chk-34.
>>
>> ------------------------------
>> *От:* Gyula Fóra <gy...@gmail.com>
>> *Отправлено:* 13 октября 2022 г. 13:19:54
>> *Кому:* Yaroslav Tkachenko
>> *Копия:* user
>> *Тема:* Re: allowNonRestoredState doesn't seem to be working
>>
>> Hi!
>>
>> If you have last-state upgrade mode configured it may happen that the
>> allowNonRestoredState config is ignored by Flink (as the last-state upgrade
>> mechanism somewhat bypasses the regular submission).
>>
>> Worst case scenario, you can suspend the deployment, manually record the
>> last checkpoint/savepoint path. Then delete the FlinkDeployment and
>> recreate it with the initialSavepointPath set to your checkpoint.
>>
>> Cheers,
>> Gyula
>>
>> On Thu, Oct 13, 2022 at 7:36 AM Yaroslav Tkachenko <ya...@goldsky.com>
>> wrote:
>>
>>> Hey everyone,
>>>
>>> I'm trying to redeploy an application using a savepoint. The new version
>>> of the application has a few operators with new uids and a few operators
>>> with the old uids. I'd like to keep the state for the old ones.
>>>
>>> I passed the allowNonRestoredState flag (using Apache Kubernetes
>>> Operator actually) and I can confirm that
>>> "execution.savepoint.ignore-unclaimed-state" is "true" after that.
>>>
>>> However, the application still fails with the following exception:
>>>
>>> "java.lang.IllegalStateException: Failed to rollback to
>>> checkpoint/savepoint s3p://<REDACTED>. Cannot map checkpoint/savepoint
>>> state for operator d9ea0f9654a3395802138c72c1bfd35b to the new program,
>>> because the operator is not available in the new program. If you want to
>>> allow to skip this, you can set the --allowNonRestoredState option on the
>>> CLI."
>>>
>>> Is there a situation where allowNonRestoredState may not work? Thanks.
>>>
>>
>> * ------------------------------ *“This message contains confidential
>> information/commercial secret. If you are not the intended addressee of
>> this message you may not copy, save, print or forward it to any third party
>> and you are kindly requested to destroy this message and notify the sender
>> thereof by email.
>> Данное сообщение содержит конфиденциальную информацию/информацию,
>> являющуюся коммерческой тайной. Если Вы не являетесь надлежащим адресатом
>> данного сообщения, Вы не вправе копировать, сохранять, печатать или
>> пересылать его каким либо иным лицам. Просьба уничтожить данное сообщение и
>> уведомить об этом отправителя электронным письмом.”
>>
>
Re: allowNonRestoredState doesn't seem to be working
Posted by Yaroslav Tkachenko <ya...@goldsky.com>.
Thanks folks, I understand this can be a limitation when redeploying. I did
try to delete my job and start it from scratch using
"initialSavepointPath"... and I got the same issue. Going to investigate
this more today.
On Thu, Oct 13, 2022 at 12:18 AM Evgeniy Lyutikov <eb...@avito.ru>
wrote:
> The problem is that changing the FlinkDeployment specification (new jar
> version, changing pod resources, etc.) for JobManager is just a restart.
>
> 2022-09-16 09:30:52,526 INFO
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Restoring
> job 00000000000000000000000000000000 from Checkpoint 34 @ 1663320593326 for
> 00000000000000000000000000000000 located at
> s3p://flink-checkpoints/k8s-checkpoint-test-k8s-deploy/00000000000000000000000000000000/chk-34.
> 2022-09-16 09:30:52,624 INFO
> org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] - Job
> 00000000000000000000000000000000 reached terminal state FAILED.
> org.apache.flink.runtime.client.JobInitializationException: Could not
> start the JobMaster.
> Caused by: java.util.concurrent.CompletionException:
> java.lang.IllegalStateException: There is no operator for the state
> f215196137eeb29b6f14c1ac14a1dc9f
> Caused by: java.lang.IllegalStateException: There is no operator for the
> state f215196137eeb29b6f14c1ac14a1dc9f
>
> After starting, it restores everything from the saved HA metadata saved in
> the configmap (jobgraph, etc.).
> The only correct method for us was to completely delete the
> FlinkDeployment object and create a new one with initialSavepointPath and
> allowNonRestoredState.
> After that, the startup log looks a little different
>
> 2022-09-16 10:30:52,624 INFO
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Restoring
> job 00000000000000000000000000000000 from Savepoint 34 @ 0 for
> 00000000000000000000000000000000 located at
> s3p://flink-checkpoints/k8s-checkpoint-test-k8s-deploy/00000000000000000000000000000000/chk-34.
>
> ------------------------------
> *От:* Gyula Fóra <gy...@gmail.com>
> *Отправлено:* 13 октября 2022 г. 13:19:54
> *Кому:* Yaroslav Tkachenko
> *Копия:* user
> *Тема:* Re: allowNonRestoredState doesn't seem to be working
>
> Hi!
>
> If you have last-state upgrade mode configured it may happen that the
> allowNonRestoredState config is ignored by Flink (as the last-state upgrade
> mechanism somewhat bypasses the regular submission).
>
> Worst case scenario, you can suspend the deployment, manually record the
> last checkpoint/savepoint path. Then delete the FlinkDeployment and
> recreate it with the initialSavepointPath set to your checkpoint.
>
> Cheers,
> Gyula
>
> On Thu, Oct 13, 2022 at 7:36 AM Yaroslav Tkachenko <ya...@goldsky.com>
> wrote:
>
>> Hey everyone,
>>
>> I'm trying to redeploy an application using a savepoint. The new version
>> of the application has a few operators with new uids and a few operators
>> with the old uids. I'd like to keep the state for the old ones.
>>
>> I passed the allowNonRestoredState flag (using Apache Kubernetes Operator
>> actually) and I can confirm that
>> "execution.savepoint.ignore-unclaimed-state" is "true" after that.
>>
>> However, the application still fails with the following exception:
>>
>> "java.lang.IllegalStateException: Failed to rollback to
>> checkpoint/savepoint s3p://<REDACTED>. Cannot map checkpoint/savepoint
>> state for operator d9ea0f9654a3395802138c72c1bfd35b to the new program,
>> because the operator is not available in the new program. If you want to
>> allow to skip this, you can set the --allowNonRestoredState option on the
>> CLI."
>>
>> Is there a situation where allowNonRestoredState may not work? Thanks.
>>
>
> * ------------------------------ *“This message contains confidential
> information/commercial secret. If you are not the intended addressee of
> this message you may not copy, save, print or forward it to any third party
> and you are kindly requested to destroy this message and notify the sender
> thereof by email.
> Данное сообщение содержит конфиденциальную информацию/информацию,
> являющуюся коммерческой тайной. Если Вы не являетесь надлежащим адресатом
> данного сообщения, Вы не вправе копировать, сохранять, печатать или
> пересылать его каким либо иным лицам. Просьба уничтожить данное сообщение и
> уведомить об этом отправителя электронным письмом.”
>
Re: allowNonRestoredState doesn't seem to be working
Posted by Evgeniy Lyutikov <eb...@avito.ru>.
The problem is that changing the FlinkDeployment specification (new jar version, changing pod resources, etc.) for JobManager is just a restart.
2022-09-16 09:30:52,526 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Restoring job 00000000000000000000000000000000 from Checkpoint 34 @ 1663320593326 for 00000000000000000000000000000000 located at s3p://flink-checkpoints/k8s-checkpoint-test-k8s-deploy/00000000000000000000000000000000/chk-34.
2022-09-16 09:30:52,624 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] - Job 00000000000000000000000000000000 reached terminal state FAILED.
org.apache.flink.runtime.client.JobInitializationException: Could not start the JobMaster.
Caused by: java.util.concurrent.CompletionException: java.lang.IllegalStateException: There is no operator for the state f215196137eeb29b6f14c1ac14a1dc9f
Caused by: java.lang.IllegalStateException: There is no operator for the state f215196137eeb29b6f14c1ac14a1dc9f
After starting, it restores everything from the saved HA metadata saved in the configmap (jobgraph, etc.).
The only correct method for us was to completely delete the FlinkDeployment object and create a new one with initialSavepointPath and allowNonRestoredState.
After that, the startup log looks a little different
2022-09-16 10:30:52,624 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Restoring job 00000000000000000000000000000000 from Savepoint 34 @ 0 for 00000000000000000000000000000000 located at s3p://flink-checkpoints/k8s-checkpoint-test-k8s-deploy/00000000000000000000000000000000/chk-34.
________________________________
От: Gyula Fóra <gy...@gmail.com>
Отправлено: 13 октября 2022 г. 13:19:54
Кому: Yaroslav Tkachenko
Копия: user
Тема: Re: allowNonRestoredState doesn't seem to be working
Hi!
If you have last-state upgrade mode configured it may happen that the allowNonRestoredState config is ignored by Flink (as the last-state upgrade mechanism somewhat bypasses the regular submission).
Worst case scenario, you can suspend the deployment, manually record the last checkpoint/savepoint path. Then delete the FlinkDeployment and recreate it with the initialSavepointPath set to your checkpoint.
Cheers,
Gyula
On Thu, Oct 13, 2022 at 7:36 AM Yaroslav Tkachenko <ya...@goldsky.com>> wrote:
Hey everyone,
I'm trying to redeploy an application using a savepoint. The new version of the application has a few operators with new uids and a few operators with the old uids. I'd like to keep the state for the old ones.
I passed the allowNonRestoredState flag (using Apache Kubernetes Operator actually) and I can confirm that "execution.savepoint.ignore-unclaimed-state" is "true" after that.
However, the application still fails with the following exception:
"java.lang.IllegalStateException: Failed to rollback to checkpoint/savepoint s3p://<REDACTED>. Cannot map checkpoint/savepoint state for operator d9ea0f9654a3395802138c72c1bfd35b to the new program, because the operator is not available in the new program. If you want to allow to skip this, you can set the --allowNonRestoredState option on the CLI."
Is there a situation where allowNonRestoredState may not work? Thanks.
________________________________
“This message contains confidential information/commercial secret. If you are not the intended addressee of this message you may not copy, save, print or forward it to any third party and you are kindly requested to destroy this message and notify the sender thereof by email.
Данное сообщение содержит конфиденциальную информацию/информацию, являющуюся коммерческой тайной. Если Вы не являетесь надлежащим адресатом данного сообщения, Вы не вправе копировать, сохранять, печатать или пересылать его каким либо иным лицам. Просьба уничтожить данное сообщение и уведомить об этом отправителя электронным письмом.”
Re: allowNonRestoredState doesn't seem to be working
Posted by Gyula Fóra <gy...@gmail.com>.
Hi!
If you have last-state upgrade mode configured it may happen that the
allowNonRestoredState config is ignored by Flink (as the last-state upgrade
mechanism somewhat bypasses the regular submission).
Worst case scenario, you can suspend the deployment, manually record the
last checkpoint/savepoint path. Then delete the FlinkDeployment and
recreate it with the initialSavepointPath set to your checkpoint.
Cheers,
Gyula
On Thu, Oct 13, 2022 at 7:36 AM Yaroslav Tkachenko <ya...@goldsky.com>
wrote:
> Hey everyone,
>
> I'm trying to redeploy an application using a savepoint. The new version
> of the application has a few operators with new uids and a few operators
> with the old uids. I'd like to keep the state for the old ones.
>
> I passed the allowNonRestoredState flag (using Apache Kubernetes Operator
> actually) and I can confirm that
> "execution.savepoint.ignore-unclaimed-state" is "true" after that.
>
> However, the application still fails with the following exception:
>
> "java.lang.IllegalStateException: Failed to rollback to
> checkpoint/savepoint s3p://<REDACTED>. Cannot map checkpoint/savepoint
> state for operator d9ea0f9654a3395802138c72c1bfd35b to the new program,
> because the operator is not available in the new program. If you want to
> allow to skip this, you can set the --allowNonRestoredState option on the
> CLI."
>
> Is there a situation where allowNonRestoredState may not work? Thanks.
>