You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Dan Hill <qu...@gmail.com> on 2021/12/08 04:48:18 UTC
Weird operator ID check restore from checkpoint fails
Hi. I noticed this warning has "operator 811d3b279c8b26ed99ff0883b7630242"
in it. I assume this should be an operator uid or name. It looks like
something else. What is it? Is something corrupted?
org.apache.flink.runtime.client.JobInitializationException: Could not
instantiate JobManager.
at org.apache.flink.runtime.dispatcher.Dispatcher.lambda$createJobManagerRunner$5(Dispatcher.java:494)
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: Failed to rollback to
checkpoint/savepoint
s3a://my-flink-state/checkpoints/ce9b90eafde97ca4629c13936c34426f/chk-626.
Cannot map checkpoint/savepoint state for operator
811d3b279c8b26ed99ff0883b7630242 to the new program, because the
operator is not available in the new program. If you want to allow to
skip this, you can set the --allowNonRestoredState option on the CLI.
at org.apache.flink.runtime.checkpoint.Checkpoints.throwNonRestoredStateException(Checkpoints.java:226)
at org.apache.flink.runtime.checkpoint.Checkpoints.loadAndValidateCheckpoint(Checkpoints.java:190)
at org.apache.flink.runtime.checkpoint.CheckpointCoordinator.restoreSavepoint(CheckpointCoordinator.java:1627)
at org.apache.flink.runtime.scheduler.SchedulerBase.tryRestoreExecutionGraphFromSavepoint(SchedulerBase.java:362)
at org.apache.flink.runtime.scheduler.SchedulerBase.createAndRestoreExecutionGraph(SchedulerBase.java:292)
at org.apache.flink.runtime.scheduler.SchedulerBase.<init>(SchedulerBase.java:249)
at org.apache.flink.runtime.scheduler.DefaultScheduler.<init>(DefaultScheduler.java:133)
at org.apache.flink.runtime.scheduler.DefaultSchedulerFactory.createInstance(DefaultSchedulerFactory.java:111)
at org.apache.flink.runtime.jobmaster.JobMaster.createScheduler(JobMaster.java:345)
at org.apache.flink.runtime.jobmaster.JobMaster.<init>(JobMaster.java:330)
at org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.createJobMasterService(DefaultJobMasterServiceFactory.java:95)
at org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.createJobMasterService(DefaultJobMasterServiceFactory.java:39)
at org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.<init>(JobManagerRunnerImpl.java:162)
at org.apache.flink.runtime.dispatcher.DefaultJobManagerRunnerFactory.createJobManagerRunner(DefaultJobManagerRunnerFactory.java:86)
at org.apache.flink.runtime.dispatcher.Dispatcher.lambda$createJobManagerRunner$5(Dispatcher.java:478)
... 4 more
Re: Weird operator ID check restore from checkpoint fails
Posted by Dan Hill <qu...@gmail.com>.
Thanks!
On Tue, Dec 7, 2021, 22:55 Robert Metzger <me...@gmail.com> wrote:
> 811d3b279c8b26ed99ff0883b7630242 is the operator id.
> If I'm not mistaken, running the job graph generation (e.g. the main
> method) in DEBUG log level will show you all the IDs generated. This should
> help you map this ID to your code.
>
> On Wed, Dec 8, 2021 at 7:52 AM Dan Hill <qu...@gmail.com> wrote:
>
>> Nothing changed (as far as I know). It's the same binary and the same
>> args. It's Flink v1.12.3. I'm going to switch away from auto-gen uids and
>> see if that helps. The job randomly started failing to checkpoint. I
>> cancelled the job and started it from the last successful checkpoint.
>>
>> I'm confused why `811d3b279c8b26ed99ff0883b7630242` is used and not the
>> auto-generated uid. That seems like a bug.
>>
>> On Tue, Dec 7, 2021 at 10:40 PM Robert Metzger <me...@gmail.com>
>> wrote:
>>
>>> Hi Dan,
>>>
>>> When restoring a savepoint/checkpoint, Flink is matching the state for
>>> the operators based on the uuid of the operator. The exception says that
>>> there is some state that doesn't match any operator. So from Flink's
>>> perspective, the operator is gone.
>>> Here is more information:
>>> https://nightlies.apache.org/flink/flink-docs-master/docs/ops/state/savepoints/#assigning-operator-ids
>>>
>>>
>>> Somehow something must have changed in your job: Did you change the
>>> Flink version?
>>>
>>> Hope this helps!
>>>
>>> On Wed, Dec 8, 2021 at 5:49 AM Dan Hill <qu...@gmail.com> wrote:
>>>
>>>> I'm restoring the job with the same binary and same flags/args.
>>>>
>>>> On Tue, Dec 7, 2021 at 8:48 PM Dan Hill <qu...@gmail.com> wrote:
>>>>
>>>>> Hi. I noticed this warning has "operator
>>>>> 811d3b279c8b26ed99ff0883b7630242" in it. I assume this should be an
>>>>> operator uid or name. It looks like something else. What is it? Is
>>>>> something corrupted?
>>>>>
>>>>>
>>>>> org.apache.flink.runtime.client.JobInitializationException: Could not instantiate JobManager.
>>>>> at org.apache.flink.runtime.dispatcher.Dispatcher.lambda$createJobManagerRunner$5(Dispatcher.java:494)
>>>>> at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>> at java.lang.Thread.run(Thread.java:748)
>>>>> Caused by: java.lang.IllegalStateException: Failed to rollback to checkpoint/savepoint s3a://my-flink-state/checkpoints/ce9b90eafde97ca4629c13936c34426f/chk-626. Cannot map checkpoint/savepoint state for operator 811d3b279c8b26ed99ff0883b7630242 to the new program, because the operator is not available in the new program. If you want to allow to skip this, you can set the --allowNonRestoredState option on the CLI.
>>>>> at org.apache.flink.runtime.checkpoint.Checkpoints.throwNonRestoredStateException(Checkpoints.java:226)
>>>>> at org.apache.flink.runtime.checkpoint.Checkpoints.loadAndValidateCheckpoint(Checkpoints.java:190)
>>>>> at org.apache.flink.runtime.checkpoint.CheckpointCoordinator.restoreSavepoint(CheckpointCoordinator.java:1627)
>>>>> at org.apache.flink.runtime.scheduler.SchedulerBase.tryRestoreExecutionGraphFromSavepoint(SchedulerBase.java:362)
>>>>> at org.apache.flink.runtime.scheduler.SchedulerBase.createAndRestoreExecutionGraph(SchedulerBase.java:292)
>>>>> at org.apache.flink.runtime.scheduler.SchedulerBase.<init>(SchedulerBase.java:249)
>>>>> at org.apache.flink.runtime.scheduler.DefaultScheduler.<init>(DefaultScheduler.java:133)
>>>>> at org.apache.flink.runtime.scheduler.DefaultSchedulerFactory.createInstance(DefaultSchedulerFactory.java:111)
>>>>> at org.apache.flink.runtime.jobmaster.JobMaster.createScheduler(JobMaster.java:345)
>>>>> at org.apache.flink.runtime.jobmaster.JobMaster.<init>(JobMaster.java:330)
>>>>> at org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.createJobMasterService(DefaultJobMasterServiceFactory.java:95)
>>>>> at org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.createJobMasterService(DefaultJobMasterServiceFactory.java:39)
>>>>> at org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.<init>(JobManagerRunnerImpl.java:162)
>>>>> at org.apache.flink.runtime.dispatcher.DefaultJobManagerRunnerFactory.createJobManagerRunner(DefaultJobManagerRunnerFactory.java:86)
>>>>> at org.apache.flink.runtime.dispatcher.Dispatcher.lambda$createJobManagerRunner$5(Dispatcher.java:478)
>>>>> ... 4 more
>>>>>
>>>>>
Re: Weird operator ID check restore from checkpoint fails
Posted by Robert Metzger <me...@gmail.com>.
811d3b279c8b26ed99ff0883b7630242 is the operator id.
If I'm not mistaken, running the job graph generation (e.g. the main
method) in DEBUG log level will show you all the IDs generated. This should
help you map this ID to your code.
On Wed, Dec 8, 2021 at 7:52 AM Dan Hill <qu...@gmail.com> wrote:
> Nothing changed (as far as I know). It's the same binary and the same
> args. It's Flink v1.12.3. I'm going to switch away from auto-gen uids and
> see if that helps. The job randomly started failing to checkpoint. I
> cancelled the job and started it from the last successful checkpoint.
>
> I'm confused why `811d3b279c8b26ed99ff0883b7630242` is used and not the
> auto-generated uid. That seems like a bug.
>
> On Tue, Dec 7, 2021 at 10:40 PM Robert Metzger <me...@gmail.com>
> wrote:
>
>> Hi Dan,
>>
>> When restoring a savepoint/checkpoint, Flink is matching the state for
>> the operators based on the uuid of the operator. The exception says that
>> there is some state that doesn't match any operator. So from Flink's
>> perspective, the operator is gone.
>> Here is more information:
>> https://nightlies.apache.org/flink/flink-docs-master/docs/ops/state/savepoints/#assigning-operator-ids
>>
>>
>> Somehow something must have changed in your job: Did you change the Flink
>> version?
>>
>> Hope this helps!
>>
>> On Wed, Dec 8, 2021 at 5:49 AM Dan Hill <qu...@gmail.com> wrote:
>>
>>> I'm restoring the job with the same binary and same flags/args.
>>>
>>> On Tue, Dec 7, 2021 at 8:48 PM Dan Hill <qu...@gmail.com> wrote:
>>>
>>>> Hi. I noticed this warning has "operator
>>>> 811d3b279c8b26ed99ff0883b7630242" in it. I assume this should be an
>>>> operator uid or name. It looks like something else. What is it? Is
>>>> something corrupted?
>>>>
>>>>
>>>> org.apache.flink.runtime.client.JobInitializationException: Could not instantiate JobManager.
>>>> at org.apache.flink.runtime.dispatcher.Dispatcher.lambda$createJobManagerRunner$5(Dispatcher.java:494)
>>>> at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>> at java.lang.Thread.run(Thread.java:748)
>>>> Caused by: java.lang.IllegalStateException: Failed to rollback to checkpoint/savepoint s3a://my-flink-state/checkpoints/ce9b90eafde97ca4629c13936c34426f/chk-626. Cannot map checkpoint/savepoint state for operator 811d3b279c8b26ed99ff0883b7630242 to the new program, because the operator is not available in the new program. If you want to allow to skip this, you can set the --allowNonRestoredState option on the CLI.
>>>> at org.apache.flink.runtime.checkpoint.Checkpoints.throwNonRestoredStateException(Checkpoints.java:226)
>>>> at org.apache.flink.runtime.checkpoint.Checkpoints.loadAndValidateCheckpoint(Checkpoints.java:190)
>>>> at org.apache.flink.runtime.checkpoint.CheckpointCoordinator.restoreSavepoint(CheckpointCoordinator.java:1627)
>>>> at org.apache.flink.runtime.scheduler.SchedulerBase.tryRestoreExecutionGraphFromSavepoint(SchedulerBase.java:362)
>>>> at org.apache.flink.runtime.scheduler.SchedulerBase.createAndRestoreExecutionGraph(SchedulerBase.java:292)
>>>> at org.apache.flink.runtime.scheduler.SchedulerBase.<init>(SchedulerBase.java:249)
>>>> at org.apache.flink.runtime.scheduler.DefaultScheduler.<init>(DefaultScheduler.java:133)
>>>> at org.apache.flink.runtime.scheduler.DefaultSchedulerFactory.createInstance(DefaultSchedulerFactory.java:111)
>>>> at org.apache.flink.runtime.jobmaster.JobMaster.createScheduler(JobMaster.java:345)
>>>> at org.apache.flink.runtime.jobmaster.JobMaster.<init>(JobMaster.java:330)
>>>> at org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.createJobMasterService(DefaultJobMasterServiceFactory.java:95)
>>>> at org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.createJobMasterService(DefaultJobMasterServiceFactory.java:39)
>>>> at org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.<init>(JobManagerRunnerImpl.java:162)
>>>> at org.apache.flink.runtime.dispatcher.DefaultJobManagerRunnerFactory.createJobManagerRunner(DefaultJobManagerRunnerFactory.java:86)
>>>> at org.apache.flink.runtime.dispatcher.Dispatcher.lambda$createJobManagerRunner$5(Dispatcher.java:478)
>>>> ... 4 more
>>>>
>>>>
Re: Weird operator ID check restore from checkpoint fails
Posted by Dan Hill <qu...@gmail.com>.
Nothing changed (as far as I know). It's the same binary and the same
args. It's Flink v1.12.3. I'm going to switch away from auto-gen uids and
see if that helps. The job randomly started failing to checkpoint. I
cancelled the job and started it from the last successful checkpoint.
I'm confused why `811d3b279c8b26ed99ff0883b7630242` is used and not the
auto-generated uid. That seems like a bug.
On Tue, Dec 7, 2021 at 10:40 PM Robert Metzger <me...@gmail.com> wrote:
> Hi Dan,
>
> When restoring a savepoint/checkpoint, Flink is matching the state for the
> operators based on the uuid of the operator. The exception says that there
> is some state that doesn't match any operator. So from Flink's perspective,
> the operator is gone.
> Here is more information:
> https://nightlies.apache.org/flink/flink-docs-master/docs/ops/state/savepoints/#assigning-operator-ids
>
>
> Somehow something must have changed in your job: Did you change the Flink
> version?
>
> Hope this helps!
>
> On Wed, Dec 8, 2021 at 5:49 AM Dan Hill <qu...@gmail.com> wrote:
>
>> I'm restoring the job with the same binary and same flags/args.
>>
>> On Tue, Dec 7, 2021 at 8:48 PM Dan Hill <qu...@gmail.com> wrote:
>>
>>> Hi. I noticed this warning has "operator
>>> 811d3b279c8b26ed99ff0883b7630242" in it. I assume this should be an
>>> operator uid or name. It looks like something else. What is it? Is
>>> something corrupted?
>>>
>>>
>>> org.apache.flink.runtime.client.JobInitializationException: Could not instantiate JobManager.
>>> at org.apache.flink.runtime.dispatcher.Dispatcher.lambda$createJobManagerRunner$5(Dispatcher.java:494)
>>> at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>> at java.lang.Thread.run(Thread.java:748)
>>> Caused by: java.lang.IllegalStateException: Failed to rollback to checkpoint/savepoint s3a://my-flink-state/checkpoints/ce9b90eafde97ca4629c13936c34426f/chk-626. Cannot map checkpoint/savepoint state for operator 811d3b279c8b26ed99ff0883b7630242 to the new program, because the operator is not available in the new program. If you want to allow to skip this, you can set the --allowNonRestoredState option on the CLI.
>>> at org.apache.flink.runtime.checkpoint.Checkpoints.throwNonRestoredStateException(Checkpoints.java:226)
>>> at org.apache.flink.runtime.checkpoint.Checkpoints.loadAndValidateCheckpoint(Checkpoints.java:190)
>>> at org.apache.flink.runtime.checkpoint.CheckpointCoordinator.restoreSavepoint(CheckpointCoordinator.java:1627)
>>> at org.apache.flink.runtime.scheduler.SchedulerBase.tryRestoreExecutionGraphFromSavepoint(SchedulerBase.java:362)
>>> at org.apache.flink.runtime.scheduler.SchedulerBase.createAndRestoreExecutionGraph(SchedulerBase.java:292)
>>> at org.apache.flink.runtime.scheduler.SchedulerBase.<init>(SchedulerBase.java:249)
>>> at org.apache.flink.runtime.scheduler.DefaultScheduler.<init>(DefaultScheduler.java:133)
>>> at org.apache.flink.runtime.scheduler.DefaultSchedulerFactory.createInstance(DefaultSchedulerFactory.java:111)
>>> at org.apache.flink.runtime.jobmaster.JobMaster.createScheduler(JobMaster.java:345)
>>> at org.apache.flink.runtime.jobmaster.JobMaster.<init>(JobMaster.java:330)
>>> at org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.createJobMasterService(DefaultJobMasterServiceFactory.java:95)
>>> at org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.createJobMasterService(DefaultJobMasterServiceFactory.java:39)
>>> at org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.<init>(JobManagerRunnerImpl.java:162)
>>> at org.apache.flink.runtime.dispatcher.DefaultJobManagerRunnerFactory.createJobManagerRunner(DefaultJobManagerRunnerFactory.java:86)
>>> at org.apache.flink.runtime.dispatcher.Dispatcher.lambda$createJobManagerRunner$5(Dispatcher.java:478)
>>> ... 4 more
>>>
>>>
Re: Weird operator ID check restore from checkpoint fails
Posted by Robert Metzger <me...@gmail.com>.
Hi Dan,
When restoring a savepoint/checkpoint, Flink is matching the state for the
operators based on the uuid of the operator. The exception says that there
is some state that doesn't match any operator. So from Flink's perspective,
the operator is gone.
Here is more information:
https://nightlies.apache.org/flink/flink-docs-master/docs/ops/state/savepoints/#assigning-operator-ids
Somehow something must have changed in your job: Did you change the Flink
version?
Hope this helps!
On Wed, Dec 8, 2021 at 5:49 AM Dan Hill <qu...@gmail.com> wrote:
> I'm restoring the job with the same binary and same flags/args.
>
> On Tue, Dec 7, 2021 at 8:48 PM Dan Hill <qu...@gmail.com> wrote:
>
>> Hi. I noticed this warning has "operator
>> 811d3b279c8b26ed99ff0883b7630242" in it. I assume this should be an
>> operator uid or name. It looks like something else. What is it? Is
>> something corrupted?
>>
>>
>> org.apache.flink.runtime.client.JobInitializationException: Could not instantiate JobManager.
>> at org.apache.flink.runtime.dispatcher.Dispatcher.lambda$createJobManagerRunner$5(Dispatcher.java:494)
>> at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> at java.lang.Thread.run(Thread.java:748)
>> Caused by: java.lang.IllegalStateException: Failed to rollback to checkpoint/savepoint s3a://my-flink-state/checkpoints/ce9b90eafde97ca4629c13936c34426f/chk-626. Cannot map checkpoint/savepoint state for operator 811d3b279c8b26ed99ff0883b7630242 to the new program, because the operator is not available in the new program. If you want to allow to skip this, you can set the --allowNonRestoredState option on the CLI.
>> at org.apache.flink.runtime.checkpoint.Checkpoints.throwNonRestoredStateException(Checkpoints.java:226)
>> at org.apache.flink.runtime.checkpoint.Checkpoints.loadAndValidateCheckpoint(Checkpoints.java:190)
>> at org.apache.flink.runtime.checkpoint.CheckpointCoordinator.restoreSavepoint(CheckpointCoordinator.java:1627)
>> at org.apache.flink.runtime.scheduler.SchedulerBase.tryRestoreExecutionGraphFromSavepoint(SchedulerBase.java:362)
>> at org.apache.flink.runtime.scheduler.SchedulerBase.createAndRestoreExecutionGraph(SchedulerBase.java:292)
>> at org.apache.flink.runtime.scheduler.SchedulerBase.<init>(SchedulerBase.java:249)
>> at org.apache.flink.runtime.scheduler.DefaultScheduler.<init>(DefaultScheduler.java:133)
>> at org.apache.flink.runtime.scheduler.DefaultSchedulerFactory.createInstance(DefaultSchedulerFactory.java:111)
>> at org.apache.flink.runtime.jobmaster.JobMaster.createScheduler(JobMaster.java:345)
>> at org.apache.flink.runtime.jobmaster.JobMaster.<init>(JobMaster.java:330)
>> at org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.createJobMasterService(DefaultJobMasterServiceFactory.java:95)
>> at org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.createJobMasterService(DefaultJobMasterServiceFactory.java:39)
>> at org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.<init>(JobManagerRunnerImpl.java:162)
>> at org.apache.flink.runtime.dispatcher.DefaultJobManagerRunnerFactory.createJobManagerRunner(DefaultJobManagerRunnerFactory.java:86)
>> at org.apache.flink.runtime.dispatcher.Dispatcher.lambda$createJobManagerRunner$5(Dispatcher.java:478)
>> ... 4 more
>>
>>
Re: Weird operator ID check restore from checkpoint fails
Posted by Dan Hill <qu...@gmail.com>.
I'm restoring the job with the same binary and same flags/args.
On Tue, Dec 7, 2021 at 8:48 PM Dan Hill <qu...@gmail.com> wrote:
> Hi. I noticed this warning has "operator
> 811d3b279c8b26ed99ff0883b7630242" in it. I assume this should be an
> operator uid or name. It looks like something else. What is it? Is
> something corrupted?
>
>
> org.apache.flink.runtime.client.JobInitializationException: Could not instantiate JobManager.
> at org.apache.flink.runtime.dispatcher.Dispatcher.lambda$createJobManagerRunner$5(Dispatcher.java:494)
> at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.IllegalStateException: Failed to rollback to checkpoint/savepoint s3a://my-flink-state/checkpoints/ce9b90eafde97ca4629c13936c34426f/chk-626. Cannot map checkpoint/savepoint state for operator 811d3b279c8b26ed99ff0883b7630242 to the new program, because the operator is not available in the new program. If you want to allow to skip this, you can set the --allowNonRestoredState option on the CLI.
> at org.apache.flink.runtime.checkpoint.Checkpoints.throwNonRestoredStateException(Checkpoints.java:226)
> at org.apache.flink.runtime.checkpoint.Checkpoints.loadAndValidateCheckpoint(Checkpoints.java:190)
> at org.apache.flink.runtime.checkpoint.CheckpointCoordinator.restoreSavepoint(CheckpointCoordinator.java:1627)
> at org.apache.flink.runtime.scheduler.SchedulerBase.tryRestoreExecutionGraphFromSavepoint(SchedulerBase.java:362)
> at org.apache.flink.runtime.scheduler.SchedulerBase.createAndRestoreExecutionGraph(SchedulerBase.java:292)
> at org.apache.flink.runtime.scheduler.SchedulerBase.<init>(SchedulerBase.java:249)
> at org.apache.flink.runtime.scheduler.DefaultScheduler.<init>(DefaultScheduler.java:133)
> at org.apache.flink.runtime.scheduler.DefaultSchedulerFactory.createInstance(DefaultSchedulerFactory.java:111)
> at org.apache.flink.runtime.jobmaster.JobMaster.createScheduler(JobMaster.java:345)
> at org.apache.flink.runtime.jobmaster.JobMaster.<init>(JobMaster.java:330)
> at org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.createJobMasterService(DefaultJobMasterServiceFactory.java:95)
> at org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.createJobMasterService(DefaultJobMasterServiceFactory.java:39)
> at org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.<init>(JobManagerRunnerImpl.java:162)
> at org.apache.flink.runtime.dispatcher.DefaultJobManagerRunnerFactory.createJobManagerRunner(DefaultJobManagerRunnerFactory.java:86)
> at org.apache.flink.runtime.dispatcher.Dispatcher.lambda$createJobManagerRunner$5(Dispatcher.java:478)
> ... 4 more
>
>