You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Sigalit Eliazov <e....@gmail.com> on 2024/08/14 10:07:19 UTC

flink operator - restart a pipeline from a manually trigger savepoint

hi,

We are trying to restart a pipeline from a save point we triggered manually
via the job manager rest api.

with the following configuration in the flinkdeployment crd:
savepointTriggerNonce: 1
initialSavepointPath: <the save point path>
upgradeMode: savepoint
this always fails with the following error

  org.apache.flink.runtime.dispatcher.StandaloneDispatcher     [] -
Ignoring JobGraph submission 'XXX' (33427c11cef9a7ff48edfd5341b34e28)
because the job already reached a globally-terminal state (i.e. FAILED,
CANCELED, FINISHED) in a previous
execution.org.apache.flink.util.FlinkException: Failed to execute job ‘XXX’.

at
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:2253)
~[flink-dist-1.18.1.jar:1.18.1]

at
org.apache.flink.client.program.StreamContextEnvironment.executeAsync(StreamContextEnvironment.java:189)
~[flink-dist-1.18.1.jar:1.18.1]

at
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:2219)
~[flink-dist-1.18.1.jar:1.18.1]

at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:?]

at
jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:?]

at
jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:?]

at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]

at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355)
~[flink-dist-1.18.1.jar:1.18.1]

at
org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)
~[flink-dist-1.18.1.jar:1.18.1]

at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:105)
~[flink-dist-1.18.1.jar:1.18.1]

at
org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.runApplicationEntryPoint(ApplicationDispatcherBootstrap.java:301)
~[flink-dist-1.18.1.jar:1.18.1]

at
org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.lambda$runApplicationAsync$2(ApplicationDispatcherBootstrap.java:254)
~[flink-dist-1.18.1.jar:1.18.1]

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
~[?:?]

at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]

at
org.apache.flink.runtime.concurrent.pekko.ActorSystemScheduledExecutorAdapter$ScheduledFutureTask.run(ActorSystemScheduledExecutorAdapter.java:172)
~[?:?]

at
org.apache.flink.runtime.concurrent.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:68)
~[flink-dist-1.18.1.jar:1.18.1]

at
org.apache.flink.runtime.concurrent.ClassLoadingUtils.lambda$withContextClassLoader$0(ClassLoadingUtils.java:41)
~[flink-dist-1.18.1.jar:1.18.1]

at
org.apache.pekko.dispatch.TaskInvocation.run(AbstractDispatcher.scala:59)
[flink-rpc-akkae0beeada-9e90-40ec-abe6-dc401bbf1f51.jar:1.18.1]

at
org.apache.pekko.dispatch.ForkJoinExecutorConfigurator$PekkoForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:57)
[flink-rpc-akkae0beeada-9e90-40ec-abe6-dc401bbf1f51.jar:1.18.1]

at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) [?:?]

at
java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
[?:?]

at java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656) [?:?]

at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594) [?:?]

at
java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
[?:?]

Caused by: org.apache.flink.runtime.client.DuplicateJobSubmissionException:
Job has already been submitted.


We are using flink operator 1.7 and flink 1.18
When using the flink operator automatic savepoint mechanism everything
works great.

Any ideas?

Thanks
Sigalit

Re: flink operator - restart a pipeline from a manually trigger savepoint

Posted by Rion Williams <ri...@gmail.com>.
Hi Sigalit,

To restart from an explicit savepoint, you need to not only set your initialSavepointPath but also the savepointRedeployNonce which should trigger the job to restart using that savepoint. 

Try setting/incrementing that property and you should see the job run from your manual savepoint.

Hope that helps,

Rion

> On Aug 14, 2024, at 6:26 AM, Sigalit Eliazov <e....@gmail.com> wrote:
> 
> 
> hi,
> 
> We are trying to restart a pipeline from a save point we triggered manually via the job manager rest api.
> 
> with the following configuration in the flinkdeployment crd:
> savepointTriggerNonce: 1
> initialSavepointPath: <the save point path>
> upgradeMode: savepoint
> this always fails with the following error
> 
>   org.apache.flink.runtime.dispatcher.StandaloneDispatcher     [] - Ignoring JobGraph submission 'XXX' (33427c11cef9a7ff48edfd5341b34e28) because the job already reached a globally-terminal state (i.e. FAILED, CANCELED, FINISHED) in a previous execution.org.apache.flink.util.FlinkException: Failed to execute job ‘XXX’.
> 	at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:2253) ~[flink-dist-1.18.1.jar:1.18.1]
> 	at org.apache.flink.client.program.StreamContextEnvironment.executeAsync(StreamContextEnvironment.java:189) ~[flink-dist-1.18.1.jar:1.18.1]
> 	at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:2219) ~[flink-dist-1.18.1.jar:1.18.1]
> 	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
> 	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
> 	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
> 	at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
> 	at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355) ~[flink-dist-1.18.1.jar:1.18.1]
> 	at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222) ~[flink-dist-1.18.1.jar:1.18.1]
> 	at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:105) ~[flink-dist-1.18.1.jar:1.18.1]
> 	at org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.runApplicationEntryPoint(ApplicationDispatcherBootstrap.java:301) ~[flink-dist-1.18.1.jar:1.18.1]
> 	at org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.lambda$runApplicationAsync$2(ApplicationDispatcherBootstrap.java:254) ~[flink-dist-1.18.1.jar:1.18.1]
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
> 	at org.apache.flink.runtime.concurrent.pekko.ActorSystemScheduledExecutorAdapter$ScheduledFutureTask.run(ActorSystemScheduledExecutorAdapter.java:172) ~[?:?]
> 	at org.apache.flink.runtime.concurrent.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:68) ~[flink-dist-1.18.1.jar:1.18.1]
> 	at org.apache.flink.runtime.concurrent.ClassLoadingUtils.lambda$withContextClassLoader$0(ClassLoadingUtils.java:41) ~[flink-dist-1.18.1.jar:1.18.1]
> 	at org.apache.pekko.dispatch.TaskInvocation.run(AbstractDispatcher.scala:59) [flink-rpc-akkae0beeada-9e90-40ec-abe6-dc401bbf1f51.jar:1.18.1]
> 	at org.apache.pekko.dispatch.ForkJoinExecutorConfigurator$PekkoForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:57) [flink-rpc-akkae0beeada-9e90-40ec-abe6-dc401bbf1f51.jar:1.18.1]
> 	at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) [?:?]
> 	at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020) [?:?]
> 	at java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656) [?:?]
> 	at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594) [?:?]
> 	at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183) [?:?]
> Caused by: org.apache.flink.runtime.client.DuplicateJobSubmissionException: Job has already been submitted.
> 
> We are using flink operator 1.7 and flink 1.18
> When using the flink operator automatic savepoint mechanism everything works great.
> 
> Any ideas?
> 
> Thanks
> Sigalit
> 
>