[jira] [Created] (FLINK-27855) Job Manager fails to recover with S3 storage and HA enabled

["Sushant (Jira)" <ji...@apache.org>]

Sushant created FLINK-27855:
-------------------------------

             Summary: Job Manager fails to recover with S3 storage and HA enabled
                 Key: FLINK-27855
                 URL: https://issues.apache.org/jira/browse/FLINK-27855
             Project: Flink
          Issue Type: Bug
          Components: Kubernetes Operator
            Reporter: Sushant


Steps to replicate
1. Enable HA and mention S3 path in flink configuration
2. Create the flink deployment and let it run for sometime to generate checkpoints
3. Delete the flink deployment
4. Recreate once again and job manager doesn’t come up complaining about S3 server cleanup

Note that above steps goes through fine if  AWS EFS is being used instead of S3 for HA


Error Traceback: 


{code:java}
2022-05-31 16:39:44,332 WARN  org.apache.flink.runtime.dispatcher.cleanup.DefaultResourceCleaner [] - Cleanup of BlobServer failed for job 00000000000000000000000000000000 due to a CompletionException: java.io.IOException: java.io.IOException: Error while cleaning up the BlobStore for job 00000000000000000000000000000000

2022-05-31 16:42:56,955 WARN  org.apache.flink.runtime.dispatcher.StandaloneDispatcher     [] - Ignoring JobGraph submission (00000000000000000000000000000000) because the job already reached a globally-terminal state (i.e. FAILED, CANCELED, FINISHED) in a previous execution.
2022-05-31 16:42:57,026 ERROR              [] - Error while processing events :
org.apache.flink.util.FlinkException: Failed to execute job
	at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:2108) ~[flink-dist-1.15.0.jar:1.15.0]
Caused by: org.apache.flink.runtime.client.DuplicateJobSubmissionException: Job has already been submitted.
	at org.apache.flink.runtime.client.DuplicateJobSubmissionException.ofGloballyTerminated(DuplicateJobSubmissionException.java:35) ~[flink-dist-1.15.0.jar:1.15.0]
2022-05-31 16:42:57,130 INFO  org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap [] - Application CANCELED:
java.util.concurrent.CompletionException: org.apache.flink.client.deployment.application.UnsuccessfulExecutionException: Application Status: CANCELED
	at org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.lambda$unwrapJobResultException$6(ApplicationDispatcherBootstrap.java:389) ~[flink-dist-1.15.0.jar:1.15.0]
	at java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source) [?:?]
Caused by: org.apache.flink.client.deployment.application.UnsuccessfulExecutionException: Application Status: CANCELED
	at org.apache.flink.client.deployment.application.UnsuccessfulExecutionException.fromJobResult(UnsuccessfulExecutionException.java:71) ~[flink-dist-1.15.0.jar:1.15.0]
	... 56 more
Caused by: org.apache.flink.runtime.client.JobCancellationException: Job was cancelled.
	at org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:146) ~[flink-dist-1.15.0.jar:1.15.0]
	at org.apache.flink.client.deployment.application.UnsuccessfulExecutionException.fromJobResult(UnsuccessfulExecutionException.java:60) ~[flink-dist-1.15.0.jar:1.15.0]
	... 56 more
{code}




--
This message was sent by Atlassian Jira
(v8.20.7#820007)