You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "zyh (Jira)" <ji...@apache.org> on 2024/10/18 07:22:00 UTC
[jira] [Created] (FLINK-36570) Fatal error occurred in the cluster entrypoint
zyh created FLINK-36570:
---------------------------
Summary: Fatal error occurred in the cluster entrypoint
Key: FLINK-36570
URL: https://issues.apache.org/jira/browse/FLINK-36570
Project: Flink
Issue Type: Bug
Components: Deployment / Kubernetes
Affects Versions: 1.17.2
Environment: * flink 1.17.2
* native kubernetes session cluster HA(3 jobmanager replica)
Reporter: zyh
I commit batch jobs to my session cluster with rest api. The jobmanager pod would restart when occurred the error.
Seems to be because the new leader elected and exist running job at the same time. Then the job send to the new leader and error.
{code:java}
2024-10-18 03:07:22,107 INFO org.apache.flink.client.deployment.application.executors.EmbeddedExecutor [] - Submitting Job with JobId=a9d339b6ba26ab51746514cc7aea0537.2024-10-18 03:07:22,546 INFO org.apache.flink.kubernetes.kubeclient.resources.KubernetesLeaderElector [] - New leader elected 57c4be1d-58f0-4c2c-89d8-11aefe1ec273 for flink-cluster-cluster-config-map.2024-10-18 03:07:22,549 ERROR org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Fatal error occurred in the cluster entrypoint.org.apache.flink.util.FlinkException: JobMaster for job 99c02051a54c77499f53f09cd4b7a0d9 failed.at org.apache.flink.runtime.dispatcher.Dispatcher.jobMasterFailed(Dispatcher.java:1360) ~[flink-dist-1.17.2.jar:1.17.2]
at org.apache.flink.runtime.dispatcher.Dispatcher.jobManagerRunnerFailed(Dispatcher.java:772) ~[flink-dist-1.17.2.jar:1.17.2]
at org.apache.flink.runtime.dispatcher.Dispatcher.lambda$runJob$6(Dispatcher.java:694) ~[flink-dist-1.17.2.jar:1.17.2]
at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) ~[?:?]
at java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source) ~[?:?]
at java.util.concurrent.CompletableFuture$Completion.run(Unknown Source) ~[?:?]
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.lambda$handleRunAsync$4(AkkaRpcActor.java:453) ~[flink-rpc-akka_27725420-d3ff-407e-864b-d8e6936565db.jar:1.17.2]
at org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:68) ~[flink-rpc-akka_27725420-d3ff-407e-864b-d8e6936565db.jar:1.17.2]
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:453) ~[flink-rpc-akka_27725420-d3ff-407e-864b-d8e6936565db.jar:1.17.2]
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:218) ~[flink-rpc-akka_27725420-d3ff-407e-864b-d8e6936565db.jar:1.17.2]
at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:84) ~[flink-rpc-akka_27725420-d3ff-407e-864b-d8e6936565db.jar:1.17.2]
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:168) ~[flink-rpc-akka_27725420-d3ff-407e-864b-d8e6936565db.jar:1.17.2]
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:24) [flink-rpc-akka_27725420-d3ff-407e-864b-d8e6936565db.jar:1.17.2]
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:20) [flink-rpc-akka_27725420-d3ff-407e-864b-d8e6936565db.jar:1.17.2]
at scala.PartialFunction.applyOrElse(PartialFunction.scala:127) [flink-rpc-akka_27725420-d3ff-407e-864b-d8e6936565db.jar:1.17.2]
at scala.PartialFunction.applyOrElse$(PartialFunction.scala:126) [flink-rpc-akka_27725420-d3ff-407e-864b-d8e6936565db.jar:1.17.2]
at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:20) [flink-rpc-akka_27725420-d3ff-407e-864b-d8e6936565db.jar:1.17.2]
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:175) [flink-rpc-akka_27725420-d3ff-407e-864b-d8e6936565db.jar:1.17.2]
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:176) [flink-rpc-akka_27725420-d3ff-407e-864b-d8e6936565db.jar:1.17.2]
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:176) [flink-rpc-akka_27725420-d3ff-407e-864b-d8e6936565db.jar:1.17.2]
at akka.actor.Actor.aroundReceive(Actor.scala:537) [flink-rpc-akka_27725420-d3ff-407e-864b-d8e6936565db.jar:1.17.2]
at akka.actor.Actor.aroundReceive$(Actor.scala:535) [flink-rpc-akka_27725420-d3ff-407e-864b-d8e6936565db.jar:1.17.2]
at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:220) [flink-rpc-akka_27725420-d3ff-407e-864b-d8e6936565db.jar:1.17.2]
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:579) [flink-rpc-akka_27725420-d3ff-407e-864b-d8e6936565db.jar:1.17.2]
at akka.actor.ActorCell.invoke(ActorCell.scala:547) [flink-rpc-akka_27725420-d3ff-407e-864b-d8e6936565db.jar:1.17.2]
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270) [flink-rpc-akka_27725420-d3ff-407e-864b-d8e6936565db.jar:1.17.2]
at akka.dispatch.Mailbox.run(Mailbox.scala:231) [flink-rpc-akka_27725420-d3ff-407e-864b-d8e6936565db.jar:1.17.2]
at akka.dispatch.Mailbox.exec(Mailbox.scala:243) [flink-rpc-akka_27725420-d3ff-407e-864b-d8e6936565db.jar:1.17.2]
at java.util.concurrent.ForkJoinTask.doExec(Unknown Source) [?:?]
at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source) [?:?]
at java.util.concurrent.ForkJoinPool.scan(Unknown Source) [?:?]
at java.util.concurrent.ForkJoinPool.runWorker(Unknown Source) [?:?]
at java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source) [?:?]
ed by: org.apache.flink.util.FlinkException: Could not suspend the job manager.
at org.apache.flink.runtime.jobmaster.JobMasterServiceLeadershipRunner.lambda$null$13(JobMasterServiceLeadershipRunner.java:438) ~[flink-dist-1.17.2.jar:1.17.2]
at org.apache.flink.runtime.jobmaster.JobMasterServiceLeadershipRunner.runIfStateRunning(JobMasterServiceLeadershipRunner.java:456) ~[flink-dist-1.17.2.jar:1.17.2]
at org.apache.flink.runtime.jobmaster.JobMasterServiceLeadershipRunner.lambda$handleAsyncOperationError$14(JobMasterServiceLeadershipRunner.java:436) ~[flink-dist-1.17.2.jar:1.17.2]
at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) ~[?:?]
at java.util.concurrent.CompletableFuture.uniWhenCompleteStage(Unknown Source) ~[?:?]
at java.util.concurrent.CompletableFuture.whenComplete(Unknown Source) ~[?:?]
at org.apache.flink.runtime.jobmaster.JobMasterServiceLeadershipRunner.handleAsyncOperationError(JobMasterServiceLeadershipRunner.java:433) ~[flink-dist-1.17.2.jar:1.17.2]
at org.apache.flink.runtime.jobmaster.JobMasterServiceLeadershipRunner.stopJobMasterServiceProcessAsync(JobMasterServiceLeadershipRunner.java:405) ~[flink-dist-1.17.2.jar:1.17.2]
at org.apache.flink.runtime.jobmaster.JobMasterServiceLeadershipRunner.runIfStateRunning(JobMasterServiceLeadershipRunner.java:456) ~[flink-dist-1.17.2.jar:1.17.2]
at org.apache.flink.runtime.jobmaster.JobMasterServiceLeadershipRunner.revokeLeadership(JobMasterServiceLeadershipRunner.java:390) ~[flink-dist-1.17.2.jar:1.17.2]
at org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService.onRevokeLeadership(DefaultLeaderElectionService.java:236) ~[flink-dist-1.17.2.jar:1.17.2]
at org.apache.flink.runtime.leaderelection.DefaultMultipleComponentLeaderElectionService.lambda$forEachLeaderElectionEventHandler$2(DefaultMultipleComponentLeaderElectionService.java:225) ~[flink-dist-1.17.2.jar:1.17.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
at java.lang.Thread.run(Unknown Source) ~[?:?]
ed by: java.util.concurrent.CompletionException: java.lang.UnsupportedOperationException: Still waiting for the leadership.
at java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source) ~[?:?]
at java.util.concurrent.CompletableFuture.uniComposeStage(Unknown Source) ~[?:?]
at java.util.concurrent.CompletableFuture.thenCompose(Unknown Source) ~[?:?]
at org.apache.flink.runtime.jobmaster.JobMasterServiceLeadershipRunner.stopJobMasterServiceProcessAsync(JobMasterServiceLeadershipRunner.java:398) ~[flink-dist-1.17.2.jar:1.17.2]
at org.apache.flink.runtime.jobmaster.JobMasterServiceLeadershipRunner.runIfStateRunning(JobMasterServiceLeadershipRunner.java:456) ~[flink-dist-1.17.2.jar:1.17.2]
at org.apache.flink.runtime.jobmaster.JobMasterServiceLeadershipRunner.revokeLeadership(JobMasterServiceLeadershipRunner.java:390) ~[flink-dist-1.17.2.jar:1.17.2]
at org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService.onRevokeLeadership(DefaultLeaderElectionService.java:236) ~[flink-dist-1.17.2.jar:1.17.2]
at org.apache.flink.runtime.leaderelection.DefaultMultipleComponentLeaderElectionService.lambda$forEachLeaderElectionEventHandler$2(DefaultMultipleComponentLeaderElectionService.java:225) ~[flink-dist-1.17.2.jar:1.17.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
at java.lang.Thread.run(Unknown Source) ~[?:?]
ed by: java.lang.UnsupportedOperationException: Still waiting for the leadership.
at org.apache.flink.runtime.jobmaster.JobMasterServiceProcess$WaitingForLeadership.getLeaderSessionId(JobMasterServiceProcess.java:71) ~[flink-dist-1.17.2.jar:1.17.2]
at org.apache.flink.runtime.jobmaster.JobMasterServiceLeadershipRunner.stopJobMasterServiceProcess(JobMasterServiceLeadershipRunner.java:414) ~[flink-dist-1.17.2.jar:1.17.2]
at org.apache.flink.runtime.jobmaster.JobMasterServiceLeadershipRunner.callIfRunning(JobMasterServiceLeadershipRunner.java:469) ~[flink-dist-1.17.2.jar:1.17.2]
at org.apache.flink.runtime.jobmaster.JobMasterServiceLeadershipRunner.lambda$stopJobMasterServiceProcessAsync$12(JobMasterServiceLeadershipRunner.java:400) ~[flink-dist-1.17.2.jar:1.17.2]
at java.util.concurrent.CompletableFuture.uniComposeStage(Unknown Source) ~[?:?]
at java.util.concurrent.CompletableFuture.thenCompose(Unknown Source) ~[?:?]
at org.apache.flink.runtime.jobmaster.JobMasterServiceLeadershipRunner.stopJobMasterServiceProcessAsync(JobMasterServiceLeadershipRunner.java:398) ~[flink-dist-1.17.2.jar:1.17.2]
at org.apache.flink.runtime.jobmaster.JobMasterServiceLeadershipRunner.runIfStateRunning(JobMasterServiceLeadershipRunner.java:456) ~[flink-dist-1.17.2.jar:1.17.2]
at org.apache.flink.runtime.jobmaster.JobMasterServiceLeadershipRunner.revokeLeadership(JobMasterServiceLeadershipRunner.java:390) ~[flink-dist-1.17.2.jar:1.17.2]
at org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService.onRevokeLeadership(DefaultLeaderElectionService.java:236) ~[flink-dist-1.17.2.jar:1.17.2]
at org.apache.flink.runtime.leaderelection.DefaultMultipleComponentLeaderElectionService.lambda$forEachLeaderElectionEventHandler$2(DefaultMultipleComponentLeaderElectionService.java:225) ~[flink-dist-1.17.2.jar:1.17.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
at java.lang.Thread.run(Unknown Source) ~[?:?]
INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Shutting KubernetesSessionClusterEntrypoint down with application status UNKNOWN. Diagnostics Cluster entrypoint has been closed externally..INFO org.apache.flink.runtime.blob.BlobServer [] - Stopped BLOB server at 0.0.0.0:6124
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)