You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@streampark.apache.org by "lx-jialin (via GitHub)" <gi...@apache.org> on 2023/02/27 03:20:17 UTC
[GitHub] [incubator-streampark] lx-jialin opened a new issue, #2369: k8s 运行报错 io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager [] - Exec Failure: HTTP 403, Status: 403 - Forbidden
lx-jialin opened a new issue, #2369:
URL: https://github.com/apache/incubator-streampark/issues/2369
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-streampark/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### Java Version
11
### Scala Version
2.11.x
### StreamPark Version
2.0.0
### Flink Version
1.6
### deploy mode
None
### What happened
在k8s 集群上 application 运行报错如下：
### Error Exception
```log
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.flink.api.java.ClosureCleaner (file:/opt/flink/lib/flink-dist-1.16.1.jar) to field java.lang.String.value
WARNING: Please consider reporting this to the maintainers of org.apache.flink.api.java.ClosureCleaner
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2023-02-27 03:00:43,121 INFO org.apache.flink.client.deployment.application.executors.EmbeddedExecutor [] - Job fc4e8c6d3349089d79a45e546f6f4966 is submitted.
2023-02-27 03:00:43,122 INFO org.apache.flink.client.deployment.application.executors.EmbeddedExecutor [] - Submitting Job with JobId=fc4e8c6d3349089d79a45e546f6f4966.
2023-02-27 03:00:45,619 WARN org.apache.flink.configuration.GlobalConfiguration [] - Error while trying to split key and value in configuration file /opt/flink/conf/flink-conf.yaml:6: "$internal.application.program-args: "
2023-02-27 03:00:45,620 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: blob.server.port, 6124
2023-02-27 03:00:45,620 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: $internal.application.main, com.flink.demo.job.Job1Main
2023-02-27 03:00:45,620 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: taskmanager.memory.process.size, 1728m
2023-02-27 03:00:45,621 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: state.checkpoints.num-retained, 1
2023-02-27 03:00:45,621 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: kubernetes.internal.jobmanager.entrypoint.class, org.apache.flink.kubernetes.entrypoint.KubernetesApplicationClusterEntrypoint
2023-02-27 03:00:45,621 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: pipeline.name, flink-my-job
2023-02-27 03:00:45,621 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: classloader.resolve-order, parent-first
2023-02-27 03:00:45,622 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.execution.failover-strategy, region
2023-02-27 03:00:45,622 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.rpc.address, flink-my-job.default
2023-02-27 03:00:45,622 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: execution.target, kubernetes-application
2023-02-27 03:00:45,622 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.memory.process.size, 1600m
2023-02-27 03:00:45,623 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: kubernetes.cluster-id, flink-my-job
2023-02-27 03:00:45,623 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.rpc.port, 6123
2023-02-27 03:00:45,623 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: taskmanager.rpc.port, 6122
2023-02-27 03:00:45,624 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: kubernetes.container.image.pull-policy, Always
2023-02-27 03:00:45,624 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: internal.cluster.execution-mode, NORMAL
2023-02-27 03:00:45,624 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: $internal.pipeline.job-id, fc4e8c6d3349089d79a45e546f6f4966
2023-02-27 03:00:45,625 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: kubernetes.container.image, 10.96.183.34:5000/streampark/streamparkflinkjob-default-flink-my-job
2023-02-27 03:00:45,625 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: pipeline.jars, local:///opt/flink/usrlib/streampark-flinkjob_flink-my-job.jar
2023-02-27 03:00:45,625 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: parallelism.default, 1
2023-02-27 03:00:45,626 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: kubernetes.namespace, default
2023-02-27 03:00:45,626 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: taskmanager.numberOfTaskSlots, 1
2023-02-27 03:00:45,626 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: kubernetes.rest-service.exposed.type, LoadBalancer
2023-02-27 03:00:45,626 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: rest.address, localhost
2023-02-27 03:00:45,718 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService [] - Starting RPC endpoint for org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager at akka://flink/user/rpc/resourcemanager_1 .
2023-02-27 03:00:45,824 INFO org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - Starting the resource manager.
2023-02-27 03:00:48,647 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] - Received JobGraph submission 'flink-my-job' (fc4e8c6d3349089d79a45e546f6f4966).
2023-02-27 03:00:48,649 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] - Submitting job 'flink-my-job' (fc4e8c6d3349089d79a45e546f6f4966).
2023-02-27 03:00:49,028 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService [] - Starting RPC endpoint for org.apache.flink.runtime.jobmaster.JobMaster at akka://flink/user/rpc/jobmanager_2 .
2023-02-27 03:00:49,123 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Initializing job 'flink-my-job' (fc4e8c6d3349089d79a45e546f6f4966).
2023-02-27 03:00:49,432 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Using restart back off time strategy NoRestartBackoffTimeStrategy for flink-my-job (fc4e8c6d3349089d79a45e546f6f4966).
2023-02-27 03:00:49,657 WARN io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager [] - Exec Failure: HTTP 403, Status: 403 - Forbidden
2023-02-27 03:00:49,739 WARN io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager [] - Exec Failure: HTTP 403, Status: 403 - Forbidden
2023-02-27 03:00:49,813 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Created execution graph b2700d2abda2be70fced98dbe895c42c for job fc4e8c6d3349089d79a45e546f6f4966.
2023-02-27 03:00:49,823 WARN io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager [] - Exec Failure: HTTP 403, Status: 403 - Forbidden
2023-02-27 03:00:49,835 WARN io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager [] - Exec Failure: HTTP 403, Status: 403 - Forbidden
2023-02-27 03:00:49,917 WARN io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager [] - Exec Failure: HTTP 403, Status: 403 - Forbidden
2023-02-27 03:00:49,928 WARN io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager [] - Exec Failure: HTTP 403, Status: 403 - Forbidden
2023-02-27 03:00:49,934 INFO org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager [] - Closing the slot manager.
2023-02-27 03:00:49,935 INFO org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager [] - Suspending the slot manager.
2023-02-27 03:00:49,935 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Running initialization on master for job flink-my-job (fc4e8c6d3349089d79a45e546f6f4966).
2023-02-27 03:00:50,013 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Successfully ran initialization on master in 77 ms.
2023-02-27 03:00:49,938 ERROR org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - Fatal error occurred in ResourceManager.
org.apache.flink.runtime.resourcemanager.exceptions.ResourceManagerException: Could not start the ResourceManager akka.tcp://flink@flink-my-job.default:6123/user/rpc/resourcemanager_1
at org.apache.flink.runtime.resourcemanager.ResourceManager.onStart(ResourceManager.java:246) ~[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.runtime.rpc.RpcEndpoint.internalCallOnStart(RpcEndpoint.java:198) ~[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor$StoppedState.lambda$start$0(AkkaRpcActor.java:622) ~[flink-rpc-akka_9af1f9d4-313a-43c3-9244-adcaf2178cb9.jar:1.16.1]
at org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:68) ~[flink-rpc-akka_9af1f9d4-313a-43c3-9244-adcaf2178cb9.jar:1.16.1]
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor$StoppedState.start(AkkaRpcActor.java:621) ~[flink-rpc-akka_9af1f9d4-313a-43c3-9244-adcaf2178cb9.jar:1.16.1]
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleControlMessage(AkkaRpcActor.java:190) ~[flink-rpc-akka_9af1f9d4-313a-43c3-9244-adcaf2178cb9.jar:1.16.1]
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:24) [flink-rpc-akka_9af1f9d4-313a-43c3-9244-adcaf2178cb9.jar:1.16.1]
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:20) [flink-rpc-akka_9af1f9d4-313a-43c3-9244-adcaf2178cb9.jar:1.16.1]
at scala.PartialFunction.applyOrElse(PartialFunction.scala:123) [flink-rpc-akka_9af1f9d4-313a-43c3-9244-adcaf2178cb9.jar:1.16.1]
at scala.PartialFunction.applyOrElse$(PartialFunction.scala:122) [flink-rpc-akka_9af1f9d4-313a-43c3-9244-adcaf2178cb9.jar:1.16.1]
at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:20) [flink-rpc-akka_9af1f9d4-313a-43c3-9244-adcaf2178cb9.jar:1.16.1]
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) [flink-rpc-akka_9af1f9d4-313a-43c3-9244-adcaf2178cb9.jar:1.16.1]
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172) [flink-rpc-akka_9af1f9d4-313a-43c3-9244-adcaf2178cb9.jar:1.16.1]
at akka.actor.Actor.aroundReceive(Actor.scala:537) [flink-rpc-akka_9af1f9d4-313a-43c3-9244-adcaf2178cb9.jar:1.16.1]
at akka.actor.Actor.aroundReceive$(Actor.scala:535) [flink-rpc-akka_9af1f9d4-313a-43c3-9244-adcaf2178cb9.jar:1.16.1]
at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:220) [flink-rpc-akka_9af1f9d4-313a-43c3-9244-adcaf2178cb9.jar:1.16.1]
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:580) [flink-rpc-akka_9af1f9d4-313a-43c3-9244-adcaf2178cb9.jar:1.16.1]
at akka.actor.ActorCell.invoke(ActorCell.scala:548) [flink-rpc-akka_9af1f9d4-313a-43c3-9244-adcaf2178cb9.jar:1.16.1]
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270) [flink-rpc-akka_9af1f9d4-313a-43c3-9244-adcaf2178cb9.jar:1.16.1]
at akka.dispatch.Mailbox.run(Mailbox.scala:231) [flink-rpc-akka_9af1f9d4-313a-43c3-9244-adcaf2178cb9.jar:1.16.1]
at akka.dispatch.Mailbox.exec(Mailbox.scala:243) [flink-rpc-akka_9af1f9d4-313a-43c3-9244-adcaf2178cb9.jar:1.16.1]
at java.util.concurrent.ForkJoinTask.doExec(Unknown Source) [?:?]
at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source) [?:?]
at java.util.concurrent.ForkJoinPool.scan(Unknown Source) [?:?]
at java.util.concurrent.ForkJoinPool.runWorker(Unknown Source) [?:?]
at java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source) [?:?]
Caused by: org.apache.flink.runtime.resourcemanager.exceptions.ResourceManagerException: Cannot initialize resource provider.
at org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager.initialize(ActiveResourceManager.java:177) ~[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.runtime.resourcemanager.ResourceManager.startResourceManagerServices(ResourceManager.java:269) ~[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.runtime.resourcemanager.ResourceManager.onStart(ResourceManager.java:241) ~[flink-dist-1.16.1.jar:1.16.1]
... 25 more
Caused by: java.util.concurrent.ExecutionException: org.apache.flink.util.concurrent.FutureUtils$RetryException: Could not complete the operation. Number of retries has been exhausted.
at java.util.concurrent.CompletableFuture.reportGet(Unknown Source) ~[?:?]
at java.util.concurrent.CompletableFuture.get(Unknown Source) ~[?:?]
at org.apache.flink.kubernetes.kubeclient.Fabric8FlinkKubeClient.watchPodsAndDoCallback(Fabric8FlinkKubeClient.java:245) ~[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.kubernetes.KubernetesResourceManagerDriver.watchTaskManagerPods(KubernetesResourceManagerDriver.java:373) ~[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.kubernetes.KubernetesResourceManagerDriver.initializeInternal(KubernetesResourceManagerDriver.java:113) ~[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.runtime.resourcemanager.active.AbstractResourceManagerDriver.initialize(AbstractResourceManagerDriver.java:92) ~[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager.initialize(ActiveResourceManager.java:171) ~[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.runtime.resourcemanager.ResourceManager.startResourceManagerServices(ResourceManager.java:269) ~[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.runtime.resourcemanager.ResourceManager.onStart(ResourceManager.java:241) ~[flink-dist-1.16.1.jar:1.16.1]
... 25 more
Caused by: org.apache.flink.util.concurrent.FutureUtils$RetryException: Could not complete the operation. Number of retries has been exhausted.
at org.apache.flink.util.concurrent.FutureUtils.lambda$retryOperation$1(FutureUtils.java:191) ~[flink-dist-1.16.1.jar:1.16.1]
at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) ~[?:?]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source) ~[?:?]
at java.util.concurrent.CompletableFuture$Completion.run(Unknown Source) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
at java.lang.Thread.run(Unknown Source) ~[?:?]
Caused by: java.util.concurrent.CompletionException: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://10.233.0.1/api/v1/namespaces/default/pods?labelSelector=app%3Dflink-my-job%2Ccomponent%3Dtaskmanager%2Ctype%3Dflink-native-kubernetes&allowWatchBookmarks=true&watch=true. Message: Forbidden.
at java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source) ~[?:?]
at java.util.concurrent.CompletableFuture.completeThrowable(Unknown Source) ~[?:?]
at java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
at java.lang.Thread.run(Unknown Source) ~[?:?]
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://10.233.0.1/api/v1/namespaces/default/pods?labelSelector=app%3Dflink-my-job%2Ccomponent%3Dtaskmanager%2Ctype%3Dflink-native-kubernetes&allowWatchBookmarks=true&watch=true. Message: Forbidden.
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:684) ~[flink-dist-1.16.1.jar:1.16.1]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:664) ~[flink-dist-1.16.1.jar:1.16.1]
at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager.lambda$run$2(WatchConnectionManager.java:126) ~[flink-dist-1.16.1.jar:1.16.1]
at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) ~[?:?]
at java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source) ~[?:?]
at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?]
at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?]
at io.fabric8.kubernetes.client.okhttp.OkHttpWebSocketImpl$BuilderImpl$1.onFailure(OkHttpWebSocketImpl.java:66) ~[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket.failWebSocket(RealWebSocket.java:570) ~[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket$1.onResponse(RealWebSocket.java:199) ~[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.kubernetes.shaded.okhttp3.RealCall$AsyncCall.execute(RealCall.java:174) ~[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.kubernetes.shaded.okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) ~[flink-dist-1.16.1.jar:1.16.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
at java.lang.Thread.run(Unknown Source) ~[?:?]
Suppressed: java.lang.Throwable: waiting here
at io.fabric8.kubernetes.client.utils.Utils.waitUntilReady(Utils.java:169) ~[flink-dist-1.16.1.jar:1.16.1]
at io.fabric8.kubernetes.client.utils.Utils.waitUntilReadyOrFail(Utils.java:180) ~[flink-dist-1.16.1.jar:1.16.1]
at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager.waitUntilReady(WatchConnectionManager.java:96) ~[flink-dist-1.16.1.jar:1.16.1]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.watch(BaseOperation.java:572) ~[flink-dist-1.16.1.jar:1.16.1]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.watch(BaseOperation.java:547) ~[flink-dist-1.16.1.jar:1.16.1]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.watch(BaseOperation.java:83) ~[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.kubernetes.kubeclient.Fabric8FlinkKubeClient.lambda$null$3(Fabric8FlinkKubeClient.java:236) ~[flink-dist-1.16.1.jar:1.16.1]
at java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
at java.lang.Thread.run(Unknown Source) ~[?:?]
2023-02-27 03:00:50,050 ERROR org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Fatal error occurred in the cluster entrypoint.
org.apache.flink.runtime.resourcemanager.exceptions.ResourceManagerException: Could not start the ResourceManager akka.tcp://flink@flink-my-job.default:6123/user/rpc/resourcemanager_1
at org.apache.flink.runtime.resourcemanager.ResourceManager.onStart(ResourceManager.java:246) ~[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.runtime.rpc.RpcEndpoint.internalCallOnStart(RpcEndpoint.java:198) ~[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor$StoppedState.lambda$start$0(AkkaRpcActor.java:622) ~[flink-rpc-akka_9af1f9d4-313a-43c3-9244-adcaf2178cb9.jar:1.16.1]
at org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:68) ~[flink-rpc-akka_9af1f9d4-313a-43c3-9244-adcaf2178cb9.jar:1.16.1]
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor$StoppedState.start(AkkaRpcActor.java:621) ~[flink-rpc-akka_9af1f9d4-313a-43c3-9244-adcaf2178cb9.jar:1.16.1]
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleControlMessage(AkkaRpcActor.java:190) ~[flink-rpc-akka_9af1f9d4-313a-43c3-9244-adcaf2178cb9.jar:1.16.1]
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:24) [flink-rpc-akka_9af1f9d4-313a-43c3-9244-adcaf2178cb9.jar:1.16.1]
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:20) [flink-rpc-akka_9af1f9d4-313a-43c3-9244-adcaf2178cb9.jar:1.16.1]
at scala.PartialFunction.applyOrElse(PartialFunction.scala:123) [flink-rpc-akka_9af1f9d4-313a-43c3-9244-adcaf2178cb9.jar:1.16.1]
at scala.PartialFunction.applyOrElse$(PartialFunction.scala:122) [flink-rpc-akka_9af1f9d4-313a-43c3-9244-adcaf2178cb9.jar:1.16.1]
at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:20) [flink-rpc-akka_9af1f9d4-313a-43c3-9244-adcaf2178cb9.jar:1.16.1]
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) [flink-rpc-akka_9af1f9d4-313a-43c3-9244-adcaf2178cb9.jar:1.16.1]
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172) [flink-rpc-akka_9af1f9d4-313a-43c3-9244-adcaf2178cb9.jar:1.16.1]
at akka.actor.Actor.aroundReceive(Actor.scala:537) [flink-rpc-akka_9af1f9d4-313a-43c3-9244-adcaf2178cb9.jar:1.16.1]
at akka.actor.Actor.aroundReceive$(Actor.scala:535) [flink-rpc-akka_9af1f9d4-313a-43c3-9244-adcaf2178cb9.jar:1.16.1]
at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:220) [flink-rpc-akka_9af1f9d4-313a-43c3-9244-adcaf2178cb9.jar:1.16.1]
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:580) [flink-rpc-akka_9af1f9d4-313a-43c3-9244-adcaf2178cb9.jar:1.16.1]
at akka.actor.ActorCell.invoke(ActorCell.scala:548) [flink-rpc-akka_9af1f9d4-313a-43c3-9244-adcaf2178cb9.jar:1.16.1]
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270) [flink-rpc-akka_9af1f9d4-313a-43c3-9244-adcaf2178cb9.jar:1.16.1]
at akka.dispatch.Mailbox.run(Mailbox.scala:231) [flink-rpc-akka_9af1f9d4-313a-43c3-9244-adcaf2178cb9.jar:1.16.1]
at akka.dispatch.Mailbox.exec(Mailbox.scala:243) [flink-rpc-akka_9af1f9d4-313a-43c3-9244-adcaf2178cb9.jar:1.16.1]
at java.util.concurrent.ForkJoinTask.doExec(Unknown Source) [?:?]
at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source) [?:?]
at java.util.concurrent.ForkJoinPool.scan(Unknown Source) [?:?]
at java.util.concurrent.ForkJoinPool.runWorker(Unknown Source) [?:?]
at java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source) [?:?]
Caused by: org.apache.flink.runtime.resourcemanager.exceptions.ResourceManagerException: Cannot initialize resource provider.
at org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager.initialize(ActiveResourceManager.java:177) ~[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.runtime.resourcemanager.ResourceManager.startResourceManagerServices(ResourceManager.java:269) ~[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.runtime.resourcemanager.ResourceManager.onStart(ResourceManager.java:241) ~[flink-dist-1.16.1.jar:1.16.1]
... 25 more
Caused by: java.util.concurrent.ExecutionException: org.apache.flink.util.concurrent.FutureUtils$RetryException: Could not complete the operation. Number of retries has been exhausted.
at java.util.concurrent.CompletableFuture.reportGet(Unknown Source) ~[?:?]
at java.util.concurrent.CompletableFuture.get(Unknown Source) ~[?:?]
at org.apache.flink.kubernetes.kubeclient.Fabric8FlinkKubeClient.watchPodsAndDoCallback(Fabric8FlinkKubeClient.java:245) ~[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.kubernetes.KubernetesResourceManagerDriver.watchTaskManagerPods(KubernetesResourceManagerDriver.java:373) ~[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.kubernetes.KubernetesResourceManagerDriver.initializeInternal(KubernetesResourceManagerDriver.java:113) ~[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.runtime.resourcemanager.active.AbstractResourceManagerDriver.initialize(AbstractResourceManagerDriver.java:92) ~[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager.initialize(ActiveResourceManager.java:171) ~[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.runtime.resourcemanager.ResourceManager.startResourceManagerServices(ResourceManager.java:269) ~[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.runtime.resourcemanager.ResourceManager.onStart(ResourceManager.java:241) ~[flink-dist-1.16.1.jar:1.16.1]
... 25 more
Caused by: org.apache.flink.util.concurrent.FutureUtils$RetryException: Could not complete the operation. Number of retries has been exhausted.
at org.apache.flink.util.concurrent.FutureUtils.lambda$retryOperation$1(FutureUtils.java:191) ~[flink-dist-1.16.1.jar:1.16.1]
at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) ~[?:?]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source) ~[?:?]
at java.util.concurrent.CompletableFuture$Completion.run(Unknown Source) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
at java.lang.Thread.run(Unknown Source) ~[?:?]
Caused by: java.util.concurrent.CompletionException: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://10.233.0.1/api/v1/namespaces/default/pods?labelSelector=app%3Dflink-my-job%2Ccomponent%3Dtaskmanager%2Ctype%3Dflink-native-kubernetes&allowWatchBookmarks=true&watch=true. Message: Forbidden.
at java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source) ~[?:?]
at java.util.concurrent.CompletableFuture.completeThrowable(Unknown Source) ~[?:?]
at java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
at java.lang.Thread.run(Unknown Source) ~[?:?]
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://10.233.0.1/api/v1/namespaces/default/pods?labelSelector=app%3Dflink-my-job%2Ccomponent%3Dtaskmanager%2Ctype%3Dflink-native-kubernetes&allowWatchBookmarks=true&watch=true. Message: Forbidden.
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:684) ~[flink-dist-1.16.1.jar:1.16.1]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:664) ~[flink-dist-1.16.1.jar:1.16.1]
at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager.lambda$run$2(WatchConnectionManager.java:126) ~[flink-dist-1.16.1.jar:1.16.1]
at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) ~[?:?]
at java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source) ~[?:?]
at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?]
at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?]
at io.fabric8.kubernetes.client.okhttp.OkHttpWebSocketImpl$BuilderImpl$1.onFailure(OkHttpWebSocketImpl.java:66) ~[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket.failWebSocket(RealWebSocket.java:570) ~[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket$1.onResponse(RealWebSocket.java:199) ~[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.kubernetes.shaded.okhttp3.RealCall$AsyncCall.execute(RealCall.java:174) ~[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.kubernetes.shaded.okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) ~[flink-dist-1.16.1.jar:1.16.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
at java.lang.Thread.run(Unknown Source) ~[?:?]
Suppressed: java.lang.Throwable: waiting here
at io.fabric8.kubernetes.client.utils.Utils.waitUntilReady(Utils.java:169) ~[flink-dist-1.16.1.jar:1.16.1]
at io.fabric8.kubernetes.client.utils.Utils.waitUntilReadyOrFail(Utils.java:180) ~[flink-dist-1.16.1.jar:1.16.1]
at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager.waitUntilReady(WatchConnectionManager.java:96) ~[flink-dist-1.16.1.jar:1.16.1]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.watch(BaseOperation.java:572) ~[flink-dist-1.16.1.jar:1.16.1]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.watch(BaseOperation.java:547) ~[flink-dist-1.16.1.jar:1.16.1]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.watch(BaseOperation.java:83) ~[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.kubernetes.kubeclient.Fabric8FlinkKubeClient.lambda$null$3(Fabric8FlinkKubeClient.java:236) ~[flink-dist-1.16.1.jar:1.16.1]
at java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
at java.lang.Thread.run(Unknown Source) ~[?:?]
2023-02-27 03:00:50,121 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Shutting KubernetesApplicationClusterEntrypoint down with application status UNKNOWN. Diagnostics Cluster entrypoint has been closed externally..
2023-02-27 03:00:50,138 INFO org.apache.flink.runtime.blob.BlobServer [] - Stopped BLOB server at 0.0.0.0:6124
2023-02-27 03:00:51,035 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - No checkpoint found during restore.
2023-02-27 03:00:51,423 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Stopping checkpoint coordinator for job fc4e8c6d3349089d79a45e546f6f4966.
```
### Screenshots
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!(您是否要贡献这个PR?)
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] wolfboys commented on issue #2369: k8s 运行报错 io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager [] - Exec Failure: HTTP 403, Status: 403 - Forbidden
Posted by "wolfboys (via GitHub)" <gi...@apache.org>.
wolfboys commented on issue #2369:
URL: https://github.com/apache/incubator-streampark/issues/2369#issuecomment-1445634641
please use English
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] yjz-enter commented on issue #2369: Could not complete the operation. Number of retries has been exhausted.
Posted by "yjz-enter (via GitHub)" <gi...@apache.org>.
yjz-enter commented on issue #2369:
URL: https://github.com/apache/incubator-streampark/issues/2369#issuecomment-1717297138
这个问题怎么解决的呢
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] wolfboys closed issue #2369: k8s 运行报错 io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager [] - Exec Failure: HTTP 403, Status: 403 - Forbidden
Posted by "wolfboys (via GitHub)" <gi...@apache.org>.
wolfboys closed issue #2369: k8s 运行报错 io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager [] - Exec Failure: HTTP 403, Status: 403 - Forbidden
URL: https://github.com/apache/incubator-streampark/issues/2369
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org