You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user-zh@flink.apache.org by xinzhuxiansheng <xi...@126.com> on 2022/03/01 07:22:15 UTC
回复： Flink UI 经常打不开作业的拓扑图
用的是Flink 1.14.2 版本， 作业的参数比较简单，除了HA相关配置，其他都是默认值，运行环境是在 Flink on Native K8s application mode部署 。 作业分别看了jobManager和taskmanager 日志， 看到 tm的业务日志及 kafka数据都正常输出，是可以确认作业在正常工作的。 今天我在钉钉群问了，还是归结于源码逻辑为什么，有时间学习下这块代码，Flink on Native K8s S3 HA的处理
你说的对，下回将log放在附件中。
| |
xinzhuxiansheng
|
|
xinzhuxiansheng@126.com
|
签名由网易邮箱大师定制
在2022年02月28日 23:54，yu'an huang<h....@gmail.com> 写道：
你好，
请问使用的Flink版本是哪一个呢，可以提供下完整的作业提交参数和运行环境吗？
看log猜测是某个UI请求而不是runtime内部出现了问题，确认下如果不访问UI中的作业拓扑图的话整个作业是否可以正常执行。
最后有一个建议把log作为邮件附件发送的话查看起来比较方便一点^_。
On 28 Feb 2022, at 11:21 PM, xinzhuxiansheng <xi...@126.com> wrote:
作业在HA或者单jobManager 写入kafka，消费kafka 都没问题 ， 不过，Flink UI 出现拓扑图打不开的情况只在HA模式下有，单jobManager一切都正常， 不知道是否是HA配置有问题，看堆栈信息，我用远程debug来调试代码，可缺少对代码上下文的理解，有些找不到头绪。 目前把理解的重心 放在 “java.util.concurrent.CompletableFuture.cancel()”，为啥有 “java.util.concurrent.CancellationException: null”
以下是我的job关于HA的参数：
-Dkubernetes.container.image.pull-policy=Always
-Dhigh-availability=org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
-Dhigh-availability.storageDir=s3://fmflinkha/
-Dcontainerized.master.env.ENABLE_BUILT_IN_PLUGINS=flink-s3-fs-hadoop-1.14.2.jar
-Dcontainerized.taskmanager.env.ENABLE_BUILT_IN_PLUGINS=flink-s3-fs-hadoop-1.14.2.jar
-Dkubernetes.jobmanager.replicas=2
| |
xinzhuxiansheng
|
|
xinzhuxiansheng@126.com
|
签名由网易邮箱大师定制
在2022年02月28日 21:48，胡伟华<hu...@gmail.com> 写道：
日志里没有 Dispatcher 、Job 的启动日志，请问作业是正常工作的吗？
如果作业不正常，可以检查下 leader 选主的逻辑，或者尝试去掉 HA，观察作业是否正常
2022年2月28日 下午8:57，xinzhuxiansheng <xi...@126.com> 写道：
你是对的，在你的提醒下 ，我看了完整的 log，确实还有一些error log
Enabling required built-in plugins
Linking flink-s3-fs-hadoop-1.14.2.jar to plugin directory
Successfully enabled flink-s3-fs-hadoop-1.14.2.jar
sed: couldn't open temporary file /opt/flink/conf/sedqVEQdn: Read-only file system
sed: couldn't open temporary file /opt/flink/conf/sedNyxXkO: Read-only file system
/docker-entrypoint.sh: line 73: /opt/flink/conf/flink-conf.yaml: Read-only file system
/docker-entrypoint.sh: line 89: /opt/flink/conf/flink-conf.yaml.tmp: Read-only file system
Starting kubernetes-application as a console application on host fm-hwm8u1o2x8-9cc6947f9-tgxd5.
Listening for transport dt_socket at address: 5006
2022-02-28 12:23:49,009 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - --------------------------------------------------------------------------------
2022-02-28 12:23:49,012 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Preconfiguration:
2022-02-28 12:23:49,012 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
RESOURCE_PARAMS extraction logs:
jvm_params: -Xmx1073741824 -Xms1073741824 -XX:MaxMetaspaceSize=268435456
dynamic_configs: -D jobmanager.memory.off-heap.size=134217728b -D jobmanager.memory.jvm-overhead.min=201326592b -D jobmanager.memory.jvm-metaspace.size=268435456b -D jobmanager.memory.heap.size=1073741824b -D jobmanager.memory.jvm-overhead.max=201326592b
logs: INFO [] - Loading configuration property: blob.server.port, 6124
INFO [] - Loading configuration property: env.java.opts.jobmanager, "-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5006"
INFO [] - Loading configuration property: kubernetes.jobmanager.replicas, 2
INFO [] - Loading configuration property: jobmanager.execution.failover-strategy, region
INFO [] - Loading configuration property: high-availability.cluster-id, fm-hwm8u1o2x8
INFO [] - Loading configuration property: jobmanager.rpc.address, localhost
INFO [] - Loading configuration property: -parallelism.default, 10
INFO [] - Loading configuration property: containerized.master.env.ENABLE_BUILT_IN_PLUGINS, flink-s3-fs-hadoop-1.14.2.jar
INFO [] - Loading configuration property: kubernetes.taskmanager.cpu, 2
INFO [] - Loading configuration property: kubernetes.cluster-id, fm-hwm8u1o2x8
INFO [] - Loading configuration property: kubernetes.service-account, flink
INFO [] - Loading configuration property: s3.secret-key, ******
INFO [] - Loading configuration property: high-availability.storageDir, s3://xx.xx/flink-ha
INFO [] - Loading configuration property: s3.endpoint, xx.xx.com
INFO [] - Loading configuration property: $internal.application.program-args, --bootstrapServers;192.168.xx.xx:9092;--topic;yzhoutp01;--groupId;yzhougid022801;--s3path;s3://xx.xx/flink-ha;--flink.conf;;--app.name;HA消费作业;--conf;'json://{"$internal.application.main":"com.yzhou.checkpoint.UDFDataConsumerSource"}'
INFO [] - Loading configuration property: kubernetes.container.image, flink:1.14.2-scala_2.11-java8
INFO [] - Loading configuration property: parallelism.default, 10
INFO [] - Loading configuration property: kubernetes.namespace, yzhou
INFO [] - Loading configuration property: taskmanager.numberOfTaskSlots, 4
INFO [] - Loading configuration property: kubernetes.rest-service.exposed.type, NodePort
INFO [] - Loading configuration property: high-availability.jobmanager.port, 6123
INFO [] - Loading configuration property: $internal.application.main, com.yzhou.checkpoint.UDFDataConsumerSource
INFO [] - Loading configuration property: taskmanager.memory.process.size, 1728m
INFO [] - Loading configuration property: kubernetes.internal.jobmanager.entrypoint.class, org.apache.flink.kubernetes.entrypoint.KubernetesApplicationClusterEntrypoint
INFO [] - Loading configuration property: pipeline.name, HA消费作业
INFO [] - Loading configuration property: classloader.resolve-order, parent-first
INFO [] - Loading configuration property: kubernetes.pod-template-file, /Users/xx.xx/Tmp/app/fm-hwm8u1o2x8/pod-template.yaml
INFO [] - Loading configuration property: execution.target, kubernetes-application
INFO [] - Loading configuration property: restart-strategy.fixed-delay.attempts, 10
INFO [] - Loading configuration property: jobmanager.memory.process.size, 1600m
INFO [] - Loading configuration property: jobmanager.rpc.port, 6123
INFO [] - Loading configuration property: taskmanager.rpc.port, 6122
INFO [] - Loading configuration property: s3.access-key, AKIDjCPQBY8JfniPinIbXbfbji1AB4sYNsK0
INFO [] - Loading configuration property: kubernetes.container.image.pull-policy, Always
INFO [] - Loading configuration property: internal.cluster.execution-mode, NORMAL
INFO [] - Loading configuration property: $internal.pipeline.job-id, 72351556fc8d911c4de6e64750fd805a
INFO [] - Loading configuration property: execution.shutdown-on-attached-exit, false
INFO [] - Loading configuration property: high-availability, org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
INFO [] - Loading configuration property: pipeline.jars, local:///opt/flink/usrlib/app.jar
INFO [] - Loading configuration property: restart-strategy, fixed-delay
INFO [] - Loading configuration property: env.java.opts.taskmanager, "-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005"
INFO [] - Loading configuration property: containerized.taskmanager.env.ENABLE_BUILT_IN_PLUGINS, flink-s3-fs-hadoop-1.14.2.jar
INFO [] - Loading configuration property: s3.path.style.access, true
INFO [] - The derived from fraction jvm overhead memory (160.000mb (167772162 bytes)) is less than its min value 192.000mb (201326592 bytes), min value will be used instead
INFO [] - Final Master Memory configuration:
INFO [] - Total Process Memory: 1.563gb (1677721600 bytes)
INFO [] - Total Flink Memory: 1.125gb (1207959552 bytes)
INFO [] - JVM Heap: 1024.000mb (1073741824 bytes)
INFO [] - Off-heap: 128.000mb (134217728 bytes)
INFO [] - JVM Metaspace: 256.000mb (268435456 bytes)
INFO [] - JVM Overhead: 192.000mb (201326592 bytes)
2022-02-28 12:23:49,013 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - --------------------------------------------------------------------------------
2022-02-28 12:23:49,013 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Starting KubernetesApplicationClusterEntrypoint (Version: 1.14.2, Scala: 2.11, Rev:6fd4b1c, Date:2021-12-14T23:22:57+01:00)
2022-02-28 12:23:49,013 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - OS current user: flink
2022-02-28 12:23:49,014 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Current Hadoop/Kerberos user:
2022-02-28 12:23:49,014 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - JVM: OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.312-b07
2022-02-28 12:23:49,014 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Maximum heap size: 989 MiBytes
2022-02-28 12:23:49,014 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - JAVA_HOME: /usr/local/openjdk-8
2022-02-28 12:23:49,015 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - No Hadoop Dependency available
2022-02-28 12:23:49,015 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - JVM Options:
2022-02-28 12:23:49,015 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -Xmx1073741824
2022-02-28 12:23:49,015 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -Xms1073741824
2022-02-28 12:23:49,015 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -XX:MaxMetaspaceSize=268435456
2022-02-28 12:23:49,015 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5006
2022-02-28 12:23:49,015 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -Dlog.file=/opt/flink/log/flink--kubernetes-application-0-fm-hwm8u1o2x8-9cc6947f9-tgxd5.log
2022-02-28 12:23:49,016 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -Dlog4j.configuration=file:/opt/flink/conf/log4j-console.properties
2022-02-28 12:23:49,016 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -Dlog4j.configurationFile=file:/opt/flink/conf/log4j-console.properties
2022-02-28 12:23:49,016 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -Dlogback.configurationFile=file:/opt/flink/conf/logback-console.xml
2022-02-28 12:23:49,016 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Program Arguments:
2022-02-28 12:23:49,017 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -D
2022-02-28 12:23:49,017 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - jobmanager.memory.off-heap.size=134217728b
2022-02-28 12:23:49,018 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -D
2022-02-28 12:23:49,018 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - jobmanager.memory.jvm-overhead.min=201326592b
2022-02-28 12:23:49,018 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -D
2022-02-28 12:23:49,018 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - jobmanager.memory.jvm-metaspace.size=268435456b
2022-02-28 12:23:49,018 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -D
2022-02-28 12:23:49,018 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - jobmanager.memory.heap.size=1073741824b
2022-02-28 12:23:49,019 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -D
2022-02-28 12:23:49,019 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - jobmanager.memory.jvm-overhead.max=201326592b
2022-02-28 12:23:49,019 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Classpath: /opt/flink/lib/flink-csv-1.14.2.jar:/opt/flink/lib/flink-json-1.14.2.jar:/opt/flink/lib/flink-shaded-zookeeper-3.4.14.jar:/opt/flink/lib/flink-table_2.11-1.14.2.jar:/opt/flink/lib/log4j-1.2-api-2.16.0.jar:/opt/flink/lib/log4j-api-2.16.0.jar:/opt/flink/lib/log4j-core-2.16.0.jar:/opt/flink/lib/log4j-slf4j-impl-2.16.0.jar:/opt/flink/lib/flink-dist_2.11-1.14.2.jar:::
2022-02-28 12:23:49,019 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - --------------------------------------------------------------------------------
2022-02-28 12:23:49,020 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Registered UNIX signal handlers for [TERM, HUP, INT]
2022-02-28 12:23:49,037 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: blob.server.port, 6124
2022-02-28 12:23:49,038 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: env.java.opts.jobmanager, "-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5006"
2022-02-28 12:23:49,038 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: kubernetes.jobmanager.replicas, 2
2022-02-28 12:23:49,038 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.execution.failover-strategy, region
2022-02-28 12:23:49,096 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: high-availability.cluster-id, fm-hwm8u1o2x8
2022-02-28 12:23:49,097 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.rpc.address, localhost
2022-02-28 12:23:49,097 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: -parallelism.default, 10
2022-02-28 12:23:49,097 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: containerized.master.env.ENABLE_BUILT_IN_PLUGINS, flink-s3-fs-hadoop-1.14.2.jar
2022-02-28 12:23:49,097 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: kubernetes.taskmanager.cpu, 2
2022-02-28 12:23:49,098 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: kubernetes.cluster-id, fm-hwm8u1o2x8
2022-02-28 12:23:49,098 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: kubernetes.service-account, flink
2022-02-28 12:23:49,098 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: s3.secret-key, ******
2022-02-28 12:23:49,098 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: high-availability.storageDir, s3://xx.xx/flink-ha
2022-02-28 12:23:49,099 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: s3.endpoint, xx.xx.com
2022-02-28 12:23:49,099 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: $internal.application.program-args, --bootstrapServers;192.168.xx.xx:9092;--topic;yzhoutp01;--groupId;yzhougid022801;--s3path;s3://xx.xx/flink-ha;--flink.conf;;--app.name;HA消费作业;--conf;'json://{"$internal.application.main":"com.yzhou.checkpoint.UDFDataConsumerSource"}'
2022-02-28 12:23:49,099 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: kubernetes.container.image, flink:1.14.2-scala_2.11-java8
2022-02-28 12:23:49,099 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: parallelism.default, 10
2022-02-28 12:23:49,099 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: kubernetes.namespace, yzhou
2022-02-28 12:23:49,100 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: taskmanager.numberOfTaskSlots, 4
2022-02-28 12:23:49,100 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: kubernetes.rest-service.exposed.type, NodePort
2022-02-28 12:23:49,100 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: high-availability.jobmanager.port, 6123
2022-02-28 12:23:49,100 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: $internal.application.main, com.yzhou.checkpoint.UDFDataConsumerSource
2022-02-28 12:23:49,100 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: taskmanager.memory.process.size, 1728m
2022-02-28 12:23:49,101 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: kubernetes.internal.jobmanager.entrypoint.class, org.apache.flink.kubernetes.entrypoint.KubernetesApplicationClusterEntrypoint
2022-02-28 12:23:49,101 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: pipeline.name, HA消费作业
2022-02-28 12:23:49,101 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: classloader.resolve-order, parent-first
2022-02-28 12:23:49,101 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: kubernetes.pod-template-file, /Users/xx.xx/Tmp/app/fm-hwm8u1o2x8/pod-template.yaml
2022-02-28 12:23:49,101 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: execution.target, kubernetes-application
2022-02-28 12:23:49,102 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: restart-strategy.fixed-delay.attempts, 10
2022-02-28 12:23:49,102 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.memory.process.size, 1600m
2022-02-28 12:23:49,102 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.rpc.port, 6123
2022-02-28 12:23:49,102 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: taskmanager.rpc.port, 6122
2022-02-28 12:23:49,102 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: s3.access-key, AKIDjCPQBY8JfniPinIbXbfbji1AB4sYNsK0
2022-02-28 12:23:49,103 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: kubernetes.container.image.pull-policy, Always
2022-02-28 12:23:49,103 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: internal.cluster.execution-mode, NORMAL
2022-02-28 12:23:49,103 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: $internal.pipeline.job-id, 72351556fc8d911c4de6e64750fd805a
2022-02-28 12:23:49,103 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: execution.shutdown-on-attached-exit, false
2022-02-28 12:23:49,103 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: high-availability, org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
2022-02-28 12:23:49,104 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: pipeline.jars, local:///opt/flink/usrlib/app.jar
2022-02-28 12:23:49,104 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: restart-strategy, fixed-delay
2022-02-28 12:23:49,104 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: env.java.opts.taskmanager, "-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005"
2022-02-28 12:23:49,104 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: containerized.taskmanager.env.ENABLE_BUILT_IN_PLUGINS, flink-s3-fs-hadoop-1.14.2.jar
2022-02-28 12:23:49,104 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: s3.path.style.access, true
2022-02-28 12:23:49,800 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Starting KubernetesApplicationClusterEntrypoint.
2022-02-28 12:23:49,818 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Install default filesystem.
2022-02-28 12:23:49,824 INFO org.apache.flink.core.fs.FileSystem [] - Hadoop is not in the classpath/dependencies. The extended set of supported File Systems via Hadoop is not available.
2022-02-28 12:23:50,117 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Install security context.
2022-02-28 12:23:50,133 INFO org.apache.flink.runtime.security.modules.HadoopModuleFactory [] - Cannot create Hadoop Security Module because Hadoop cannot be found in the Classpath.
2022-02-28 12:23:50,140 INFO org.apache.flink.runtime.security.modules.JaasModule [] - Jaas file will be created as /tmp/jaas-8100113544211743196.conf.
2022-02-28 12:23:50,149 INFO org.apache.flink.runtime.security.contexts.HadoopSecurityContextFactory [] - Cannot install HadoopSecurityContext because Hadoop cannot be found in the Classpath.
2022-02-28 12:23:50,151 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Initializing cluster services.
2022-02-28 12:23:51,258 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils [] - Trying to start actor system, external address 10.168.xx.xx:6123, bind address 0.0.0.0:6123.
2022-02-28 12:23:53,108 INFO akka.event.slf4j.Slf4jLogger [] - Slf4jLogger started
2022-02-28 12:23:53,215 INFO akka.remote.RemoteActorRefProvider [] - Akka Cluster not in use - enabling unsafe features anyway because `akka.remote.use-unsafe-remote-features-outside-cluster` has been enabled.
2022-02-28 12:23:53,216 INFO akka.remote.Remoting [] - Starting remoting
2022-02-28 12:23:53,609 INFO akka.remote.Remoting [] - Remoting started; listening on addresses :[akka.tcp://flink@10.168.xx.xx:6123]
2022-02-28 12:23:54,063 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils [] - Actor system started at akka.tcp://flink@10.168.xx.xx:6123
2022-02-28 12:23:55,630 WARN org.apache.hadoop.metrics2.impl.MetricsConfig [] - Cannot locate configuration: tried hadoop-metrics2-s3a-file-system.properties,hadoop-metrics2.properties
2022-02-28 12:23:55,718 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl [] - Scheduled Metric snapshot period at 10 second(s).
2022-02-28 12:23:55,719 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl [] - s3a-file-system metrics system started
2022-02-28 12:23:55,909 WARN org.apache.hadoop.util.NativeCodeLoader [] - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2022-02-28 12:23:58,602 INFO org.apache.flink.runtime.blob.FileSystemBlobStore [] - Creating highly available BLOB storage directory at s3://xx.xx/flink-ha/fm-hwm8u1o2x8/blob
2022-02-28 12:24:01,010 INFO org.apache.flink.runtime.blob.BlobServer [] - Created BLOB server storage directory /tmp/blobStore-39b216f8-1ad6-4e24-9132-2166dfa40ce0
2022-02-28 12:24:01,018 INFO org.apache.flink.runtime.blob.BlobServer [] - Started BLOB server at 0.0.0.0:6124 - max concurrent requests: 50 - max backlog: 1000
2022-02-28 12:24:01,107 INFO org.apache.flink.runtime.metrics.MetricRegistryImpl [] - No metrics reporter configured, no metrics will be exposed/reported.
2022-02-28 12:24:01,114 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils [] - Trying to start actor system, external address 10.168.xx.xx:0, bind address 0.0.0.0:0.
2022-02-28 12:24:01,223 INFO akka.event.slf4j.Slf4jLogger [] - Slf4jLogger started
2022-02-28 12:24:01,297 INFO akka.remote.RemoteActorRefProvider [] - Akka Cluster not in use - enabling unsafe features anyway because `akka.remote.use-unsafe-remote-features-outside-cluster` has been enabled.
2022-02-28 12:24:01,297 INFO akka.remote.Remoting [] - Starting remoting
2022-02-28 12:24:01,321 INFO akka.remote.Remoting [] - Remoting started; listening on addresses :[akka.tcp://flink-metrics@10.168.xx.xx:38165]
2022-02-28 12:24:01,407 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils [] - Actor system started at akka.tcp://flink-metrics@10.168.xx.xx:38165
2022-02-28 12:24:01,430 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService [] - Starting RPC endpoint for org.apache.flink.runtime.metrics.dump.MetricQueryService at akka://flink-metrics/user/rpc/MetricQueryService .
2022-02-28 12:24:01,639 INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Upload directory /tmp/flink-web-813c1c10-1aeb-4c95-b010-4ecdb6fb3829/flink-web-upload does not exist.
2022-02-28 12:24:01,697 INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Created directory /tmp/flink-web-813c1c10-1aeb-4c95-b010-4ecdb6fb3829/flink-web-upload for file uploads.
2022-02-28 12:24:01,731 INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Starting rest endpoint.
2022-02-28 12:24:02,028 INFO org.apache.flink.runtime.webmonitor.WebMonitorUtils [] - Determined location of main cluster component log file: /opt/flink/log/flink--kubernetes-application-0-fm-hwm8u1o2x8-9cc6947f9-tgxd5.log
2022-02-28 12:24:02,028 INFO org.apache.flink.runtime.webmonitor.WebMonitorUtils [] - Determined location of main cluster component stdout file: /opt/flink/log/flink--kubernetes-application-0-fm-hwm8u1o2x8-9cc6947f9-tgxd5.out
2022-02-28 12:24:02,608 INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Rest endpoint listening at 10.168.xx.xx:8081
2022-02-28 12:24:02,620 INFO org.apache.flink.kubernetes.kubeclient.resources.KubernetesLeaderElector [] - Create KubernetesLeaderElector fm-hwm8u1o2x8-restserver-leader with lock identity a9d2cf8d-1a03-437d-8215-20b08ad4813f.
2022-02-28 12:24:02,624 INFO org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService [] - Starting DefaultLeaderElectionService with KubernetesLeaderElectionDriver{configMapName='fm-hwm8u1o2x8-restserver-leader'}.
2022-02-28 12:24:02,624 INFO org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapSharedInformer [] - Starting to watch for yzhou/fm-hwm8u1o2x8-restserver-leader, watching id:25697182-adb4-409e-bcb5-11c172b02bcd
2022-02-28 12:24:02,625 INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Web frontend listening at http://10.168.xx.xx:8081.
2022-02-28 12:24:02,718 INFO org.apache.flink.runtime.util.config.memory.ProcessMemoryUtils [] - The derived from fraction jvm overhead memory (172.800mb (181193935 bytes)) is less than its min value 192.000mb (201326592 bytes), min value will be used instead
2022-02-28 12:24:02,908 INFO org.apache.flink.kubernetes.kubeclient.resources.KubernetesLeaderElector [] - Create KubernetesLeaderElector fm-hwm8u1o2x8-dispatcher-leader with lock identity a9d2cf8d-1a03-437d-8215-20b08ad4813f.
2022-02-28 12:24:02,909 INFO org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService [] - Starting DefaultLeaderElectionService with KubernetesLeaderElectionDriver{configMapName='fm-hwm8u1o2x8-dispatcher-leader'}.
2022-02-28 12:24:02,909 INFO org.apache.flink.runtime.resourcemanager.ResourceManagerServiceImpl [] - Starting resource manager service.
2022-02-28 12:24:02,909 INFO org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapSharedInformer [] - Starting to watch for yzhou/fm-hwm8u1o2x8-dispatcher-leader, watching id:2c14bb82-cd22-4c7b-8a4f-838e37efee81
2022-02-28 12:24:02,910 INFO org.apache.flink.kubernetes.kubeclient.resources.KubernetesLeaderElector [] - Create KubernetesLeaderElector fm-hwm8u1o2x8-resourcemanager-leader with lock identity a9d2cf8d-1a03-437d-8215-20b08ad4813f.
2022-02-28 12:24:02,910 INFO org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService [] - Starting DefaultLeaderElectionService with KubernetesLeaderElectionDriver{configMapName='fm-hwm8u1o2x8-resourcemanager-leader'}.
2022-02-28 12:24:02,910 INFO org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapSharedInformer [] - Starting to watch for yzhou/fm-hwm8u1o2x8-resourcemanager-leader, watching id:cd6a0293-f546-4510-aa68-2481bf4f69b9
2022-02-28 12:24:02,913 INFO org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService [] - Starting DefaultLeaderRetrievalService with KubernetesLeaderRetrievalDriver{configMapName='fm-hwm8u1o2x8-resourcemanager-leader'}.
2022-02-28 12:24:02,913 INFO org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapSharedInformer [] - Starting to watch for yzhou/fm-hwm8u1o2x8-resourcemanager-leader, watching id:994bd8c2-fe3a-436c-8116-83ce42656c87
2022-02-28 12:24:02,913 INFO org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService [] - Starting DefaultLeaderRetrievalService with KubernetesLeaderRetrievalDriver{configMapName='fm-hwm8u1o2x8-dispatcher-leader'}.
2022-02-28 12:24:02,913 INFO org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapSharedInformer [] - Starting to watch for yzhou/fm-hwm8u1o2x8-dispatcher-leader, watching id:5071b799-e48b-4705-9e30-5f1b190b867d
2022-02-28 12:24:03,109 INFO org.apache.flink.kubernetes.kubeclient.resources.KubernetesLeaderElector [] - New leader elected b2035865-bcec-4803-9a6c-7267d1b1ba54 for fm-hwm8u1o2x8-dispatcher-leader.
2022-02-28 12:24:03,109 INFO org.apache.flink.kubernetes.kubeclient.resources.KubernetesLeaderElector [] - New leader elected b2035865-bcec-4803-9a6c-7267d1b1ba54 for fm-hwm8u1o2x8-restserver-leader.
2022-02-28 12:24:03,207 INFO org.apache.flink.kubernetes.kubeclient.resources.KubernetesLeaderElector [] - New leader elected b2035865-bcec-4803-9a6c-7267d1b1ba54 for fm-hwm8u1o2x8-resourcemanager-leader.
2022-02-28 12:26:02,178 ERROR org.apache.flink.runtime.rest.handler.job.JobExceptionsHandler [] - Exception occurred in REST handler: Job b2c4f20005144514a7995a33b8fe03f3 not found
2022-02-28 12:27:34,854 WARN io.fabric8.kubernetes.client.dsl.internal.WatcherWebSocketListener [] - Exec Failure java.net.SocketTimeoutException sent ping but didn't receive pong within 30000ms (after 5 successful ping/pongs)
2022-02-28 12:27:15,149 ERROR org.apache.flink.runtime.rest.handler.job.checkpoints.CheckpointingStatisticsHandler [] - Unhandled exception.
java.util.concurrent.CancellationException: null
at java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2276) ~[?:1.8.0_312]
at org.apache.flink.runtime.rest.handler.legacy.DefaultExecutionGraphCache.getExecutionGraphInternal(DefaultExecutionGraphCache.java:98) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.legacy.DefaultExecutionGraphCache.getExecutionGraphInfo(DefaultExecutionGraphCache.java:67) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.job.AbstractExecutionGraphHandler.handleRequest(AbstractExecutionGraphHandler.java:81) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.AbstractRestHandler.respondToRequest(AbstractRestHandler.java:83) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.AbstractHandler.respondAsLeader(AbstractHandler.java:195) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.lambda$channelRead0$0(LeaderRetrievalHandler.java:83) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at java.util.Optional.ifPresent(Optional.java:159) [?:1.8.0_312]
at org.apache.flink.util.OptionalConsumer.ifPresent(OptionalConsumer.java:45) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.channelRead0(LeaderRetrievalHandler.java:80) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.channelRead0(LeaderRetrievalHandler.java:49) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.router.RouterHandler.routed(RouterHandler.java:115) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.router.RouterHandler.channelRead0(RouterHandler.java:94) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.router.RouterHandler.channelRead0(RouterHandler.java:55) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.FileUploadHandler.channelRead0(FileUploadHandler.java:238) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.FileUploadHandler.channelRead0(FileUploadHandler.java:71) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [flink-dist_2.11-1.14.2.jar:1.14.2]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_312]
2022-02-28 12:35:11,614 ERROR org.apache.flink.runtime.rest.handler.job.metrics.JobVertexWatermarksHandler [] - Unhandled exception.
java.util.concurrent.CancellationException: null
at java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2276) ~[?:1.8.0_312]
at org.apache.flink.runtime.rest.handler.legacy.DefaultExecutionGraphCache.getExecutionGraphInternal(DefaultExecutionGraphCache.java:98) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.legacy.DefaultExecutionGraphCache.getExecutionGraphInfo(DefaultExecutionGraphCache.java:67) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.job.AbstractExecutionGraphHandler.handleRequest(AbstractExecutionGraphHandler.java:81) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.AbstractRestHandler.respondToRequest(AbstractRestHandler.java:83) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.AbstractHandler.respondAsLeader(AbstractHandler.java:195) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.lambda$channelRead0$0(LeaderRetrievalHandler.java:83) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at java.util.Optional.ifPresent(Optional.java:159) [?:1.8.0_312]
at org.apache.flink.util.OptionalConsumer.ifPresent(OptionalConsumer.java:45) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.channelRead0(LeaderRetrievalHandler.java:80) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.channelRead0(LeaderRetrievalHandler.java:49) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.router.RouterHandler.routed(RouterHandler.java:115) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.router.RouterHandler.channelRead0(RouterHandler.java:94) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.router.RouterHandler.channelRead0(RouterHandler.java:55) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.FileUploadHandler.channelRead0(FileUploadHandler.java:238) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.FileUploadHandler.channelRead0(FileUploadHandler.java:71) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [flink-dist_2.11-1.14.2.jar:1.14.2]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_312]
2022-02-28 12:35:46,286 ERROR org.apache.flink.runtime.rest.handler.job.JobDetailsHandler [] - Unhandled exception.
java.util.concurrent.CancellationException: null
at java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2276) ~[?:1.8.0_312]
at org.apache.flink.runtime.rest.handler.legacy.DefaultExecutionGraphCache.getExecutionGraphInternal(DefaultExecutionGraphCache.java:98) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.legacy.DefaultExecutionGraphCache.getExecutionGraphInfo(DefaultExecutionGraphCache.java:67) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.job.AbstractExecutionGraphHandler.handleRequest(AbstractExecutionGraphHandler.java:81) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.AbstractRestHandler.respondToRequest(AbstractRestHandler.java:83) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.AbstractHandler.respondAsLeader(AbstractHandler.java:195) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.lambda$channelRead0$0(LeaderRetrievalHandler.java:83) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at java.util.Optional.ifPresent(Optional.java:159) [?:1.8.0_312]
at org.apache.flink.util.OptionalConsumer.ifPresent(OptionalConsumer.java:45) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.channelRead0(LeaderRetrievalHandler.java:80) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.channelRead0(LeaderRetrievalHandler.java:49) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.router.RouterHandler.routed(RouterHandler.java:115) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.router.RouterHandler.channelRead0(RouterHandler.java:94) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.router.RouterHandler.channelRead0(RouterHandler.java:55) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.FileUploadHandler.channelRead0(FileUploadHandler.java:238) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.FileUploadHandler.channelRead0(FileUploadHandler.java:71) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [flink-dist_2.11-1.14.2.jar:1.14.2]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_312]
2022-02-28 12:35:47,305 ERROR org.apache.flink.runtime.rest.handler.job.JobDetailsHandler [] - Unhandled exception.
java.util.concurrent.CancellationException: null
at java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2276) ~[?:1.8.0_312]
at org.apache.flink.runtime.rest.handler.legacy.DefaultExecutionGraphCache.getExecutionGraphInternal(DefaultExecutionGraphCache.java:98) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.legacy.DefaultExecutionGraphCache.getExecutionGraphInfo(DefaultExecutionGraphCache.java:67) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.job.AbstractExecutionGraphHandler.handleRequest(AbstractExecutionGraphHandler.java:81) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.AbstractRestHandler.respondToRequest(AbstractRestHandler.java:83) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.AbstractHandler.respondAsLeader(AbstractHandler.java:195) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.lambda$channelRead0$0(LeaderRetrievalHandler.java:83) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at java.util.Optional.ifPresent(Optional.java:159) [?:1.8.0_312]
at org.apache.flink.util.OptionalConsumer.ifPresent(OptionalConsumer.java:45) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.channelRead0(LeaderRetrievalHandler.java:80) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.channelRead0(LeaderRetrievalHandler.java:49) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.router.RouterHandler.routed(RouterHandler.java:115) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.router.RouterHandler.channelRead0(RouterHandler.java:94) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.router.RouterHandler.channelRead0(RouterHandler.java:55) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.FileUploadHandler.channelRead0(FileUploadHandler.java:238) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.FileUploadHandler.channelRead0(FileUploadHandler.java:71) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [flink-dist_2.11-1.14.2.jar:1.14.2]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_312]
2022-02-28 12:42:13,537 ERROR org.apache.flink.runtime.rest.handler.job.JobDetailsHandler [] - Unhandled exception.
java.util.concurrent.CancellationException: null
at java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2276) ~[?:1.8.0_312]
at org.apache.flink.runtime.rest.handler.legacy.DefaultExecutionGraphCache.getExecutionGraphInternal(DefaultExecutionGraphCache.java:98) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.legacy.DefaultExecutionGraphCache.getExecutionGraphInfo(DefaultExecutionGraphCache.java:67) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.job.AbstractExecutionGraphHandler.handleRequest(AbstractExecutionGraphHandler.java:81) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.AbstractRestHandler.respondToRequest(AbstractRestHandler.java:83) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.AbstractHandler.respondAsLeader(AbstractHandler.java:195) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.lambda$channelRead0$0(LeaderRetrievalHandler.java:83) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at java.util.Optional.ifPresent(Optional.java:159) [?:1.8.0_312]
at org.apache.flink.util.OptionalConsumer.ifPresent(OptionalConsumer.java:45) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.channelRead0(LeaderRetrievalHandler.java:80) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.channelRead0(LeaderRetrievalHandler.java:49) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.router.RouterHandler.routed(RouterHandler.java:115) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.router.RouterHandler.channelRead0(RouterHandler.java:94) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.router.RouterHandler.channelRead0(RouterHandler.java:55) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.FileUploadHandler.channelRead0(FileUploadHandler.java:238) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.FileUploadHandler.channelRead0(FileUploadHandler.java:71) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [flink-dist_2.11-1.14.2.jar:1.14.2]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_312]
2022-02-28 12:42:17,268 ERROR org.apache.flink.runtime.rest.handler.job.JobDetailsHandler [] - Unhandled exception.
java.util.concurrent.CancellationException: null
at java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2276) ~[?:1.8.0_312]
at org.apache.flink.runtime.rest.handler.legacy.DefaultExecutionGraphCache.getExecutionGraphInternal(DefaultExecutionGraphCache.java:98) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.legacy.DefaultExecutionGraphCache.getExecutionGraphInfo(DefaultExecutionGraphCache.java:67) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.job.AbstractExecutionGraphHandler.handleRequest(AbstractExecutionGraphHandler.java:81) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.AbstractRestHandler.respondToRequest(AbstractRestHandler.java:83) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.AbstractHandler.respondAsLeader(AbstractHandler.java:195) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.lambda$channelRead0$0(LeaderRetrievalHandler.java:83) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at java.util.Optional.ifPresent(Optional.java:159) [?:1.8.0_312]
at org.apache.flink.util.OptionalConsumer.ifPresent(OptionalConsumer.java:45) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.channelRead0(LeaderRetrievalHandler.java:80) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.channelRead0(LeaderRetrievalHandler.java:49) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.router.RouterHandler.routed(RouterHandler.java:115) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.router.RouterHandler.channelRead0(RouterHandler.java:94) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.router.RouterHandler.channelRead0(RouterHandler.java:55) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.FileUploadHandler.channelRead0(FileUploadHandler.java:238) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.FileUploadHandler.channelRead0(FileUploadHandler.java:71) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [flink-dist_2.11-1.14.2.jar:1.14.2]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_312]
2022-02-28 12:42:21,958 ERROR org.apache.flink.runtime.rest.handler.job.JobDetailsHandler [] - Unhandled exception.
java.util.concurrent.CancellationException: null
at java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2276) ~[?:1.8.0_312]
at org.apache.flink.runtime.rest.handler.legacy.DefaultExecutionGraphCache.getExecutionGraphInternal(DefaultExecutionGraphCache.java:98) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.legacy.DefaultExecutionGraphCache.getExecutionGraphInfo(DefaultExecutionGraphCache.java:67) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.job.AbstractExecutionGraphHandler.handleRequest(AbstractExecutionGraphHandler.java:81) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.AbstractRestHandler.respondToRequest(AbstractRestHandler.java:83) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.AbstractHandler.respondAsLeader(AbstractHandler.java:195) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.lambda$channelRead0$0(LeaderRetrievalHandler.java:83) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at java.util.Optional.ifPresent(Optional.java:159) [?:1.8.0_312]
at org.apache.flink.util.OptionalConsumer.ifPresent(OptionalConsumer.java:45) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.channelRead0(LeaderRetrievalHandler.java:80) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.channelRead0(LeaderRetrievalHandler.java:49) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.router.RouterHandler.routed(RouterHandler.java:115) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.router.RouterHandler.channelRead0(RouterHandler.java:94) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.router.RouterHandler.channelRead0(RouterHandler.java:55) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.FileUploadHandler.channelRead0(FileUploadHandler.java:238) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.FileUploadHandler.channelRead0(FileUploadHandler.java:71) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [flink-dist_2.11-1.14.2.jar:1.14.2]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_312]
2022-02-28 12:43:55,653 ERROR io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector [] - Exception occurred while acquiring lock 'ConfigMapLock: yzhou - fm-hwm8u1o2x8-dispatcher-leader (a9d2cf8d-1a03-437d-8215-20b08ad4813f)'
io.fabric8.kubernetes.client.KubernetesClientException: Operation: [get] for kind: [ConfigMap] with name: [fm-hwm8u1o2x8-dispatcher-leader] in namespace: [yzhou] failed.
at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:64) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:72) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.getMandatory(BaseOperation.java:205) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.get(BaseOperation.java:167) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.get(BaseOperation.java:90) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.ConfigMapLock.get(ConfigMapLock.java:55) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.tryAcquireOrRenew(LeaderElector.java:134) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.lambda$acquire$0(LeaderElector.java:82) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.lambda$loop$3(LeaderElector.java:198) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_312]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_312]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_312]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_312]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_312]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_312]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_312]
Caused by: java.net.SocketTimeoutException: timeout
at org.apache.flink.kubernetes.shaded.okio.Okio$4.newTimeoutException(Okio.java:232) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.kubernetes.shaded.okio.AsyncTimeout.exit(AsyncTimeout.java:276) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.kubernetes.shaded.okio.AsyncTimeout$2.read(AsyncTimeout.java:243) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.kubernetes.shaded.okio.RealBufferedSource.indexOf(RealBufferedSource.java:358) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.kubernetes.shaded.okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:230) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.kubernetes.shaded.okhttp3.internal.http1.Http1ExchangeCodec.readHeaderLine(Http1ExchangeCodec.java:242) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.kubernetes.shaded.okhttp3.internal.http1.Http1ExchangeCodec.readResponseHeaders(Http1ExchangeCodec.java:213) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.kubernetes.shaded.okhttp3.internal.connection.Exchange.readResponseHeaders(Exchange.java:115) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.kubernetes.shaded.okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:94) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.kubernetes.shaded.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.kubernetes.shaded.okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:43) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.kubernetes.shaded.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.kubernetes.shaded.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.kubernetes.shaded.okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:94) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.kubernetes.shaded.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.kubernetes.shaded.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.kubernetes.shaded.okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.kubernetes.shaded.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.kubernetes.shaded.okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:88) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.kubernetes.shaded.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.kubernetes.shaded.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at io.fabric8.kubernetes.client.utils.BackwardsCompatibilityInterceptor.intercept(BackwardsCompatibilityInterceptor.java:133) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.kubernetes.shaded.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.kubernetes.shaded.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at io.fabric8.kubernetes.client.utils.TokenRefreshInterceptor.intercept(TokenRefreshInterceptor.java:42) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.kubernetes.shaded.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.kubernetes.shaded.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at io.fabric8.kubernetes.client.utils.ImpersonatorInterceptor.intercept(ImpersonatorInterceptor.java:68) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.kubernetes.shaded.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.kubernetes.shaded.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at io.fabric8.kubernetes.client.utils.HttpClientUtils.lambda$createApplicableInterceptors$6(HttpClientUtils.java:290) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.kubernetes.shaded.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.kubernetes.shaded.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.kubernetes.shaded.okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:229) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.kubernetes.shaded.okhttp3.RealCall.execute(RealCall.java:81) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.retryWithExponentialBackoff(OperationSupport.java:585) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:558) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:521) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:488) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:470) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleGet(BaseOperation.java:830) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.getMandatory(BaseOperation.java:200) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
... 13 more
2022-02-28 12:47:30,763 WARN io.fabric8.kubernetes.client.dsl.internal.WatcherWebSocketListener [] - Exec Failure java.net.SocketTimeoutException sent ping but didn't receive pong within 30000ms (after 32 successful ping/pongs)
2022-02-28 12:47:30,763 ERROR org.apache.flink.runtime.rest.handler.job.JobDetailsHandler [] - Unhandled exception.
java.util.concurrent.CancellationException: null
at java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2276) ~[?:1.8.0_312]
at org.apache.flink.runtime.rest.handler.legacy.DefaultExecutionGraphCache.getExecutionGraphInternal(DefaultExecutionGraphCache.java:98) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.legacy.DefaultExecutionGraphCache.getExecutionGraphInfo(DefaultExecutionGraphCache.java:67) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.job.AbstractExecutionGraphHandler.handleRequest(AbstractExecutionGraphHandler.java:81) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.AbstractRestHandler.respondToRequest(AbstractRestHandler.java:83) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.AbstractHandler.respondAsLeader(AbstractHandler.java:195) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.lambda$channelRead0$0(LeaderRetrievalHandler.java:83) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at java.util.Optional.ifPresent(Optional.java:159) [?:1.8.0_312]
at org.apache.flink.util.OptionalConsumer.ifPresent(OptionalConsumer.java:45) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.channelRead0(LeaderRetrievalHandler.java:80) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.channelRead0(LeaderRetrievalHandler.java:49) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.router.RouterHandler.routed(RouterHandler.java:115) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.router.RouterHandler.channelRead0(RouterHandler.java:94) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.router.RouterHandler.channelRead0(RouterHandler.java:55) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.FileUploadHandler.channelRead0(FileUploadHandler.java:238) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.FileUploadHandler.channelRead0(FileUploadHandler.java:71) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [flink-dist_2.11-1.14.2.jar:1.14.2]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_312]
2022-02-28 12:47:34,206 ERROR io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector [] - Exception occurred while acquiring lock 'ConfigMapLock: yzhou - fm-hwm8u1o2x8-resourcemanager-leader (a9d2cf8d-1a03-437d-8215-20b08ad4813f)'
io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.LockException: Unable to update ConfigMapLock
at io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.ConfigMapLock.update(ConfigMapLock.java:109) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.tryAcquireOrRenew(LeaderElector.java:156) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.lambda$acquire$0(LeaderElector.java:82) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.lambda$loop$3(LeaderElector.java:198) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_312]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_312]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_312]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_312]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_312]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_312]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_312]
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: PUT at: https://10.96.0.1/api/v1/namespaces/yzhou/configmaps/fm-hwm8u1o2x8-resourcemanager-leader. Message: Operation cannot be fulfilled on configmaps "fm-hwm8u1o2x8-resourcemanager-leader": the object has been modified; please apply your changes to the latest version and try again. Received status: Status(apiVersion=v1, code=409, details=StatusDetails(causes=[], group=null, kind=configmaps, name=fm-hwm8u1o2x8-resourcemanager-leader, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=Operation cannot be fulfilled on configmaps "fm-hwm8u1o2x8-resourcemanager-leader": the object has been modified; please apply your changes to the latest version and try again, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Conflict, status=Failure, additionalProperties={}).
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:673) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:612) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:560) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:521) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleUpdate(OperationSupport.java:347) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleUpdate(OperationSupport.java:327) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleUpdate(BaseOperation.java:781) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.lambda$replace$1(HasMetadataOperation.java:183) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:188) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:130) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:41) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.ConfigMapLock.update(ConfigMapLock.java:107) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
... 10 more
| |
xinzhuxiansheng
|
|
xinzhuxiansheng@126.com
|
签名由网易邮箱大师定制
在2022年02月28日 20:49，胡伟华<hu...@gmail.com> 写道：
Hi, xinzhuxiansheng
这个异常日志看起来不是根本原因，方便提供下全量 JobManager 日志吗？
2022年2月28日 下午7:50，xinzhuxiansheng <xi...@126.com> 写道：
我的环境:
Flink on Native K8s Application mode，并且用S3协议来做HA部署， 部署后，作业正常工作，可Flink UI 无法正常打开 作业的拓扑图，页面会提示内部错误
jobmanager会提示很多 类似的异常：
2022-02-28 08:46:21,171 ERROR org.apache.flink.runtime.rest.handler.job.JobExceptionsHandler [] - Unhandled exception.
java.util.concurrent.CancellationException: null
at java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2276) ~[?:1.8.0_312]
at org.apache.flink.runtime.rest.handler.legacy.DefaultExecutionGraphCache.getExecutionGraphInternal(DefaultExecutionGraphCache.java:98) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.legacy.DefaultExecutionGraphCache.getExecutionGraphInfo(DefaultExecutionGraphCache.java:67) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.job.AbstractExecutionGraphHandler.handleRequest(AbstractExecutionGraphHandler.java:81) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.AbstractRestHandler.respondToRequest(AbstractRestHandler.java:83) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.AbstractHandler.respondAsLeader(AbstractHandler.java:195) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.lambda$channelRead0$0(LeaderRetrievalHandler.java:83) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at java.util.Optional.ifPresent(Optional.java:159) [?:1.8.0_312]
at org.apache.flink.util.OptionalConsumer.ifPresent(OptionalConsumer.java:45) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.channelRead0(LeaderRetrievalHandler.java:80) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.channelRead0(LeaderRetrievalHandler.java:49) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.router.RouterHandler.routed(RouterHandler.java:115) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.router.RouterHandler.channelRead0(RouterHandler.java:94) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.handler.router.RouterHandler.channelRead0(RouterHandler.java:55) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.FileUploadHandler.channelRead0(FileUploadHandler.java:238) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.FileUploadHandler.channelRead0(FileUploadHandler.java:71) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [flink-dist_2.11-1.14.2.jar:1.14.2]
有人知道这是咋回事么 ？
| |
xinzhuxiansheng
|
|
xinzhuxiansheng@126.com
|
签名由网易邮箱大师定制